Rendering (computer graphics)

• 1. What Rendering Is

  ›

  Rendering, in computer graphics, is the process of generating an image from input data such as 3D models. The word itself predates computers. In one of its older senses, it described what an artist did when depicting a real or imaginary thing on paper or canvas. The finished artwork was also called a rendering. Today, to render commonly means to use a computer to generate an image from a precise specification, often created by artists working in interactive 3D modeling software.

  A software application or component that performs rendering is called a rendering engine, render engine, graphics engine, or simply a renderer. The images produced range from still photographs to frames for films and video games. Rendering draws from optics, the study of visual perception, mathematics, and software engineering. Its applications include video games, simulators, visual effects for films and television, design visualization, and medical diagnosis.

  Two fundamentally different modes exist. Real-time rendering generates and displays images immediately, fast enough to give the impression of motion or animation. Offline rendering, sometimes called pre-rendering, generates frames for later viewing, allowing slower processes that produce higher quality. Interactive applications like games must primarily use real-time rendering, though they may incorporate pre-rendered content alongside it.

  Historically, this field was called image synthesis. Today that term more likely refers to AI image generation. The term neural rendering is sometimes applied when a neural network is the primary means of generating an image, while still providing some degree of control over the output.

• 2. Chasing Photorealism

  ›

  A large proportion of computer graphics research has worked toward producing images that resemble photographs. Fundamental techniques that make photorealism possible were invented in the 1980s, but at the end of that decade, photorealism for complex scenes was still considered a distant goal. Today it is routinely achievable for offline rendering, though it remains difficult for real-time rendering.

  Realistic rendering must simulate how light travels from light sources, is reflected, refracted, and scattered by objects in a scene, passes through a camera lens, and finally reaches a film or sensor. The physics used in these simulations is primarily geometrical optics, where particles of light follow lines called rays. In some situations, such as rendering thin films like the surface of soap bubbles, the wave nature of light must also be taken into account.

  Effects that may need to be simulated include shadows with sharp or soft edges; reflections in mirrors and rough surfaces; refraction, the bending of light when it crosses boundaries between materials like air and glass; volumetric effects such as absorption and scattering through partially transparent substances; and caustics, the bright patches sometimes with distinct filaments that result when light is reflected or refracted before illuminating an object.

  For true photorealism, the camera must also be simulated. The thin lens approximation allows combining perspective projection with depth of field and bokeh emulation. Motion blur is often simulated when film or video frames are being rendered. Realistic materials such as marble, plant leaves, and human skin require simulating subsurface scattering, where light travels into the material, is scattered, and then exits again. Mathematical descriptions of how surfaces reflect light, called reflectance models or bidirectional reflectance distribution functions, are the formal basis for all of this work.

• 3. Rasterization and Real-Time Graphics

  ›

  Real-time rendering relies on high-performance rasterization algorithms that process a list of shapes and determine which pixels each shape covers. Three-dimensional animated films were rendered by rasterization before ray tracing and path tracing became practical enough to take over that role.

  The rasterization pipeline combines geometry processing with pixel processing that computes the RGB color values placed in the framebuffer for display. Its main tasks include determining which pixels are covered by each shape, blending colors and depths defined at vertices using methods like barycentric coordinates, determining which parts of shapes are hidden by other shapes, running a pixel shader or fragment shader for each covered pixel, smoothing edges through anti-aliasing, and blending overlapping transparent shapes.

  Hidden surface removal has a long history. Early computer graphics used the painter's algorithm, which sorts shapes by depth and renders them from back to front. Depth sorting was later incorporated directly into scanline rendering. The z-buffer algorithm performs depth comparisons by storing a depth value for each pixel in the framebuffer. A pixel is only covered by a shape if that shape's z value indicates it is closer to the camera than what the buffer currently holds. The z-buffer requires additional memory, which was an expensive resource when it was invented, but simplifies the rasterization code and permits multiple rendering passes.

  Until relatively recently, Pixar used rasterization for rendering its animated films. Its Reyes rendering system in RenderMan software was optimized for rendering very small, pixel-sized polygons, and incorporated stochastic sampling techniques more typically associated with ray tracing. Today, even low-power mobile processors typically incorporate 3D graphics acceleration features.

• 4. Ray Tracing and Path Tracing

  ›

  One of the simplest ways to render a 3D scene is to test whether a ray starting at the viewpoint intersects any geometric shape in the scene. Early ray casting experiments include the work of Arthur Appel in the 1960s. Appel rendered shadows by casting an additional ray from each visible surface point toward a light source, and tried rendering the density of illumination by casting random rays from the light source toward an object, plotting the intersection points in a way similar to the later technique called photon mapping.

  Classical ray tracing, also called Whitted-style or recursive ray tracing, extends the basic approach to render mirrors and transparent objects. When a ray traced backward from the camera hits a mirror, the reflection formula from geometric optics determines where the reflected ray came from. When it hits a transparent surface, Snell's law computes the refracted direction. The result is a branching tree of rays that must be traced recursively. Films such as Shrek 2 and Monsters University used distribution ray tracing or path tracing to precompute indirect illumination before rendering with rasterization.

  Path tracing was proposed and named in 1986 by Jim Kajiya in the same paper that introduced the rendering equation. Kajiya observed that much of the complexity of distributed ray tracing could be avoided by tracing only a single path from the camera at a time. Monster House, the first feature film rendered entirely using path tracing, was not released until 20 years later. The Arnold renderer, first released in 1998, proved that path tracing was practical for film rendering and that demand existed for unbiased, physically based rendering in the film industry. The popular open source software Blender uses path tracing in its Cycles renderer.

  Advances in GPU technology have made real-time ray tracing possible in games, though it is currently almost always used in combination with rasterization. Ray tracing support is included in recent versions of graphics APIs used by games, including DirectX, Metal, and Vulkan. Ray tracing has also been used to render simulated black holes and the appearance of objects moving at close to the speed of light, by taking spacetime curvature and relativistic effects into account.

• 5. Radiosity and Global Illumination

  ›

  Radiosity, named after the radiometric quantity of the same name, is a method for rendering objects illuminated by light bouncing off rough or matte surfaces. This type of illumination is called indirect light, environment lighting, or diffuse interreflection, and rendering it realistically is the problem of global illumination. Rasterization and basic ray tracing can only roughly approximate indirect light, typically by adding a uniform ambient lighting amount chosen by the artist.

  The original radiosity method was first proposed in 1984. In this classical form, surfaces and lights in the scene are split into pieces called patches, a process called meshing that makes it a finite element method. The fractions of light transferred between patches are called form factors or view factors, terms first used in engineering to model radiative heat transfer. These fractions are assembled into a matrix equation that is solved using methods from linear algebra.

  The large size of the matrices required for classical radiosity causes problems for realistic scenes. Practical implementations may use Jacobi or Gauss-Seidel iterations, which is equivalent to simulating light propagation one bounce at a time until the remaining light is insignificant. An important improvement is hierarchical radiosity, which uses a coarser mesh for simulating light transfer between distant surfaces and adaptively subdivides patches as needed, allowing radiosity to be applied to much larger and more complex scenes.

  For real-time rendering, precomputed irradiance values can be stored in a texture called an irradiance map or stored as vertex data for 3D models. This feature was used in architectural visualization software to allow real-time walkthroughs of building interiors after computing the lighting. Radiosity is considered a physically based method, though it often assumes all surfaces are opaque and perfectly Lambertian, which reduces realism and limits its applicability compared to path tracing.

• 6. Machine Learning and Neural Approaches

  ›

  As of 2003, machine learning techniques were being used in other areas of computer graphics, including texture synthesis, fitting surfaces to point clouds, and fitting curves to motion capture data. The use of machine learning for rendering expanded as GPUs capable of quickly evaluating neural networks, especially convolutional neural networks, became available.

  One of the most widely used applications of machine learning in rendering is denoising of path traced images. In practice, most path tracing now likely uses such a denoiser. Neural networks trained on pairs of noisy and low-noise images can use data such as surface normals and albedo in addition to the rendered image to reduce blurring and preserve textures. They are effective at removing the scattered bright pixels called fireflies that frequently occur in path tracing. Denoisers specialized for real-time rendering can work with very low sample counts and improve temporal coherence to reduce flickering.

  Neural radiance fields, or NeRFs, encode radiance at all points in a volume and use evaluation of this approximation as the primary rendering method. For any useful resolution, the amount of data in a radiance field is so large that it must be approximated, typically using neural networks trained on video frames or photographs of a scene taken from different angles. Algorithms related to neural networks have also been used to represent scenes as 3D Gaussians, using fuzzy partially-transparent blobs of varying dimensions and orientations rather than points. This technique, called 3D Gaussian splatting, appeared in 2023 on the chronology of rendering techniques.

  By combining denoising and hardware ray tracing acceleration, it is now practical to use path tracing for real-time rendering. Spatiotemporal reservoir resampling, known as ReSTIR, aims to improve the quality of real-time path tracing by reusing samples from previous frames and adjacent pixels. Eric Veach's 1997 PhD thesis summarized and expanded much of the earlier work on path sampling methods, helping raise interest in path tracing across the computer graphics research community.