Volumetric Rendering : "The How-to"
Rendering a 3D model is straightforward. With some vertices, we can make a mesh of triangles, add a bit of color, and get an object. Simple.
3d render of a dolphin, courtesy of Hans IT Academy
But what happens when our data isn't a mesh of vertices, but a grid of millions of values describing what's inside a 3D space? We can't really ask each value to fake it till it makes it as a ‘vertex’ in this competitive world.
To understand that, we need to first understand how volumetric data is structured, how it can be represented as a 3D texture, and how a cool rendering trick uses that texture to visualize the data.
In this blog, we'll walk through these concepts using a simple volumetric dataset, exploring how the 3D Texture Slicing method works and how it helps us visualize an entire volume.
Volumetric Render of a fish, courtesy of Nvidia
How is Volumetric data structured ?
Let’s begin our adventure at how our data is expected to look like, that is, be structured at the base level. But which door do we go knocking?
Every book begins with an index or a table of contents. The aim of this is for you to gauge the material of the book without having to read it first. This index could be called the ‘header’ of the book.
Similarly in the field of computer science, prefacing blocks of data with a ‘Header’ is pretty common.
From image formats like PNG, JPEG to network protocols like TCP/IP, every standard makes specifications for headers that describe the incoming data to the program reading it.
The header for volumetric data sets, located in the starting bytes, gives us basic information like the dimensions, the size of bytes and how to interpret them.
Simplified example of header data and content in dataset
However our file here (one we’re discussing about in this blog) has purely binary data with no header whatsoever, for the purpose of handling it simply. We learn from the source and file name that it is meant to be interpreted as a 256x256x256 byte array.
So it is just a Grid of Blocks but having 256 x 256 x 256 values in it, where each block only has a single value to represent material Density at the point.
For a deeper look into what Volume rendering is as a concept and why it is relevant, visit our previous blog What in the Graphics is Volumetric Rendering?! - What The Graphics!
Mesh Rendering vs Volume Rendering
To further discover the novel nature of volume rendering we must take a quick look at what our run-of-the-mill mesh rendering looks like.
While studying Computer Graphics, we normally start with writing programs to draw simple Geometry Primitives i.e different shapes.
The triangle is famously (in-famously for Vulkan programmers) known as the “Hello World” of computer graphics programming.
To draw this triangle, we sent our program the 3 vertices / points.
Using the vertices our Program draws the edges of the Triangle.
It is by the combination of these very triangles that we draw all the myriad shapes of the universe such as squares, rectangles, polygons, etc.
So far here’s our Triangle.
With only vertices, we could build a world full of incredibly complex geometry. And yet this engineering marvel of a world remains devoid of life until we breathe color into it - and this we do by passing color data for the coordinates onto our GPU.
Bee.glb file source opened in Microsoft 3D Viewer
You can see how this bee model is made up of a mesh with Triangles and Vertices.
While it might be tempting to hasten into Volume rendering already now that we’ve covered the basics of meshes, we must take one last detour that will only prove to be beneficial for us in the long run. And our stop is - textures.
A texture is nothing but an image, with a catch - it's meant to be wrapped over an object.
Ever opened up your game folder for GTA VICE CITY and come across this monstrosity?
Tommy Vercetti skin texture, courtesy of The Textures Resource
Poor Tommy, what did the gangs do to him? Fret not, for this is what he really looks like, before being wrapped on a human-looking mesh.
Model for Tommy Vercetti, courtesy of Wikimedia
Textures essentially enable us to load an image and apply it over the shape we are drawing.
The above example shows how a simple Cube looks when Texture is applied over it.
To make this work, we provide texture coordinates, commonly called UV coordinates, alongside our vertices. These coordinates tell the GPU which parts of the image should appear at which points on the geometry.
Without them, our carefully chosen image would have a hard time finding its way onto the right parts of the model.
Working with 2D images as textures surely suffices when it comes to mesh rendering and getting the model looking all prim and proper. However this 2D approach is perilous for the volume rendering territory.
But in what way on God’s green earth could we make our textures to fit in volumes??
It’s about time we make them go 3D.
What is 3D Texture:
Instead of storing image data across just a width and height, a 3D texture stores values across width, height, and depth. You can think of it as a stack of 2D images, with each layer representing a different depth in the volume.
Unlike a regular 2D texture, which is mapped onto a surface, a 3D texture lets us sample values from within a three-dimensional space.
And no, it doesn't always have to come from a volumetric dataset. We can also generate a 3D texture using mathematical functions, random values, or procedural noise.
For example, noise functions can generate patterns that vary smoothly throughout 3D space. These are useful for creating effects such as fire, clouds, and other procedural visuals.
This is a simple example of inferno effect created using Noise concept which uses 3D Array data to create 3D Texture.
We can also consider the 3D Texture as a 2D texture map which smoothly changes along its depth and blends from one texture to next.
For our volume rendering application, however, we'll use the voxel values from our dataset to create the 3D texture. Instead of generating the values ourselves, we'll use the data we've already got.
Set of 2D textures which change over R dimension Reference
Rendering Volumetric Data:
There are several ways to render volumetric data. We'll start with 3D texture slicing, a relatively simple technique that helps explain the fundamentals of volume rendering.
It's an older approach, but that doesn't make it useless for learning. Sometimes the easiest way to understand a complicated idea is to start with a method that lets us see each of its moving parts.
3D Texture Slicing:
3DTexture Slicing is one of the simplest methods for Volume rendering.
The basic idea is simple: we create a 3D texture from our voxel data, then render a series of 2D polygons through that texture. These polygons act as proxy geometry. They don't represent the actual surface of our object. Instead, they give us a way to sample and display the data inside the volume.
There's some geometry and mathematics involved in positioning these slices, but we'll focus on the underlying idea first.
Imagine a cube enclosing our entire volume. Inside it, we place a series of flat polygons, or slices, that pass through the volume.
These slices are oriented perpendicular to the viewing direction, so they face the viewer. Each slice is made up of triangles, just like the geometry we discussed earlier.
In the example above, the red box represents our bounding box, defined between -0.5 and +0.5 along each axis. The triangles inside it represent the slices. At this stage, they're just geometry with no texture applied.
The number of slices determines how many planes we use to sample the volume. With only a few slices, we're taking a rather rough look at the data. Increase the count, and we can represent the volume with much greater detail.
Applying the 3D Texture
Now let's give those slices something to work with.
We apply the 3D texture to our proxy geometry. Each slice samples the texture at the appropriate positions, retrieving values from the volume and using them to determine what gets drawn.
When we render a single slice, we see only a portion of the volume. But when we render many slices in sequence and blend their contributions together, they combine to produce the appearance of a 3D object.
Notice that we're not drawing the actual surface of the volumetric object using its own vertices and colours. We're rendering textured planes that sample the data and combine to create the final image.
That's the key difference between this technique and ordinary mesh rendering.
So.. How does it render in the end?
1. Calculate the slice geometry.
During initialization, we calculate the vertices of the slices based on the viewing direction, the volume's bounding box, and the desired number of slices.
2. Update the geometry when needed.
Because the slices need to remain perpendicular to the viewing direction, rotating the camera may require us to recalculate their vertices. Changing the slice count or modifying the bounding box can also require geometry updates.
3. Bind the 3D texture and render the slices.
We bind our 3D texture and draw the proxy geometry. Each slice samples a portion of the texture using its texture coordinates.
4. Blend the slices together.
As the slices are rendered, their contributions are blended to produce the final image. The blending method and sampling settings affect how the volume appears.
5. View the result.
With enough slices and suitable sampling and blending settings, the volume becomes recognizable as a 3D object.
That's 3D texture slicing in a nutshell. We're using ordinary polygons and a 3D texture to visualize data that doesn't have an explicit mesh surface.
ahh.. My fooot 😣
There are other approaches to volumetric rendering, each with its own advantages. While our current example uses 3D Texture Slicing, techniques like Marching Tetrahedra and Dual Contouring take a different approach. Instead of rendering the volume directly using a 3D texture, they extract surfaces from the volumetric data and generate a conventional 3D mesh made up of vertices and triangles.
We explored these approaches while building our own open-source volume rendering engine, implementing different techniques and experimenting with how volumetric data can be visualized. We also worked with colormaps, which map data values to colours to make variations in the volume easier to interpret.
Our Volumetric engine is available on Github, as an application for Windows.
Different Formats of Volumetric Data
Like Raw binary data, there are other Volumetric Data formats which are widely used.
1. DICOM -
DICOM stands for Digital Imaging and Communications in Medicine. It's a standard used for storing, exchanging, and managing medical imaging information.
Hospitals and medical imaging software use DICOM for data from CT scanners, MRI machines, ultrasound systems, and other imaging equipment.
Unlike our headerless raw dataset, a DICOM file can contain information about the image dimensions, pixel data, acquisition parameters, depending on the file and its metadata. For instance, if the DICOM file was obtained from a medical scan, it would also contain metadata like patient details or test metrics.
In other words, it carries more than just the numbers that make up the image. It can also carry important context about where those numbers came from.
CT scan of a jawline, Source: Class 3 malocclusion - Medimodel
2. OpenVDB:
OpenVDB is a little different. It isn't simply a file format; it's an open-source software library and data structure for working with volumetric data.
It's widely used in computer graphics and visual effects, particularly for representing sparse volumes such as smoke, fire, explosions, and clouds.
Instead of storing every possible point in a dense 3D grid, OpenVDB uses a hierarchical structure that can represent large, mostly empty volumes efficiently.
That's especially useful for effects where most of the space contains no meaningful data. There's little point in spending memory on millions of empty cells if the interesting part is just a cloud of smoke.
JangaFX - Real-Time VFX Simulations
There are other formats and tools as well, including NRRD (Nearly Raw Raster Data), VTK (Visualization Toolkit), and MRB (Medical Reality Bundle, used by 3D Slicer).
Each comes with its own structure, metadata, and intended use. Choosing a format and rendering technique depends on the application, the type of data, related domain, and what we want to achieve with it.
We've gone from a grid of raw numbers to a 3D texture, then used a stack of textured polygons to visualize the entire volume.
3D texture slicing may be an older technique, but it's a useful starting point for understanding the fundamentals of volume rendering. It shows how familiar concepts from computer graphics, such as textures, polygons, sampling, and blending, can be combined to visualize something that doesn't begin as a conventional 3D model.
In healthcare, doctors use 3D visualizations of CT and MRI scans to examine organs, identify abnormalities, and plan procedures. In industrial inspection, volumetric imaging can help engineers examine internal structures and detect defects in components without cutting them open. And in VFX, artists use volumetric techniques to create realistic smoke, fire, clouds, and explosions.
Volumetric rendering can get mathematically involved, but the underlying idea is fascinating: we can turn millions of values describing a three-dimensional space into something we can actually see and explore.
And that's a good place to start with the volumetric journey.