Why the future of 3D graphics is in high-resolution textures

This article investigates the importance of high-resolution textures in achieving the best visual quality for gaming, AR, and VR applications and what role texture streaming systems such as Granite SDK play in this ecosystem. We will investigate how modern real-time engines can benefit from high-resolution textures, how these textures can be created cost-effectively, and how these textures can be visualized in real-time without sacrificing performance or increasing texture memory use.

Download Granite SDK whitepaper

Challenges in modern physics-based game engines

Textures play an important role in modern physically based game engines such as Unreal and Unity. Although modern game engines are moving more and more towards dynamic scenes where everything is calculated on the fly using complex shaders, textures still remain an essential building block of the look and feel of modern renderers.

Physically based lighting – offloading the material system

The increased use of physically based rendering in game engines has resulted in more textures being used by the material system since the raised number of material parameters needs to be driven by high-resolution input textures to reflect detailed and varied materials.

The Unreal 4 engine has an advanced shading model which in all but the simplest cases requires at least three inputs to be specified (base color, roughness, normal). However, when using the full potential of the shading model, several additional maps may be used such as displacement, metallic and self-illumination maps. Finally, Unreal 4’s flexible node-based shader interface makes it very easy for artists to freely use many additional maps to achieve custom shading effects.

Similar to Unreal 4, the Unity engine also requires several high-resolution maps when using Unity’s physically based renderer. Unity’s default shader supports up to 9 different parameters that can be specified as textures. Additional features such as displacement mapping may even further increase the number of textures used by the engine.

Baked Information – preventing visual lighting artefacts

Besides textures defining the separate materials in the scene, modern renderers also use textures for a lot of additional information. For example, Unreal 4 stores the following additional information on textures.

  • Light maps
  • Shadow masks (pre-calculated shadows for use with real-time lighting)
  • Ambient occlusion maps
  • Reflection probes
  • One important drawback of such pre-baked maps is that while the regular material textures can be tiled and used on many different object instances in the scene, baked maps are necessarily unique for every object and surface in the scene. To prevent this constraint from stressing the texture budget too much, engines usually resort to using very low resolution baked lightmaps. However, this may lead to visual artifacts in the lighting. Using higher resolution textures for lightmaps may easily increase the visual quality of complex scenes as shown in the screenshot below.

    Figure 2: Notice how the quality of the shadows cast by the wooden sun blinds in this architectural visualization is insufficient for the default quality settings. Doubling the resolution of the baked textures resolves the quality issues.

    Producing Detailed Material Textures

    While increasing the texture resolution of baked textures by simply updating the baking parameters, increasing the resolution of artist-authored material textures is obviously not so trivial. However, producing high-resolution textures to drive modern renderers has never been easier. There are many tools out there that allow authoring high-resolution textures cost efficiently.

    Scanning and Photogrammetry – speed-up the digitalization process while keeping the details

    Perhaps the most obvious use of high-resolution textures is using them to represent all the detail and uniqueness present in real-life scenes. Through the use of 3D scanning and/or photogrammetry it becomes possible to capture such real-life scenes in an automated way. Although some manual clean-up of the captured data is usually still required, it is much faster than manually modelling the whole scene based on traditional photographs and plans. Capturing tools are also likely to improve even further in the future, allowing real-life scenes to be captured with little or no manual work.

    Nu Reality Desert Home is a perfect example of the photorealism that is achievable when combining scanned environments with high-resolution textures.

    Painting textures From Scratch

    Several tools allow artists to directly paint on a 3D model. The most well know texture painter’s tool Photoshop has had 3D mesh painting support for several versions now while tools aimed specifically at texture creation such as Mari have added extensive feature sets aimed at VFX content production. One notable feature of Mari here is its focus on very high-resolution textures.

    Finally, there are a number of tools that are specifically aimed at creating physically based material textures for modern real-time renderers. Notable examples are Allegorithmic’s Substance Painter and Quixel’s DDO Painter. They work usually at a higher conceptual level than the aforementioned painting tools. Instead of modifying pixels the artist immediately modifies materials such as “leather”, “wood” and “gunmetal”. The tool then automatically updates any underlying texture maps such as color and normal maps. These tools are also specifically tailored to real-time physically based material workflows and shaders. Furthermore, they allow semi-procedurally compositing the material layers based on 3D model parameters such as edges, curvature, … These together with other procedural techniques allow artists to quickly create unique materials tailored to the underlying 3D model.

    To handle the large amount of textures that are typical for VFX-quality UDIM textures (e.g. the Robot in the Rise demo by Nurulize has 700 4K maps), only the subparts of the UDIM patches that are actually visible at a given time can be loaded. This way, you can see the textures at the highest quality without any issues.

    Advanced geometry-based tools such as for example curvature masking (shown here in Substance Painter) allow quickly adding a worn look to the uniform’s exposed edges by combining two materials.

    Using High-Resolution Textures in Real-Time

    High-resolution textures are no longer a future technology but a reality and are found more and more in games and applications for the current generation of platforms and engine technologies. Managing these high-resolution textures becomes a critical factor. Fortunately, technology and tools are here to help create the best and most stunning high-resolution experiences.

    Traditional Texture Streaming – visual artefacts and limitations of the memory cache

    Texture streaming is the process of loading texture data from disk in the background while playing a game. Only a small subset of the textures is actually kept in memory. The streaming system decides automatically when to load specific textures from disk, or the program explicitly requests textures to load or a combination of both.

    For example, Unreal has its own texture streaming system that automatically loads textures based on a number of parameters, with the most important factor being the distance to the camera. Unreal will load a new mipmap of a texture once the camera gets within a certain distance of that texture. Such a system enables large open world games. However, having dense scenes is challenging. Dense here means many objects closely placed together or many textures per object. If too many textures are used in one specific area of the virtual world, all of those textures get close to the camera at a specific time and need to be loaded by the streaming system. This can result in visual artifacts like popping and blurring. While it is possible to set the size of the memory cache higher, this solution obviously has its limits.

    High-Resolution Advanced Streaming – 8K to 256K textures within reach

    Traditional texture streaming systems don’t scale with larger textures. Once you start to use higher resolution single textures such as 8192x 8192 pixels and up, the size of a single mipmap becomes so large that streaming it in becomes a big burden on the streaming system. To avoid loading full-mipmaps of a texture, a new approach is needed. Without going too deep into the technical details, advanced streaming systems such as Granite will divide all of the mipmaps of textures into small texture tiles, usually 128x128 pixels. While running a game, the system will automatically figure out which tiles are actually visible by the camera and load only those tiles, as only parts of a texture mipmap are visible in most cases. For example, only the top is visible from the airplane wing in the image below. While in a typical streaming engine all of the textures of the plane would be loaded (8K diffuse, 8K normal and 8K specular), an advanced system will only load the tiles that are actually needed. This results in much fewer data to load, allowing you to use less texture memory which achieves a higher effective texture resolution.

    One major benefit from the use of virtual texturing is the possibility to use much larger textures, even up to 256Kx256K. A texture of that size compressed with DXT1 would occupy 32GB of memory, and loading that texture would take minutes. But for example with Granite, only 32MB of VRAM for that texture needs to be allocated, and the loading time is unnoticeable.

    One thing to take into consideration is that Virtual Texture sampling adds a small performance cost compared to regular texture sampling. For example, Granite for Unreal limits the amount of VT textures to 16 per material with a maximum of 4 different UV coordinates to sample those textures. This way, we can make sure that the performance impact of the VT samples is almost unmeasurable in most practical scenarios (obviously, regular textures are still available if 16 textures are not enough for a particular material).

    Compression – Minimize disc usage while maintaining maximum quality

    Although texture streaming solves the problem of the run-time memory use of the application, one other major challenge when adopting large textures is the memory use on disk. This is another area where a mature streaming system like Granite has done all the hard work. Granite supports an in-house developed custom (de-)compressor which can deliver high-speed decompressed results in popular GPU formats such as BC5 and BC7. In addition to a custom codec (which adds a small amount of extra loss on top of the basic BCN block compression), Granite also contains support for streaming BCN data directly giving results visually indistinguishable compared to traditional game textures.

    It is important to note that Granite is not ideally suited as a stand-alone texture compression product. The texture compression is mainly there to help alleviate the burden of using a lot of high-resolution textures. However, when choosing between popping-free high quality streamed results or saved disc storage, quality will usually be the preferred choice.

    Integrated or standalone solutions provide easy transition.

    Switching to a tiled representation for all textures and using a stronger compression format requires some changes to how textures are managed and imported in the engine. Fortunately, tools such as Granite provide an easy to use, yet powerful, toolchain. These are available as pre-integrated tools for game engines or as stand-alone GUI based tools with CLI support which can be deployed on cooking servers or other automated processes.

    For popular 3rd party engines, such as Unreal, a pre-integrated version of Granite will automatically take care of importing textures in Granite streamable format, and adjust materials for rendering from it.

    For example, to stream a UE4 texture through Granite, the only step to take is to replace the original texture sample node with the GraniteStreamNode. Granite automatically picks up all UE4 textures behind each GraniteStreamNode, builds the corresponding Tile Set and updates the material to use the Granite sampling methods. The integration enables streaming in the editor and automatically adds all Tile Set files to the final game build. More information on UE4 integration can be found here.

    Adopting high-resolution textures in a controllable way

    Granite SDK delivers on the promise to continuously improve quality, use memory more efficiently and deliver tools that are easier to use.

    High-resolution texture scalability for AR and Mobile

    On mobile, scalability can be improved in a number of areas, making it easier to target memory constrained systems as well as very high-end professional GPUs with Gigabytes of texture memory from a single set of tile sets and textures.

    Augmented Reality applications such as our own AR Horse, benefit greatly from high-resolution real-time texture streaming. With mobile AR, we can now anchor virtual objects to the physical world and view these blended worlds through our smartphone and tablet. However, to blend these worlds believably, we need the virtual objects to have the same level of detail as the physical world. This is extremely challenging. It requires a tremendous amount of artistic skill and technology to create these virtual objects. And it requires technology that can actually handle and render these objects.

    Smaller storage footprint, more quality

    The Granite optimization technology allows automatically optimizing textures based on their actual use in the scene. Textures authored at a too-high resolution (i.e. which aren’t ever visible in the application) will be automatically optimized so they don’t take up storage space and the Tile Set becomes smaller on disk. In some customer applications, we have noticed that only 10% of texture data is ever visible in the application at runtime.

    For example, our experimental Tile Set optimizer was executed on Epic’s A Boy and His Kite demo. This demo has a two minute long cinematic showing a 100-square-mile landscape with many photogrammetry-captured textured objects. Our optimizer removed all texture detail that isn’t shown in the cinematic. From a 4.5GB Tile Set, we went to a 230MB Tile Set. We replaced the original tile sets with the culled version, replayed the cinematic, and the output was visually identical.

    The benefits are clear, loading times decrease significantly, the game can be played instantly and download time is drastically reduced.

    Conclusion, use high-resolution textures with the current state-of-the-art tools and workflows

    Large textures can efficiently be created and used in real-time rendering applications through the use of texture streaming systems. A number of tools, specifically aimed at artists, exist which can be used to create high-resolution physically based material textures. These tools allow high-resolution textures in real-time engines using established techniques on current generation hardware and engines while lowering the impact on your production workflow.

    Download Granite SDK whitepaper

    Leave your information