Guide to the world of NVIDIA technology: DLSS 3 and Frame Generation – extra frames for smoother gaming – Chapter 1

With the latest generation GeForce RTX 40 Nvidia also introduced the DLSS 3 frame generation technology – or if you prefer DLSS 3 Frame Generation, which inserts a frame generated with the help of AI between the frames rendered by the graphics card. This is another option to significantly increase the frame rate with the use of artificial intelligence, even if the game does not manage to render the image satisfactorily quickly with the graphics settings used, because it is limited by the hardware performance.

Interestingly, the game doesn’t have to interfere with the creation of cutscenes. Image generation thus helps raise frame rates, regardless of whether the game is held back by the processor or the graphics card. This is especially useful in situations where the processor is at fault. If it does not perform the necessary calculations fast enough to send the necessary data to the graphics card, you will no longer achieve significantly higher frame rates either by reducing the details in the graphics settings or by reducing the resolution.

How to get an extra frame?

Image generation is an evolutionary leap from the original DLSS. With it, the game helps to achieve higher frame rates while maintaining high image quality by reconstructing the samples needed to render a high-quality image in native resolution from images with a lower resolution (that is, with a lower number of samples). It obtains the required number of samples by taking them not from one but from several images.

NVIDIA goes even further when it comes to image generation. It uses the last two images in high quality to insert an additional image rendered using artificial intelligence between them.

The beginning of the process can be the same as with DLSS – from multiple frames rendered with a lower number of samples and a motion vector map, the game renders the frame in native resolution. The last two of these images are processed by the optical flow accelerator, which builds a map of the optical flow in the scene. And it will use DLSS together with the finished images to generate a new intermediate frame.

Why does DLSS need it? When you want to insert an extra one between two frames, you need to know what moved between those frames in the image and where. The motion vector map serves the same purpose, but it is not perfect. It contains things that enter the image only during rendering. These can be elements that are calculated aside due to optimizations and applied to the scene only afterwards using maps, such as shadows, reflections or lights.

An example is the shadow of a motorcycle driving down the road in the following image. The neural network receives a map of motion vectors from the game during image processing. It captures the direction in which the objects in the scene are moving. DLSS learns from the map that the road under the motorcycle is going backwards. And he will want to move her in this direction. Along with the texture of the asphalt, he may also want to move the shadow that the motorcycle casts on the road to the rear. This creates a ghost that trails behind the rider similar to the following illustration:

Only asphalt should run properly. The bike is fast, but not fast enough to outrun its own shadow. And this is exactly what the optical flow accelerator can detect when analyzing images. It compares the two images, looks for matches, and generates another map in near real-time that records the movement of pixel regions in the image. In doing so, it is already based on ready-made renders, the analysis of which determines where the movement in the image differs from the movement vector map.

It can then combine the two maps into one, which more accurately captures what has moved in the image and what, on the contrary, should not move, and transmits the information to the neural network. Thanks to this, the movement between frames can be estimated more accurately and a higher quality intermediate frame can be generated.

In games, it is necessary to do such an analysis really quickly. If at 100 frames per second the analysis of the images took longer than the rendering, their generation would not be very valid. In order to achieve the desired quality at the same time, the latest generation GeForces have enhanced computing units that perform the analysis (the Optical Flow Accelerator). And this is exactly what NVIDIA states as the reason why only the latest generations of GeForce from the RTX 40 family handle the generation of DLSS 3 images, and you cannot count on it in older models.

There is one more problem you have to deal with when generating a snapshot. If you insert a new frame between two finished ones, it means that you have to delay the rendering of the last rendered frame, show an intermediate frame instead, and you can render the most recent frame after it. To at least partially compensate for this NVIDA delay, developers integrate Reflex technology into games with image generation at the same time. We will discuss this in more detail, for now I will only briefly remind you that it helps to better synchronize the work of the graphics card and the processor and shorten the queue for image processing. The scene calculation processor waits to supply the graphics card with “last minute” rendering data just before the card can start rendering a new frame.

DLSS 3 Frame Generation thus benefits from a combination of Nvidia’s software and hardware technologies – DLSS super sampling to increase frame rates, reflex to shorten the response and frame generation itself on the software side and on the hardware side from cores for tensor calculations, an optical flow accelerator and NVIDIA training supercomputers neural networks that DLSS uses.

We will look at what you can expect from this technology in practice on the next page.