This is going to be a very quick look at the 9th generation HEVC NVENC encoding engine found on the Nvidia’s flagship RTX 5090 compared to 8th generation found on a RTX 4080 Super.
Encoding Only
For this test we are only going to be looking at the quality of the encoder, not the decoder in any way. In other words, we will be using a raw video stream as input to do an encode, instead of providing a familiar format like a mp4 file which would also require decoding. When using both it’s called transcoding, which we are also purposefully avoiding.
The Contenders
The Challenge
Every type of benchmarking we have done on this site focuses on high resolution videos with the aim of maximum quality, and we’re sticking to that formula today. We will be using the 1 minute 24 second long Butterfly video in 2160p HDR format. It has been converted to a raw yuv file in y4m format.
ffmpeg -y -i '.\Butterfly HDR Rec.2020 HEVC.mp4' -map 0:v:0 -strict -1 butterfly.y4m
The Setup
Hardware
- CPU: AMD 9800X3D
- Motherboard: Asus ProArt X870E-CREATOR WIFI
- Memory: 96GB DDR5 V-Color ECC 5600MHz
- Disk: 8TB WD BLACK SN850X
- PSU: Be-Quiet Pro 13 1300W
Software
- OS: Windows 11 Pro – 10.0.26100
- Nvidia Driver: 572.16
- FFmpeg: 7.1 built by BtbN
The Parameters
For every encode, we will be passing the same parameter set to ffmpeg to the hevc_nvenc codec. We are using the slowest preset, in the newly released Ultra High Quality mode.
f'"{ffmpeg}" -hwaccel cuda -hwaccel_output_format cuda -i "{butterfly_raw}" -map 0:v -c:v hevc_nvenc -preset p7 -b:v {bitrate} -bsf:v hevc_metadata=transfer_characteristics=16:colour_primaries=9:video_full_range_flag=0:matrix_coefficients=9 -multipass fullres -lookahead_level 3 -g:v 120 -keyint_min 120 -b_ref_mode middle -temporal-aq 1 -no-scenecut 1 -tune:v uhq -rc vbr -rc-lookahead 250 -profile:v main10 -pix_fmt p010le -fps_mode cfr -f hevc -y "{output}"'
The Results
VMAF Averages
Across the board, the average VMAF score has increased with the newest 9th generation NVENC in HEVC. The largest gains are at smaller bitrates, and the gap closes as they increase.
This results in a bdrate of -5.59%, meaning we can use five and half percent less bitrate to achieve the same results.
VMAF Lowest Frames
The most incredible part about the encodes is the improvement on the worst frames. This had an inverse relationship than the average, with the largest gains being with the larger bitrates. Jumping five VMAF points is huge!
Bitrates
| Target (k) | RTX 4080 Super | RTX 5090 FE |
| 2,000 | 2,440.62 | 2433.45 |
| 4,000 | 4,579.86 | 4578.86 |
| 6,000 | 6,656.18 | 6677.72 |
| 8,000 | 8,646.42 | 8645.85 |
| 10,000 | 10,544.21 | 10531.34 |
The bitrates were impressively close to each other, but is clear that there has been no improvement to staying close to the requested bitrate.
The Conclusion
The newest 9th Generation NVENC has a nice bump in video quality from the previous gen. This is obviously not a video encoding specific card like the AMD MA35D or the Nvidia L4, but it’s great to see high quality encoding available for the everyday user, as this NVENC will be available across most of the 5xxx range.
The Raw Numbers
RTX 4080 Super RTX 5090 FE
Bitrate K VMAF Bitrate K VMAF
2440.62 68.189246 2433.45 69.873222
4579.86 81.81946 4578.86 82.921356
6656.18 87.720977 6677.72 88.231568
8646.42 90.713858 8645.85 91.034569
10544.21 92.615648 10531.34 92.852886
20083.53 96.63489 20025.33 96.769932
