Staxrip - what settings to use for HQ Nvidia x264 encoding with GPU

i usually encode with staxrip using the standard x264 encoder which users the CPU.,
Some of the main Settings i currently use for x264 encoding 1080p are Below and i get some dam good results from them and would like to try and match these settings below but using Nvidia GPU x264.

Usually use the Bitrate or 2 pass setting using slower or very slow preset and a bitrate of 8000 - 10,000.
High Level 4.1 , 8bit
Uneven multi hexagon search , subme 10.
B adapt = optimal , B frames 3 , Ref frames=4, Lookahead 80
Colour primary= BT 709
With an MKV container and Ac3 640 audio.

But now i have a new PC with a Good GPU, and i want to start using the GPU to do the encoding rather then all the CPU. So i have looked at the settings a bit and dont know if i can apply the same settings there, and there are some which i do not know what they should be, if someone can help me use the correct settings.
by default some of the settings i would normal use are not selected, so i am a little confused as to why and if its ok for me to change them or are they unselected for a reason. That where your help comes in.

Here are some screenshots .

I’ve been playing with Traxrip and discovered the tools it uses for NVidia H.264 and H.265 is the same as those used in HandBrake. You just have a different user interface.

What you’ll have to remember is instead of using software for rendering, when you use NVidia you are using the GPU hardware for rendering. Therefore the NVidia GPU driver has already been optimised regarding a balance of performance and quality.
You shouldn’t really have to tweak much to get good results.

Here are the settings i used for encoding in Traxrip (Nvidia H.265).


I prefer to still do the decoding in software and only use the GPU hardware for encoding.
The only tweak I made was selecting the quality profile.

Below is how the encoding used the hardware.

To me, the quality was good to my eyes.
Give it a try.

ok ill give it a try .
just a few questions, Due to me not being familiar with win10 and now in the process of changing the way i encode from CPU to GPU, i have these questions.

i notice your GPU is 99% is this totally normal to be this high ?

Also in the screenshot of yours above , in the decoder tab, did u see that it says NVenc in there , should it be set to that, or leave on avisyth/vapoursynth, or is the decoding part of it not really necessary to put it to NVenc.

also i did an encode the other day with CPU and i notice the power usage in task manager for staxrip is marked as "very High " , just like to know a bit of whats normal as these specs were never displayed in win7 so a little bit of a learning curve for me . will do GPU encode soon, and hopefully from then on .

Edit , just did a test encode with small file using all the defaults with Nvidia x264 encoder, , getting this error below. staxrip doesnt engage in encoding ,

Error Video encoding (

Video encoding returned error exit code: 1 (0x1)

It’s unclear what the exit code means, in case it’s a Windows system error then it possibly means:

Incorrect function

--------------------------- Video encoding ---------------------------

NVEnc 5.04

“C:\Media programs 1\StaxRip x64\StaxRip\StaxRip-x64-\Apps\Encoders\NVEnc\NVEncC64.exe” --cqp 18 --preset quality --level 4.1 --lookahead 32 --videoformat ntsc --colormatrix bt709 --colorprim bt709 --transfer bt709 --colorrange auto --no-deblock -i “C:\Tempo\Marv - 1920 X 1080 ( 37 Sec )_temp\Marv - 1920 X 1080 ( 37 Sec )_new.vpy” -o “C:\Tempo\Marv - 1920 X 1080 ( 37 Sec )_temp\Marv - 1920 X 1080 ( 37 Sec )_new_out.h264”

C:\Tempo\Marv- 1920 X 1080 ( 37 Sec )_temp\Marv - 1920 X 1080 ( 37 Sec )_new_out.h264

“???o” is only supported with avsw/avhw reader.
failed to initialize file reader(s).
Failed to open input file.

StaxRip.ErrorAbortException: Video encoding returned error exit code: 1 (0x1)

It’s unclear what the exit code means, in case it’s a Windows system error then it possibly means:

Incorrect function

Just did another test encode, read below

in that screenshot of yours above, where it says “Decoder” i asked if it should actually be NVenc or leave it at the default which is --> avisnth/vapoursynth, leaving it at default gave me that error list above, so i changed it to NVenc and it worked, but there are 2 options there for NVenc, one is Hardware , and the other is software , do u know which i should have it on. i know yours worked as is , without changing it.

Yes perfectly normal during an GPU encode. If you look closely at my screenshot above, you can see that only the ‘video encoder’ engine is engaged. Probably more important is the PC is only using 17% CPU, and none of the raster processing parts of the GPU. This will leave the PC free to be used for other things whilst encoding. Which I believe was one of the things you wanted to achieve when buying your new PC.

I used it exactly as in my screenshot above.

Regarding your errors.
It would appear you don’t have the HEVC extension pack installed on Win10.
The pack is not included in Win10. You can get it from the Microsoft Store, but it isn’t free.
The good news is, at least on Win10 Pro, is it only costs £1.
Just search for HEVC in the MS store. It’s the Microsoft version you want.

i just did a bit more testing before buying that codec,
i am actually able to do Nvidia x265 encoding , but its the Nvidia x264, that is giving me that error, which is what i was testing when it gave me that error listed above.
Hevc - thats for x265 encoding correct ? where as i am doing x264 encoding , so i would not need that HEVC video extension to do x264 encoding, is this correct.

Your graphics card has an NVidia Turing TU106 GPU.

If you’re using the GPU for encoding, both H.264 and H.265 use HEVC, but should be handled by the NVidia driver.
If you want to do the decoding in software then you require the HEVC extension pack.
You may also need the extension pack to play back videos encoded using HEVC depending on the application used for playback and the codecs installed on your system.

i can playback all hevc stuff already , thats the thing i am not understanding, i can encode in 265 and play it back no issues. even if i download some sample in hevc i can play them all ok.
in that second screenshot when i selected NVenc hardware it worked , also when i selected NVenc software it worked. so would this mean i dont need that pack from windows ?

It sounds like you have some piece of software that already has a HEVC playback codec already installed. Maybe this is causing a problem with Avisynth?

All I can say is the MS HEVC extension pack works with everything I use.

What app do you use for video playback?
Try playing an H.265 HEVC encoded video on Windows media player

i use Pot player, and VLC player also works playing it.

windows media player classic that is already installed on this new pc plays it too.

but i have not installed the other main windows media player, as i never use it .

should i see if i have the latest avisynth ,

Try playing an HEVC encoded video in Windows media player. Both H.265 and H.264. If it can play them it should work with Avisynth.

yes i just installed Windows media player , it it also plays everything .

Then I have no idea why Staxrip fails with an error when decoding with Avisynth. The version I have was the one in the latest package
It works perfectly well here with the MS extension pack. Not saying you should go ahead and purchase it. Just saying it works here.

If you’re happy doing the decoding on the GPU as well, then its something you don’t have to worry about.

ill see how i go with it, if all fails ill look at buying that hevc.

so from that second screenshot where it has the 3 options with the red lines pointing to the decoder, which one should i select , they do all work.

also is selecting one of them any different to not selecting one

Only the HVEnc Hardware will use the GPU
Try the different ones and see which gives you the best result in quality without having to much of an impact in speed.
Decoding an 1080P video should be quite easy to handle using software on your CPU.
Its the encoding part you really want to offload to the GPU.

If your going to use H.264 (Nvidia H.264) the encoding quality will be awesome, but it does create quite a large file. So also try H.265 NVidia as a quality comparison. H.265 will make a smaller file, but will compress the video more.

ok ill see how i go, didnt think one little error can cause this much problem.

so that error code its giving , is that definately related to HEVC.

Its the only thing I can think of.
I’m assuming you installed the later VC distribution library DLL file when you first run Traxrip on your new PC? It should have prompted you to do so.

im sure i would of , in programs and features , the version i have are
c++ 2010,
2015-2019(x64) , 14.27.29016
2015-2019 (x86) 14.26.28720 is the latest i have.

Sounds right.
Just play with Traxrip and see how it goes.
I’m sure you find a setting that you’re happy with.

ok , thanks.

hi Dee,

found this page, and under VUI in Staxrip i unselected colourrange from auto to undefined , and now it works.