Absolutely not - This option in nero and the booktype are 2 different things Leaving that option checked will NOT affect your booktype setting, it's simply an option to create compatibility borders and padding your DVD ROM to make it look bigger than 1GB for compatibility issues, your booktype will remain what you set it to.
I have personally tested this myself and can confirm it. Here's how I set things up.
I'm using 811S (real retail) with HSOP and official booktype 1.4 from LiteON. I set booktype to DVD-ROM, once it says that the change is done I close the booktype program ( you don't need to leave it open ) I then open up nero, make sure the dvd high compatibility mode is enabled, and I burn away, nero will determine for you if this option will be used or not, so you might as well leave it on.
Someone reported earlier that this option is for XBOX and also works for non dvd video - so the best thing for you is to run some tests with small dvd video discs (< 1GB) and see if it plays on different DVD players. If you have no compatibility problems you might as well disable it, if it works for you, and it will take much less time to complete a burn.
I may have made a mistake earlier, the option is dvd high compatibility, and is not strictly reserved for dvd video if I'm correct, so sorry about the confusion.