Recode (and sometimes other programs) won't compress past a certain point. What you can do is run 2 passes on it. Go ahead and compress it as much as you can and have it output to files somewhere on your harddrive. Then run it again against those files and it will bring it down to fit on a disc.
Example: 8GB down to 5GB, 5GB down to 4.36GB to DVDR.
You may also be able to unselect some audio tracks (foreign language, commentary) of some extras (trailers, making of.., etc.) and get it to fit with 1 pass. But, I have had a couple of discs that I had to run thru twice to get them to fit.