Your question really belongs in our Movie Copy forum, so I'm going to move it in there.
It is quite possible that this movie has an advanced protection scheme that your current programs cannot handle well. I suggest getting the trial for DVDFab and use that to decrypt and rip the movie to the hard drive.
While you have the trial, the entire program will work for you, so you could use it to compress as well. When the trial ends, there is one section within DVDFab that will continue to function as a dvd decrypter and ripper. It is called DVDFab HD Decrypter. The only caveat to using it is that the free section isn't updated quite as often, and sometimes will not work with the very newest releases.
So, try DVDFab and see if it works for this movie. Once the trial ends, you can still use the program to rip to the hard drive, then compress with something like DVDShrink, which is still a good transcoder, just not effective against advanced protections.
Another free option is BDLot DVD ISO Master. It rips the movie to the hard drive, removing protections, but outputs as an ISO file. You can open an ISO file directly with DVDShrink by clicking Ctrl I.