>>>>Is there a setting in DVD Decrypter to rip as an ISO?<<<<
YES, and that is what you need to use, if you want to do dual layer exact backups.
Look at the menus at the top. One is "Mode." Under that, choose "ISO READ."
Then, once it has been ripped, you go back to that same menu, and choose "ISO WRITE." Navigate to the correct directory, find the ".MDS" file, and choose it.
You were ripping in file mode, which will not work with dual layer backups.