I don’t use Mac’s so I’ll be of little help. If you are converting to mp4 straight from the disc, you might want to do an intermediate step instead. Rip to the hard drive without conversion. You should have the complete DVD on the hard drive in its original format. Check to make sure you have correctly ripped from the disc by playing back the episodes in question, then convert from that point.
Once decrypted and on the hard drive, you should also be able to convert with another tool if necessary. Handbrake has a version for Mac’s.