Which will provide better quality rips?

The quality of the video and audio are not determined by these factors. MKV and MP4 are container formats, which can hold many different types of video codecs, and the video can be encoded at virtually any level of bitrate per measure of time, which is what will determine the overall quality of the video.

AC3 and AAC are different types of audio, but are generally comparable to one another, and they are both lossy formats. (though True HD AC3 is sometimes seen in blu ray and it is a lossless format). AAC is the default format for many types of portable players, like iPads for example. AC3 is the format most often seen in dvd video.

Which you choose depends, in part, on the type of device you will be playing the video.

Because AAC is more advanced lossy audio format than AC3, it will produce better audio quality than AC3 with [U][B]same bitrate[/B][/U], especially on low bitrates. With that said AC3 is more widely supported, so like Kerry said depends on what you are using it on.