There are only two ways of reducing the size of your movies. You can remove sections that you don't need, like foreign soundtracks, directors commentary and other types of extras. Or you can use compression to reduce the size of the movies (and you might need to do a combination of these two things). Compression is always going to compromise the video quality. If done well, you might not notice the lesser quality.
What kind of format do you want for the output? Since you have mkv in your thread title I assume you mean H264 encoded mkv files, but it would help to be sure about that.
There are any number of good, free programs for this process, but the free ones need decrypted video to start with. So always rip the movie to the hard drive before converting the video.
Depending on the output format you want, look into BD Rebuilder, MultiAVCHD, and Handbrake. All can produce .mkv files using the X264 encoder....if that is what you are after.
Edit: Forgot a few other tools, like RipBot264, StaxRip and BluRip.