You'll need two programs for this, assuming you are starting with a commercially made Blu Ray movie. The first is AnyDVD HD, which is a rather expensive decryption program, but the only one that can deal with blu ray movies with BD+ protection.
The second is BD-Rebuilder, which will do the compression for you. You can download it from the first post in this thread at doom9.org: http://forum.doom9.org/showthread.php?t=143716 Read the first post carefully to make sure you have all the necessary other parts installed correctly (ffdshow, Haali Matroska splitter, AviSynth). All of these are free to use.
The H264 codec has excellent compression qualities, but you will always lose some detail in compression. You probably won't notice it in playback.