I've moved your question out of the BD Rebuilder guide and into this new thread.
When you start seeing blocky artifacts in dark areas of your encodes, it means you are pushing the limits on the encoder. You are probably using too little bitrate for the amount of compression necessary, in order to hit a specific size output.
You haven't mentioned your target size. Are you compressing blu ray movies to fit onto a single layer dvd? If so, you might try reducing the resolution. In Settings-->Setup, put a check mark in the box to change resolution from 1080p to 720p for extra large blu ray movies when going to dvd-5 size. I would also recommend doing a movie only backup, instead of trying to save the menus and extras.
You may not need to do this for every encoding job, so be aware of the settings that you choose before going to the next conversion.
Another thing that can affect the looks of the video is an improperly calibrated tv screen. This can be part of the problem if you are using a large screen lcd or plasma.