I think you're possibly looking at this from slightly different angle to me.
Apologies if you're already aware of a lot of this but I thought you'd find it interesting.
Whether it's transcoding or a full re-encode is in my mind not dependant on the software you're using, the resolution, or the eventual codec, it depends on where you get the parameters from for the compression process and how they're used.
Transcoding retains the existing motion vectors and compresses by re-quantising the DCT (Discrete Cosine Transformation) coefficients.
Quantisation is the process of lessening the accuracy of the bitstream to save data and you can set this at different levels for a corresponding variation in compression (which is basically what the 'Q-value' is that you sometimes see referred to, usually in connection with discussions regarding encoders).
The DCT coefficients describe an individual frame based on the macroblocks that make it up and the motion vectors determine the relative position of these macroblocks between frames.
As transcoding doesn't recalculate the motion vectors they end up out of sync with the corresponding DCT coefficients.
This is the primary reason for the higher levels of pixelation or blocking for transcoded video compared to fully re-encoded video and is also why it's more pronounced at higher levels of compression since the higher the compression, the more and more these values get out of sync.
Amongst other things a full re-encode generates new DCT coefficients and from these new motion vectors so these are always fully in sync which yields better quality as a result and this is especially noticeable with rapidly changing scenes.
Converting between codecs using transcoding algorithms is an example of format shifting transcoding.
What I'm not sure of is whether the methods used by DVDFab for Blu-ray to DVD fall into this category or not.
The change in resolution would require recalculation of the motion vectors but this, in theory, could be carried out in a number of ways so they could be transformed, or resampled or even completely recalculated from scratch.
If the existing values are processed or transformed in some way then it's transcoding and that's the reason for the lack of quality.
If they're completely recalculated along with all the other relevant data then that's re-encoding and the reason for the lack of quality as you say must be down to the settings.
Without knowing the exact methods used though it's very difficult to speculate.