the “smearing” effect is GMC and/or use of non-compatible mpeg-4 matrices, or mixing certain ones in 1st/2nd pass.
quartel pixel resolution (q-pel) is not supported in most players (new mediatek chips do), player will simply not load file if encoded with q-pel enabled (this is not software issue, but hardware, chip related and no updates will fix that). GMC is not supported in most either and those that support it have certain limits, eg warp 3 won’t play.
b-frames should work ok on most with enabled packed bitstream, even though advanced simple profile (ASP), one should test for a specific player and see how it handles them.
audio desync and “shuttering” often would be caused by improper audio/video interleaving, especially for AC3 (easily fixable … no need to re-encode)
480kbps video is way too low for full screen playback, no matter what. u need higher bitrate and proper inclusion of filters that would help compression at a minimum loss of detail. further make sure you deinterlace if interlaced source and for R1 in most cases you would need to perform IVTC.
i’d recommend the use of xvid since open source and plenty of enthusiasts out there, as well as guides and help. it is fully mpeg-4 compatible and will play in divx players. still this is a personal preference.
probably easiest program for decen quality is auto gordian-knot, for better requires somewhat more reading and learning but this does quite a good job. places like doom9 have easy and comprehensive step by step guides.
hope this helps some.