512 up are you sure? I have yet to see a domestic ISP exceed 364 up let alone get to 512. What do you have your connection limits set to? When giving advise on ed2k side of things it is recommended to keep max limits to 80% of line bandwidth, to allow for overhead created. Allowing the full bandwidth can actually slow your speeds down.
Is your hardware a modem or router? I know you say modem in original post, but I have seen people confuse the two before. I am shooting in the dark as I don't use torrent, but with using and advising on the use of clients on the ed2k network, there can be some similarities, it is recommended to reduce conections to less than 250 for routers as otherwise you risk flooding the router.
When you say you have tried other firewalls alongside torrent. Was ZA still installed?
If it is happening with other firewalls, then it sounds like you may have bandwidth throttling, this can happen with the common/default ports used with p2p clients, even more so if your ISP has a fair use policy. Try changing the ports.
Again everything I am suggesting is from my experience on ed2k, therefore some may not even be appropriate for torrents.