Just to check; you are trying to make the change before the burn and not after (because it only works before), right?
Okay, um, it's likely an ASPI issue, as that's really the only thing that can cause the official booktype utility to fail. I've never been good at handling them, but I think you should be able to find some info on troubleshooting this with a forum search.
Another possible workaround is to use DVD Decrypter's booktype setting tool, assuming that you can manage to get a copy of it, as I've heard of cases where that would work even when the official tool didn't. I haven't personally tried this out, though.
As for Nero and Nero CD-DVD Speed's booktype controls... the last time I tried them (about half a year ago), they were buggy and didn't really work. Maybe they've fixed it, but I never use their booktype controls.