No, you don't work with ISO files using zip/compression utilities. If you want to burn to another similar sized disk, you can do this directly with ImgBurn. If you simply want to access the files within the ISO, you mount it in a virtual drive, like the free one from Slysoft called Virtual CloneDrive.
tsMuxeR and ClownBD will not re-encode, but they don't have the fine control you want over the menu structure. I tend to eliminate menus whenever possible, so this is not something I do, even with dvds.
If there are some authoring tools for working with blu ray menus, they are either commercial or simply unknown to me. Someone else with more knowledge of the subject may come forward with more information for you however.
You might want to look through the available tools for blu ray listed at videohelp.com.