I'm no expert, but the documentation states this:
Can I format a disc with UDF?
It is not possible to format a disc with the NeroAPI.
However, I see that the ISOTrack BurnOptions allow you to specify creating a UDF track. you might want to investigate that option.
As far as the file lengths. Have you tried to specify the following?
How do I create a CD that is Windows readable with long file names?
Use all of the following options: