Jdobbs has mentioned in his bug thread over at Doom9 that he is going to put in a new setting for Alternate output. This will be H265 video codec in an MKV file container.
In that post at Doom9, jdobbs has a link to a DirectShow filter you will need in your system in order for this to work.
H265 support should show up in the next version of BD Rebuilder, so try it out if you are interested. Just be aware that H265 encoding is extremely slow compared to what we are used to seeing when using H264.