Hello there,
I haven’t heard of any Mac software that can rip Blu-ray discs.
Normally the Blu-ray file format is a container for videos in either H.264, MPEG-2, or VC-1 format. If your ripping software allows unpacking this container without further decoding/coding, then H.264 output should be alright. Uncompressed AVI should be safest though, if possible.