I don't have any balloon mollies, only lyretails (
Poecilia sphenops) but I can give you a bit of info about livebearers in general, somebody else will doubtless come and expand on it
.
Mollies cannot breed with platies. I think you can get platy/swordtail hybrid (both are
Xiphophorus species) and you can certainly get variatus platy/maculatus platy because as far as I know my females are hybrids. But no platy/molly.
As far as I know, balloon mollies can hybridise with sailfin mollies, correct me if I'm wrong.
Sexing livebearers is not hard. Apart from the fact that the females (if they have had access to a male even once) are almost perpetually pregnant, the males have a modified anal fin that they use to fertilise the females that makes it easy to tell if you know what to look for. Look at the bottom of the fish, about two-third of the way towards the tail. You'll see two fins hanging down side by side. These are the pectoral fins. Behind them is the vent, and behind that is another fin. In the female (when she is relaxed and hovers in the water) that fin hangs in a loose triangle, point down. In the male, it is longish and points backwards. This penis-like organ is called a gonopodium - if you type 'gonopodium' into Google Images, you'll probably get the pictures that I don't have right now.
Hope that helps.