Commonest cause of things like this is ammonia poisoning (or nitrite poisoning, by the same process). I'm not sure if you mean that the 5.5 gallon is the main tank or the tank you moved the sick fish into, but it's quite a small tank, particularly for mollies, which get quite big and have an appetite to match, producing a large amount of waste in the process, which then feeds the ammonia cycle.
We see these problems in tanks that are too small, so toxins build up between water changes, or in tanks that have immature or underperforming filters. It takes a while for the filter bacteria to colonise the filter properly, a process known as cycling. There are a few articles on the forum explaining the ways to best do it.
If the main tank is the 5.5 gallon then your problem is that it's essentially too small a space to keep mollies in, you're looking at 4 times that size as a minimum for them. Toxin build up will be essentially uncontrollable in that volume of water, and it'll happen again I'm afraid.