I really hate to be giving bad news, but we all must learn what is needed and best for our fish. Some of these fish, in fact all of them except for the swordtails, are shoaling fish. These species live in large groups, and they have inherent needs for a group of their own species. Sometimes it may be as simple as security, "safety in numbers," while in other species there are hierarchies, interactive behaviours, etc. We know that when a shoaling fish is denied an adequate sized group of its own, it will suffer stress which then leads to other problems with its health (the immune system is severely weakened by stress), and very frequently increased aggression because that is just about the only way the fish can deal with something it needs being denied it. These needs are part of the species' DNA, so we cannot change them.
You have space with a 55g fortunately, so increasing the numbers of the shoaling fish is your best option. Before doing that, you should consider carefully the natural behaviours of the individual species; you may find that one or more will not bee compatible for long. The fact that you do not see aggression does not mean it is not present. Fish usually display aggressive tendencies physically, but sometimes they will only release pheromones (read by others in that species) and allomones (read by other species) and these can severely stress fish that are targeted.
Tiger Barbs for example are naturally aggressive and tend to fin nip sedate or long-finned fish. In a group less than 10 they may nip ach other; in larger groups this usually is under control, but not when it comes to sedate fish or fish with long fins, which are like waving a red flag in front of a bull. This same aspect applies to the Black Tetra, if this is the Black Widow or Black Skirt Tetra. This fish too is naturally prone to nip fins, and needs 8+ with more always being better.
The rainbowfish, what species is this? They are shoaling and need a group, but they are different sizes with some different needs respecting water parameters.
Also, what is the GH and pH of the source water? Some fish here (swordtails) need moderately hard water, the others are more soft water (rainbowfish depend upon the species) but with more adaptability, depending upon the actual GH.
