i have a pair of 2 swordtails, and the male follows the stronger and bigger femaleThey have a pecking order. If you get 50 fish and put them together, a group will hang out together and dominate the others, often harassing the smaller weaker fish.
If you get 5 or 6 fish, sometimes they all get along and everyone is fine, but other times a dominant fish will bully some of them and try to drive them out of the area.
If you get 2 or 3 and they are getting along well, that is also fine for them.