Definitely either get a bigger tank or rehome them. Platies are naturally territorial, especially the males, and will not hesitate to be aggressive towards their tankmates.
Fish you can fit in a five gallon include bettas, tetras, raspboras, and certain species of livebearers like guppies.