Hmmm...How much of a water change are you doing and how frequently? Is it possible to test your tank with another kit?
The concern is that levels of ammonia of 4-8 are quite toxic. Even if the fish seem to be doing alright it's likely that they would be stressed or that their lifespans shortened. I think the most important thing is to establish whether you are getting false reads or not by testing with another kit.
If you are not getting false reads, we need to figure out what's causing the high ammonia levels. It could be anything from an uncycled tank (not enough nitrosomonas colonies to convert ammonia to nitrite), a dead fish hidden someplace in the tank, overfeeding, and so forth. Finally, it would be important to bring down the ammonia with water changes.
If I had to guess given the information you've provided, I would say that the tank never completely went through a full nitrogen cycle and colonies of nitrosomonas and nitrobacter never grew in, and that the high levels of ammonia may be inhibiting further bacterial growth.
All is not lost! Zebras are a very hardy species, and swordtails from what I understand are fairly hardy as well. I'm not sure about barbs...neons aren't the ideal species to cycle a tank as from what I've heard they are hit or miss in terms of hardiness. The book I have suggests they don't tolerate high levels of ammonia well.
I'd also encourage you to read the pinned topic in Beginner Questions entitled "Avoiding and Treating New Tank Syndrome" by Alien Anna. It's a terrific read and it may address a lot of the questions you have right now. Good luck! Keep us posted!