You are asking the question "Why are they doing this nipping behavior?"
I didn't understand this at first even though people had mentioned the answer and the words were sitting there grinning at me. Its the business of the "minimum shoal" that people keep mentioning. Its very common in tetras especially. In their natural habitat these neons live in shoals of millions of fish and move in huge swirls that look like a shining snake of glittering scales (each individual neon) at times. By the time you get them down to a handful of fish, their natural instincts tell them they've been separated from the shoal and "they are going to die unless they find their way back" (I'm projecting human thoughts there but you get the idea.) There's just little way around it, the genetics of these fish, shoaling fish, makes them stressed more and more the closer they get to being a lone fish. The "minimum shoal size" is people's opinions about just how small a group you can get down to and still have them enough not stressed that they will live ok.
Another angle I like to imagine it from (and I don't know if this is true, its just my take on it) is that with these shoaling behavior fish, they will see the food and survival possibilities as "OK" as long as they are out in enough space with enough same-species companions. At some point as their numbers and space contract they switch their behavior to "survival mode" and see their same-species companions as competition for what they expect to be not enough food. Thus if they can kill their companion, they may survive.
On the other topic, that of the neons sensitivity and need for stable water chems, the way I see to reconcile your previous experience of them living in your small tank and what davo is saying (which I agree with) is that "lack of stability" is just that, its random. If the environment is not quite right then sometimes with some fish it will work, sometimes it won't. You've now experienced it both ways.
~~waterdrop~~