I think we have to step back and look at what shoaling is. We don't see a lot of it in our tanks, because it is a defence mechanism. Most of the time, we see tetras dispersed and feeding, rather than running together to confuse a predator. Yesterday, I added some juvenile cardinals to a 75g that already has a good number, including the parents of the ones I added. The juvies, who have only ever seen each other and some young Ancistrus, ran for the peace lily roots when they entered the big tank. Then they cautiously swam down to where the larger cardinals were, and were ignored.
I have three male Hyphessobrycon negodagua (because I could only get three and was hoping for at least one female so I could expand the group), and they own the tank. One began to mildly harass a small cardinal, and shoaling happened. The wee fish must have released chemical distress signals (it was never in real danger, but was scared) and the loose group of cardinals suddenly became a shoal. It wasn't for long, but the troublemaker tetra wandered off. This morning, the small cardinals are acting just like the big ones, spread out, keeping contact but very relaxed.
They make sure they can see each other, and react to colour and visual signals. They remain aware of where each other is at all time. I guess if you keep an eye on Fred, and he gets eaten....
There are two things I'm not sure I buy here. One is how we use shoaling. I think these fish are not individuals as we like to think of them. They're social animals that depend on each other for security. Too few, and they're insecure. 4? 6? 10? There are credible studies that say the more the merrier. But I have uncomfortable questions whether we're using numbers like 10 for our convenience.
With glo-fish, a good part of their communication system has been destroyed. We've all seen how the use of colour is controlled by fish, and signals pass between them through their "chromatophores". With glo-fish, we've gutted that communication method by blotting out their colours They can still communicate chemically, but to a point, we have cut their tongues out. So how does having a group matter? Their ability to communicate has to be affected by the overlay of 'pretty' colours gene spliced into them.
I also don't think mixed species social groups are a problem. They form in nature all the time. They aren't always predictable, but when they do work with groups, you can't deny what you see. I've had groups of 20 and 10+ tetras of different species form identifiable groups together. I've even kept pygmy Corys and the lookalike tetras that shoal with them. I think we're thinking a bit rigidly here, even if the fundamental idea that small fish aren't individuals is sound.
No answers, just things maybe worth considering if people get into it.