Those fish, in the wild, swim in shoals of thousands. It's very hard to prove absolutely, but the hypothisis is that unless there are more fish of their own kind around, the fish may be suffering from constant low level stress. He's basically swimming round wondering where everyone else is, and in the wild that would mean he could be picked off by a predator, or that all the other fish have sensed some danger and swum away.
How can we tell whether a fish is 'happy' just by looking? It's impossible really; we just have to try and understand the life the fish evolved to have and give it those conditions. Keeping shoaling fish alone is, while not 'cruel', not really fair to it.