Yes, you do need more cardinals, at this stage. When you have had a group of a shoaling species, and a few years later they begin to die, the need to replace is much different than it is when the fish are young. Shoaling fish must have a group; they will be stressed out if not, and this leads to various health issues that otherwise wouldn't occur. The fish is stressed, and this only worsens. The lone cardinal's behaviour is indicative that he feels the need for others, but obviously he can't find them, so he may try to make do, but this is not the same thing to the fish. Fish communicate within a species by chemical signals called pheromones as well as by sight, and this is an essential part of a healthy fish.
I would aim for at least six cardinals, but more would be better, say 7-9. I don't know your tank size, but given that shoaling fish will always be healthier and "happier" with more of them, you need to provide as many as you can.
Same holds for the Black Phantom; here I would say 7-9 (again depends on your tank space). Male and female are easier to ID, and 4 males/3 females (5 males/4 females, etc) works nicely.
On another aspect, the death of the other cardinals. These are sensitive fish (so are the Phantoms, though perhaps not as much), and acclimation can be important. Also, the tank seems to have been new, was it cycled somehow?
Byron.