Agree with everything Gary has said.
But on top of that, I could see this going wrong very, very quickly. There almost certainly aren't any dosing guides for treating fish with Xanax (alprazolam) so you would be shooting in the dark for a safe dosage, and that's a very dangerous game to play. Small fish have small bodies, and so it takes much less to kill them, in terms of toxin concentrations. The difference between a safe dose and a lethal one could be less than a gram, or even just a few milligrams, of medication. That's extraordinarily easy to slip up on, and you could be risking the lives of your fish for a just a chance at making them bolder.
As far as ethics go... I'm of the mind that animals should not be medicated unless there is a reasonable certainty that it would be beneficial for them. There are plenty of calming drugs given to dogs and cats which are extremely beneficial when they have chronic anxiety (or acute anxiety in unavoidable situations like a vet clinic), but fish don't have chronic anxiety unless their environment is stressful in some way. A new fish in a new tank will be able to overcome their initial stress in a day or two just fine on their own, as long as their environment is set up in a way that helps make them comfortable. There's no need to medicate them for stress, and doing so could potentially put your fish under more stress, or kill them.