it's completely unfair of you to chase him round and stress him out because of it. also bear in mind your other fish will be scared from you chasing round the tank too. stressing your fish out will leave them prone to disease.
I can't remember the link, but researchers kept a male betta alive for 7 years. Part of hte daily routine was "exercise" by chasing the fish with a net. Considering the normal life expectancy of the fish, this doesn't seem to have done a great harm.
Fish will naturally encounter things to run away from in the wild. I would be extremely surprised to see the odd chasing with a hand cause a fish to succumb to a disease. Hell, every time I walk or move near my big tank the bala sharks and chalceus all get stressed and start flying around the tank. There don't seem to be any problems there.