I have done it before without any problems but I can't guarantee that it will be the same for you. I have some pretty hardy fish. I would say that it is stressful...more so than your average water change for example. If you are going to do it I wouldn't do a lot and I wouldn't take a long time doing it. HTH
I would just do it along with a water change so u don't stress fish out twice in the same week - kill 2 birds with 1 stone, u know (it's just a saying! ... to all the bird owners out there)
It will definitely cause a little more stress since you're messing with their shelters and areas that define territories, but they should get over it just fine.
edit: the water will also be lower because of the water change (before u add new water) - this can help tremendously if you have a 55g or a tall tank of any size.
i agree let them settle for a week first, when u do it be warned, if u have gravel anything thats trapped will get stirred up big time and it will go cloudy... just be carefull u dont squash a fish in the process, i have done this before with a loach and a peice of driftwood. wasnt nice