Not expensive as hobbies go- as long as you're happy to stick to the basics. No more than playing an instrument or being part of a sports team or whatever- it always adds up. It may be that you can get the fish hobby to add up to even more than most others if you really try, but that's personal choice.
Still, I don't think it's a hobby you should take up unless you can set aside a little extra for emergencies. After all, it is about being responsible for living creatures. It does get to me when I hear people who've spent their last penny on lots of expensive fish and then can't afford the test kit or grumble about the cost of dechlorinator. It's like any other animal, you've got to plan ahead to make sure you can afford to keep them.