Considering that goldfish in the laboratory setting have been taught how to navagiate complex mazes and perform simple tricks, I'd say they're far from dumb. The 3 second memory thing is just stupid folk-knowledge with no factual basis; anything that can be trained into peforming tasks or recognizing daily schedules can not possibly have so brief a memory. They have amazing social intelligence, can peform many skillful tasks, are known to have owner-recognicion. They are also remarkably tame and friendly due to their long history of domestication (since 800AD in fact), and many enjoy interacting with pets or being stroked by a human owner. If you think goldfish are stupid and unworthy of humane care and a long, healthy lifespan, I pity you; you are missing out on a wonderful fish by being so narrow-minded towards them.
Also, consider the fact that we should not be questioning if an animal's intelligence makes it worthy of life; in suffering, all fish are equals. Saying that a fish that is less intelligent than another does not deserve compassionate care would be like saying an infant or profoundly retarded adult human can be mistreated because they're "stupid."