Don't just do the water changes "several times a week" do the water changes in response to the ammonia or nitrite being too high (anything readable on the ammonia/nitrite test kit means that you should do a water change. The concentration determines how big the change needs to be. For example, if your reading is very light, less than the 0.25ppm, you could guess that it is about a 0.10-0.15ppm. Doing a 50% water change would be appropriate in this case to drop the ammonia to 0.05-0.07ppm. If your reading is 0.25ppm or 0.50ppm you really need to do as large a water change as possible. The stress of a water change is nothing compared to the stress of the toxin in the water. With a 0.50ppm reading, you would need to do a 90% water change to bring that level back to the "trace" level or at least close to it (0.05ppm). Ultimately, you want the ammonia to be at trace levels (or as close as you can keep it), which means that the ammonia test kit never shows any color change. Depending on which brand kit you are using, what color you want. For the API kit, you want a nice bright yellow with no tinge of green for ammonia, and a nice light blue with no noticeable trace of purple for nitrite.
If you don't have a test kit, please go out and get one. You need a test kit for ammonia, nitrite, and nitrate. You also want to complete water changes for excess nitrate as well. Generally, you want to keep your nitrate a fairly low number, but it isn't nearly as bad as ammonia or nitrite. Nitrate should be kept as close to 20-40ppm as possible with water changes. With goldfish that will be a little more difficult, because they are so messy. But, at the same time, that just means that your water changes need to be more often or larger when you do them. Fish can tolerate very high nitrates by comparison to ammonia or nitrite, but it is still dangerous to the fish if it isn't dealt with at all.
You are doing the right things, but you need to do water changes more frequently and larger.
One final thought. Whenever your fish are acting odd, before posting here, complete a large water change. That is basic fishkeeping first aid. When in doubt, do a water change. If you are keeping up with them the basic water parameters should be the same as the tap water, so there won't be a massive change for them to deal with. Meanwhile, a simple thing like fresh water is often enough for the fish to be able to handle whatever the problem is on their own. They have good immune systems. For the most part, it is only when the fish is stressed (like ammonia/nitrite poisoning, excess nitrates, etc.) that leaves them susceptible to many issues.
