The basic tests are ammonia, nitrite, nitrate and pH. Tests for GH and KH are not necessary unless you are adjusting these parameters [I'll come back to this momentarily]. API puts out a "Master Combo" liquid test kit that has all four necessary tests, and it is a good initial investment.
On the GH/KH...once you know these numbers for your source water (municipal tap or well) they are unlikely to alter very much in the aquarium unless you are targeting them somehow. For example, using a substrate composed of calcareous rock such as limestone, marble, aragonite, coral, etc. would increase the GH by dissolving minerals into the water continually. And conversely, the normal biological processes will tend to acidify and soften the water. The extent to which all this occurs depends upon the initial GH and KH, and regular partial water changes work to keep it stable. I would up your water change volume to about half the tank, once a week; water changes are the single most important maintenance aspect of a healthy aquarium.
Once the tank is biologically established (for a few months), tests can be less than initially. Testing the nitrates and pH periodically is beneficial, as changes from the "norm" in either of these can be signs of trouble. Once the tank is cycled, I never test ammonia and nitrite except if I detect some fish issue, and this is one of the first steps to pinning it down.
It is also beneficial to test your source water on its own for ammonia, nitrite and nitrate, just in case any of these are present. And pH of course. The GH and KH you should be able to ascertain from the municipal water authority (unless you are on a private well); check their website.
To the initial questions...I would clean the filter more regularly to reduce the brown sludge (this is organics). Provided the water flow is not being impeded, it should be OK, but I have found over the years than keeping this low does seem to benefit long-term with water stability.
The flashing of the fish could be due to several things, including water issues as you were asking about. If tests are fine (once we have numbers we can tell this) it would more likely be a parasite or protozoan. More data will be needed to assess this further.
Byron.