Fish are ectothermic. This means they do not have the ability to generate or maintain a specific internal body temperature (animals that can, which are the birds and mammals, are termed endothermic) so they rely on the temperature of the water in which they live. Keep in mind that this water is continually entering the fish via osmosis through every cell on its body, so the temperature of the water is the internal temperature of the fish.
Water has a high specific heat capacity, which means that temperature changes are resisted and occur relatively slowly. Daily variance in most aquatic environments is only 3-4 degrees C (5-7 F).Seasonal temperature changes in aquatic environments occur slowly over a matter of months. Therefore fish have evolved in an environment where the temperature remains relatively stable and any changes that occur do so slowly. [Read more in The Manual of Fish Health, pp. 40+].
The reason this is crucial is because the various processes of the physiology of the fish (applies to all animals) is driven by temperature. Each species of freshwater fish has evolved over thousands of years to function in a very limited and specific set of environmental factors, of which temperature is one. As soon as the temperature moves outside the specific range for the species, it has difficulty maintaining normal life processes. Goldfish obviously have an enormous temperature range; they can manage in summer heat (though their metabolism adjusts for this) and in the winter the pond can freeze over and again their metabolism slows to just keep them alive. But fish evolved to live in water that has a variability in temperature of no more than a very few degrees throughout the year are not able to achieve this, and it takes its toll. We generally say the fish are weakened by such negatives.
When the temperature moves outside the preferred range for a continuous period or changes rapidly within the accepted range then it causes stress to the fish. This is true as a general statement, because fish species do vary in their ability to withstand temperature change. Read more in the cited book.
Fish weakened by stress have a less effective immune system, so they are more prone to disease. Their whole physiology can be impacted. It is best to simply think of it as causing severe difficulty to the fish in maintaining its essential life processes day to day. The fish is having to use considerably more energy to maintain life processes that should be relatively easy to handle. The effect of all this negative is cumulative.