Most fish have excellent colour vision, (if they didn't - they'd all be black and white - why have vast tracts of expensive to run colour producing cells if nobody can see them?!!??!!??!!).
Like our own eyes, the colour receptors need a lot of light to work, because they only respond to a small part of the spectrum.
When it gets dark, there is not enough light for the colour receptors to work. Consider, when you wake up at night, is your bedroom all coloured, but just dim, or is it black, white and shades of grey?
There is a second group of cells in the eye that simply respond to light intensity, not colour. A white object reflects more light than a black object - eyes, (and I deliberatley make no distinction between yours, mine, or a Cardinals), see the low light world in monochrome.
Now consider your pattern of spots and stripes which looks so good in the day. At night, although it may not be vividly coloured to whatever is out there trying for a meal, it does show up as marked contrast changes.
By contracting the pigment cells, fish reduce the contrast they offer to night time predators, they grey down into a grey background.