It does depend a lot on what you are keeping. Tropical open water fish get the full glare of the tropical sun, which is pretty intense. Actinic lamps are designed to replicate the open water conditions over coral reefs where the water is a little deeper and has absorbed much of the red end of the spectrum, thus you can have full sun, deep water conditions indoors and in a shallow tank.
Most of the common smaller fish are not open water though, they frequent undergrowth and plants where the light is lower and they have a lot of escape options. Others are rainforrest stream species, and from experience, I know it can be pretty dark on the rainforrest floor at any time of day. If it is these we are talking about then yes, in very bright light in relatively bare tanks, they will be stressed.
I would get rid of the Actinic lamp(s) simply because they create such an un-natural picture when looking at the tank, but that is a matter for personal æsthetics.