I would water change the minerals back and dilute the leaf compounds, but when I want a higher KH (alkalinity), I keep a DIY pantyhose bag of crushed coral in the filter. I don't have to use very much to keep pH at 7.4 (KH 4), only 1/2 tablespoon per 30 gallons. I don't like to let the pH go lower than 7.2 because of the reasons someone mentioned, the nitrogen cycle is inhibited at that pH level. Any ammonia in the tank is not processed, but kept in a safe state as ammonium. So when you do a water change with higher pH water, the pH shoots up and ammonium converts to toxic ammonia. Plus the fluctuating pH is stressful for the fish. So if you want to maintain a tank at pH 4, you would have to pre-condition your new water at every water change or use RO water. If your not planning to keep a soft water tank longterm, remove some of the leaves and slowly bring the KH up with water changes.
Have you tested your tap? Usually a couple water changes is enough to bring levels back up. But are you intending to keep it that low? If you add less leaves, it won't alter so much. Plus I can't think of any options for a 5 gallon with low pH.