Yes, you should be concerned about inbreeding. Unless the fish were wild caught there's a very good chance they were related to begin with. And even if they are wild caught there's a small chance they are related. That doesn't mean it's good to keep breeding related fish. Ideally you want to mix the bloodlines up each time you make a pair.
Inbreeding will encourage good and bad traits and whilst the good traits may be desirable, the bad ones certainly aren't. Bad traits can be health issues, deformed fins or body parts, or a genetic weakness to certain things (usually diseases). Adding new bloodlines help to prevent genetic problems from establishing and destroying a species.
