Often, the father will eat eggs that are unfertilized or have fungus. Also, the father also will often "mouth" the eggs. Some experts believe the father's saliva might contain a fungecide and that he is cleaning them.
I might be tempted to leave him in if I thought he was just mouthing them, or selectively thinning them. But that's just me. YMMV.