Definitely refrain from feeding him what he won't eat... it will only foul the water. The maximum I'd feed a betta is 6 pellets throughout the day... remember, their stomach is about as big (or small, I should say) as their eye. Bettas are prone to bloat because their special pellets swell up with water inside their stomachs, so I try to feed my betta only 2 pellets twice a day. Some people pre-soak their pellets, but this will leach all the water-soluble vitamins out of them. So your best option is to feed your betta small amounts multiple times per day.
The conditions your betta is in also tell you about how much you should feed him. For instance, the temperature and size of the tank determined how fast they burn through energy. If your betta is hanging out in a heated 5 gallon where he is getting lots of exercise, you'll want to feed him a little more. If he is in a small, chilly cup, feed him less.
Since every fish is different, there is no 'perfect amount' to feed a betta. 2-6 pellets per day (depending on the factors above) should cover it.
Don't forget to mix his diet up a little too! I'd say 1 bloodworm/brine shrimp = 1 pellet, for reference.