Sometimes rabbits are just like that, one of mine hates anyone putting their hands in his cage when he is in there and often attacks the sweeping brush lol. I would make sure you keep stroking him, otherwise he wont let you touch him at all after a while. :)