Should I give up on him?

There's this guy I've known for a while now but recently since we're in the same homeroom this year I've actually started having conversations with him. It's just us and a couple of other people and this one girl told me her friend that we should go out. Sometimes he'll call me annoying and tell me how stupid I am but he doesn't seem surprised when I tell him I get straight A's. Some days when it's just us he'll be real nice and when I try to get close he won't push me away. But then the next day he calls me annoying again... I don't know why he does this..