It may be that he is very nervous and when you are talking to him, he may direct his body towards you, even touch you on the shoulder, arm. He may run into you far more often than by chance.
Guys aren't that subtle, you don't have to start reading body language to figure out a guy. If you express an interest in him and he's interested back, you'll know.