It's outside the scope of this answer to go into the theory, but a 3 phase motor by convention normally rotates anticlockwise when viewed from the 'driven' end.
To check phase sequence, the usual method is to use a phase sequence indicator. Originally these were like a miniature three phase motor encased in a hand-held device. If the flying leads were connected RYB, BRY, or YBR, the disc in the psi would rotate anticlockwise. If the flying leads were connected RBY, BYR, or YRB the disc would rotate clockwise. Nowadays, the mechanical aspect of the phase sequence indicator has been repalced with solid state devices that indicate on one of two leds if the sequence is c/w or ac/w.
To check phase sequence you look for which phase out of the three phases peaks first. You could probably put each phase on an oscilloscope and see which one peaks first. Phase sequence is usually taken as positive anticlockwise from the reference voltage ( the phase that peaks the first).