I recommend this blog-post http://www.hessmer.org/blog/2011/01/30/quadrature-encoder-too-fast-for-arduino-with-solution/ . Read no more!
We have a rotary encoder (this one) in our behavior setup. The arduino code posted on their website is mysterious (unless you know all the details). We’ve used it and it works!
Following is from the manufacturer:
Essentially if A is leading B then encoder is counting counter-clockwise; else (if B is reading A) then we are going clockwise. So essentially, I attached a rising edge triggered callback on Arduino PIN 2 (one of the encoder wire goes here). In the callback I read the value of PIN3 (where the other encoder wire is connected) and use this logic. Note that since callback is triggered on rising edge and we do nothing on falling edge, we get 1200 counts per cycle (rather than 2400). This fits our requirement. The header file with implementation is here. This Arduino does other things as well, so the main file is not linked here.