Hi Barry,
The AD2S82A has a variable resolution. The resolution you are using is 12-bit, that means that the set up is fluctuating 7-bit from the 12-bit output. There are many possible factors that may affect. Can you check on your set up the quality of the sinusoid signal and the amplitude. As these will greatly affect the decoding of RDC's position and velocity.
By the way the component values on your compensation circuit was base from the computation from the datasheet?
Regards,
Jonathan