OIP3 was measured in dBV, not dBm, because the ADRF6516's output was driving an 8:1 balun, which presented a 400 load to the output.
Can you provide a schematic of your test setup (I didn't quite understand what exactly you were doing), input signal level as read on the signal generator providing the RF tone to the ADL5380 in dBm, output signal levels as read by the spectrum analyzer in dBm, the dBc number you are reading from you main output tones to your IMD tones on the spectrum analyzer, and the gain settings on the ADRF6516?
Thanks,
Joel