Hi, Donald,
Per your testing setup, here are the comments:
1. As for the IIP3 shown in adl5387 datasheet, we usually do the two tones measurement setup. i.e. having two closely spaced sine waves input into RF port of adl5387 after they go through a frequency combiner. Measure the 2f_1 - f_2 at the output. The IIP3 in the datasheet is obtained based on measuring the intermodulation products. What you are measuring is third -order harmonics which could be a little differerent.
2. I tried your single-tone testing setup using clean sigGen as LO source. Under such testing setup, the data i got was still consistent with the datasheet for both IIP2 and IIP3.
3. Based on 1, 2, my suggetion would be:
(1) try to use two tones testing setup instead of single tone for IIP3 measurement.
(2) try to use clean sigGen for LO instead of ad9516 to verify,
(3) check why the output power level of fundenmental wave in the figure is a little higher than we expect (18.7 dBm?).
Hope you find above information helpful.
Peng