You can use a sine wave but it it must have a minimum drive level (see Figure 22). Even when you drive the LO input with a sine wave, that signal gets turned into a square wave inside the AD8333 before being applied to the mixers. So harmonics of the LO are going to be present in both cases and they will mix with harmonics of your input signal.
Now you could back drive the LO input with a backed off sine wave. If you make the sine wave small enough, it will still be a sine wave when it hits the mixer. This turns the device into a linear multiplier (output amplitude is proportional to both the mixer input signal and to the lo level. However, driving the LO in this manner will result in significantly higher output noise. In general mixer cores like to be driven by square waves with sharp edges.