I assume, you have populated the same loop filter components when using single-tone and using QDUC mode.
What is then your PDCLK? please ensure you follow the correct sample rate when in QDUC mode. It is dependent on dac clock and CCI interpolation factor (R): PDCLK = Fs/(2R).