When my DDS output frequency is a multiple of fs/2 (i.e., fs/4, fs/8, etc.), my SFDR performance is much worse than expected if noise shaping is used. Why?
The simple answer to this is that the effect is a mathematical artifact caused by the fact that the output frequency is harmonically related to the system clock. Dithering is a technique to spread the phase error energy across the spectrum, whereas in this case the harmony means there is no phase error and noise shaping is introducing harmonic distortion.
If you need an output frequency which is exactly an integer multiple of fs/2, noise shaping should be disabled.