In the data sheet, why is the minimum jitter tolerance in Figure 2-2: "Compliance to XAUI Jitter Tolerance Specification" lower than the total and deterministic jitter tolerance specs? Which is correct?
They are both valid. The plot of XAUI jitter tolerance in the characterization report shows Sinusoidal jitter tolerance. The data sheet currently specifies only deterministic and total jitter tolerance.
Sinusoidal jitter is unique as it is somewhat trackable by the receiver PLL. As seen in the graph, at very low modulation frequencies the receiver can track the slowly changing edge, even though this jitter would completely close a persistent eye scope shot.