This will occur when the user design drives invalid combinations into the row, col, row_sel, col_sel or block_size ports. If this occurs and you are using the Interleaver/Deinterleaver v6.0 core simulation model, you will see unknown values driven on the outputs of the Serial Interleaver Deinterleaver core.
For instance, if you set "Number of Selectable Rows" to 3, then you will get a 2-bit input port called row_sel because 2 bits are needed to represent 3 values. This leaves 1 extra value that can be specified ('b11) which does not correspond to a set of selectable rows. Setting row_sel at run time to 'b00, 'b01 and 'b10 is fine. Setting it to 'b11 will result in X's from the simulation model, which results in "indeterminate data" in MATLAB also known as NAN (Not A Number).
To avoid this behavior, it is important to only drive the Interleaver/De-interleaver core with valid values. For details on driving the inputs to this core, see the Serial Interleaver / De-interleaver datasheet, available at the core product page: