Mapping RGB data is not specific to the Xilinx SDI core, so how the SDI protocol is mapped is more relevant.
The core does not distinguish whether the data is YCbCr or RGB or XYZ, or if it is 422, 444, 10b, or 12b. It is simply transporting SDI streams.
For information on SDI stream formatting, start by looking at the SMPTE273, SMPTE425-X and 2081-X and ST2082-X families of documentation.
For example if looking for single link 3G-SDI information, ST 425-1 shows how to map all compatible video formats, including RGB formats. ST 425-3 is for dual link 3G-SDI. ST 425-5 is for quad link 3G-SDI. For information on single link 6G-SDI, the mapping document is ST 2081-10 (but you also need ST 425-3). For information on 12G-SDI, the mapping document is ST 2082-10 (but you also need ST 425-5).
In addition to the SDI specification documentation, you can also look at XAPP1014. The interface to the SMPTE SD/HD/3G-SDI core is the same as it was in XAPP1014.
Table 5-10 documents how the different formats are stripped across the different outputs:
For code examples, See at the SMPTE372_tx.v file in the Full_Featured_TX folder of xapp1014_c6_GTP_SDI_TX.zip. It contains some code and comments that can be of help in understanding the proper mapping of RGB data onto the ports for the reference design in XAPP1014.
Note: Other Xilinx SMPTE XAPPs might also contain code that would be useful when trying to map SDI Streams to the Xilinx SMPTE SDI cores.