The LogiCORE IP SMPTE SD/HD/3G-SDI (SMPTE SDI) can support any resolution. 

The Video format does not matter for the Rx and Tx, as long as the data is legal SDI values and contains legal embedded SDI EAV and SAV sequences.
 

In the SMPTE SDI Rx the transport detection function only works for standard resolutions. 

If a non-standard resolution is used, the rx_mode_locked signal should be asserted and everything should function normally except for the transport detection. 

When a non-standard resolution is received, the rx_t_locked signal will not be asserted and none of the rx_t_* ports will be valid. 

You will need to use your own logic to handle these cases.