This known issue affects the Video Timing Controller v5.00.a and v5.01.a when both VTC Generation and VTC Detection are enabled.
To work around this issue, you should set CONTROL register bit 12 (marked as Reserved) to '1'. This register can always be set to '1' without any side affects. The need to set bit 12 has been fixed in the Video Timing Controller v5.01.a Rev2, or later pCore patch, and in the Video Timing Controller v6.0.
Generator Vertical Blank Offset Source Select
The CONTROL register bit 17 which is marked as reserved in the Product Guide for the Video Timing Controller v5.01, and the v5.01.a as reserved is actually the Generator Vertical Blank Offset Source Select register.
If you want the Video Timing Controller Detection and Generation to be synchronized, set it to '0'.
If you want the Video Timing Controller Detection and Generation to operate independently, set it to '1'.
For a detailed list of LogiCORE IP Video Timing Controller Release Notes and Known Issues, see (Xilinx Answer 32754).