The Video Common has an API for creating a custom resolution table, and registering it.
Here is an example of how to register a specific resolution. Both 1 and 2 can be placed wherever the constant declarations are made, such as a .h file. However, 3 will need to be done at run time.
1) Assign Mode ID Enumeration. These must be greater than the XVIDC_VM_CUSTOM constant.
2) Create a custom table with entries for each of the custom resolutions that the user wants to support.
3) The user will then need to register this table in their application. This can be done prior to initializing peripherals.