In SysGen v3.1, the XtremeDSP hardware Co-Sim flow creates a new library for the run-time Co-Sim block that is named after the original design. It retrieves the design name from the "sysgeninterface description" file. The Matlab open_system call causes an error when the system name is greater than 63 characters. Therefore, if the top-level entity name is 64 or more characters, the co-Sim flow will fail when it tries to create a new library.
This has been fixed in System Generator 3.1 service pack 1