System Generator does not support testbench generation for the Multiple-Subsystem Generator design flow. The ISE project generated by System Generator with this flow uses black box netlists for each subsystem. Post-implementation simulation can be run from this project without making any changes, other than adding a testbench.
To perform a behavioral HDL simulation, use the following process.
Because different System Generator subsystems can contain the same IP core blocks, it becomes necessary to compile the different subsystems to different libraries for simulation. Otherwise, the IP core HDL entities may be in conflict which will cause errors.
1) Generate the VHDL code from System Generator to your output netlist directory
2) In Matlab, cd into the output /netlist directory and use the xlSwitchLibrary command to change default library name for one or more subsystems as follows:
xlswitchlibrary('top_level_0_mydesign_subsystem0', 'work', 'subsystem0_work')
Where 'top_level_0_mydesign_subsystem0' is the name of the directory for one of your subsystems.
3) Create a new HDL based ISE project
4) Go to the Libraries tab of Project Navigator
5) Create a new library called "subsystem0_work"
6) Right click on the new library and select "Add Sources" and select the HDL files subsystem0.vhd and subsystem0_cw.vhd
7) Edit the System Generator top level VHDL file (design_name.vhd) for each subsystem with a new library.
a) Add the library declaration before the top level entity declaration:
library subsystem0_work;
use subsystem0_work.conv_pkg.all;
b) Modify the component instantiation for each subsystem as follows:
top_level_0: entity subsystem0_work.subsystem0_cw
8) Repeat steps 2 - 7 for any additional subsystems
9) Add the rest of the HDL source files to your project. This will include the top level .vhd from System Generator and any other subsystem files (note there are 2 per subsystem). This may also include a behavioral simulation model for shared memories at the top level.
10) Copy any .mif files from the subsystem directories to the top level ISE project directory
11) Create a test bench and associate it to the top-level.
You should now be able to simulate your design without any conflicts.