Starting in 11.3, System Generator has beta support for 64-bit Windows operating systems.
The 11.1 installer does not allow you to install System Generator. Therefore, you must first install the ISE design tools and then run the 11.3 Standalone DSP Tools installer. To obtain the 11.4 Standalone DSP Tools installer, you need to submit a WebCase:
http://www.xilinx.com/support/clearexpress/websupport.htm
System Generator for DSP does not have native 64-bit support, so it must be run with the 32-bit version of the ISE design tools and MATLAB. This requires that your PATH and XILINX environment variables point to the 32-bit nt folder inside the ISE design tools installation area.
For MATLAB, this requires the 32-bit installation of a supported MATLAB release.