Module-Based Partial Reconfiguration

The Module-based Partial Reconfiguration flow is based on the Xilinx Modular Design methodology. The designer should read and understand "Modular Design" in Chapter 4 of this reference guide before proceeding.

Defining Reconfigurable Modules

Partial Reconfiguration involves defining distinct portions of an FPGA design to be reconfigured while the rest of the device remains in active operation. These portions are referred to as reconfigurable modules.

The reconfigurable module height is always the full height of the device

The reconfigurable module width ranges from a minimum of four slices to a maximum of the full-device width in four-slice increments.

Horizontal placement must always be on a four-slice boundary; the leftmost placement being x = 0, 4, 8, …

All logic resources encompassed by the width of the module are considered part of the reconfigurable module's bitstream frame. This includes slices, TBUFs, block RAMs, multipliers, IOBs, and most importantly, all routing resources.

Clocking logic (BUFGMUX, CLKIOBs) is always separate from the reconfigurable module. Clocks have separate bitstream frames.

IOBs immediately above the top edge and below the bottom edge of a reconfigurable module are part of the specific reconfigurable module's resources.

If a reconfigurable module occupies either the leftmost or rightmost slice column, all IOBs on the specific edge are part of the specific reconfigurable modules resources.

To help minimize problems related to design complexity, the number of reconfigurable modules should be minimized (ideally, just a single reconfigurable module, if possible). This said, the number of slice columns divided by four is the only real limit to the number of defined reconfigurable module regions.

A reconfigurable module's boundary cannot be changed. The position and region occupied by any single reconfigurable module is always fixed.

Reconfigurable modules communicate with other modules, both fixed and reconfigurable, by using a special bus macro (described in the "Bus Macro Communication" section).

The implementation must be designed so that the static portions of the design do not rely on the state of the module under reconfiguration while reconfiguration is taking place. The implementation should ensure proper operation of the design during the reconfiguration process. Explicit handshaking (e.g., module ready/not-ready) logic may be required.

The state of the storage elements inside the reconfigurable module are preserved during and after the reconfiguration process. Designs can take advantage of this fact to utilize "prior state" information after a new configuration is loaded. On the other hand, it is not possible to utilize the FPGA devices global set/reset (GSR) logic to independently initialize the state of the reconfigurable module. If set/reset initialization is required for the reconfigurable module, user-defined set/reset signals should be defined in the source HDL.

A layout of a design with two modules that are reconfigurable (shaded) is shown in the following figure.

Creating a Design for Partial Reconfiguration

Creating a Partial Reconfiguration design requires the creation and implementation of the design within a set of specific guidelines. The Partial Reconfiguration flow utilizes a modified form of the Xilinx Modular Design process. For more details on modular design, see "Modular Design" in Chapter 4.

Design Entry - Write and synthesize HDL code in conformance with Partial Reconfiguration guidelines.

Initial Budgeting - Design the floorplan, constrain the logic, and create timing constraints for the top-level design and each module.

Run Active Implementation (NGDBUILD, MAP, PAR, etc.) for:

Each module of the reconfigurable module.

Each configuration of a particular reconfigurable module.

Assembly Phase Implementation:

Minimum - Full design (initial power-up configuration).

Recommended - Every possible combination of device configurations of fixed and reconfigurable modules for simulation and/or verification purposes.

Verify design (static timing analysis, functional simulation).

Visually inspect design using FPGA Editor to ensure no unexpected routing crosses module boundaries. Though the software enforces this rule, it is still important to manually check this result.

Create bitstream for full design (initial power-up configuration).

Create individual (or partial) bitstreams for each reconfigurable module.

Download the device with initial power-up configuration.

Reprogram reconfigurable modules as needed with individual (or partial) bitstreams.

Recommended Project Directory Structure

Xilinx highly recommends the recommended project directory structure be created and followed. Since most designers are new to the modular design flow, such a structure greatly helps to organize the files created during each major phase of the design and implementation process.

This application note assumes a directory structure conforming to the form outlined in the following figure.

Design Entry/HDL Coding/Synthesis Process Details

The HDL code for the design should be kept in the HDL folder of the project structure. Further subdirectories should be kept for the top-level design as well as each module. Likewise, synthesis projects and results for the top-level and each module should be kept in corresponding subdirectories of the Synthesis folder.

To conform to the requirements of Partial Reconfiguration, the HDL coding and synthesis process follows some general structural rules.

Overall structure should be a top-level design with each functional module defined as a "black-box" level of hierarchy. Logic at the top level should be limited to I/Os, clocking logic, and the instantiations for the bus macros. There should be no other logic in the top-level design.

For new users of Partial Reconfiguration, it is highly recommended to minimize the number of reconfigurable modules in a design. Ideally, a single reconfigurable module would be most easily handled. However, this is only a recommendation, not a limit imposed by the implementation tools.

Each module, whether a reconfigurable module or fixed, should be a self-contained block of the logical hierarchy. Declaring port definitions as input or output are required for all module ports.

Bus macros are used as fixed data paths for signals going between a reconfigurable module and another module as shown in the following figure. The HDL code should ensure that any reconfigurable module signal that is used to communicate with another module does so only by first passing through a bus macro. There are device-specific versions of the bus macro. Be sure to instantiate the version compatible to the chosen device. Each bus macro provides for 4 bits of intermodule communication. As many bus macros as needed must be instantiated to match the number of bits traversing the boundaries of the reconfigurable modules. As an example, if reconfigurable module A communicates via 32 bits to module B, then eight (32/4) bus macros will need to be instantiated to define the data paths between modules A and B. The details of bus macro usage are described in the "Bus Macro Communication" section of this chapter.

Figure 5-3: Bus Macro Used for Intermodule Signals

If a signal passes through a reconfigurable module connecting the two modules on either side of the reconfigurable module, bus macros must be used to make that connection. This effectively requires creation of an intermediate signal that is defined in the reconfigurable module. The signal cannot be actively used during the time the reconfigurable module is being configured.

For the sake of simplicity, especially for new users of Partial Reconfiguration, Xilinx highly recommends keeping clock design as straightforward as possible. Avoid designing clocks to use in one variation of a reconfigurable module, but not in others. Though this can be done via a clock template structure (see the "Clock Template" section), it is an advanced technique that those who are new to Partial Reconfiguration should avoid. Use of clocking logic, such as DCM, is also best avoided.

All defined clocks must use dedicated global routing resources. Bitstream frames for global clocks are separate from those bitstream frames defining the CLBs. The Partial Reconfiguration flow depends on this separation to keep clocks functional during reconfiguration. Do not define clocks that use "local" (non-global) resources.

Reconfigurable modules must not directly share any signals with other modules, except clocks. This includes resets, constants (VCC, GND), enables, etc.

The top-level is synthesized with I/O insertion enabled, producing a top-level netlist.

Each module is synthesized with I/O insertion disabled, producing a module-level netlist for each module.

These guidelines are very similar to those specified for the Modular Design flow.

The "Modular Design" chapter of this guide has further details and examples of proper HDL coding styles and synthesis techniques. To help verify proper practice for the HDL coding and synthesis phase, consult "Checklist for Top-Level HDL Design" for a top-level HDL design checklist and a module HDL design checklist.

Bus Macro Communication

To facilitate communication across reconfigurable module boundaries, yet still conform to the Partial Reconfiguration requirement that routing resources across such boundaries be completely fixed and static, the use of a special bus macro is required.

In the following diagram, the left half "A" is a module and the right-half "B" is another module. Either A, B, or both could be partially reconfigurable. To support communication between modules A and B, a special bus macro is used. Partial Reconfiguration requires the signals used as communication paths between or through reconfigurable modules must use fixed routing resources. That is, the routing resources used for such intermodule signals must not change when a module is reconfigured. As shown in the following figure, the bus macro is a fixed routing bridge between the two sides, facilitating reliable communication. It is a pre-routed hard macro used to specify the exact routing channels and will not change from compilation to compilation.

For each of the different design implementations, there is absolutely no variation in the bus macro routing.

Route locking is required because if any of the designs choose a different routing for the bus macro, it will not align properly with other designs and the communication between the two halves is effectively broken.

The current implementation of the bus macro uses eight 3-state buffers (TBUFs) hooked up in an arrangement that allows one bit of information to travel either left-to-right or right-to-left, using one TBUF longline per bit as shown in the preceding figure. Each row of the device can support four bits of a bus macro. The bus macro position exactly straddles the dividing line between design A and B, using four columns of TBUFs on the A side, and four columns of TBUFs on the B side. With the Virtex-II architecture, only two columns on either side are used. Design A only connects to the TBUFs in the two or four columns on the Design A side. Likewise, Design B only connects to the TBUFs in the two or four columns on the Design B side. The "fixed bridge" that is pre-routed is comprised of the TBUF output longlines to ensure reliable communication between the two sides.

The bus macro must be physically locked in such a way as to straddle the boundary line between A and B, and it must be locked in exactly the same position for all compilations. The process of locking the bus macros to proper locations is described in the "Initial Budgeting Phase Details" section. The bus macro can be wired so that signals can go in either direction (left-to-right or right-to-left). It is strongly recommended that once direction is defined, it should not change for that particular FPGA design. Bus macro signals should neither be bidirectional nor reconfigurable.

The number of bus macro communication channels is limited by the number of horizontal longline routing resources available in each CLB tile.

Implementation Using Modular Design

As defined by the Modular Design flow, the Partial Reconfiguration Implementation process is broken down into three main phases:

Initial Budgeting Phase - Creating the floorplan and constraints for the overall design.

Active Module Phase - Implementing each module through the place and route process.

Final Assembly Phase - Assembling individual modules together into a complete design.

Initial Budgeting Phase Details

Initial budgeting operations should be done in the top or initial folder of the recommended project directory structure.

The output of the Initial Budgeting phase is a .ucf file containing all placement and timing constraints. Each module is implemented using this .ucf file, in addition to any module-specific constraint requirements.

For Partial Reconfiguration, there is a significant difference from the standard modular design flow. Floorplanning in the Partial Reconfiguration flow does not involve the port-placement process. In Partial Reconfiguration designs, all reconfigurable module inputs and outputs connect to either primary I/Os, global logic, or bus macros. No signals going to or from a reconfigurable module will load or source any element in another module without first passing through a bus macro. Unlike a standard modular design, a Partial Reconfiguration design does not have intermodule ports. In fact, if pseudo-drivers or pseudo-loads are found when viewing the design in the Floorplanner, the design violates the criteria that all intermodule signals must utilize a bus macro channel. If this occurs, re-examine the HDL source and correct the problem.

As with Modular Design, the product of the Initial Budgeting phase is a .ucf file describing all of the following:

The area-based floorplan for each module.

Fixed location constraints for all top-level (context) logic.

Fixed location constraints for all bus macro instantiations.

Any global level timing constraints.

Active Module Phase Details

Up to this point, the design has been synthesized, floorplanned, and constrained. Now implementation (place and route) of all modules, both fixed and partially reconfigurable, for the design can begin. Each module will be implemented separately, but always in the context of the top-level logic and constraints. Bitstreams will be generated for all reconfigurable modules.

This section gives an overview of how to independently implement each module. Again, this process is in conformance with the flow described for Modular Design.

Copy the .ucf file created during the Initial Budgeting phase (top or initial folder) to the Active Implementation directories for each module (Active/*).

In each active module working directory, augment the local copy of the .ucf file with any module-level timing constraints required to specify the performance requirements for that module.

The Constraints Editor can be used to create module-level timing constraints. Run ngdbuild, map, par, bitgen, and pimcreate for each module according to the script in the example design described in "Saving Block RAM (BRAM) Contents with SaveData". This will result in a placed and routed module as well as a module-specific bitstream.

The PimCreate process "publishes" the routed design and associated files to the PIMs folder. This will be used during the Final Assembly phase later in the implementation process.

Optionally, run netgen if module-level simulation is to be done. See "NetGen" chapter for more information.

Using FPGA Editor, visually inspect the routed design to verify that routing does not expand beyond the module boundary except, of course, for signals traversing to other modules via the bus macro structures.

The following shows a view in FPGA Editor of such a routed design.

Figure 5-6: FPGA Editor Routed Design for Reconfigurable Module

Repeat steps 1 through 4 for each module in the design. To help verify proper practice during the Active Module implementation phase, consult "Checklist for Top-Level HDL Design" for more information.

Final Assembly Overview

The Final Assembly phase is the process of combining each of the individual modules back into a complete FPGA design. The placement and routing achieved during the Active Implementation phase for each module will be preserved, thereby, maintaining the performance of each module.

The steps of the Final Assembly phase are to be run in the Top/Assemble<n> directories of the recommended project directory structure. Where <n> corresponds to each possible combination of fixed and reconfigurable modules.

At the very minimum, at least one Final Assembly must be run ( Top/Assemble<1>). The Partial Reconfiguration flow requires that the initial bitstream loaded into the FPGA device be a complete design. This is required so that all global, non-reconfigurable logic is placed and locked down, and that only reconfigurable portions of the design will change during Partial Reconfiguration. However, it is highly recommended that all module combinations are compiled into unique assemblies (top or assemble <1 ..n>). Compiling each possible combination allows simulation and verification that each combination functions as intended. Do the following:

Copy the .ucf file created during the initial budgeting phase (top or initial folder) to the final assembly directory for each full design combination (top or assemble<n>).

Run ngdbuild, map, par, and bitgen according to the example Final Assembly script in "Saving Block RAM (BRAM) Contents with SaveData".

This will result in a placed and routed design as well as a full-design bitstream. Optionally, run netgen if simulation is to be done. See "NetGen" in Chapter 24.

Using FPGA Editor, visually inspect the routed design to verify that local module routing does not expand beyond the module boundaries except, of course, for signals traversing to other modules via the bus macro structures.

Repeat steps 1 through 4 for each possible combination of fixed and reconfigurable modules in the design. To help verify proper practice during the active module implementation phase, consult the "Checklist for Top-Level HDL Design" section.

Creating Module-Based Partial Reconfiguration Bitstreams

The -g ActiveReconfig:Yes option is required for Active Partial Reconfiguration, meaning that the device remains in full operation while the new partial bitstream is being downloaded. If the ActiveReconfig:Yes is not specified, or the -g ActiveReconfig:No is specified, then the partial bitstream contains the Shutdown and AGHIGH commands used to deassert DONE. All I/Os and internal routing should be high impedance, and writing to registers should be disabled. The changes described in this chapter can be done safely with the -g ActiveReconfig option set to Yes. Additionally, the -g Persist:Yes switch is required when utilizing Partial Reconfiguration through the SelectMAP mode. This switch allows the SelectMAP pins to persist after the device is configured, which allows the SelectMAP interface to be used for reconfiguration.

A Partial Reconfiguration bitstream can be created with any other BitGen option, including the -b option (create .rbt file) or any -g options specifying configuration options, except for encryption. A device that has been configured with an encrypted bitstream cannot be partially reconfigured. Similarly, a device cannot be partially reconfigured with an encrypted bitstream.

At present, multiple bitstreams created for a fully assembled design require one bitstream (at a minimum) for the initial configuration of the device and one for every partially reconfigurable module variation. As an example, if Module A is reconfigurable, and four possible configurations for Module A are designed, bitstreams for Modules A1, A2, A3, and A4 need to be created. This can be done by running the BitGen program within each module directory in the Modular Design. See "BitGen" in Chapter 16.

This produces a configuration file (designA1.bit) that only configures the frames encompassed by the AREA GROUP for A1 in the modular design. When downloading this file, the full bitstream configuration must already be programmed into the device.