This Answer Record contains information that will be incorporated into the next revision of UG197.
Users of the Endpoint Block Plus Wrapper for PCI Express do not need to implement the information below; this is already done for the user inside the wrapper.
Information on the Virtex-5 Endpoint Block Plus Wrapper can be found at:
The integrated block for PCI Express provides llk_rx_ch_posted_available, llk_rx_ch_nonposted_available, and llk_rx_ch_completion_available (llk_rx_*_available) signals and llk_rx_preferred_type signal at the transaction layer interface (TLI) to allow the user to implement both strict ordering and relaxed ordering rules. Relaxed ordering allows completion packets to bypass older posted or non-posted packets available in the receive buffers. For more information on relaxed ordering, see section 2.4 of the PCI Express Base Specification. The LogiCORE Endpoint Block Plus for PCI Express uses these signals to implement relaxed ordering when used in "Completion Streaming Mode" to achieve high performance on completions.
The user must take certain precautions when allowing completion packets to bypass older posted or non-posted packets. It is important to note that these precautions are already implemented in the LogiCORE Endpoint Block Plus for PCI Express when used in "Completion Streaming Mode." The following describes how these precautions are implemented in the Block Plus core.
When older posted and non-posted packets are bypassed by completion packets, the user must ensure that any given completion packet is allowed to pass any given non-posted packet only if it is within a 64 packet window from the non-posted packet. User should keep track of when a completion packet arrives relative to a non-posted packet waiting to be drained and ensure that it is within a 64 packet window from the non-posted packet before allowing it to pass. The user can keep track of this by monitoring the llk_rx_nonposted_available signal, llk_rx_preferred_type signal and the credit status information through the management interface. The LogiCORE Endpoint Block Plus for PCI Express implements the above logic.
The user should then switch from draining completions to draining posted or non-posted packets whenever the above requirement is in danger of being violated. If the above requirement is not met, it could result in several undesirable effects such as older non-posted packets passing older posted packets, complete blocking of posted or non-posted packets. This requirement must be met only when the user allows completions to bypass posted and non-posted packets while draining packets from the Receive Buffers. An example below illustrates the above requirement and suggested work-around with a traffic pattern where posted and non-posted packets are sprinkled inside a stream of completions:
1P, 10C, 2NP,50C, 1P, 10C, 1NP, 90C, 2NP ...
For the purpose of the illustration, each packet is described with the type of packet, followed by the sequence number assigned to it by the receive logic. The above sequence would be as follows:
P-1, C-2,...,C-11, NP-12, NP-13, C-14,...,C-63, P-64, C-65, ... C-74, NP-75, C-76,...C-165, NP-166, NP-167
and the following statements will hold true:
1. C-77 (12+65) cannot pass NP-12, C-78 (13+65) cannot pass NP-13, and so forth.
2. If one of the above conditions is violated and C-130 is allowed to pass P-1(1+129), then NP-12 will look younger than P-1 and could be read out ahead of P-1.
For the above example, the user can drain the packets in the following sequence without violating the requirement:
C-1,...,C-76, P-1,NP-12,C-77,NP-13,C-78,...,C-139, P-64, NP-75,C-140,...,C-165, NP-166, NP-167, etc.
However, if the posted packets are large and completions are very short, the completion buffer is at risk of overflow while the posted packet is being drained. The risk is higher if multiple posted or non-posted packets have to be drained before switching back to draining completions. As the risk of completion buffer overflow is dependent on the traffic pattern, the user has three potential solutions to ensure safe operation:
1. Applies to predictable traffic pattern with uniform packet sizes and is used with "Completion Streaming" mode. In such a scenario, the user should switch from draining completions to draining posted/non-Posted packets whenever there is danger of build up of posted/non-posted packets (to avoid completion buffer overflow while draining posted/non-posted packets), or when there is a danger of completion passing a non-posted packet outside the 64 packet window.
2. In all other cases that use "Completion Streaming" mode, the user should restrict the flow control credits for posted packets and non-posted packets to 1 header each. In addition, the user should give preference to draining posted and non-posted packets whenever there is a packet of that type available in the Receive Buffer. This will control the transmission of posted and non-posted packets from the partner device and will always guarantee safe operation independent of the traffic pattern.
3. This is an alternative solution to "2" , which can be used with and without "Completion Streaming" mode. The user should turn off infinite completion credits by setting the attribute INFINITECOMPLETIONS in the Integrated block to FALSE. The user should then switch to draining posted/non-posted packets whenever there is danger of a completion passing a non-posted packet outside the 64 packet window. However, turning off infinite completion credits will result in a non-compliant solution. The user should choose the option that best suits their application. LogiCORE Endpoint Block Plus for PCI Express implements all three solutions and provides them as a user selectable option.