A5E-QSFP-Fabric-Example¶

Overview¶

The A5E QSFP example builds on the base A5E reference design and adds the qsfp_fabric_subsys subsystem, which drives a 40 Gigabit Ethernet link out of the FPGA fabric to the QSFP+ cage.

The link runs over four GTS transceiver lanes at 10.3125 Gbps each. A packet client in the fabric is the traffic source and sink: it generates Ethernet frames, hands them to the MAC, and counts the frames that come back on the receive side. Everything the example needs is in the FPGA, so the only host software involved is memtool to reach the registers over the lightweight bridge.

The data path is set up for external loopback. Frames return only when the QSFP path is looped outside the FPGA, either with a QSFP loopback module in the cage or with two boards cabled together.

This example is provided for the MitySBC platform only.

QSFP Fabric Subsystem¶

The subsystem contains fourteen blocks. The link and datapath:

intel_eth_e40_0 - the Low Latency 40G Ethernet IP, the MAC and PCS for the link
alt_e40_packet_client_0 - the packet generator, the counters and the shared register window, wrapping Altera's example design packet client
intel_systemclk_gts_0 - the GTS system PLL, which turns the 156.25 MHz reference clock into the 322.265625 MHz transceiver system clock

The software interface. Three of the PIOs sit behind a small adapter that splits the PIO's register bits out into the individual named signals:

qsfp_control_pio_0 - a 4-bit output PIO driving the MAC and packet client resets
qsfp_control_adapter_0 - the adapter behind it
qsfp_status_pio_0 - a 6-bit input PIO reporting lane, PCS and PLL status
qsfp_status_adapter_0 - the adapter behind it
reset_status_pio_0 - a 3-bit input PIO reporting the MAC's reset acknowledges
reset_status_adapter_0 - the adapter behind it
qsfp_in_pio_0 - a 2-bit input PIO for the QSFP module's present and interrupt pins
qsfp_out_pio_0 - a 3-bit output PIO for the QSFP module's select, reset and low-power pins

The bridges that connect it to the rest of the design:

lwclock_bridge_0 - brings the HPS user0 clock into the subsystem
lwbus_mm_bridge_0 - the subsystem's slave on the HPS lightweight bridge, in front of every register below
reset_bridge_mmclk - brings the fabric reset in on that clock domain

                      ┌─────────────────────────────────┐
    HPS lightweight   │       lwbus_mm_bridge_0         │
    bridge  ─────────►│      (all registers below)      │
    0x30000000        └───┬────────────┬────────────┬───┘
                          │            │            │
                          ▼            ▼            ▼
                 ┌──────────────┐ ┌──────────┐ ┌──────────┐
                 │ alt_e40_     │ │ status,  │ │ QSFP in  │
                 │ packet_      │ │ control, │ │ and out  │
                 │ client_0     │ │ reset    │ │ PIOs     │
                 │              │ │ PIOs     │ │          │
                 │  packet gen  │ └────┬─────┘ └────┬─────┘
                 │  counters    │      │            │
                 │  CSR window  │      │ resets     │ MODPRSN, INTN,
                 └──────┬───────┘      │ and        │ MODESELN,
                        │ Avalon-ST    │ status     │ RESETN, LPMODE
                        │ TX / RX      │            │
                        ▼              ▼            │
                   ┌─────────────────────────────┐  │
    QSFP_CLK_P ───►│                             │  │
    156.25 MHz     │      intel_eth_e40_0        │  │
         │         │   40GbE MAC + PCS, 4 lanes  │  │
         │    ┌───►│                             │  │
         │    │    └──────┬────────────▲─────────┘  │
         │    │ system    │ TX         │ RX         │
         ▼    │ clock     │ 4 lanes    │ 4 lanes    ▼
    ┌─────────┴──────┐    │            │         sideband
    │ intel_systemclk│    │            │           pins
    │ _gts_0         │    ▼            ▲            │
    │ 322.265625 MHz │ QSFP_TX_P/N  QSFP_RX_P/N     │
    └────────────────┘    │            ▲            │
                          ▼            │            ▼
                      ┌────────────────────────────────────┐
                      │             QSFP+ cage             │
                      └──────────┬─────────────▲───────────┘
                                 └─────────────┘
                                external loopback

What This Project Offers¶

This project provides:

A working 40 Gigabit Ethernet instantiation with four serial lanes routed to the QSFP+ cage.
A packet generator and packet counters in fabric, so the link can be exercised with no host software beyond memtool.
Status registers covering lane stability, PCS readiness, alignment lock, system PLL lock and reset acknowledge.
QSFP module sideband control through fabric PIOs, covering module present, interrupt, select, reset and low-power mode.
A starting point for attaching your own Avalon-ST traffic source in place of the packet client.

Link Capabilities¶

Property	Value
Speed	40 Gbps
Lanes	4 at 10.3125 Gbps
Reference clock	156.25 MHz
MAC statistics	Enabled
TX CRC insertion	Enabled
Forward error correction	Not enabled
Auto-negotiation and link training	Not enabled
Flow control	Not enabled
Link fault signalling	Not enabled
Synchronous Ethernet	Not enabled

Because auto-negotiation and link training are not built in, the far end has to be running at the same fixed rate. That is what a loopback module or a second board with this same design gives you.

Address Map¶

The subsystem sits on the lightweight bridge at 0x30000000. Within that, the packet client presents one 64 KB register window, the reconfiguration interface presents an 8 MB window, and the PIOs sit just above the first of those.

Absolute Address	Size	Block	Access
`0x30000000`	64 KB	Packet client CSR window, see below	RW
`0x30010400`	16 B	QSFP output PIO	RW
`0x30010410`	16 B	QSFP input PIO	R
`0x30010420`	16 B	Control PIO (resets)	RW
`0x30010430`	16 B	Status PIO	R
`0x30010440`	16 B	Reset status PIO	R
`0x30800000`	8 MB	Transceiver reconfiguration interface, see the GTS Transceiver PHY User Guide	RW
`0x20000500`	16 B	GTS refclk fail status PIO, outside this subsystem	R
`0x20000520`	16 B	GTS refclk on PIO, outside this subsystem	W

The CSR window at 0x30000000 is shared between two clients:

Absolute Address	Client
`0x30000000` - `0x300037FC`	Ethernet IP status port, the MAC, PCS and statistics registers
`0x30004000` - `0x300040FC`	Packet client registers, listed below

A read to an address inside the 64 KB window that neither client claims returns 0x12345678. That is the read arbiter's timeout value rather than register content.

Ethernet IP status port¶

The status port decodes the whole range above, but only part of it is populated. An offset the IP does not implement reads 0xDEADC0DE, which distinguishes it from the 0x12345678 returned for an address outside the port altogether.

The registers themselves are described in the Low Latency 40G Ethernet IP User Guide, which gives them as word offsets. Multiply by four for the byte address used here, so the guide's offset 0x800 is at 0x30002000.

The two statistics blocks are the ones worth knowing about, because they are the MAC's own view of the link and are independent of the packet client's counters:

Absolute Address	Block
`0x30002000`	Transmit statistics
`0x30002400`	Receive statistics

MAC statistics are enabled in this design, and both blocks include a frame-size histogram: running the generator in fixed-length mode advances exactly one of its bins and leaves the rest alone, which makes it a quick way to confirm what size frames are actually on the wire.

Packet Client Registers¶

Absolute Address	Register	Access	Description
`0x30004000`	Scratch	RW	Read/write scratch register, no hardware effect.
`0x30004004`	Identifier	R	Reads `0x434C4E54`, the ASCII string `CLNT`.
`0x3000400C`	RX status	R	Bit 0 = RX PCS ready, bit 1 = RX alignment marker lock, bit 2 = RX block lock.
`0x30004010`	TX packet count	R	Frames transmitted since the counters were last cleared.
`0x30004014`	RX packet count	R	Frames received since the counters were last cleared.
`0x30004018`	RX error count	R	Received frames flagged with an error.
`0x3000401C`	Counter control	RW	Bit 0 = hold the TX/RX/error counters in reset, bit 1 = clear the packet client's dropped-block counter, which is not itself readable here.
`0x30004020`	Packet length range	RW	Bits [13:0] = start length in bytes, bits [29:16] = end length. Reads `0x25800040` after reset, that is 64 and 9600 bytes.
`0x30004024`	Packet count limit	RW	Number of frames to send when the generator is set to stop after a fixed count. Reads `0x0000000A` after reset.
`0x30004040`	Generator control	RW	See the bit definitions below. Reads back `0x00000006` after reset.
`0x30004044`	Destination MAC low	RW	Lower 32 bits of the generated frames' destination address.
`0x30004048`	Destination MAC high	RW	Upper 16 bits, in bits [15:0].
`0x3000404C`	Source MAC low	RW	Lower 32 bits of the generated frames' source address.
`0x30004050`	Source MAC high	RW	Upper 16 bits, in bits [15:0].

Generator control (0x30004040)¶

Bit	Name	Description
0	reserved	Not used by the generator.
1	`gen_disable`	1 stops the packet generator, 0 runs it. Set at reset, so the generator is idle until software clears it.
2	reserved	Not used by the generator.
3	`client_loopback`	1 takes the transmitter's data from received frames instead of from the generator, so this board echoes what the far end sends.
5:4	`pattern_mode`	Frame length pattern: 00 random, 01 fixed, 10 incrementing.
6	`end_sel`	0 runs until stopped, 1 stops after the packet count limit is reached.
7	`ipg_sel`	0 uses a random inter-packet gap, 1 sends frames back to back.

Bit 1 is inverted with respect to the rest: the generator runs when it is clear. The reset value of the whole register is 0x06, so the design comes up with the generator held off.

The three frame length modes use the packet length range register differently. Fixed mode sends every frame at the start length. Incrementing mode ramps the length from start to end one byte per frame and then wraps back to start. Random mode ignores the register entirely and picks each length between a fixed 64 and 9600 bytes, so the start and end lengths only take effect in the other two modes.

With end_sel set, the count restarts on the transition of bit 1 from set to clear, so a second counted run needs the generator stopped and started again rather than a second write of the same value.

Client loopback is for a two-board setup, where the far end supplies the traffic. On a single board with an external loopback it produces nothing: the transmitter's only source is then its own receiver, which has nothing to give it. Switching it on mid-run stops traffic and counts the frames that were in flight as receive errors.

Status and Control PIOs¶

Status PIO (0x30010430)¶

Bit	Name	Description
0	`tx_lanes_stable`	All four transmit lanes are stable.
1	`rx_pcs_ready`	The receive PCS has completed its startup.
2	`rx_block_lock`	The receiver has block lock on every lane.
3	`rx_am_lock`	The receiver has alignment marker lock, so the four lanes are deskewed.
4	`syspll_lock`	The GTS system PLL is locked.
5	`csr_rst_n`	The MAC's control and status reset, active low. 1 means the MAC is out of reset.

Reset status PIO (0x30010440)¶

Bit	Name	Description
0	`csr_rst_ack_n`	Active-low acknowledge for the control and status reset.
1	`tx_rst_ack_n`	Active-low acknowledge for the transmit reset.
2	`rx_rst_ack_n`	Active-low acknowledge for the receive reset.

These bits are active-low acknowledges, so all three high is the normal running state. They are mainly a debug aid when a reset does not release.

Note that the receive bits shared with the RX status register at 0x3000400C are not in the same order in both places: that register has alignment marker lock below block lock, and the status PIO has them the other way round.

Control PIO (0x30010420)¶

Bit	Name	Description
0	`csr_reset`	Assert the Ethernet IP's control and status reset.
1	`tx_reset`	Assert the Ethernet IP's transmit reset.
2	`rx_reset`	Assert the Ethernet IP's receive reset.
3	`pc_reset`	Assert the packet client's reset.

All four are active high and all four are clear at reset, so the design comes up with nothing held in reset. Writing a 1 asserts the corresponding reset and holds it until the bit is cleared again. Each of the first three drives a matching acknowledge low in the reset status PIO, and shows up in the status PIO as well:

Written	Reset	Status PIO	Reset status PIO
`0x1`	`csr_reset`	`0x10`	`0x06`
`0x2`	`tx_reset`	`0x30`	`0x05`
`0x4`	`rx_reset`	`0x31`	`0x03`
`0x8`	`pc_reset`	`0x3F`	`0x07`

Clearing the bit restores 0x3F and 0x07 in every case, with no further action needed: the link relocks on its own.

pc_reset is the odd one out. It resets the packet client rather than the Ethernet IP, so it leaves both status registers untouched and has no acknowledge of its own. What it does do is halt the packet client: the transmit and receive counts freeze while it is held and resume when it is released. It does not restore the packet client's registers to their reset values, so the generator control register keeps whatever was last written to it.

QSFP input PIO (0x30010410)¶

Bit	Name	Description
0	`QSFP_MODPRSN`	Module present, active low. 0 means a module is seated in the cage.
1	`QSFP_INTN`	Module interrupt, active low.

QSFP output PIO (0x30010400)¶

Bit	Name	Description
0	`QSFP_MODESELN`	Module select for the module's two-wire management interface, active low.
1	`QSFP_RESETN`	Module reset, active low.
2	`QSFP_LPMODE`	Low power mode request, active high.

The reset value is 0x00000003: the module is deselected, held out of reset and not in low power mode. All three bits are writable and read back, and none of them disturbs the link. The design brings these pins under software control but includes no controller for the module's two-wire management interface, so reading a module's identification EEPROM is not part of this example.

GTS refclk fail status PIO (0x20000500)¶

This PIO exposes gts_rst_seq_left.o_refclk_fail_status[7:0], the status of the GTS input reference clock buffer protection. It sits at 0x20000500, outside the 0x30000000 block used by the rest of this subsystem, because the GTS reset sequencer lives at the top level of the design rather than inside qsfp_fabric_subsys. Only the left-hand sequencer is instantiated for this design. The GTS hardware monitors each enabled refclk input buffer and automatically turns a buffer off when it no longer detects a toggling clock at the input (the detector covers 25 MHz to 380 MHz; a clock outside that range is also treated as absent). A bit value of 1 means the corresponding refclk buffer was enabled at configuration but has since been turned off due to the absence of a clock at its input. A bit value of 0 means the buffer has not been turned off.

NOTE - A refclk buffer that has been turned off by the protection mechanism does not turn back on by itself. On ES silicon, the register method must be used to turn a refclk buffer back on. On production devices, either the register method or the status and control ports method may be used. However, the design is currently only configured to support the register method, which is driven from gts_left_refclk_on_pio, a 10-bit write-only PIO at 0x20000520. See section 3.7.5 of the GTS Transceiver PHY User Guide for more details

Clocking and Reset¶

The subsystem takes two clocks from outside:

lwclock_bridge_0 carries the 100 MHz HPS user0 clock, from hps_subsys.intel_agilex_5_soc_0_h2f_user0_clk. It clocks the lightweight bridge slave, every register in this subsystem and the Ethernet IP's status and reconfiguration ports.
QSFP_CLK_P is the 156.25 MHz transceiver reference clock. It feeds both the GTS system PLL and the Ethernet IP's reference clock input.

Because the register clock comes from the HPS, the HPS must be out of reset and clocking the fabric before any of these registers respond.

On the MitySBC the 156.25 MHz reference comes from the U27 Si5338 clock generator, output 0. That is the generator's factory configuration, so this example needs no clock generator programming during boot.

Reset comes from fabric_reset_controller_0.reset_out at the top level and enters the subsystem through reset_bridge_mmclk. Software can additionally assert the Ethernet IP's individual resets and the packet client's reset through the control PIO.

Board Interface Signals¶

Signal	Direction	Purpose
QSFP_TX_P/N[3:0]	output	the four transmit lanes
QSFP_RX_P/N[3:0]	input	the four receive lanes
QSFP_CLK_P	input	156.25 MHz transceiver reference clock
QSFP_MODPRSN	input	module present, active low
QSFP_INTN	input	module interrupt, active low
QSFP_MODESELN	output	module select, active low
QSFP_RESETN	output	module reset, active low
QSFP_LPMODE	output	low power mode request

The five sideband signals are 3.3 V LVCMOS. The reference clock is CML and the eight serial pins are high speed differential. All eight lanes and the reference clock are on GTS bank 1A, which is why the refclk buffer protection status for this example is read from the left-hand sequencer's PIO.

Scope of This Example¶

The traffic source in this design is a packet generator, not a real Ethernet stack. The MAC presents a plain Avalon-ST interface on both sides, so the packet client can be replaced with another traffic source that produces the same stream format.

That makes it a good base for:

bringing up and characterising a QSFP module, cable assembly or optical link
measuring the 40 Gigabit transmit and receive path on a new carrier design
attaching a custom protocol or offload engine in the fabric to a 40 Gigabit port

There is no Linux network interface in this example. The MAC is not connected to the HPS Ethernet controllers and there is no device tree overlay or kernel driver for it, so it will not appear in ip link.

Building the Example¶

Compile the FPGA design¶

Refer to Building the FPGA and Bootloader for building the FPGA design. Navigate into the mitysbc-a5e-ref-qsfp-fabric example project before compiling the design.

Ensure to flash the resulting a5e.hps.jic onto the hardware
Ensure to replace the a5e.core.rbf on the SD card

Using the Example¶

Hardware setup¶

Program a design that includes this QSFP fabric example.
Create the external transmit-to-receive path. Either fit a QSFP+ loopback module in the cage, or cable two boards together with a QSFP+ cable, both running this design.
Boot Linux on the HPS so memtool can access the lightweight bridge.

The link comes up on its own once the FPGA is configured and the external path is in place. Nothing has to be started to bring the link up; the commands below only start and observe traffic.

Optional pre-check¶

Before starting traffic, confirm the register window is alive and the link side is up:

memtool 0x30004004 1
# Expected: 0x434C4E54, the packet client identifier. Anything else means
# the design on the board is not this example.

memtool 0x30010430 1
# Status PIO. Expected healthy value: 0x0000003F.

memtool 0x30010440 1
# Reset status PIO. Expected healthy value: 0x00000007.

memtool 0x30010410 1
# QSFP input PIO. Expected 0x00000002 with a module seated:
# module present is active low, module interrupt is inactive.

memtool 0x20000500 1
# GTS refclk fail status. Expected 0x00000000, all refclk buffers on.

Typical signs of a healthy path are:

The status PIO reads 0x3F: lanes stable, PCS ready, block lock, alignment marker lock, PLL locked and the MAC out of reset.
The reset status PIO reads 0x07, all three reset acknowledges released.
GTS refclk fail status reads 0x00000000, no refclk buffer has been turned off.

The three receive bits only assert once the receiver sees a valid signal, so with nothing looping the transmit side back they stay clear and the status PIO reads a lower value. The system PLL lock and reset bits are unaffected by that, since they do not depend on traffic.

Note that memtool takes and prints hexadecimal, so every value on this page is hexadecimal whether or not it carries an 0x prefix.

Start packet generation¶

The generator is held off at reset by bit 1 of the generator control register. Clear that bit to start it:

memtool 0x30004040=0x84

In this example, 0x84 means:

bit 1 clear = run the packet generator
bits [5:4] = 00, random frame lengths between 64 and 9600 bytes
bit 6 clear = keep running until stopped
bit 7 set = no inter-packet gap, so the link runs at line rate

For a run that stops on its own, set the count and the mode while the generator is still off, then clear bit 1 to start it:

memtool 0x30004024=0x1000
memtool 0x30004040=0xC6
memtool 0x30004040=0xC4

That sends 0x1000 frames and stops, and the counters settle at 0x00001000 transmitted and 0x00001000 received. A counted run is the easiest way to prove the loopback path is lossless, since transmit and receive should end on exactly the same number.

Verify TX/RX packet activity¶

Read the transmit count, the receive count and the receive error count:

memtool 0x30004010 1
memtool 0x30004014 1
memtool 0x30004018 1

To watch all three at once, read the three words starting at 0x30004010:

watch memtool 0x30004010 3

Expected behavior:

0x30004010 increments as frames are transmitted by the packet client.
0x30004014 increments as those frames return through the external loopback path.
0x30004018 stays at zero during a clean run.
Receive should track transmit closely in steady state.

The counters are 32-bit and wrap. At 40 Gbps that takes minutes rather than hours, so compare successive reads rather than treating a value as a running total.

Two reads a known time apart also give the link rate. With fixed 512-byte frames and no inter-packet gap the transmit count advances by about 9.4 million frames per second, which at the standard 20 bytes of preamble and inter-packet gap per frame is 40 Gbps, or very close to 10 Gbps of payload on each of the four lanes.

To clear the counters, pulse bit 0 of the counter control register:

memtool 0x3000401C=1
memtool 0x3000401C=0

Stop packet generation¶

memtool 0x30004040=0x06

That restores the register's reset value, which holds the generator off.

Troubleshooting¶

The packet client identifier does not read back¶

If memtool 0x30004004 returns something other than 0x434C4E54, the FPGA is not running this design or the lightweight bridge is not up. Two return values narrow it down: 0x12345678 means the register window responded but no client claimed that address, and 0xDEADC0DE means the Ethernet IP's status port claimed it but does not implement it. Either one points at a wrong offset rather than a dead bridge.

The system ID at 0x20000000 does not distinguish this design from the other examples built from the same commit, so the packet client identifier is the register to check.

Transmit counts increase but receive counts stay at zero¶

The generator is running but frames are not coming back. Check the external path first: a QSFP loopback module has to be fully seated, and a two-board setup needs both ends programmed with this design. Then read the status PIO at 0x30010430; if block lock and alignment marker lock are clear, the receiver is not seeing a valid signal at all.

The status PIO never reaches 0x3F¶

Read the GTS refclk fail status at 0x20000500 first. A non-zero value means a reference clock buffer was turned off because its clock stopped, and it will not turn back on by itself. If the refclk status is clean but the system PLL lock bit stays clear, the 156.25 MHz reference is missing or out of tolerance.

The module present bit reads 1 with a module fitted¶

QSFP_MODPRSN is active low, so 1 means no module detected. Reseat the module. This bit only reports the cage's presence contact; it says nothing about whether the module is working.

References¶

Files (0)

Go to top

Project

General

Profile

Mity CPU Platforms » MitySBC-Agilex5

Wiki