A5E-DMA-Example¶

Overview¶

The DMA example works by demonstrating the following:

The FPGA XorShift Data Generator generates a data pattern which is written into the HPS's DDR through the RX mSGDMA.
- The HPS verifies this data that is written to ensure that the expected data is received.
  - This demonstrates the data path from the FPGA to the HPS
The HPS generates an xorshift data pattern which it writes into the HPS DDR
- The FPGA TX mSGDMA reads this data and passes it to the XorShift Data Checker to verify the expected data is received.
  - This demonstrates the data path from the HPS to the FPGA

DMA Subsystem¶

The example moves data in both directions between the FPGA fabric and the HPS DDR, using an mSGDMA engine per direction and an XorShift pattern generator and checker to prove the data arrives intact.

┌──────────────────────────────┐         ┌────────────────────────┐
│  XorShift Data Generator     │         │       RX mSGDMA        │
│        (pattern source)      ├────────►│     (FPGA → HPS DDR)   │
└──────────────────────────────┘         └───────────┬────────────┘
                                                     │
                                                     │ AXI writes
                                                     ▼
        ┌────────────────────────────────────────────────────────┐
        │                          HPS DDR                       │
        │   (shared memory via FPGA-HPS interconnect bridge)     │
        └────────────────────────────────────────────────────────┘
                                                     │
                                                     │ AXI reads
                                                     ▼
┌──────────────────────────────┐         ┌────────────────────────┐
│  XorShift Data Checker       │◄────────┤       TX mSGDMA        │
│     (pattern verification)   │         │     (HPS DDR → FPGA)   │
└──────────────────────────────┘         └────────────────────────┘

The subsystem contains:

dma_write_msgdma_0 - the RX engine, at 0x20010040. It writes fabric data into HPS DDR and backs /dev/cl_msgdma_rx.
dma_read_msgdma_0 - the TX engine, at 0x20010000. It reads HPS DDR and streams to fabric, and backs /dev/cl_msgdma_tx.
dma_xorshift_generator_0 - the fabric-side pattern source checked by the HPS on the RX path.
dma_xorshift_checker_0 - the fabric-side pattern checker for data the HPS sends on the TX path.
dma_cct_bridge_0 - an ACE5-Lite Cache Coherency Translator in front of the F2H bridge.
dma_iopll_0 and the clock/reset bridges that give the subsystem its own stream clock domain.

The mSGDMA engines reach the HPS DDR through one of the Agilex 5 HPS FPGA-to-HPS bridge interfaces. As released, this example uses the FPGA-to-HPS (F2H) bridge. The FPGA-to-SDRAM (F2SDRAM) bridge is the alternative, and the two sections below describe what changes if you rework the design to use it. Each has different coherency behavior and therefore a different device-tree configuration. Refer to the Agilex 5 SoC FPGA Technical Reference Manual ("FPGA-to-HPS, HPS-to-FPGA, and FPGA-to-SDRAM Bridges" material) for the full description of each interface.

FPGA-to-HPS (F2H) Bridge - cache coherent (default configuration)¶

The F2H bridge is an ACE5-Lite interface that routes FPGA transactions through the HPS Cache Coherency Unit (CCU). Because traffic passes through the CCU, the FPGA can issue coherent accesses to HPS DDR: reads observe data that is still resident in the A55/A76 caches, and writes are reflected into the caches, with no software cache maintenance required.

To drive the correct ACE5-Lite coherency/snoop attributes, this design instantiates an ACE5-Lite Cache Coherency Translator (CCT) inside the DMA subsystem, in front of the F2H bridge. On the software side, the device-tree overlay marks each mSGDMA node dma-coherent and assigns it an SMMU stream ID via the iommus property:

dma_read: dma-controller@10000 {
    compatible = "altr,socfpga-msgdma";
    ...
    iommus = <&smmu 0x101>;
    dma-coherent;
};

dma_write: dma-controller@10040 {
    compatible = "altr,socfpga-msgdma";
    ...
    iommus = <&smmu 0x100>;
    dma-coherent;
};

With dma-coherent present, the kernel treats the buffers as coherent and the DMA mapping layer emits no cache maintenance, because the hardware coherency path keeps the CPU caches and DDR in sync.

FPGA-to-SDRAM (F2SDRAM) Bridge - non-coherent¶

The F2SDRAM bridge connects the FPGA more directly to the HPS hard memory controller, bypassing the CCU. It is non-coherent: the CPU caches are not snooped, so software must perform cache maintenance around every DMA transfer.

To use the F2SDRAM path, remove both the dma-coherent and iommus lines from each mSGDMA node in the overlay:

dma_read: dma-controller@10000 {
    compatible = "altr,socfpga-msgdma";
    ...
    /* no iommus property */
    /* no dma-coherent property */
};

With those two properties removed, the kernel marks the buffers non-coherent and the DMA mapping layer performs the required CPU cache maintenance (clean before the FPGA reads, invalidate after the FPGA writes) when the buffers are mapped and unmapped. This is handled by the kernel, not by the cl-msgdma driver: the driver attaches each dma-buf to the mSGDMA device and lets the DMA API decide what maintenance a given transfer needs.

Note: selecting the F2SDRAM path is not purely a device-tree change - the FPGA design must also be built to route the mSGDMA masters to the F2SDRAM port. The overlay edit above is the software-visible half of that configuration.

When wiring the FPGA design to the F2SDRAM port, the f2sdram_adapter_256_hw.tcl bridge adapter must be placed between the F2SDRAM bridge port and the mSGDMA masters. Both the F2SDRAM port and the adapter are AXI4 interfaces; the adapter is an AXI4-to-AXI4 pass-through that forces the AXI sideband signals (notably the awuser/aruser user bits, along with awcache/arcache and awprot/arprot) to the fixed values the F2SDRAM port expects for non-coherent access. The interconnect auto-generated for the mSGDMA masters leaves those user bits at 0, which the F2SDRAM port mishandles - so without the adapter the mSGDMA descriptors complete but data is silently dropped on writes or returned scrambled on reads.

What This Project Offers¶

This project provides:

A complete bidirectional DMA path between fabric and HPS DDR, exercised end to end by a supplied application.
Hardware pattern generation and checking, so a failure is reported by the design rather than inferred from a hexdump.
A cache-coherent path through the F2H bridge and the HPS SMMU, with no software cache maintenance required.
A dma-buf based kernel interface (cl-msgdma) that ships in the prebuilt image and loads itself.
A worked example of driving the Altera mSGDMA from userspace, usable as a starting point for your own transfers.

Building the Example¶

Prerequisites¶

MitySOM-A5E, MitySOM-A5E Mini or MitySBC-A5E Development Kit
Intel Quartus Prime Pro 26.1
The MitySOM-A5E Yocto SDK toolchain, for cross-compiling the application
The prebuilt SD card image, which supplies the kernel, the cl-msgdma driver and the device tree overlay

A local checkout of linux-socfpga is optional. The application needs one header from it, which the build can either take from a local tree or fetch on its own. See Build the userspace application.

Compile the FPGA design¶

Refer to Building the FPGA and Bootloader (MitySOM) or Building the FPGA and Bootloader (MitySBC) for building the FPGA design. Navigate into the mitysom-a5e[-mini]-ref-dma or mitysbc-a5e-ref-dma example project before compiling the design.

Ensure to flash the resulting a5e.hps.jic onto the hardware
Ensure to replace the a5e.core.rbf and boot.scr on the SD card

Generate the FPGA design headers¶

Still in the FPGA project directory, run the following to generate the header files that describe the FPGA design

make generate_headers

Where the driver and overlay live¶

The kernel side of this example is not part of the FPGA reference project. Both pieces live in the Linux kernel source and ship in the prebuilt SD card image, so there is nothing to compile or copy for either of them:

Component	Path in linux-socfpga	On the target
dma-buf shim driver	`drivers/misc/cl-msgdma.c`	`/lib/modules/<release>/kernel/drivers/misc/cl-msgdma.ko`
Driver ABI header	`include/uapi/linux/cl_msgdma.h`	used when building the application
Device tree overlay	`arch/arm64/boot/dts/intel/socfpga_agilex5_mity_a5e_devkit_dma_example.dtso`	`/boot/socfpga_agilex5_mity_a5e_devkit_dma_example.dtbo`

Browse them at gitweb to see how the pieces fit together: the overlay declares the two mSGDMA controllers and a cl,msgdma node for each direction, and the driver binds to those nodes and exposes the character devices.

If you want to modify the driver or the overlay and redeploy, follow Linux_Kernel for building and installing a kernel from source.

Build the userspace application¶

Source the SDK environment and build:

source /path/to/toolchain/environment-setup-armv8-2a-mitysom-linux
cd software/dma_example/userspace

Supply exactly one of the following so the build can obtain linux/cl_msgdma.h, which defines the driver ioctl interface:

# (a) you already have a linux-socfpga tree checked out
make KERNEL_SRC=/path/to/linux-socfpga

# (b) fetch just that one header from the kernel repository
make KERNEL_BRANCH=socfpga-6.18.2-lts

Option (b) transfers a single file and needs no kernel checkout, but requires outbound access to git://support.criticallink.com (port 9418). Behind a firewall that blocks it, use option (a). A commit id cannot be used in place of a branch name, because the git daemon only resolves named refs. If both variables are set, KERNEL_SRC wins. To list the available branches:

git ls-remote --heads git://support.criticallink.com/home/git/linux-socfpga.git

Using the Example¶

Deploy¶

Artifact	Destination	Supplied by
`a5e.hps.jic`	QSPI flash	FPGA build
`a5e.core.rbf`	`/lib/firmware/`	FPGA build
`boot.scr`	SD card FAT partition	FPGA build
`socfpga_agilex5_mity_a5e_devkit_dma_example.dtbo`	`/boot/`	prebuilt image
`cl-msgdma.ko`	`/lib/modules/<release>/`	prebuilt image
`dma_example`	`/root/`	you

Boot and verify¶

Boot the SD card and log in as root. Confirm the overlay was applied by looking for the two fabric mSGDMA controllers. The two 10dx0000 entries are the HPS DMA controllers and are always present:

root@mity-a5e:~# ls /sys/bus/platform/devices/ | grep dma-controller
10db0000.dma-controller
10dc0000.dma-controller
20010000.dma-controller
20010040.dma-controller

If the 20010000 and 20010040 entries are missing, check the U-Boot console for the line applying /boot/socfpga_agilex5_mity_a5e_devkit_dma_example.dtbo.

The driver loads itself, because the overlay's cl,msgdma nodes match the module's device tree alias. There is nothing to insmod or modprobe. Confirm the character devices exist:

root@mity-a5e:~# ls /dev/cl_msgdma_*
/dev/cl_msgdma_rx  /dev/cl_msgdma_tx

Run the application¶

root@mity-a5e:~# ./dma_example
Run Configuration:
  buf_size:   4194304
  num_bufs:   8
  num_iters:  1000
  rx_enabled: 1
  tx_enabled: 1
  verify:     1
  bandwidth:  0
  rx_seed:    0xa5a5a5a5
  tx_seed:    0x5a5a5a5a
==========================
Starting DMA Test
Allocating RX buffers
Allocating TX buffers
Queuing all RX buffers
Setting up XorShift generator
Setting up XorShift checker
Priming TX buffers
Starting the RX DMA Engine
Starting the TX DMA Engine
Running DMA loop
Read 1000 buffers from DMA
Data RX check passed
Wrote 1000 buffers to DMA
Data TX check passed
Test PASSED

The final Test PASSED line is the all-good signal. Any Data RX check failed or Data TX check failed line, or a final Test FAILED, indicates a data mismatch.

The application can be run repeatedly with no intervention between runs. Closing the device stops the DMA engine, resets the dispatcher and releases the buffers, so every run starts from a clean engine.

Command-line options¶

Usage: ./dma_example [options]
  -s, --buf-size BYTES   buffer size, multiple of PAGE_SIZE  [default: 4194304]
  -i, --iters N          buffers to push through each direction  [default: 1000]
  -n, --num-bufs N       ring buffers per direction (> 0)  [default: 8]
  -r, --rx-seed HEX      xorshift seed for RX pattern  [default: 0xa5a5a5a5]
  -t, --tx-seed HEX      xorshift seed for TX pattern  [default: 0x5a5a5a5a]
  -R, --rx-only          run RX phase only
  -T, --tx-only          run TX phase only
  -N, --no-verify        skip CPU-side xorshift verify/generate (raw bandwidth)
  -B, --bandwidth        print per-direction throughput on completion
  -q, --quiet            suppress progress logging

A few notes on the most useful flags:

--bandwidth adds an Elapsed summary line with the measured per-direction throughput. What that figure actually measures depends on the other flags, so see Throughput below before reading anything into it.
--no-verify skips the CPU-side xorshift generate and check. It performs no data check and prints no PASS/FAIL verdict, so confirm correctness separately with a default (verifying) run on the same build.
--rx-only / --tx-only exercise a single direction: --rx-only runs only the FPGA to HPS DDR path, --tx-only runs only the HPS DDR to FPGA path.
--quiet suppresses the progress logging, leaving only the Elapsed summary (when --bandwidth is set) and the final PASS/FAIL verdict, which is handy for scripted test loops.

Throughput¶

Three flag combinations produce an Elapsed line, and each measures something different. Reading a figure from the wrong one is the easiest mistake to make here.

Invocation	What is being measured
`--bandwidth`	The whole loop, including the CPU-side xorshift generate and check. The CPU work dominates by a wide margin, so this is a measure of pattern generation and verification, not of the DMA path.
`--bandwidth --no-verify`	Both DMA directions running at once. They share the F2H bridge and are timed from a common start, so the two figures are not independent throughputs and the slower one is held up by the other.
`--bandwidth --no-verify --rx-only` (or `--tx-only`)	One direction with the bridge to itself. This is the only combination that characterises a single path.

Measured one direction at a time on a MitySOM-A5E Mini development kit:

Direction	Measured
RX, the FPGA writing into HPS DDR	3.08 GB/s
TX, the FPGA reading from HPS DDR	3.14 GB/s

Both sit just under the ceiling this design sets for itself. The mSGDMA engines have a 256-bit data path and are clocked from dma_iopll_0 at 100 MHz, so the fabric can move at most 32 bytes x 100 MHz = 3.2 GB/s. The figures above are 96% and 98% of that.

The limit is the fabric datapath rather than the memory. The HPS LPDDR4 interface is 32 bits wide at 2133 MT/s, about 8.5 GB/s, so the DDR has well over twice the bandwidth this design is able to ask of it. The F2H bridge data width is fixed at 256 bits and is not configurable, so the clock is the only lever: raising the dma_iopll_0 output frequency would raise the ceiling proportionally, subject to timing closure in the fabric.

Both engines draw on that same 3.2 GB/s budget, so exercising both directions at once does not add to it. A concurrent run divides it between them and the combined rate stays close to 3.2 GB/s, which is why the per-direction figures from a bidirectional run come out lower than the same directions measured alone.

Note on TX throughput on R0 silicon: the TX direction is a coherent read through the F2H bridge, and on R0 (ES) Agilex 5 devices it is degraded by a known erratum, Degraded HPS EMIF performance with 2MB L3 cache. The RX direction is not affected and reaches the same rate on R0 as on later silicon. Because both directions share the bridge, a bidirectional run on an R0 part also drags the RX figure well below what RX alone achieves, so measure a single direction when characterising either path. This is a silicon limitation, not a defect in the design or the example.

Troubleshooting¶

/dev/cl_msgdma_rx and /dev/cl_msgdma_tx do not exist¶

Confirm the overlay was applied (see Boot and verify), then check dmesg for the probe messages. A healthy boot looks like this:

[    7.023699] altera-msgdma 20010040.dma-controller: Altera mSGDMA driver probe success
[    7.060454] cl-msgdma write-cl-msgdma: registered /dev/cl_msgdma_rx
[    7.069656] cl-msgdma write-cl-msgdma: CL mSGDMA driver probe success
[    7.138500] altera-msgdma 20010000.dma-controller: optional resource resp not defined
[    7.195517] altera-msgdma 20010000.dma-controller: Altera mSGDMA driver probe success
[    7.477417] cl-msgdma read-cl-msgdma: registered /dev/cl_msgdma_tx
[    7.492628] cl-msgdma read-cl-msgdma: CL mSGDMA driver probe success

-EPROBE_DEFER lines may also appear and are harmless: the shim can probe before the altera-msgdma engine has registered its channel, and is retried automatically. What matters is that a CL mSGDMA driver probe success line follows for each of the two nodes.

Confirm the running kernel carries the driver:

root@mity-a5e:~# ls /lib/modules/$(uname -r)/kernel/drivers/misc/cl-msgdma.ko

fatal error: linux/cl_msgdma.h: No such file or directory¶

Raised when building the application: neither KERNEL_SRC nor KERNEL_BRANCH was supplied, or the branch given predates the driver.

A repeated run hangs or fails its data check¶

This should not happen on a current kernel. It indicates a kernel that predates the altera-msgdma terminate_all fix, so the engine is not being reset between runs. Read each dispatcher's status and control registers. On an idle engine they read:

root@mity-a5e:~# memtool 0x20010000 2      # TX engine, /dev/cl_msgdma_tx
0x20010000:  00000002 0000001C

root@mity-a5e:~# memtool 0x20010040 2      # RX engine, /dev/cl_msgdma_rx
0x20010040:  0000000A 0000001C

STOPPED (bit 5) or STOPPED_ON_EARLY (bit 8) set in the status word, or STOP (bit 0) set in control, means the engine is latched off and every subsequent transfer will fail. On such a kernel, unbind and rebind both controllers between runs:

for d in 20010000.dma-controller 20010040.dma-controller; do
    echo $d > /sys/bus/platform/drivers/altera-msgdma/unbind
    echo $d > /sys/bus/platform/drivers/altera-msgdma/bind
done

dmesg module names look reversed relative to the device nodes¶

This is expected. The probe messages name the engines from the FPGA point of view, while the character devices are named from the HPS point of view, so they appear swapped:

write-cl-msgdma registers /dev/cl_msgdma_rx (the FPGA writes HPS DDR)
read-cl-msgdma registers /dev/cl_msgdma_tx (the FPGA reads HPS DDR)

Files (0)

Go to top

Project

General

Profile

Mity CPU Platforms » MitySOM-A5

Wiki