Mity CPU Platforms » MitySOM-A5

Hi Maxim,

Thanks for the detailed write-up, and thanks especially for attaching the archive. I pulled your project apart and diffed it against our reference design, and I think I found your problem. Let me answer your example design questions first, because the answer to those turns out to be the same thing that's breaking your loopback.

First things first: our example design is not a loopback.

This is the key thing to understand before anything else. Our mSG-DMA example is two completely independent one-way data streams that just happen to share the same bridge. They never exchange data with each other:

1. RX path (FPGA to Linux). The XorShift Generator in the fabric makes the data, the Write mSG-DMA converts that stream into memory writes, the data lands in DDR, and the Linux application reads it back and checks it. Seed is 0xA5A5A5A5.
2. TX path (Linux to FPGA). The Linux application generates the pattern into DDR, the Read mSG-DMA pulls it out of DDR and converts it back into a stream, and the XorShift Checker in the fabric verifies it. Seed is 0x5A5A5A5A.

So when you see two different buffer addresses and two different seeds in Linux, that is correct and expected. They're two different tests running at the same time in opposite directions. The Read DMA is NOT reading back what the Write DMA just wrote, even though on Signal Tap it really looks like it is.

Which brings us to the address question.

Why both DMAs show 0xFFC0.0000

The address you're seeing on the bus is not a physical DDR address. It's a virtual address, and there's a translation block sitting in between.

The Agilex 5 has an ARM SMMU (System Memory Management Unit, which is basically an MMU for peripherals instead of for the CPU). It sits between the FPGA-to-HPS bridge and memory. Every transaction coming out of the fabric carries a StreamID, and the SMMU uses that StreamID to pick which translation table to apply. Each StreamID effectively gets its own private address space.

In our design the StreamIDs are set on the ACE5-Lite Cache Coherency Translator, and they're split by channel:

• Read channel (AR): ARMMUSID_OVERRIDE = 257, which is 0x101
• Write channel (AW): AWMMUSID_OVERRIDE = 256, which is 0x100

You can see the matching half of this in our device tree overlay (software/dma_example/kernel_dts/dma_example.dtso):

dma_read:   iommus = <&smmu 0x101>;
dma_write:  iommus = <&smmu 0x100>;

So the Read DMA and the Write DMA are pointed at two completely different SMMU translation tables. Same address number, different address space, different physical DDR pages behind it.

As for why they both land on exactly 0xFFC0.0000: our kernel shim sets a 32-bit DMA mask (dma_set_mask_and_coherent(dev, DMA_BIT_MASK(32)) in cl_msgdma.c), so each device's address window tops out at 4 GB. Linux allocates these virtual addresses from the top down, and the default buffer size in the example application is 4 MB. 4 GB minus 4 MB is 0xFFC0.0000. They both get the same number because they each start from the top of their own empty window. It's a coincidence of the allocator, not a collision.

Your burst observation lines up too. The datapath is 256 bits (32 bytes) and MAX_BURST_COUNT is 128 (0x80) in our design, so each burst moves 128 x 32 = 4096 bytes, which is exactly the 0x1000 address step you were seeing.

So the short answer to "how do I relate FPGA address space to Linux address space" is that when the SMMU is in the path, you can't, and you shouldn't try to. Linux hands the driver an address out of that device's own translation domain, the driver programs it into the descriptor, and the SMMU sorts out the rest.

Now, your all zeros problem

I diffed your archive against our reference design. Your Avalon-ST loopback wiring is correct (I can see the source-to-sink tie with the ready signal going back the other way in a5e_top.sv around line 129, and that all looks right), and your cache attribute settings are all intact. So none of that is the problem.

What I did find is that the SMMU path is switched off in your hardware. Four settings differ from ours, and they're the four that matter:

                            ours    yours
ENABLE_MMU                     1        0     (on the ACE5-Lite translator)
ARMMUSID_OVERRIDE            257        0     (on the ACE5-Lite translator)
AWMMUSID_OVERRIDE            256        0     (on the ACE5-Lite translator)
f2s_SMMU                    true    false     (on the HPS component)

With f2s_SMMU set false on the HPS, the FPGA-to-HPS port doesn't even have StreamID sideband signals on it. I checked, and there are no SID ports on that interface in your generated IP at all. So your fabric has no way to tell the SMMU which translation table to use.

My guess is this happened because you re-instantiated the ACE5-Lite Cache Coherency Translator when you moved it into hps_subsys (yours is named ace5lite_cache_coherency_translator_0, ours is dma_cct_bridge_0). A freshly added instance comes up with the catalog defaults, and the catalog defaults have the MMU support switched off.

Here's why that gives you zeros. Assuming you're still running our dma_example.dtso, those iommus lines tell Linux "this device sits behind the SMMU," so Linux hands your driver a translated address (the 0xFFC0.0000 style address) and expects the SMMU to convert it. Your hardware then puts that address on the bus with no StreamID attached, so nothing ever translates it. The transaction goes off to a physical address that has nothing to do with your buffer, reads come back as zeros, and writes disappear into the weeds.

That's exactly why Signal Tap looks perfect. Your fabric is doing everything right. The addresses, the bursts, the packet boundaries are all correct. The data just never reaches the memory you think it's reaching, because Linux and your FPGA disagree about what kind of address is on the bus.

The fix

Set these four back in Platform Designer and rebuild the FPGA image:

• On the HPS component: f2s_SMMU = 1
• On the ACE5-Lite Cache Coherency Translator: ENABLE_MMU = 1, ARMMUSID_OVERRIDE = 257, AWMMUSID_OVERRIDE = 256

Then leave your device tree overlay alone. The iommus lines in it are already correct, and once the hardware actually emits StreamIDs the two sides will line up. Just keep the numbers matched: 257 decimal is 0x101 and pairs with the read DMA, 256 decimal is 0x100 and pairs with the write DMA. If those ever drift apart again you'll be right back to zeros, for exactly the reason above.

One more thing worth knowing. The StreamIDs get applied on the shared bridge per channel, not per DMA engine. Every read that goes through that bridge is tagged 0x101 and every write is tagged 0x100. That works fine for our example and for your loopback because exactly one engine reads and exactly one engine writes. If you ever add a second reading master behind that same bridge, though, both would land in the same SMMU translation domain and step on each other.

Once you've rebuilt, dmesg | grep -i smmu is a good sanity check after a test run. Translation faults or a bad StreamID complaint means the two sides still disagree somewhere.

A couple of smaller things I noticed

Neither of these is your bug, but while I was in there:

• Your MAX_BURST_COUNT is 32 where ours is 128, and your DATA_FIFO_DEPTH is 128 where ours is 1024, on both mSG-DMAs. That's purely a throughput thing and it'll work fine as is, but you'll leave a fair amount of DDR bandwidth on the table. Might as well bump those while you're in there rebuilding.
• Your loopback is a real loopback, unlike our example, so keep in mind the Read DMA and the Write DMA still live in separate SMMU address spaces. Reading out of buffer A and writing into buffer B is exactly what you want here, so that's fine, just don't expect the two address numbers on Signal Tap to line up.

Give that a shot and let me know if the data comes through. If you're still seeing zeros after the rebuild, please post your modified application and any changes you made to the cl_msgdma kernel module and I'll take a look at the software side!

- Mike