!!! UNDER CONSTRUCTIION !!!¶

PCIe Hard IP¶

This page exists as a starting point for people interested in using the PCIe interface on the Critical Link Development Board. This page contains links to all the necessary portions (FPGA, kernel, dtb, etc.) for getting the system up and running quickly. This page also documents specifics on the PCIe Hard IP and pitfalls that may occur.

Example¶

Currently we are using a ported version of the RocketBoards.org PCIe Root Port With MSI example. The original unmodified source can be found at http://www.rocketboards.org/foswiki/view/Projects/PCIeRootPortWithMSI.

Fabric Resource Utilization¶

For the example provided in this Wiki we have recorded the following Cyclone V fabric utilization based upon the 110KLE sized Cyclone V SoC:

Resource	Available	Used	% Used
ALM	41,910	10,230	24%
Total registers	N/A	14,246	N/A
Memory (M10K & MLAB)	5.6Mbit	2.7Mbit	48%
DSP Blocks	112	0	0%
Transceivers TX & RX	6	4	67%
PLLs	12	1	8%
DLLS	4	1	25%

Altera PCIe Hard IP Documentation: [[http://www.altera.com/literature/ug/ug_c5_pcie.pdf]]

Pre-Built Quick Test¶

Required Files¶

For those who want to perform quick verification of, or tests with, the PCIe the necessary binaries can be found in this section:

1. FPGA (note this was compiled to run on a 5CSXFC6C6U23C7): fpga.rbf
2. 3.12 Kernel (with PCIe and SATA/AHCI support compiled in): uImage
3. Device Tree Blob: socfpga_mityarm5csx_devkit.dtb
4. Preloader: TODO

Steps for Updating the SD Card¶

1. It is assumed you are starting with an SD card distributed with the dev board that already has a file system, kernel, u-Boot, preloader, etc.
   1. If you need to create such the SD card from scratch follow these steps first: TODO
2. Update the Preloader by following the steps outlined in the ARM Software FAQs
3. Boot the dev board and make the following changes in the root file system:
   1. Replace /boot/socfpga_mitysom5csx_devkit.dtb with socfpga_mityarm5csx_devkit.dtb
   2. Replace /boot/uImage with uImage
   3. Replace /home/root/mityarm_5csx_dev_board.rbf with fpga.rbf
4. See section "U-Boot Changes" below for details on modifications needed to U-Boot.

Source¶

For those interested in the source for this example we have the following:

1. The Quartus II 13.1 project source: test_pcie.tar
   1. Pay special attention to the section Verilog Qsys Generation below.
2. See Building U-Boot and Preloader for details on rebuilding the Preloader once the FPGA has been successfully built.
3. Our latest 3.12 kernel branch can be pulled from git://support.criticallink.com/home/git/linux-socfpga.git. See Building the Linux Kernel for details on building the kernel and dtb.
   1. Make sure menuconfig is called when rebuilding the kernel and flag PCI_ALTERA is set
      1. This can be found via the menuconfig GUI under Bus support -> PCI host controller drivers
   2. Example DTS entry (add the following within the soc{} section before generating the dtb):
      

                       pcie_0: pcie@0x0 {                                              
                             compatible = "altr,pcie-root-port-1.0";                  
                             reg = < 0xc0000000 0x20000000                            
                                     0xFF240000 0x00004000  >;                        
                             reg-names = "Txs", "Cra";                                
                             interrupt-parent = < &intc>;                             
                             interrupts = < 0 40 4 >;                                 
                             interrupt-controller;                                    
                             #interrupt-cells = < 1 >;                                
                             altr,msi = <&msi0>;                                      
                             device_type = "pci";                                     
                      };                                                              
      
                      msi0: msi@0xFF200000 {                                          
                             compatible = "altr,msi-1.0";                             
                             reg = < 0xFF200000 0x00000010                            
                                     0xFF200010 0x00000080 >;                         
                             reg-names = "csr", "vector_slave";                       
                             interrupt-parent = < &intc>;                             
                             interrupts = < 0 42 4 >;                                 
                             msi-controller = < 1 >;                                  
                             num-vectors = < 32 >;                                    
                      };        
      

4. See section "U-Boot Changes" below for details on modifications needed to U-Boot.
5. Please also read through the potential pitfalls and additional details listed below as they may come into play when rebuilding the FPGA and/or kernel.
6. See the section above Steps for Updating the SD Card.

Potential Pitfalls and Additional Details¶

Documentation Discrepancies¶

The Cyclone V Hard IP for PCI Express User Guide and altera_pcie.c driver from RocketBoards contend on the location of TLP Data Registers SOP and EOP bits. See Table 8-32 in the User Guide and the RP_RXCPL_EOP definition in the kernel driver. The kernel driver seems to be correct, as changing the bits to match the documentation causes the driver to not work at all...

No Card Present System Lockup¶

If the PCIe kernel drivers are enabled, but no card is plugged into the development board the system will lock up during boot. The system seems to think there is a child on the PCIe bridge and will try to talk to it. On the 21st attempted TLP communication with this child the system will lock up when reading the TLP registers. Digging into the FPGA it seems this read request never makes it across the clock crossing bridge to the CRA memory mapped registers.

This is a known issue that has been mentioned on the updated RocketBoards.org project page: http://www.rocketboards.org/foswiki/view/Projects/PCIeRootPortWithMSI

Verilog Qsys Generation¶

When building the FPGA project if the Qsys file needs to be generated, make sure that the Synthesis output is set to Verilog. Specifying VHDL will result in a PCIe hard core that does not function (the CRA registers cannot be accessed).

Credit Allocation Selection¶

The credit allocation scheme for the PCIe hard core can be changed when compiling. If it is set to anything but "balanced" this causes kernel panics when booting. Perhaps there is another setting (maybe in the kernel or DTS?) that needs to be changed in concert with this?

U-Boot Changes¶

In order to make sure the PCIe is reset properly a PIO was added to manually reset the interface. The environment variable initpcie was added to U-Boot to make sure this reset happens.

initpcie=mw 0xFF202000 1;sleep 1; mw 0xFF202000 0

This should be run after loading the FPGA, and after bridge_enable_handoff, but before loading the kernel. The easiest way to do this if to modify bootcmd and add "run initpcie;" after "run bridge_enable_handoff;"

Meeting Timing¶

When porting the RocketBoards.org project to the Critical Link 5CSX development board there were issues with the system meeting timing. The RocketBoards.org project is targeted at a 5CSXFC6D6, while the Critical Link MitySOM has a 5CSXFC6C6. We were only able to successfully port the example project and not have timing errors if we started with the cv_soc_rp_simple_design.tar.gz.

Changes Between FPGA and Kernel Patch¶

We manually applied the RocketBoards.org kernel patch to the 3.12 kernel (their patch is targeted at the 3.9 kernel). We also had success applying their patch to the 3.11 kernel.

The main issue to be aware of are the PCIe DTS entries. The examples in the kernel Documentation do not match the RocketBoards.org example FPGA project. The kernel DTS example has the CRA located at offset 0x40000 (i.e. HPS address 0xFF240000) on the lw h2f axi bridge, while the FPGA design has the CRA set to offset 0x20000 (i.e. HPS address 0xFF220000).

The other DTS issue to be aware of is specification of the IRQ for the PCIe core. In the RocketBoards.org example the interrupts are specified as

interrupts = < 0 43 4 >; 

However, they need to be specified as

interrupts = < 0 40 4 >;

in order to work with the example FPGA project.

The DTS example on this page is coherent with the FPGA files also linked on this page, so this should not be an issue.

Marvel 88SE9230 PCIe to SATA¶

We tested this part using the IOCREST SATA PCI Express Card. It performed well and allowed us to achieve a 100 MB/s throughput sustained transfer rate when writing 100 GB worth of data to an mSATA SSD. 1-5 GB transfers were performed at 140 MB/s. We think we might be able to improve these numbers with a newer file system or some tweaks to our system.

It is important to note that this chip can only support one drive. Port Multiplier FIS support is not in the 3.12 kernel we tested with (and not even in Ubuntu 13.10 when tested on a PC).