Project

General

Profile

FPGA programming problems

Added by Dario Russo almost 4 years ago

Hi,
I’m working with MitySOM-5CSX-H6-42A development kit and I would like to solve some annoying problems. After generating a new rbf file for programming the FPGA and uploading it to the SD card through sftp, it is necessary to reboot the board at least two times before the new configuration is effectively loaded. Moreover, programming the FPGA through linux command often causes a freeze of the processor and requires a reset. Why is this happening? It seems that it still has the old rbf somewhere and only replaces it after few reboots.

Dario


Replies (11)

RE: FPGA programming problems - Added by Michael Williamson almost 4 years ago

Hi Dario,

In your FPGA project, did you modify any of the HPS peripheral settings over your standard load? For example, did you enable or disable a UART or SPI port, or change the SDRAM bridge configurations?

If you did, you will need to rebuild the preloader and reinstall it and reboot. The preloader sets up the IO boundary between the ARM memory subsystem and the FPGA interconnect. If you don't keep that consistent with your FPGA image, you will have problems. There is no way to hot-swap a bitstream that requires the FPGA/ARM/DDR boundaries to be in a different configuration. This is a common problem.

If none of the FPGA <-> ARM or FPGA <-> DDR peripheral interconnect has changed, then here are some other things that I have had trouble with when trying to install / load an FPGA image on the fly:

1) Watch any interrupts that could be generated by the FPGA fabric. If you are generating interrupts from the FPGA and have a linux kernel interrupt handler installed and then reset/reload the bitstream, it is likely that the interrupt connection can activate. I have seen the linux kernel driver try to handle FPGA interrupts during the load and basically not function properly with random behavior.

2) Ensure nothing on the ARM is trying to access any of the FPGA memory bridges (hps2fpga lightweight bus, etc.) during the load. This can also create hangs.

3) You will need to disable the DDR bridges exposed to the FPGA if you reconfigure while linux is running. I believe there are sysfs entries in linux to enable/disable these crossbars. If you reprogram and FPGA image that uses the DDR interface (e.g., fpga2sdram), it is possible that during the bitstream load the bridge interface could be put into an unknown state and lock up the DDR bridge controller (causing trouble for the ARM).

In general, we try to avoid "hot-swapping" FPGA bitstreams and load our target bitstream during uBoot.

-Mike

RE: FPGA programming problems - Added by Dario Russo almost 4 years ago

Hi Mike,
thank you for the detailed answer. I don’t used interruputs and UART or SPI, but I use the sdram bridge. The preloader should be updated (I will check it to be sure). I agree with you, it is better to change the rbf file and reboot the system so that uboot loads the FPGA bitsteam.. However, the problem of updating the rbf file is present even if I don’t use the linux command (but it is necessary to reboot the system at least twice).

RE: FPGA programming problems - Added by Dario Russo almost 4 years ago

I checked the preloader and uboot and are updated. I noticed that in the bootlog the bridges are not initialized:

altera_hps2fpga_bridge soc:fpgabridge@0: fpga bridge [hps2fpga] registered as device hps2fpga
altera_hps2fpga_bridge soc:fpgabridge@0: init-val not specified
altera_hps2fpga_bridge soc:fpgabridge@1: fpga bridge [lwhps2fpga] registered as device lwhps2fpga
altera_hps2fpga_bridge soc:fpgabridge@1: init-val not specified
altera_hps2fpga_bridge soc:fpgabridge@2: fpga bridge [fpga2hps] registered as device fpga2hps
altera_hps2fpga_bridge soc:fpgabridge@2: init-val not specified

Is it possibile that the device tree is missing something?

RE: FPGA programming problems - Added by Daniel Vincelette almost 4 years ago

Hi Dario,

If you are still seeing it take multiple boots to load the "new" rbf from u-boot then it shouldn't be a device-tree issue. Also if you're using u-boot to program it will handle opening the correct bridges, so those warnings can be ignored.

1) Are you changing any of the SDRAM bridge parameters or enabling/disabling any of the bridges between build to build? If so that is handled by u-boot/preloader.

2) Do you only see this "old" version persist with reboots? If you copy over your new rbf and do a full power off do you still see the old rbf being loaded?

3) During reboot do you see the yellow FPGA config light toggle?

4) Could you post a full boot log with you updating the rbf and the multiple boots it takes to load the new "one"?

5) How are you verifying which rbf is loaded? Are you using a qsys sys id core?

Thank you Dario, having to reboot multiple times definitely sounds frustrating and I'd like to help you get this resolved.

Dan

RE: FPGA programming problems - Added by Dario Russo almost 4 years ago

Hi Dan,
yes, I still have the multiple boots problem.
1) I'm not changing the SDRAM bridge configuration between builds. I'm only changing the vhdl code.
2) If I copy a new rbf file and reboot the system, the fpga will be programmed with the old configuration. Sometime it is necessary to reboot the system twice, sometimes more. In some cases, I had to use the linux command and then reboot the system.
3) Yes the yellow led toggles, so uboot programs the FPGA.
4) Attacched you can find the boot log.
5) I simply used a register in the FPGA fixed at a value (that I manually change over compilations). I read this register from HPS and display its value on the terminal.

Thank you Dan.

Dario

RE: FPGA programming problems - Added by Daniel Vincelette almost 4 years ago

Hi Dario,

Thank you for the bootlog and answering my questions.

If you are seeing the yellow led toggle in u-boot then the FPGA is reloaded from what is on the file system. There is a small chance that the file system didn't fully write out the RBF change if you don't cleanly reboot the SOM (using the linux reboot command) and on reboot linux fixes the file system from the journal so that's why it works on next boot.

Have you done a md5sum of the RBF either in linux or uboot (preferably in uboot) after you update it and compare it with the file on your PC? This will help us verify that the file system correctly sees the updated image.

Can you send a continuous bootlog of the following:
  • Boot old RBF
  • Booting new RBF but still sees old RBF
  • Booting new RBF and you see it actually load correctly

Also can you also post your u-boot environment by breaking into u-boot and running the "print".

With best regards,
Dan

RE: FPGA programming problems - Added by Dario Russo almost 4 years ago

Hi Dan,
thank you very much for helping me with this problem.
Attacched you can find the boot logs for the three cases required.
The checksum verification it is ok by comparing the linux checksum with the PC file. Instead there is no md5sum command in uboot. Can you give me an example for uboot checksum?
I think you are right, it seems that rebooting the system with the linux command after changing the rbf file works.

Best regards,
Dario

RE: FPGA programming problems - Added by Daniel Vincelette almost 4 years ago

Hi Dario,

My apologizes for the later reply, it was a long holiday weekend in the states.

So by running "reboot" after copying over the RBF fixed your problem?

Dan

RE: FPGA programming problems - Added by Dario Russo almost 4 years ago

Hi Dan,
Don't worry, I really appreciate your help.
Yes, running “reboot” works and it is the only way to change the FPGA configuration without rebooting the system multiple times. Indeed, even if I change the rbf and then I turn the board off and on, the FPGA configuration doesn’t change. As you said, the problem is that the rbf is not completely written by the file system. Is there any way to fix it?

Best regards,
Dario

RE: FPGA programming problems - Added by Daniel Vincelette almost 4 years ago

Hi Dario,

You could also try using the "sync" linux command after you copy the rbf over. That should flush anything out to the SD card. I normally do a sync and a reboot after updating the RBF. Seeing as the filesystem is read/write doing a "nice" power off is always recommended to make sure you don't get into situations like you current are with files partially written or worse file systems corrupted. A "nice" power is with a "poweroff" or "reboot" command.

Dan

RE: FPGA programming problems - Added by Dario Russo almost 4 years ago

I will do that.
Thank you very much Dan.

Dario

    (1-11/11)
    Go to top
    Add picture from clipboard (Maximum size: 1 GB)