Mity CPU Platforms » MitySOM-5CSX Altera Cyclone V » MitySOM-5CSX Embedded Vision Developer's Kit for Basler dart BCON

You don't by chance have an Arria10 "VDK" or equivalent? What would be the most equivalent solution you have?

Answering my own question -- it looks like this: https://www.criticallink.com/product/mitycam-c50000/

But, I'd still like to know what is possible with the Cyclone V VDK

Hi Ed,

Regarding an Arria10 "VDK"or equivalent, the MityCAM-C50000 is an evaluation platform for the CMV50000 CMOS image sensor from ams / CMOSIS. We also have evaluations platforms for some Canon sensors that can be found on https://www.criticallink.com/product-cats/imaging/ (you can use the Filter on the left of the page to only show Arria 10 platforms).

Regarding OpenVINO and FPGA acceleration, Intel seems to currently only have support for their PCIe based systems (https://software.intel.com/en-us/openvino-toolkit/hardware). I have found an example of running OpenVINO on a Cyclone V SoC, but it looks like it is only running on the ARM (i.e. no FPGA acceleration). It seems to be taking advantage of OpenVINO for Raspberry Pi and targeted at a particular development board. No one here has personally gotten OpenVINO running on the Cyclone V yet, so I cannot say how useful or helpful this might be, but it might be worth a look. https://rocketboards.org/foswiki/Projects/OpenVINOForSoCFPGA

The other option to consider is Tensorflow Lite. Like with OpenVINO this will be running only on the ARM cores, however, an engineer here has gotten an included demo from a Tensorflow Lite Yocto meta layer to run on the Cyclone V, so we know this is possible.

Thanks,
Greg

Unfortunately, running tensorflow lite on the ARM core would not meet the goals of the project.

Also, if I recall correctly, Tensorflow is used to train a model. What I am interested in is a way to execute a pre-trained model on the FPGA hardware, with a video stream from the camera.

Camera Video Stream => Inference Engine => Output

The output would be a list of identified objects and bounding boxes, suitable for identification, localization and then tracking.

Currently, I'm looking at the following capabilities to see if one ore more of them would meet our goals

- OpenCL -- Implement a CNN (of some architecture)
- PyOpenCL (just a Python interface to OpenCLL)
- PipeCNN -- This is an example implementation I'm looking at
- PlaidML -- also from Intel, seems interesting, and possibly relevant?
-

Hi Ed,

I understand that Tensorflow Lite being CPU only in this instance seems like it is a deal breaker for you. However, please realize that Tensorflow Lite is a portion of Tensorflow that is focused on inferencing on embedded devices. While it is targeted more to run on cell phone level processors (i.e. quard or octo core Cortex-A7X or Cortex A5X cores, potentially with AI acceleration hardware built in v.s. the dual core Cortex A9s on the Cyclone V SoC) there may be something you could take away from the large body of work that has been done here regarding running quantzied models on embedded devices. There is also another subset of Tensorflow called Tensorflow Lite Micro, which is targeted at inferencing on Cortex-M level processors. There may also be useful information here to consider.

We have some experience at Critical Link with using Intel's FPGA SDK for OpenCL, but not specifically for implementing CNNs.

PipeCNN seems interesting, however, the mention in the Performance section that reads "The performance of PipeCNN is no longer comparable to the state-of-the-art designs" makes me wonder what roadblocks you might experience with pursuing this avenue. The examples they have seem to be only doing object detection and not the bounding box (SSD) output that it seems you need. I would question whether the statement I quoted is implying that they do not currently support the necessary layer types to perform SSD output.

PlaidML seems interesting as well, however, it seems to be targeted at x86 processors with GPU acceleration. You may get more out of looking into Tensorflow Lite or Tensorflow Lite Micro as they target ARM.

Depending on your requirements, you may need to consider a mix of currently available solutions (e.g. trying to use OpenCL FPGA to accelerate Tensorflow Lite inferencing or custom FPGA implementations of certain parts of your selected inferencing network).

Let us know if you have any further questions.

Thanks,
Greg

Thank you for the advice Greg.

A hybrid approach does seem to be the best way to approach this, as you describe, a combination of Tensorflow lite and some custom FPGA fabric.

My first goal is to get some OpenCl logic working on my Vision Developer KIt. Any tips there would be appreciated.

There is no "easy" path here that I see.

Have you looked into HLS instead of OpenCL? If you are familiar with FPGA
/ Intel / Altara Platform Designer, the HLS converts c++ to VHDL that fits
into that framework fairly easily (compared to making a OpenCL design).

To do OpenCL, you need to create an OpenCL platform BSP, which takes a
little effort to put together.

-Mike

On Mon, Mar 16, 2020 at 12:48 PM <redmine@criticallink.com> wrote:

No, I had not looked at that.
I'll take a look at that as well.

Do you have a BSP for the VDK? Or, that start of one?

I will need to check. I don't think I have one that put aside resources
for the current camera pipeline.

-Mike

On Mon, Mar 16, 2020 at 1:01 PM <redmine@criticallink.com> wrote:

I wanted to add this link:

https://software.intel.com/en-us/articles/pipecnn-on-intel-fpga-acceleration-of-machine-learning-workloads

It does not sound like PipeCNN is "fast", but it does appear to meet the spirit of what I am attempting to do, and so it seems like a worthwhile project to review for the learning if nothing else.

I'll need to:
- Get a BSP
- Get OpenCL toolchain setup
- Incorporate the output into a format I can connect into the VDK image flow

Michael -- I am currently pursuing the OpenCL approach. It seems the most viable.

We are looking at OpenCL implementations of darknet, so my goal will be to see if/how much of a darknet implementation I could get working on the Cyclone V.

Intel has various implementations of code to FPGA. They all use a "host" of some sort, and often that can be a Host computer like a PC/Linux machine, it can also be the Arm core. I'm shooting for the ARM core as "Host", with acceleration in the FPGA, probably built with OpenCL.

I could probably build some of that acceleration with HSL, as well, or just hand-code it in SystemVerilog (My preferred language).

I'll need to create a custom BSP for all of this. I'd like to not start at zero, so ANY support you can provide in building the BSP will be much appreciated. If you have an existing one that could be modified, that's probably the best help.
I'm currently "working my way" through all of the documents to learn how this all works....from OpenCL, to BSP generation, to custom BSP building... etc.

My Immediate goal would be to try and get the "Optical Flow" design example working on the VDK. That would exercise the necessary parts from my perspective... and give me a working framework to try and hook a darknet implementation into.

Wish me luck......

Hi Ed:

Have you looked at our OpenCL BSP reference kit for our MitySOM-5CSX DevKit?

https://support.criticallink.com/redmine/projects/mityarm-5cs/wiki/MitySOM-5CSX_DevKit_OpenCL_BSP

It's a little old, but you could probably adapt/use this pretty quickly to
get started.

It's been a little while since we've done stuff with OpenCL for this
platform.... HLS has been a lot easier to use.

-Mike

On Wed, Mar 18, 2020 at 12:20 PM <redmine@criticallink.com> wrote:

I was just looking through the code on that. I've already cloned the git repo.

I'll use that as a starting point. It won't have any of the code related to the VDK, i.e. the Basler LVDS input logic, and the image processing flow that is contained in the VDK. I'll need to merge those two together I guess...

Hopefully, this will all be more clear as I get a better understanding of how this all works.

There is one key part that I'm not able to wrap my head around just yet. If you have some knowledge here, it would help greatly.

What I know is that I can write an OpenCL kernel that performs some work. I can write a host program with an ARM target that runs on the SOC. This host program seems "not unlike" the demo_app for the VDK

What is fuzzy is that, the BSP builds an FPGA program that incorporates the BLVDS image decoding, some pipelines and Vision IP from Altera that convert the image format into the altera version, put it into memory, read it out, process it, and eventually send it to HDML.
With OpenCL, the 'host' program loads an OpenCL Kernel into the FPGA, this is a file named *.aocx.

The SOC runs independent of the FPGA, so in the current VDK, the SOC loads the FPGA code, compiled from the fpga/vdk_single (or fpga/vdk_single_ssd, etc).

I'm wondering if the OpenCL kernel is mutually exclusive to the fpga code on the VDK.. if so, then I'd have to figure out how to implement the entire vision pipeline in the OpenCL kernel???
Is there a way for the FPGA framework, built with the Platform Designer (i.e. FPGA/vdk_single) to live WITH the OpenCL kernel? Such that I could route the image pipeline through the OpenCL kernel?

This is the part I'm unclear about, and it's a pretty critical bit...I'd love to hear some thoughts on this from someone more knowledgable than I.

You can, but you have to implement all of that stuff as part of the "BSP" if you don't want to implement it all in OpenCL (which, for the RAW LVDS inputs, is not possible I don't think).

I have only played around a little with that, and it was really not a nice experience. This is why we've gone the HLS route.

I think you need to "wrap" the current VDK pipeline together with the HPS, etc., in the OpenCL BSP to accomplish what you need.

Not sure if that makes sense... we tried this with another camera project, it was hard to maintain.

-Mike

I found an interesting reference to a presentation here:
https://www.globallogic.com/wp-content/uploads/2016/10/Combination-of-OpenCL-and-Video-Processing-Design-in-SoC_.pdf

It shows using both together, even multiple kernels. Unfortunately, it does not provide any more detail. It also discusses using the Altera VIP, which is used for the VDK.
I have not had a chance to dig into it further yet, or possibly contact the company for advice. Might do that later.

I'm currently reading section 10 of the "Intel FPGA SDK for OpenCL Pro Edition: Programming Guide"
This section talks about combining HLS code with OpenCL. It appears you can write code in a variety of languages, compile to object files, and then combine together into a library, that eventually goes into an OpenCL Kernel. This is a promising approach in my mind.

Hey Mike, i've been reading manuals for the past few days, and learned quite a lot.
I see now how the BSP and Kernel fit together -- the BSP can contain a partition where the OpenCL Kernel is inserted.
The BSP can contain all of the code to decode the BLVDS stream, and even put the data into memory.

There are a couple of ways the OpenCL Kernel can access the data, process it, and generate an output. It seems like "Channels" would be a good approach.

So, my current task is to build a suitable BSP for the VDK.
For that, I can start with:
- Reference Platform for Cyclone V --- That is pretty low level, would prefer not to do that
- MitySOM-5CSX DevKit OpenCL BSP --- This is a generic BSP for the MitySOM, but does not have any of the VDK stuff in it
- The VDK BSP --- modify this to include the OpenCL kernel partition, and then work on the Channels/Streaming part

My gut tells me that the VDK BSP would be a better starting point.
I have the 'mitysom5csx-dart-evdk' already, but it isn't clear that there is a BSP in there --

How could I use the VDK BSP as a starting point, is that available somewhere?

Mike -- I have several examples of Cyclone V BSP's that incorporate an OpenCL kernel. It would be VERY helpful to have a starting "BSP" for the VDK, because then I can use the other examples as a reference and add the OpenCL kernel into the project.
I think that is by far the smoothest approach.

Without that BSP I'll have to reverse engineer the VDK BSP from the * MitySOM-5CSX DevKit OpenCL BSP*.
Totally possible, but probably not as smooth as I'd like...

Hi Ed,

If you are looking for an FPGA project / BSP with VDK + OpenCL architected, I don't have one handy. I only have the reference FPGA project that works with the example ARM programs.

If you need that FPGA project, it should be available (let me know, I thought there was a git repo on github for it posted on the wiki for this).

I am not certain what you mean by reverse engineering the VDK BSP. All of the source code (except the Intel VIP HDL) should be available for that FPGA.

-Mike

I was looking for the one that is for the current VDK, so it would just have the ARM code.
I do have the code from the VDK files on the wiki. There isn't a 'BSP' there, but there are several versions of the FPGA code.
I guess I was looking for something with a board_spec.xml file in it, but that may not exist.

For the current VDK, I think that the FPGA code is downloaded from the SD card, from a .rbf file saved to the flash... so, there isn't a BSP at all, just a linux kernel, demo_app, and an FPGA .rbf.

If I'm right about the above, then I would need to create that board_spec.xml file (which should not be terribly difficult..), and the rest of the parts for a VDK BSP, then modify the quartus project to include the OpenCL placeholder ad OpenCL interface.
There are examples of those parts in the 5CSX.

Does this sound accurate?? If so, I think I know my path forward.

The FPGA code for the VDK is on github here:

https://github.com/CriticalLink/mitysom5csx-dart-evdk/tree/master/fpga

You want the vdk_single if you are using just one camera interface.

The board_spec.xml is part of an OpenCL BSP, and that does not exist for the VDK (though you could probably start with the one from the stock MitySOM-5CSX OpenCL BSP).

What needs to happen is someone needs to merge the code from above with the OpenCL BSP from the MitySOM-5CSX to create a new OpenCL BSP board definition.

Normally I would help you out, but at the moment with the situation here with the COVID-19 (we are in NYS) we are a bit overwhelmed. Sorry about that.

-Mike

Thanks Mike,

That confirms what I've been thinking about it also, so that is the approach I'll take -- merge the two projects into a VDK BSP.

Thanks, and please stay safe out there in NYC..