Project

General

Profile

MitySOM-AM57x Wiki » DSP Development »

Building TI Linux SDK and IPC examples on Linux

The process below was done on a 64-bit x86-based Linux host running Ubuntu 18.04.4 LTS using ti-processor-sdk 06.03.00.106.

TI has a good rundown on the different communication techniques available between the ARM and DSPs. This page is going to focus on some of the IPC and BigData IPC examples.
https://software-dl.ti.com/processor-sdk-linux/esd/docs/06_03_00_106/linux/Foundational_Components_IPC.html#multiple-ways-of-arm-dsp-communication

You will need the TI Processor SDK and the TI RTOS Processor SDK for most projects using the ARM and DSP. This guide is intended to supplement TI's documentation, not replace it. It is recommended to read through the referenced documentation.

Useful references:
TI Linux IPC
TI RTOS IPC
TI IPC Training

Downloading and Installing the SDKs from TI

Download PROCESSOR-SDK-LINUX-AM57X and PROCESSOR-SDK-RTOS-AM57X from the https://www.ti.com/tool/PROCESSOR-SDK-AM57X

Install TI Processor SDK for Linux. Follow instructions here: http://software-dl.ti.com/processor-sdk-linux/esd/docs/06_03_00_106/linux/Overview/Download_and_Install_the_SDK.html.

After installation, set up the host environment. Link is here: http://software-dl.ti.com/processor-sdk-linux/esd/docs/06_03_00_106/linux/Overview/Run_Setup_Scripts.html.
Note that the instructions refer to a script called sdk-install.sh, which is not in the installation. Skip this script (but note the information)
Proceed to running the ./setup.sh script. At the end of the script, the script loops until an EVM is detected. Just break (ctrl-c) from the script.

Install TI RTOS Processor SDK according to the steps here: http://software-dl.ti.com/processor-sdk-rtos/esd/docs/06_03_00_106/rtos/index_overview.html.

Note: TI recommends using the same version of Linux SDK as RTOS SDK. As such we are renaming the default rtos install location to include this version number for future sanity. This will mean code composer will need to be told of this location if you plan on using it

 $ chmod +x ti-processor-sdk*.bin
 $ ./ti-processor-sdk-linux-am57xx-evm-06.03.00.106-Linux-x86-Install.bin
 Destination Folder [/home/jcormier/ti-processor-sdk-linux-am57xx-evm-06.03.00.106]: $HOME/ti-processor-sdk-linux-am57xx-evm-06.03.00.106
 $ ./ti-processor-sdk-rtos-am57xx-evm-06.03.00.106-Linux-x86-Install.bin --prefix $HOME/ti-processor-sdk-rtos-am57xx-evm-06.03.00.106
 Destination Folder [/home/jcormier/ti]: $HOME/ti-processor-sdk-rtos-am57xx-evm-06.03.00.106

Building the IPC libraries

Build the IPC libraries and examples by following steps here: http://software-dl.ti.com/processor-sdk-linux/esd/docs/06_03_00_106/linux/Foundational_Components_IPC.html#ipc-for-am57xx

  1. Setup environment
    export TI_SDK_PATH=$HOME/ti-processor-sdk-linux-am57xx-evm-06.03.00.106
export PATH=$TI_SDK_PATH/linux-devkit/sysroots/x86_64-arago-linux/usr/bin/:$PATH
export TI_RTOS_PATH=$HOME/ti-processor-sdk-rtos-am57xx-evm-06.03.00.106
export IPC_INSTALL_PATH=$TI_RTOS_PATH/ipc_3_50_04_08
# Env variables for ipc_bios compile
export SDK_INSTALL_PATH=$TI_RTOS_PATH
export TOOLS_INSTALL_PATH=$TI_RTOS_PATH
  2. Build the Linux IPC library and examples
    cd $TI_SDK_PATH
make ti-ipc-linux
make ti-ipc-linux-examples
cd $TI_RTOS_PATH/processor_sdk_rtos_am57xx_6_03_00_106/
source setupenv.sh
make ipc_bios

Reloading DSP/IPU firmware key

To load/reload firmware on a live system, you need to unbind and bind the omap remoteproc driver with the name of the processor's rproc device name. Capturing these below for easier reference.

Processor Device Name Remote Proc Number MultiProc id
DSP1 40800000.dsp remoteproc2 4
DSP2 41000000.dsp remoteproc3 3
IPU1 58820000.ipu remoteproc0
IPU2 55020000.ipu remoteproc1 1

Example:

echo 40800000.dsp > /sys/bus/platform/drivers/omap-rproc/unbind
echo 40800000.dsp > /sys/bus/platform/drivers/omap-rproc/bind

Running the ex02_messageq example

Reference: http://software-dl.ti.com/processor-sdk-linux/esd/docs/06_03_00_106/linux/Foundational_Components_IPC.html#ipc-for-am57xx

Steps to run (assuming booted from tisdk-rootfs-image):

  • PC: Create directory on devkit and copy build example files
    ssh root@<devkit> mkdir -p ipc-starter/ex02_messageq
cd $TI_RTOS_PATH/ipc_3_50_04_08/examples/DRA7XX_linux_elf/ex02_messageq/
scp */bin/debug/{server_dsp*,app_host,server_ipu*} root@<devkit>:ipc-starter/ex02_messageq/
  • devkit: Stop opencl daemon which interacts with DSPs. Optionally you can disable it if you don't plan on using opencl by replacing `stop` with `disable`
    systemctl stop ti-mct-daemon.service
  • devkit: load new firmware on DSPs/IPUs
    ln -sf /home/root/ipc-starter/ex02_messageq/server_dsp1.xe66 /lib/firmware/dra7-dsp1-fw.xe66
echo 40800000.dsp > /sys/bus/platform/drivers/omap-rproc/unbind
echo 40800000.dsp > /sys/bus/platform/drivers/omap-rproc/bind

ln -sf /home/root/ipc-starter/ex02_messageq/server_dsp2.xe66 /lib/firmware/dra7-dsp2-fw.xe66
echo 41000000.dsp > /sys/bus/platform/drivers/omap-rproc/unbind
echo 41000000.dsp > /sys/bus/platform/drivers/omap-rproc/bind

ln -sf /home/root/ipc-starter/ex02_messageq/server_ipu1.xem4 /lib/firmware/dra7-ipu1-fw.xem4
echo 58820000.ipu > /sys/bus/platform/drivers/omap-rproc/unbind
echo 58820000.ipu > /sys/bus/platform/drivers/omap-rproc/bind

ln -sf /home/root/ipc-starter/ex02_messageq/server_ipu2.xem4 /lib/firmware/dra7-ipu2-fw.xem4
echo 55020000.ipu > /sys/bus/platform/drivers/omap-rproc/unbind
echo 55020000.ipu > /sys/bus/platform/drivers/omap-rproc/bind
  • devkit: run example arm app
    cd /home/root/ipc-starter/ex02_messageq/
./app_host DSP1
./app_host DSP2
./app_host IPU1
./app_host IPU2

Note: During testing, the IPU1 firmware refused to load, more testing is required. See attachment:ipc_examples.txt

Running the ipc tests

Reference: https://software-dl.ti.com/processor-sdk-linux/esd/docs/06_03_00_106/linux/Foundational_Components_IPC.html#ipc-tests

NOTE: The ipc tests comes pre-compiled on the tisdk-rootfs-image. Files located at /lib/firmware/ipc/ti_platforms_evmDRA7XX_*/ and the ARM apps are in the system path.

  • PC: Create directory on devkit and copy build example files
    ssh root@<devkit> mkdir -p ipc-tests
scp -r $TI_RTOS_PATH/ipc_3_50_04_08/packages/ti/ipc/tests/bin/* root@<devkit>:ipc-tests/
scp -r $TI_RTOS_PATH/ipc_3_50_04_08/linux/src/tests/.libs/* root@<devkit>:ipc-tests/
  • devkit: Stop opencl daemon which interacts with DSPs. Optionally you can disable it if you don't plan on using opencl by replacing `stop` with `disable`
    systemctl stop ti-mct-daemon.service
  • devkit: load new firmware on DSPs/IPUs
    ln -sf /home/root/ipc-tests/ti_platforms_evmDRA7XX_dsp1/messageq_single.xe66 /lib/firmware/dra7-dsp1-fw.xe66
echo 40800000.dsp > /sys/bus/platform/drivers/omap-rproc/unbind
echo 40800000.dsp > /sys/bus/platform/drivers/omap-rproc/bind
  • devkit: run example arm app
    root@mitysom-am57x:~# ./ipc-tests/MessageQApp 1 4
Using numLoops: 1; procId : 4
Entered MessageQApp_execute
Local MessageQId: 0x80
Remote queueId  [0x40080]
Exchanging 1 messages with remote processor DSP1...
MessageQ_get #1 Msg = 0xb64007c0
Exchanged 1 messages with remote processor DSP1
Sample application successfully completed!
Leaving MessageQApp_execute

root@mitysom-am57x:~# MessageQBench 1000 8 4
Using numLoops: 1000; payloadSize: 8, procId : 4
Entered MessageQApp_execute
Local MessageQId: 0x80
Remote queueId  [0x40080]
Exchanging 1000 messages with remote processor DSP1...
DSP1: Avg round trip time: 137 usecs
Leaving MessageQApp_execute

Note: DSP1 is procId: 4, DSP2 is procId: 3

Note: As of this posting (03/08/2021) the sdk-linux docs have the wrong MessageQBench arguments. Reported to TI

Timing the IPC latency using MessageQBench

Test: Send 3 doubles from the DSP to the ARM

Note: MessageQBench times messages from ARM->DSP->ARM, with messages sent one at a time

Assuming the setup time is static: The example is sending ~7,462 messages a second, double that for both directions.

(2.54s-1.2s)/10000 = 134 us
1/(134us) = 7462

Changing it to floats (12 bytes) resulted in ~10k messages a second.

Also ran two copies of the benchmark to each DSP resulted in the same timings, so we could likely get higher throughput numbers if both DSPs are sending data, assuming the DSP calculations aren't the bottleneck.

Show log...

Building and running the big-data-ipc example

The example itself is described here: http://software-dl.ti.com/processor-sdk-rtos/esd/docs/06_03_00_106/rtos/index_examples_demos.html#big-data-ipc-example. The steps to run the example are in section 9.7.4.2.

NOTE: We tested this example for throughput and only got ~15MB/s. There is currently a pending e2e post to try and get this resolved. https://e2e.ti.com/support/processors/f/processors-forum/986073/am5728-ipc-big-data-example-gives-low-throughput

NOTE: The big data example comes pre-compiled on the tisdk-rootfs-image. Files located at /usr/bin/simple_buffer_example/release/app_host

  • Setup the environment
  • PC: Build the big data example
    cd $TI_SDK_PATH
make big-data-ipc-demo
  • PC: Create directory on devkit and copy build example files
    ssh root@<devkit> mkdir -p ipc-starter/big-data
cd $TI_SDK_PATH/example-applications/big-data-ipc-demo-linux-01.03.00.00/host_linux/simple_buffer_example/
scp */bin/DRA7XX/release/{server_dsp*,app_host} root@<devkit>:ipc-starter/big-data
  • devkit: Stop opencl daemon which interacts with DSPs. Optionally you can disable it if you don't plan on using opencl by replacing `stop` with `disable`
    systemctl stop ti-mct-daemon.service
  • devkit: load new firmware on DSPs/IPUs
    ln -sf /home/root/ipc-starter/big-data/server_dsp.xe66 /lib/firmware/dra7-dsp1-fw.xe66
echo 40800000.dsp > /sys/bus/platform/drivers/omap-rproc/unbind
echo 40800000.dsp > /sys/bus/platform/drivers/omap-rproc/bind
  • devkit: run example arm app
    cd /home/root/ipc-starter/big-data/
./app_host DSP1

Note: The Shutdown message sent from the ARM causes the DSP to stop running. Reloading DSP firmware was enough to get it back working

tail /sys/kernel/debug/remoteproc/remoteproc2/trace0
[      9.547] [t=0x00000001:47d3f6e9] Server: Message received...1
[      9.547] [t=0x00000001:47d50dbc] ti.sdo.ipc.SharedRegion: ERROR: line 536: assertion failure: A_overlap: Shared region overlaps
[      9.547] ti.sdo.ipc.SharedRegion: line 536: assertion failure: A_overlap: Shared region overlaps
[      9.547] xdc.runtime.Error.raise: terminating execution

Summary of how the big data example works:

ARM Code:
  • Create 16MB shared memory region using CMEM and SharedRegion
  • Create a Heap which is used to split this shared memory into trackable chunks to send over to the DSP
  • SEND: Send the shared memory pointer to DSP so it can setup its SharedRegion to match
  • SEND: Send 2 no-op messages
    The no-op messages are priming the pump so to speak. The ARM app is set up to only send more messages when it receives one. So sending 2 no-op messages ensures there are 3 messages in flight at a time, one for DSP to process, one for ARM, and one in waiting. To try and keep all processors active.
  • RECV: Get a response MSG from DSP
    • If the message is a BIGDATA message, then validate DSP count pattern and free the buffer
  • SEND: For every message received, we send a BIGDATA message allocated from the Heap filled with the ARM count pattern to DSP
  • For the last 3 messages, send 2 no-ops and then 1 shutdown message
DSP Code:
  • RECV: Get a message from ARM
    • If the message is a SETUP message, setup SharedRegion using info from ARM
    • If the message is a BIGDATA message, then validate the ARM count pattern and replace it with a DSP count pattern. Send the message back to ARM

The Heap is only accessed directly by the ARM code. The buffers acquired from the Heap are only accessed by one processor at a time so no locks are required.

Note: The example is designed around the expectation that the ARM is sending data to the DSP to operate on and then it gets returned. If the DSP generated data on its own and then sends it to ARM, it may be beneficial for the DSP to own the Heap management

Note: Updated Big Data example to allow the number of messages and the buffer size to be adjusted by command line arguments. https://github.com/jcormier/big-data-ipc-example/commits/benchmark

Creating a custom project using an example as a starting point

TODO

Errors

  • Failed to get a pool fitting a size 0x1000000
    root@mitysom-am57x:~/ipc-starter/big-data# /usr/bin/simple_buffer_example/release/app_host DSP1                                                                                                        [112/2022]
--> main:
--> Main_main:
--> App_create:
App_create: Host is ready
<-- App_create:
--> App_exec:
CMEM_init success
CMEM Error: getPool: Failed to get a pool fitting a size 0x1000000
CMEM_getPool failed
Segmentation fault (core dumped)

    Ensure that the opencl ti-mct-daemon service is stopped
    systemctl stop ti-mct-daemon.service

Go to top
Add picture from clipboard (Maximum size: 1 GB)