Project

General

Profile

MitySOM-AM57x Wiki » DSP Development »

Building TI Linux SDK and IPC examples on Linux SDK 9.03

The process below was done on a 64-bit x86-based Linux host running Ubuntu 24.04 using ti-processor-sdk 09_03_00_00 . Please use an acceptable version of Ubuntu such as 24.04 with the ti-processor-sdk 09_03_00_00.

TI has a good rundown on the different communication techniques available between the ARM and DSPs. This page is going to focus on some of the IPC and BigData IPC examples.
https://software-dl.ti.com/processor-sdk-linux/esd/AM57X/09_03_06_05/exports/docs/linux/Foundational_Components_IPC.html#multiple-ways-of-arm-dsp-communication

You will need the TI Processor SDK and the TI RTOS Processor SDK for most projects using the ARM and DSP. This guide is intended to supplement TI's documentation, not replace it. It is recommended to read through the referenced documentation.

TI Linux IPC
TI RTOS IPC
TI IPC Training

Prerequisites

Download PROCESSOR-SDK-LINUX-AM57X and PROCESSOR-SDK-RTOS-AM57X.

PROCESSOR-SDK-LINUX-AM57X
  • Download ti-processor-sdk-linux-am57xx-evm-09_03_06_05-Linux-x86-Install.bin
PROCESSOR-SDK-RTOS-AM57X
  • Download processor_sdk_rtos_am57xx_09_03_00_00-linux-x64-installer.tar.gz

Building the IPC libraries

  • Export the following variables
    • Replace <PROCESSOR_SDK_INSTALL_DIR> and <RTOS_SDK_INSTALL_DIR> with the installation path to the respective SDKs.
      export TI_SDK_PATH=<PROCESSOR_SDK_INSTALL_DIR>/ti-processor-sdk-linux-am57xx-evm-09_03_06_05
export PATH=$TI_SDK_PATH/linux-devkit/sysroots/x86_64-arago-linux/usr/bin/arm-oe-linux-gnueabi:$PATH
export TI_RTOS_PATH=<RTOS_SDK_INSTALL_DIR>/processor_sdk_rtos_am57xx_09_03_00_00
export IPC_INSTALL_PATH=$TI_RTOS_PATH/ipc_3_52_00_00
# Env variables for ipc_bios compile
export SDK_INSTALL_PATH=$TI_RTOS_PATH
export TOOLS_INSTALL_PATH=$TI_RTOS_PATH
export XDC_INSTALL_PATH=$TI_RTOS_PATH/xdctools_3_55_02_22_core
export BIOS_INSTALL_PATH=$TI_RTOS_PATH/bios_6_76_03_01
export LINUX_SYSROOT_DIR=$TI_SDK_PATH/linux-devkit/sysroots/armv7at2hf-neon-oe-linux-gnueabi
  • Source the setupenv script
    $ cd <RTOS_SDK_INSTALL_DIR>/processor_sdk_rtos_am57xx_09_03_00_00/
$ source processor_sdk_rtos_am57xx_09_03_00_00/setupenv.sh
  • TI has released an RTOS sdk with examples that do not build. Due to this we have to make the following hacks
    $ vim ipc_3_52_00_00/examples/DRA7XX_linux_elf/ex02_messageq/host/makefile


    @@ -99,7 +99,6 @@
 endif

 #  ======== toolchain macros ========
-ifndef LINUX_SYSROOT_DIR
 CC = $(TOOLCHAIN_PREFIX)gcc
 AR = $(TOOLCHAIN_PREFIX)ar
 LD = $(TOOLCHAIN_PREFIX)gcc
@@ -107,10 +106,8 @@
 CPPFLAGS =
 LDFLAGS = -L$(IPC_INSTALL_DIR)/linux/src/api/.libs/ \
     -L$(IPC_INSTALL_DIR)/linux/src/utils/.libs \
-    -L$(IPC_INSTALL_DIR)/linux/src/transport/.libs
-else
-LD = $(CC)
-endif
+    -L$(IPC_INSTALL_DIR)/linux/src/transport/.libs \
+    --sysroot $(LINUX_SYSROOT_DIR)

 CFLAGS += -c -MD -MF $@.dep
 ARFLAGS = cr
@@ -119,11 +116,9 @@

 CFLAGS += -Wall -ffloat-store -fPIC -Wunused -pthread -Dfar= $(CCPROFILE_$(PROFILE)) \
     -I. -I..
-ifdef LINUX_SYSROOT_DIR
-CFLAGS += -I$(LINUX_SYSROOT_DIR)
-else
+CFLAGS += --sysroot $(LINUX_SYSROOT_DIR)
 CFLAGS += -I$(IPC_INSTALL_DIR)/linux/include -I$(IPC_INSTALL_DIR)/packages
-endif
+CFLAGS += -mfpu=neon -mfloat-abi=hard

 LDFLAGS += $(LDPROFILE_$(PROFILE)) -Wall -Wl,-Map=$@.map
  • Attached is the updated makefile file. If it is easier you can copy this into the directory specified above to overwrite the file
    $ vim ipc_3_52_00_00/products.mak


    @@ -37,7 +37,7 @@

 # Optional: recommended to install all dependent components in one folder.
 #
-DEPOT = _your_depot_folder_
+DEPOT = $(TI_SDK_PATH)

 # Platform to build for
 #   Supported platforms (choose one):
@@ -60,9 +60,9 @@

 # Set up required cross compiler path for IPC Linux configuration and build
 #
-TOOLCHAIN_LONGNAME = arm-none-linux-gnueabi
-TOOLCHAIN_INSTALL_DIR = $(DEPOT)/_your_arm_code_gen_install_
-TOOLCHAIN_PREFIX = $(TOOLCHAIN_INSTALL_DIR)/bin/$(TOOLCHAIN_LONGNAME)-
+TOOLCHAIN_LONGNAME = arm-oe-linux-gnueabi
+TOOLCHAIN_INSTALL_DIR = $(DEPOT)/linux-devkit/sysroots/x86_64-arago-linux/usr/
+TOOLCHAIN_PREFIX = $(TOOLCHAIN_INSTALL_DIR)/bin/arm-oe-linux-gnueabi/$(TOOLCHAIN_LONGNAME)-

 # Path to Linux Kernel - needed to build the IPC user libraries
 #
@@ -97,8 +97,8 @@

 # Path to required dependencies for IPC BIOS builds
 #
-XDC_INSTALL_DIR = /nightlybuild/repo_manifests/scripts/jenkins/processor_sdk_rtos_am57xx_09_03_00_00/xdctools_3_55_02_22_core
-BIOS_INSTALL_DIR = /nightlybuild/repo_manifests/scripts/jenkins/processor_sdk_rtos_am57xx_09_03_00_00/bios_6_76_03_01
+XDC_INSTALL_DIR = $(TI_RTOS_PATH)/xdctools_3_55_02_22_core
+BIOS_INSTALL_DIR = $(TI_RTOS_PATH)/bios_6_76_03_01
 PDK_INSTALL_DIR = ${PDK_INSTALL_PATH}

 # Do you want to build SMP-enabled libraries (if supported for your target)?
@@ -110,14 +110,14 @@
 #
 ti.targets.elf.C64P =
 ti.targets.elf.C64T =
-ti.targets.elf.C66   = /nightlybuild/repo_manifests/scripts/jenkins/processor_sdk_rtos_am57xx_09_03_00_00/ti-cgt-c6000_8.3.2
+ti.targets.elf.C66   = $(TI_RTOS_PATH)/ti-cgt-c6000_8.3.2
 ti.targets.elf.C674 =

 ti.targets.arm.elf.Arm9 =
 ti.targets.arm.elf.A8F =
 ti.targets.arm.elf.A8Fnv =
 ti.targets.arm.elf.M3 =
-ti.targets.arm.elf.M4 = /nightlybuild/repo_manifests/scripts/jenkins/processor_sdk_rtos_am57xx_09_03_00_00/ti-cgt-arm_18.12.5.LTS
+ti.targets.arm.elf.M4 = $(TI_RTOS_PATH)/ti-cgt-arm_18.12.5.LTS
 ti.targets.arm.elf.M4F =
 ti.targets.arm.elf.R5F =

@@ -125,5 +125,5 @@
 ti.targets.arp32.elf.ARP32_far =

 gnu.targets.arm.A8F =
-gnu.targets.arm.A15F = /nightlybuild/repo_manifests/scripts/jenkins/processor_sdk_rtos_am57xx_09_03_00_00/gcc-arm-none-eabi-7-2018-q2-update
+gnu.targets.arm.A15F = $(TI_RTOS_PATH)/gcc-arm-none-eabi-7-2018-q2-update
 gnu.targets.arm.A53F =
  • Attached is the updated products.mak file. If it is easier you can copy this into the directory specified above to overwrite the file

WARNING: Be careful copy pasting these changes as ghost characters may cause issues when trying to build the example

Note: TI has yet to offically release a method for building these examples. For now these hacks are the an acceptable way to build. See: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1528563/processor-sdk-am57x-ipc-example-fails-to-create-and-execute-app/5923150

  • Build the ex02_messageq example
    $ cd ipc_3_52_00_00/examples
$ make DRA7XX_linux_elf/ex02_messageq/ HOSTOS="linux" PLATFORM=$IPC_PLATFORM XDC_INSTALL_DIR="$XDC_INSTALL_PATH" BIOS_INSTALL_DIR="$BIOS_INSTALL_PATH" IPC_INSTALL_DIR="$IPC_INSTALL_PATH" $IPC_TOOLS_PATHS

Reloading DSP/IPU firmware key

To load/reload firmware on a live system, you need to unbind and bind the omap remoteproc driver with the name of the processor's rproc device name. Capturing these below for easier reference.

Processor Device Name Remote Proc Number MultiProc id
IPU1 58820000.ipu remoteproc0
IPU2 55020000.ipu remoteproc1 1
DSP1 40800000.dsp remoteproc2 3
DSP2 41000000.dsp remoteproc3 4

Example:

root@am57xx-evm:~# echo 40800000.dsp > /sys/bus/platform/drivers/omap-rproc/unbind
root@am57xx-evm:~# echo 40800000.dsp > /sys/bus/platform/drivers/omap-rproc/bind

Running the ex02_messageq example

  • Copy the host and remote processor binaries to the SOM
    $ scp <RTOS_SDK_INSTALL_DIR>/processor_sdk_rtos_am57xx_09_03_00_00/ipc_3_52_00_00/examples/DRA7XX_linux_elf/ex02_messageq/host/bin/debug/app_host <RTOS_SDK_INSTALL_DIR>/processor_sdk_rtos_am57xx_09_03_00_00/ipc_3_52_00_00/examples/DRA7XX_linux_elf/ex02_messageq/dsp1/bin/debug/server_dsp1.xe66 <RTOS_SDK_INSTALL_DIR>/processor_sdk_rtos_am57xx_09_03_00_00/ipc_3_52_00_00/examples/DRA7XX_linux_elf/ex02_messageq/ipu2/bin/debug/server_ipu2.xem4 root@<IP_ADDRESS>:/home/root/ex02_messageq/
  • Load the DSP and IPU firmwares
    root@am57xx-evm:~# ln -sf /home/root/ex02_messageq/server_dsp1.xe66 /lib/firmware/dra7-dsp1-fw.xe66
root@am57xx-evm:~# echo 40800000.dsp > /sys/bus/platform/drivers/omap-rproc/unbind
root@am57xx-evm:~# echo 40800000.dsp > /sys/bus/platform/drivers/omap-rproc/bind

root@am57xx-evm:~# ln -sf /home/root/ex02_messageq/ipc-starter/ex02_messageq/server_dsp2.xe66 /lib/firmware/dra7-dsp2-fw.xe66
root@am57xx-evm:~# echo 41000000.dsp > /sys/bus/platform/drivers/omap-rproc/unbind
root@am57xx-evm:~# echo 41000000.dsp > /sys/bus/platform/drivers/omap-rproc/bind

root@am57xx-evm:~# ln -sf /home/root/ex02_messageq/server_ipu2.xem4 /lib/firmware/dra7-ipu2-fw.xem4
root@am57xx-evm:~# echo 55020000.ipu > /sys/bus/platform/drivers/omap-rproc/unbind
root@am57xx-evm:~# echo 55020000.ipu > /sys/bus/platform/drivers/omap-rproc/bind
  • Link the app_host file type to the ARM
    root@mitysom-am57x:~# ln -sf /lib/ld-linux-armhf.so.3 /lib/ld-linux.so.3
  • Host the new firmware and demonstrate inter-processor communication
    root@am57xx-evm:~# ./ex02_messageq/app_host DSP1
root@am57xx-evm:~# ./ex02_messageq/app_host DSP2
root@am57xx-evm:~# ./ex02_messageq/app_host IPU2
    • Example output
      --> main:
[ 1219.776184] omap-iommu 40d01000.mmu: 40d01000.mmu: version 3.0
[ 1219.782165] omap-iommu 40d02000.mmu: 40d02000.mmu: version 3.0
--> Main_main:
--> App_create:
App_create: Host is ready
<-- App_create:
--> App_exec:
App_exec: sending message 1
App_exec: sending message 2
App_exec: sending message 3
App_exec: message received, sending message 4
App_exec: message received, sending message 5
App_exec: message received, sending message 6
App_exec: message received, sending message 7
App_exec: message received, sending message 8
App_exec: message received, sending message 9
App_exec: message received, sending message 10
App_exec: message received, sending message 11
App_exec: message received, sending message 12
App_exec: message received, sending message 13
App_exec: message received, sending message 14
App_exec: message received, sending message 15
App_exec: message received
App_exec: message received
App_exec: message received
<-- App_exec: 0
--> App_delete:
<-- App_delete:
<-- Main_main:
<-- main:

TODO: Figure out if IPU1 has special firmware loaded

Running the ipc tests

  • Build the ipc bios examples
    $ cd $TI_RTOS_PATH/processor_sdk_rtos_am57xx_09_03_00_00/
$ make ipc_bios -j$(nproc)
  • Copy the build example files
    $ scp -r $TI_RTOS_PATH/ipc_3_52_00_00/packages/ti/ipc/tests/bin/* root@<IP_ADDRESS>:/home/root/ipc-tests/
  • load new firmware on DSPs/IPUs
    root@mitysom-am57x:~# ln -sf /home/root/ipc-tests/ti_platforms_evmDRA7XX_dsp1/messageq_single.xe66 /lib/firmware/dra7-dsp1-fw.xe66
root@mitysom-am57x:~# echo 40800000.dsp > /sys/bus/platform/drivers/omap-rproc/unbind
root@mitysom-am57x:~# echo 40800000.dsp > /sys/bus/platform/drivers/omap-rproc/bind
  • devkit: run example arm app
    root@mitysom-am57x:~# MessageQApp 1 4
[19864.018829] omap-iommu 55082000.mmu: 55082000.mmu: version 2.1
[19864.055328] omap-iommu 40d01000.mmu: 40d01000.mmu: version 3.0
[19864.061248] omap-iommu 40d02000.mmu: 40d02000.mmu: version 3.0
Using numLoops: 1; procId : 4
Entered MessageQApp_execute
Local MessageQId: 0x80
Remote queueId  [0x40080]
Exchanging 1 messages with remote processor DSP1...
MessageQ_get #1 Msg = 0xb6500808
Exchanged 1 messages with remote processor DSP1
Sample application successfully completed!
Leaving MessageQApp_execute

root@mitysom-am57x:~# MessageQBench 1000 8 4
Using numLoops: 1000; payloadSize: 8, procId : 4
Entered MessageQApp_execute
Local MessageQId: 0x80
Remote queueId  [0x40080]
Exchanging 1000 messages with remote processor DSP1...
DSP1: Avg round trip time: 139 usecs
Leaving MessageQApp_execute

Note: DSP1 is procId: 4, DSP2 is procId: 3

Note: As of 03/08/2021 the sdk-linux docs have the wrong MessageQBench arguments. Reported to TI

Timing the IPC latency using MessageQBench

Test: Send 3 doubles from the DSP to the ARM

Note: MessageQBench times messages from ARM->DSP->ARM, with messages sent one at a time

Assuming the setup time is static: The example is sending ~7,462 messages a second, double that for both directions.

(2.43s-1.15s)/10000 = 128 us
1/(128us) = 7812.5

Changing it to floats (12 bytes) resulted in ~10k messages a second.

Also ran two copies of the benchmark to each DSP resulted in the same timings, so we could likely get higher throughput numbers if both DSPs are sending data, assuming the DSP calculations aren't the bottleneck.

Show log...

Building and running the big-data-ipc example (NOT WORKING STILL IN PROGRESS)

This example was dropped from TI's supported examples back in SDK 08_02_00_04.

  • Clone to big-data-ipc-example git repository
    $ git clone <working creating gitlab repo>
$ cd TODO
  • Examine the following commits: 1395dda7724556635dc4c2bddaba023a1b7cb6b7, 4370b18a911fa0a599a5c410282f15e6aaceab78, 467abf2e3e7b7ebca2e3ff9893fe53209b882796. These contain the major changes needed to get this example to build.
    $ git log -p 1395dda7724556635dc4c2bddaba023a1b7cb6b7
$ git log -p 4370b18a911fa0a599a5c410282f15e6aaceab78
$ git log -p 467abf2e3e7b7ebca2e3ff9893fe53209b882796
  • Export the variables described in Building the ipc library
  • Build the project
    $ make host_linux
  • Create a project directory called big_data on the board
    root@mitysom-am57x:~# mkdir big_data
  • Copy the release binaries to the devkit
    scp host_linux/simple_buffer_example/dsp/bin/DRA7XX/release/server_dsp.xe66 root@<IP_ADDRESS>:/home/root/big_data/
scp host_linux/simple_buffer_example/host/bin/DRA7XX/release/app_host root@<IP_ADDRESS>:/home/root/big_data/
  • Sym link the dsp server binary to dsp1 firmware
    root@mitysom-am57x:~# ln -sf /home/root/big_data/server_dsp.xe66 /lib/firmware/dra7-dsp1-fw.xe66
root@mitysom-am57x:~# echo 40800000.dsp > /sys/bus/platform/drivers/omap-rproc/unbind
root@mitysom-am57x:~# echo 40800000.dsp > /sys/bus/platform/drivers/omap-rproc/bind
  • Link the app_host file type to the ARM
    root@mitysom-am57x:~# ln -sf /lib/ld-linux-armhf.so.3 /lib/ld-linux.so.3
  • Run the ARM host program to send and recieve big data messages from dsp1
    root@mitysom-am57x:~# ./big_data/app_host DSP1 10 16
root@mitysom-am57x:~# cat /sys/kernel/debug/remoteproc/remoteproc2/trace0

    Note: Although we specify 10 data messages, a total of 16 messages are expected. The first message shares the memory pointer with the DSP to allow it to configure its SharedRegion accordingly. The following two messages are no-ops. After that, the 10 actual big data messages are transmitted followed by two more no-ops, and a shutdown message.

Show log...

Summary of how the big data example works:

ARM Code:
  • Create 16MB shared memory region using CMEM and SharedRegion
  • Create a Heap which is used to split this shared memory into trackable chunks to send over to the DSP
  • SEND: Send the shared memory pointer to DSP so it can setup its SharedRegion to match
  • SEND: Send 2 no-op messages
    The no-op messages are priming the pump so to speak. The ARM app is set up to only send more messages when it receives one. So sending 2 no-op messages ensures there are 3 messages in flight at a time, one for DSP to process, one for ARM, and one in waiting. To try and keep all processors active.
  • RECV: Get a response MSG from DSP
    • If the message is a BIGDATA message, then validate DSP count pattern and free the buffer
  • SEND: For every message received, we send a BIGDATA message allocated from the Heap filled with the ARM count pattern to DSP
  • For the last 3 messages, send 2 no-ops and then 1 shutdown message
DSP Code:
  • RECV: Get a message from ARM
    • If the message is a SETUP message, setup SharedRegion using info from ARM
    • If the message is a BIGDATA message, then validate the ARM count pattern and replace it with a DSP count pattern. Send the message back to ARM

The Heap is only accessed directly by the ARM code. The buffers acquired from the Heap are only accessed by one processor at a time so no locks are required.

Note: The example is designed around the expectation that the ARM is sending data to the DSP to operate on and then it gets returned. If the DSP generated data on its own and then sends it to ARM, it may be beneficial for the DSP to own the Heap management

Note: Updated Big Data example to allow the number of messages and the buffer size to be adjusted by command line arguments. https://github.com/jcormier/big-data-ipc-example/commits/benchmark

Go to top
Add picture from clipboard (Maximum size: 1 GB)