In this blog post, we are going to demonstrate how we can leverage Xilinx FPGA to quickly deploy a simple video application using AI.
The tutorial will be split into 2 parts. First, we will create a custom hardware platform and a Linux environment in order to support a Deep Learning Processing Unit (DPU) instance. The idea here is to deploy a pre-trained model and to run it onto the hardware (HW). Note: We are not going to go through the AI science path through which we would try to train and optimize our neural network (NN).
The second part will cover how to create an application in Vitis using a DPU HW accelerator, the deployment to the target board, and the testing phase.
The system and HW requirements to follow this tutorial are as follow:
Host requirements
OS: Ubuntu 20.04.2 LTS
Vivado/Vitis: 2021.1
Petalinux: 2021.1
Target requirements
Board: ZCU104
USB camera
Ethernet Connection
Displayport compatible display
Our goal in the first part is to build the base project and Linux infrastructure so we can use it in the next phase to prepare and deploy the application. Note: Most of these steps come from a Xilinx Tutorial called Vitis-Platform-Creation.
Step 1 - create a Vivado platform compatible with DPU for integration later on.
First, we need to create a Vivado project. We will need to source the environment and then open Vivado. We are going to use the GUI flow so we can know what goes on behind the hood instead of just using a pre-built script for the ZCU104. We are also going to create a directory to hold our entire project:
source <Vitis_Install_Directory>/settings64.sh
cd ~
mkdir zcu104_custom_platform_2021
cd zcu104_custom_platform_2021
vivado &Create a new project
Once Vivado is started we need to create a new project:
Create a new project
• Select File->Project->New, Click Next.
• In Project Name dialog set Project name to zcu104_custom_platform.
• In location go to your new directory (zcu104_custom_platform_2021) Click Next.
• Enable Project is an extensible Vitis platform. Click Next.
• Select Boards tab and then select Zynq UltraScale+ ZCU104 Evaluation Board. Click Next.
• click FinishWe now have a project created for the zcu104 board.
Create a Block Design
In this step, we are going to create a platform that will be able to support the integration of a DPU bloc later on, in Vitis. We would also be able to connect to other accelerators if needed.
• In Project Manager, under IP INTEGRATOR, select Create Block Design.
• (Optional) Change the design name to system.
• Click OK.Now add an MPSoC.
• Add MPSoC IP and run block automation to configure it.
• Run it with apply board preset selected
The MPSoC has now been configured to use the board preset for the ZCU104, which includes DDR configuration, input clock, reset GPIO, etc. Note that in case you are using a custom board, you would have to provide the configurations specific to that board.
Design customization
Now we need to create a design that will be compatible with the Vitis compiler tool later on. Historically, I have mostly been doing RTL designs but it seems like what we can do now is create a generic platform in Vivado and then use a custom-built accelerator (that's now reusable through projects!) in RTL/HLS/Opencl and add them later on in the software flow as needed. This will generate a binary file that can be loaded in the FPGA. The process uses the V++ linker which combines various accelerators with a target platform in our case the generic zcu104.
Configure the MPSOC interface
Enable AXI HPM0 LPD to control the AXI Interrupt Controller
In the block diagram, double-click the Zynq UltraScale+ MPSoC block.
Select PS-PL Configuration > PS-PL interfaces > Master interface.
Enable the AXI HPM0 LPD option.
Expand the arrow before AXI HPM0 LPD. Check the AXI HPM0 LPD Data width settings and keep it as default 32.
Disable AXI HPM0 FPD and AXI HPM1 FPD
Click OK to finish the configuration.
Note:
We use AXI HPM0 LPD mainly for control purposes. It will thus read and write 32 bits control registers. If the interface was more than 32 bits, AXI Interconnect or SmartConnect would do AXI bus width conversion using PL logic. This would cost logical resources and introduce unnecessary latency.
We reserve AXI HPM0 FPD and AXI HPM1 FPD for kernel usage. Disabling them from the block diagram can prevent auto connection by accident. We can export the unused AXI interfaces in Platform Setup, no matter if it's visible in the block diagram or not.
Reset and clock creation
Add the clocking wizard block to generate three clocks:
Right-click Diagram view and select Add IP.
Search for and add a Clocking Wizard from the IP Search dialog.
Double-click the clk_wiz_0 IP block to open the Re-Customize IP dialog box.
Click the Output Clocks tab.
Enable clk_out1 through clk_out3 in the Output Clock column. Set the Requested Output Freq as follows:
clk_out1 to 100 MHz.
clk_out2 to 200 MHz.
clk_out3 to 400 MHz.
At the bottom of the dialog box set the Reset Type to Active Low.
Processor System Reset:
Right-click Diagram view and select Add IP.
Search for and add a Processor System Reset from the IP Search dialog
Rename the reset block to proc_sys_reset_1 so that it's easy to understand the relationship between reset modules and the clock signals.
Select the proc_sys_reset_1 block, type Ctrl-C and Ctrl-V to replicate two modules. They are named as proc_sys_reset_2 and proc_sys_reset_3 by default.
Connect Clocks and Resets:
Click Run Connection Automation, which will open a dialog that will help connect the proc_sys_reset blocks to the clocking wizard clock outputs.
Enable All Automation on the left side of the Run Connection Automation dialog box.
Select clk_in1 on clk_wiz_0, and set the Clock Source to /zynq_ultra_ps_e_0/pl_clk0.
For each proc_sys_reset instance, select the slowest_sync_clk, and set the Clock Source as follows:
proc_sys_reset_1 with /clk_wiz_0/clk_out1
proc_sys_reset_2 with /clk_wiz_0/clk_out2
proc_sys_reset_3 with /clk_wiz_0/clk_out3
On each proc_sys_reset instance, select the ext_reset_in, set Board Part Interface to Custom, and set the Select Manual Source to /zynq_ultra_ps_e_0/pl_resetn0.
Make sure all checkboxes are enabled and click OK to close the dialog and create the connections.
Connect all the dcm_locked signals on each proc_sys_reset instance to the locked signal on clk_wiz_0.
Add the AXI Interrupt Controller
As for the clock, the Interrupt configuration of the platform will be exported in the PFM.IRQ property. They will be available to the V++ linker so they can connect the kernel module with the platform if needed.
Right-click Diagram view and select Add IP, search and add AXI Interrupt Controller IP. It's instantiated as axi_intc_0.
Double click the AXI Interrupt Controller block, change Interrupt Output Connection to Single so that it can be connected to the PS IRQ interface.
Click OK.
Connect AXI Interfaces of axi_intc_0 to AXI HPM0 LPD of PS
Click Run Connection Automation
Review the settings (axi_intc_0 is enabled, s_axi is to be connect to /zynq_ultra_ps_e_0/M_AXI_HPM0_LPD)
Set Clock Source for Slave Interface and Clock Source for Master Interface to /clk_wiz_0/clk_out2(200 MHz)
Click OK
If you regenerate the layout it will look like this
Step 2 - Platform Configuration
As an RTL designer, I have had to deal with Vivado IPI for the last couple of years, but rarely to define parameters for a downstream tool. The platform creation is mostly that. Defining an interface so that the Vitis Linker can understand the design and connect the hardware accelerator accordingly.
Clock
Go to the Platform Setup tab.
If it's not opened yet, use menu Window -> Platform Setup to open it.
Click Clock tab.
Enable all clocks under clk_wiz_0: clk_out1, clk_out2, clk_out3
Change their ID to 0, 1 and 2
Set a default clock: click Is Default for clk_out2
After everything is set up, it should report Info: No problem with Clock interface.
Interrupt
Go to the Platform Setup tab
Go to the Interrupt tab
Enable intr under axi_intc_0
AXI Interfaces
Go to the AXI Port tab in Platform Setup
Under zynq_ultra_ps_e_0, enable M_AXI_HPM0_FPD and M_AXI_HPM1_FPD. Keep the Memport and sptag default to M_AXI_GP and empty.
Note: The V++ linker will instantiate the AXI Interconnect automatically to connect between PS AXI Master interfaces and slave interfaces of acceleration kernels. One AXI Master interface will connect up to 16 kernels.
Under zynq_ultra_ps_e_0, multi-select all AXI slave interfaces: press Ctrl and click S_AXI_HPC0_FPD, S_AXI_HPC1_FPD, S_AXI_HP0_FPD, S_AXI_HP1_FPD, S_AXI_HP2_FPD, S_AXI_HP3_FPD.
Right-click the selections and select enable.
Change Memport of S_AXI_HPC0_FPD and S_AXI_HPC1_FPD to S_AXI_HP because we won't use any coherent features for these interfaces.
Type in simple sptag names for these interfaces so that they can be selected in the V++ configuration during the linking phase. HPC0, HPC1, HP0, HP1, HP2, HP3.
Under ps8_0_axi_periph, click M01_AXI, press Shift and click M07_AXI to multi-select master interfaces from M01_AXI to M07_AXI.
Right-click the selection and click on Enable.
Keep the Memport and sptag default to M_AXI_GP and empty.
Enable EMULATION
Select the PS instance zynq_ultra_ps_e_0 in the block diagram
Check the Block Properties window.
Scroll down to SELECTED_SIM_MODEL property. Change it from RTL to TLM in order to use the TLM model.
Export Hardware XSA
Validate the block design (ignore the critical warning about /axi_intc_0/intr)
Create a top module wrapper for the block design
In the Source tab, right-click system.bd in Design Sources group
Select Create HDL Wrapper.
Select Let Vivado manage wrapper and auto-update.
Click OK to generate a wrapper for block design.
Generate pre-synth design
Select Generate Block Design from Flow Navigator
Select Synthesis Options to Global. It will skip IP synthesis during generation.
Click Generate.
Export the platform
Click menu File -> Export -> Export Platform to launch the Export Hardware Platform wizard. This wizard can also be launched by the Export Platform button in Flow Navigator or Platform Setup window.
Click Next on the first information page.
Select Platform Type: Hardware and Hardware Emulation, click Next. If you skipped the emulation setup previously, select Hardware here.
Select Platform State: Pre-synthesis, click Next
Input Platform Properties and click Next. For example,
Name: zcu104_custom_platform
Vendor: Xilinx
Board: zcu104
Version: 0.0
Description: This platform provides high PS DDR bandwidth and three clocks: 100MHz, 200MHz, and 400MHz.
Fill in XSA file name: zcu104_custom_platform and keep the export directory as default.
Click Finish.
zcu104_custom_platform.xsa will be generated. The export path is reported in the Tcl console.
You now have a platform description ready for your project. As you might have noticed, we haven't synthesized or implemented anything yet. This will be done in the other steps.
If you want to speed up the process here is a TCL file doing all the previous steps for you.
note: you will need to convert back to tcl file first
Step 3 - Create a Petalinux project for our Hardware
Now that we have a hardware description of our project, we can create the Petalinux project which targets our HW configuration.
First, setup the petalinux environnement
source <petaLinux_tool_install_dir>/settings.shCreate a PetaLinux project named zcu104_custom_plnx and configure the HW with the XSA file we created before:
petalinux-create --type project --template zynqMP --name zcu104_custom_plnx
cd zcu104_custom_plnx
petalinux-config --get-hw-description=<vivado_design_dir>A Petalinux config menu will be launched and set to use the ZCU104 device tree in this configuration window.
Select DTG Settings->MACHINE_NAME
Modify it to zcu104-revc.
Select OK -> Exit -> Exit -> Yes to close this window.After this step, your directory hierarchy should look like this:
- zcu104_custom_platform # Vivado Project Directory
- zcu104_custom_plnx # PetaLinux Project Directory
Customize Root File System, Kernel, Device Tree and U-boot
Using your prefered editor modify the user rootfs config files as follow
<your_petalinux_project_dir>/project-spec/meta-user/conf/user-rootfsconfig
Replace the content with the following package
CONFIG_gpio-demo
CONFIG_peekpoke
CONFIG_xrt
CONFIG_dnf
CONFIG_e2fsprogs-resize2fs
CONFIG_parted
CONFIG_resize-part
CONFIG_packagegroup-petalinux-vitisai
CONFIG_packagegroup-petalinux-self-hosted
CONFIG_cmake
CONFIG_packagegroup-petalinux-vitisai-dev
CONFIG_xrt-dev
CONFIG_opencl-clhpp-dev
CONFIG_opencl-headers-dev
CONFIG_packagegroup-petalinux-opencv
CONFIG_packagegroup-petalinux-opencv-dev
CONFIG_mesa-megadriver
CONFIG_packagegroup-petalinux-x11
CONFIG_packagegroup-petalinux-v4lutils
CONFIG_packagegroup-petalinux-matchboxNow we need to go into the config tool and Enable the selected rootfs packages
petalinux-config -c rootfsSelect User Packages
select the name of all rootfs libraries listed above.
Enable OpenSSH/ package management and disable dropbear (optional)
In the RootFS configuration window, go to the root directory by selecting Exit once.
Go to Image Features.
Disable ssh-server-dropbear and enable ssh-server-openssh
enable package-management and debug_tweaks options
Click Exit.
Go to Filesystem Packages-> misc->packagegroup-core-ssh-dropbear and disable packagegroup-core-ssh-dropbear.
Go to the Filesystem Packages level by selecting Exit twice.
Go to console -> network -> openssh and enable openssh, openssh-sftp-server, openssh-sshd, openssh-scp.
Go to root level by selecting Exit four times.
(optionnal) Add GIT and VIM
Go to Filesystem Packages-> console-> utils-> git
Enable git, Git-bash-completion
Go to Filesystem Packages-> console-> utils-> vim
Enable vim
Exit and save the changes.
Disable CPU IDLE in kernel config (Recommended during debugging).
petalinux-config -c kernelEnsure the following items are TURNED OFF by entering 'n' in the [ ] menu selection:
CPU Power Management > CPU Idle > CPU idle PM support
CPU Power Management > CPU Frequency scaling > CPU Frequency scaling
Exit and Save.
Update the Device tree
Edit the following device tree file with some basic changes:
project-spec/meta-user/recipe-bsp/files/system-user.dtsi.
/include/ "system-conf.dtsi"
/ {
chosen {
bootargs = "earlycon console=ttyPS0,115200 clk_ignore_unused root=/dev/mmcblk0p2 rw rootwait cma=512M";
};
};
&sdhci1 {
no-1-8-v;
disable-wp;
};Build PetaLinux Images
Now that the configuration is complete we need to build the project. This step can take a lot of time so you might want to grab a cup of coffee.
petalinux-buildOnce the build process is complete you will also need to create the sysroot self-installer.
petalinux-build --sdkStep 4 - Create the Vitis Platform
Now that we have a Petalinux project ready and an HW description, it is time to create the Vitis platform project inside in which we will create the accelerated application and build the supporting hardware file.
First, let's create a directory that will hold the Vitis platform project. It should be located in your base project directory.
mkdir zcu104_custom_pkg
cd zcu104_custom_pkg
mkdir pfmYour directory structure should look like this
- zcu104_custom_platform # Vivado Project Directory
- zcu104_custom_plnx # PetaLinux Project Directory
- zcu104_custom_pkg # Platform Packaging Directory
- pfm # Platform Packaging SourceNow we need to install the sysroot inside our platform
cd <base dir>/zcu104_custom_plnx/images/linux
./sdk.sh -d ~/zcu104_custom_platform_2021/zcu104_custom_pkg/Create Boot and sd_card directory
cd zcu104_custom_pkg/pfm
mkdir boot
mkdir sd_dirPrepare the boot component. Copy the following file to zcu104_custom_pkg/pfm/boot
zynqmp_fsbl.elf
pmufw.elf
bl31.elf
u-boot-dtb.elf: rename to u-boot.elf
system.dtbPrepare sd_dir directory. Copy the following file to cu104_custom_pkg/pfm/sd_dir
boot.scr
system.dtbCreate a Vitis Platform
Now that the project directory structure is complete it's time to create the project.
cd zcu104_custom_pkg
source ~/Xilinx/Vitis/2021.1/settings64.shImportant - At this point, the process will differ from the provided example by Xilinx since on my desktop the new platform project doesn’t work. We are going to use a console flow to create the platform and then add it to the project. You should find a source example here.
xsct xsct_create_pfm.tcl
vitis ¬e: you will need to convert back to tcl file first
We will review the configuration in the next step.
Create a Vitis Accelerated Application
In Vitis:
File -| New application projectAdd the newly created platform to the list
Select the platform and click next.
Now, let’s create a vector addition to test the workflow.
Name the application: vadd
Click Next
Select the xrt domain
Set sysroot path to : <full_pathname_to_zcu104_custom_pkg>/sysroots/cortexa72-cortexa53-xilinx-linux
Set Root Fs to : zcu104_custom_plnx/images/linux/rootfs.ext4
Set Kernel Image to : zcu104_custom_plnx/images/linux/Image
Click Next
Now Select the Vector addition example and click next
We will add the Platform project for review
Click file import ->Eclipse workspace
Let's review the project
Double click on the platform.spr
Click on xrt
On the BIF file line: Click browse and select Generate BIF
On the left pane select the zcu104_custom platform and build the project.
In the Explorer window double click the vadd.prj file to open it, change the Active Build configuration from Emulation-SW to Hardware.
Now, select the vadd_system and build the project. This could take a while since we recompile the hardware to add a vector accelerator. Once it is done, we have our first accelerated application! You can review the code to understand what the application will do.
The important part is located inside :
vadd_system/vadd/src/vadd.cppYou will find the code required to load the Binary file with the accelerator in Linux and then pass data to it.
In vadd_system/vadd_kernels/src/krnl_vadd.cpp you will find the HLS source code of the accelerator being generated and integrated into our project.
SD card preparation
Now we need to flash an SD card to test our application on the target board.
Write the sd_card.img image file located in :
~/zcu104_custom_pkg/vadd_system/Hardware/packageusing your favorite image write tool (i.e. BalenaEtcher on windows)
Booting the board
Once the board is booted, connect to it using the serial port or an ssh connection as you prefer.
Go to the FAT32 partition
cd /mnt/sd-mmcblk0p1/Execute the application by providing the new binary with the accelerator
./vadd binary_container_1.xclbinIf everything went as planned you should see:
INFO: Reading binary_container_1.xclbin
Loading: 'binary_container_1.xclbin'
TEST PASSED
Congratulation, you just completed the first part of the tutorial. You now have a working platform, running petalinux with an example vector accelerator. In the next part, we are going to build on that platform to include a DPU (deep learning processing unit) and start using some accelerated AI applications, live from the board.