As of now, Vitis AI runtime libraries are provided in a docker container. While it is possible to copy the sources for facedetect into the runtime docker and compile it there, this tutorial demonstrates an alternative approach which allows building in petalinux without using the docker for the build (though the runtime docker is still needed on the host, at least the first time the repository is created). The approach is to copy the dependencies out of the docker into a local repository that is fetched by a Yocto recipe during the petalinux build. The makefile target is shown below:
The primary tasks of this target are: 1) prepare the directory that’s shared between docker and the host 2) execute the scripts/copy_libs.sh scripts inside the docker container and 3) cleanup. The scripts/copy_libs.sh script simply copies a number of libraries and header files out of the cross compile environment in the docker container into repos/Vitis-AI/Rebuild-Model-Zoo/libs directory. Once complete, the makefile organizes it into repos/vitis-ai-pfm. In a later step, Petalinux recipes will pick up these files and insert them into the target’s rootfs.
Now that the .xsa file is available, a petalinux project can be created to generate the linux image, boot files, and sysroot.
This target does several things:
The petalinux configuration is in petalinux/configs/config_2019.2 and contains the following interesting modifications
The rootfs configuration is in petalinux/configs/rootfs_config_2019.2 and contains the following interesting modifications
A number of device tree modifications are necessary to set up the image capture pipeline and also the Zynq OpenCL framework. The device tree customizations are in petalinux/devicetree/system-user_2019.2.dtsi.
The MIPI RX Subsystem and the Demosaic are connected together normally with appropriate settings for the design
The output of the Demosaic feeds a video capture node for the Framebuffer Write IP
Normally, the input of the MIPI CSI2 RX Subsystem will be connected to an upstream V4L2 node for the image sensor driver which would complete the pipeline. In this case, the camera sensor driver is not so simple because of the serializer/deserializer in the I2C path that need to be configured and managed. The bringupcam userspace application takes the place of this driver and handles all the I2C accesses to the serializer/deserializer and image sensor. However, the MIPI CSI2 RX subsystem input cannot be left unconnected in the devicetree or else a /dev/media node will not be correctly created. One of 3 options were considered to solve this problem:
Ultimately, option 3 above was chosen for the cleanest implementation. This driver is provided as a kernel patch in repos/dummy_src_driver.
This dummy_src driver needs a device tree node as well. Since it’s implementation is based off the Xilinx TPG driver, the properties are similar.
Finally, the ZOCL driver needs to be informed about the interrupts that are available to the AXI Interrupt controller
The bringupcam application is modular software service whose purpose is to perform I2C configuration of an image sensor, serializer, and deserializer on the FMC card. It looks for hardware.conf file in /etc in the target filesystem. It is a relatively large codebase and the implementation/operation is beyond the scope of this article. The source is provided for those who are interested.
petalinux/recipes/autostart is a Yocto recipe that inserts some initialization scripts into the root filesystem that run automatically. petalinux/recipes/autostart/autostart.bb is the recipe itself shown here
This recipe is picked up by Petalinux and processed by Yocto back end during the build. The SRC_URI variable indicates the two files that are installed into the rootfs (autostart.sh and loginenv.sh). INITSCRIPT_NAME and INITSCRIPT_PARAMS are instructions for how update-rc.d is to handle inserting autostart.sh as an init script. The do_install() callback handles the install/copying of the scripts into /etc/init.d and /etc/profile.d in the target rootfs.
The autostart.sh script itself is shown below
Note that the dpu.xclbin is copied from the first partition on the SD card into /usr/lib. When the facedetect application runs, XRT locates and loads it.
The facedetect application is the primary application for this tutorial. The main() function is shown below
The OpenCV VideoCapture class is used to grab frames from a Gstreamer capture pipeline. By default, v4l2src uses /dev/video0. The cur_frame Mat object is passed to the DPU through the ml_task (which is itself a xilinx::ai::FaceDetect object). The output of the face detection algorithm is passed to process_result() function which draws bounding boxes using OpenCV rect() function. The final output frames are written to OpenCV VideoWriter class to a Gstreamer display pipeline that renders the display using the Mali GPU onto the Displayport Interface (glimagesink plugin). XR24 is the source code that corresponds to the RGB pixel format that is used. See https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/videodev.html#videodev
petalinux/recipes/vitis-ai simply contains a bitbake recipe which is shown below
Note that the FILESEXTRAPATHS_prepend and SRC_URI point to repos/vitis-ai-pfm created previously. There are a number of dependencies and configurations to package the libraries. The runtime dependencies are then installed into the target rootfs.
Brian Wiec is a Field Applications Engineer in the Detroit area serving Xilinx’s Automotive customer base supporting applications in ADAS, autonomous driving, infotainment, and powertrain control. He has worked at Xilinx for eight years, both in the field and factory support roles with experience in video, signal processing, and embedded systems design/implementation. Brian is always happy to partner with customers to help them solve their technical challenges and enjoys participating in their innovations. In his free time, Brian likes spending time with his family, hiking, listening to music, playing hockey, and watching college football (Go Blue!).