Introduction

About this Document

Related Version

The following AI Library version is related to this document.
Table 1. Vitis AI Library Packet List
No Packet Name Version
1 vitis_ai_library_r1.2.0_video.tar.gz r1.2
2 vitis_ai_library_r1.2.0_image.tar.gz r1.2
3 vitis-ai-runtime-1.2.0.tar.gz r1.2
4 vitis_ai_2020.1-r1.2.0.tar.gz r1.2
5 xilinx_model_zoo_zcu102-1.2.0-1.aarch64.rpm r1.2
6 xilinx_model_zoo_zcu104-1.2.0-1.aarch64.rpm r1.2
7 xilinx_model_zoo_u50_1.2.0_amd64.deb r1.2
8 xilinx_model_zoo_u280_u50lv9e_1.2.0_amd64.deb r1.2
9 xilinx_model_zoo_u50lv10e_1.2.0_amd64.deb r1.2
10 alveo_xclbin-1.2.1.tar.gz r1.2
11 sdk-2020.1.0.0.sh 2020.1

Intended Audience

The users of Vitis AI libraries are as follows:

  • Users who want to use Xilinx’s models to quickly build applications.
  • Users who use their own models that are retrained by their own data under the Vitis AI library support network list.
  • Users who have custom models, similar to the model supported by the Vitis AI libraries, and use the Vitis AI’s post processing library.
Note: If you have custom models that are completely different from the model supported by the AI Library or has a special post-processing part, they can also use our samples and libraries implementation for reference.

Document Navigation

This document describes how to install, use, and develop with the AI Library.

  • Chapter 1 is an introduction to the AI Library. This chapter provides a clear understanding of the AI Library in general, its framework, supported networks, supported hardware platforms and so on.
  • Chapter 2 describes how to install the AI Library and run the example. The information in this chapter will help quickly set up the host and target environments, compile and execute the AI Library related examples.
  • Chapter 3 describes, in detail, each model library supported by the AI Library. This chapter provides an understanding of the model libraries supported by the AI Library, the purpose of each library, how to test the library with images or videos, and how to test the performance of the library.
  • Chapter 4 describes, in detail, how to develop applications with AI Library. This chapter provides an understanding of the following:
    • Development using Vitis API
    • Development using your models
    • Customizing pre-processing
    • Using the configuration file as pre-processing and post-processing parameters
    • Using the post-processing library in AI Library
    • Implementing your post-processing code
  • Chapter 5 describes how to set up a test environment and run the application demos. There are two application demos provided with the Vitis AI Library.
  • Chapter 6 describes how to find the programming APIs.
  • Chapter 7 describes, in detail, the performance of the Vitis AI library on different boards.

Overview

The Vitis AI Library is a set of high-level libraries and APIs built for efficient AI inference with Deep-Learning Processor Unit (DPU). It is built based on the Vitis AI Runtime with unified APIs, and it fully supports XRT 2020.1.

The Vitis AI Library provides an easy-to-use and unified interface by encapsulating many efficient and high-quality neural networks. This simplifies the use of deep-learning neural networks, even for users without knowledge of deep-learning or FPGAs. The Vitis AI Library allows users to focus more on the development of their applications, rather than the underlying hardware.

For the intended audience for the AI Library, please refer to the About this Document section.

Block Diagram

The Vitis™ AI Library contains four parts: the base libraries, the model libraries, the library test samples, and the application demos.

The base libraries provide the operation interface with the DPU and the post-processing module of each model. dpu_task is the interface library for DPU operations. xnnpp is the post-processing library of each model, with build-in modules such as optimization and acceleration.

The model libraries implement most of the neural network deployment which are open source. They include common types of networks, such as classification, detection, segmentation, and so on. These libraries provide an easy-to-use and fast development method with a unified interface, which are applicable to the Xilinx models or custom models. The library test samples are used to quickly test and evaluate the model libraries. The application demos show users how to use AI Library to develop applications. The Vitis AI Library block diagram is shown in the following figure.

Figure 1: Vitis AI Library Block Diagram

Features

The Vitis AI Library features include:

  • A full-stack application solution from top to buttom
  • Optimized pre- and post-processing functions/libraries
  • Open-source model libraries
  • Unified operation interface with the DPU and the pre-processing and post-processing interface of the model
  • Practical, application-based model libraries, pre-processing and post-processing libraries, and application examples

Vitis AI Library 1.2 Release Notes

This section contains information regarding the features and updates of the Vitis™ AI Library 1.2 release.

Key Features And Enhancements

This AI Library release includes the following key features and enhancements.

New Cloud Board Support
Alveo U50lv and U280 cards are new supported by this release.
New Model Libraries
The following new model libraries are supported.
  • face recognition
  • plate detection
  • plate recognition
  • medical segmentation
Pytorch Model Support
3 pytorch models are supported.
Add 6 Caffe Models
6 new caffe models are supported.

Changes

  1. The installation mode of the target for the edge is changed and the rpm format package is used.
  2. meta.json file in the model has been deprecated.

Compatibility

  • Vitis AI Library 1.2 has been tested with the following images.
    • xilinx-zcu102-dpu-v2020.1-v1.2.0.img.gz
    • xilinx-zcu104-dpu-v2020.1-v1.2.0.img.gz

Model Support

The following models are supported by this version of the Vitis AI Library.

Table 2. Model Supported by the AI Library
No. Neural Network ZCU102/ZCU104 U50/U50lv/U280 Application
1 inception_resnet_v2_tf Y Y Image Classification
2 inception_v1_tf Y Y
3 inception_v3_tf Y Y
4 inception_v4_2016_09_09_tf Y Y
5 mobilenet_v1_0_25_128_tf Y N/A
6 mobilenet_v1_0_5_160_tf Y N/A
7 mobilenet_v1_1_0_224_tf Y N/A
8 mobilenet_v2_1_0_224_tf Y N/A
9 mobilenet_v2_1_4_224_tf Y N/A
10 resnet_v1_101_tf Y Y
11 resnet_v1_152_tf Y Y
12 resnet_v1_50_tf Y Y
13 vgg_16_tf Y Y
14 vgg_19_tf Y Y
15 ssd_mobilenet_v1_coco_tf Y N/A Object Detection
16 ssd_mobilenet_v2_coco_tf Y N/A
17 ssd_resnet_50_fpn_coco_tf Y Y
18 yolov3_voc_tf Y Y
19 mlperf_ssd_resnet34_tf Y N/A
20 resnet50 Y Y Image Classification
21 resnet18 Y Y
22 inception_v1 Y Y
23 inception_v2 Y Y
24 inception_v3 Y Y
25 inception_v4 Y Y
26 mobilenet_v2 Y N/A
27 squeezenet Y Y
28 ssd_pedestrain_pruned_0_97 Y Y ADAS Pedestrian Detection
29 ssd_traffic_pruned_0_9 Y Y Traffic Detection
30 ssd_adas_pruned_0_95 Y Y ADAS Vehicle Detection
31 ssd_mobilenet_v2 Y N/A Object Detection
32 refinedet_pruned_0_8 Y Y
33 refinedet_pruned_0_92 Y Y
34 refinedet_pruned_0_96 Y Y
35 vpgnet_pruned_0_99 Y Y ADAS Lane Detection
36 fpn Y Y ADAS Segmentation
37 sp_net Y Y Pose Estimation
38 openpose_pruned_0_3 Y Y
39 densebox_320_320 Y Y Face Detection
40 densebox_640_360 Y Y
41 face_landmark Y Y Face Detection and Recognition
42 reid Y Y Object tracking
43 multi_task Y Y ADAS
44 yolov3_adas_pruned_0_9 Y Y Object Detection
45 yolov3_voc Y Y
46 yolov3_bdd Y Y
47 yolov2_voc Y Y
48 yolov2_voc_pruned_0_66 Y Y
49 yolov2_voc_pruned_0_71 Y Y
50 yolov2_voc_pruned_0_77 Y Y
51 facerec_resnet20 Y Y Face Recognition
52 facerec_resnet64 Y Y
53 plate_detection Y Y Plate Recognition
54 plate_recognition Y Y
55 FPN_Res18_Medical_segmentation Y Y Medical Segmentation
56 refinedet_baseline Y Y Object Detection
57 resnet50_pt N/A Y Image Classification
58 squeezenet_pt N/A Y
59 inception_v3_pt N/A Y
  1. No1-No19 neural network models are trained based on the Tensorflow framework.
  2. No20-No56 neural network models are trained based on the Caffe framework.
  3. No57-No59 neural network models are trained based on the Pytorch framework.

Device Support

The following platforms and EVBs are supported by the Vitis AI Library1.2.

Table 3. Edge Device Support
Platform EVB Version
Zynq UltraScale+ MPSoC ZU9EG Xilinx ZCU102 V1.1
Zynq® UltraScale+™ MPSoC ZU7EV Xilinx ZCU104 V1.0
Table 4. Cloud Device Support
Accelerator Cards
Xilinx Alveo U50
Xilinx Alveo U50lv
Xilinx Alveo U280

Limitations

  • Some neural networks with mobilenet as the backbone are not supported on U50, U50lv and U280.
  • Pytorch models are not supported for edge devices.
  • Due to the limitation of Docker environment, the Multi-Task demos cannot run in DRM mode on the cloud devices.

Vitis AI Library 1.1 Release Notes

This section contains information regarding the features and updates of the Vitis™ AI Library 1.1 release.

Key Features And Enhancements

This AI Library release includes the following key features and enhancements.

Support for the cloud
Alveo U50 card is supported by this release.
New DPU support
DPU is supported which can by used for the cloud.
New Open Source Library
The xnnpp library is open source in this release, which shows how to do the pre-processing and post-processing for the neural networks.
Model Library Update
The new model library unifies the interface between the cloud and edge.

Changes

The installation mode of the host for the edge is changed and the original Docker installation mode is no longer used.

Compatibility

  • Vitis AI Library 1.1 has been tested with the following images.
    • xilinx-zcu102-dpu-v2019.2-v2.img
    • xilinx-zcu104-dpu-v2019.2-v2.img

Model Support

The following models are supported by this version of the Vitis AI Library.

Table 5. Model Supported by the AI Library
No. Neural Network ZCU102/ZCU104 U50 Application
1 inception_resnet_v2_tf Y Y Image Classification
2 inception_v1_tf Y Y
3 inception_v3_tf Y Y
4 inception_v4_2016_09_09_tf Y Y
5 mobilenet_v1_0_25_128_tf Y N/A
6 mobilenet_v1_0_5_160_tf Y N/A
7 mobilenet_v1_1_0_224_tf Y N/A
8 mobilenet_v2_1_0_224_tf Y N/A
9 mobilenet_v2_1_4_224_tf Y N/A
10 resnet_v1_101_tf Y Y
11 resnet_v1_152_tf Y Y
12 resnet_v1_50_tf Y Y
13 vgg_16_tf Y Y
14 vgg_19_tf Y Y
15 ssd_mobilenet_v1_coco_tf Y N/A Object Detection
16 ssd_mobilenet_v2_coco_tf Y N/A
17 ssd_resnet_50_fpn_coco_tf Y Y
18 yolov3_voc_tf Y Y
19 mlperf_ssd_resnet34_tf Y N/A
20 resnet50 Y Y Image Classification
21 resnet18 Y Y
22 inception_v1 Y Y
23 inception_v2 Y Y
24 inception_v3 Y Y
25 inception_v4 Y Y
26 mobilenet_v2 Y N/A
27 squeezenet Y Y
28 ssd_pedestrain_pruned_0_97 Y Y ADAS Pedestrian Detection
29 ssd_traffic_pruned_0_9 Y Y Traffic Detection
30 ssd_adas_pruned_0_95 Y Y ADAS Vehicle Detection
31 ssd_mobilenet_v2 Y N/A Object Detection
32 refinedet_pruned_0_8 Y Y
33 refinedet_pruned_0_92 Y Y
34 refinedet_pruned_0_96 Y Y
35 vpgnet_pruned_0_99 Y Y ADAS Lane Detection
36 fpn Y Y ADAS Segmentation
37 sp_net Y Y Pose Estimation
38 openpose_pruned_0_3 Y Y
39 densebox_320_320 Y Y Face Detection
40 densebox_640_360 Y Y
41 face_landmark Y Y Face Detection and Recognition
42 reid Y Y Object tracking
43 multi_task Y Y ADAS
44 yolov3_adas_pruned_0_9 Y Y Object Detection
45 yolov3_voc Y Y
46 yolov3_bdd Y Y
47 yolov2_voc Y Y
48 yolov2_voc_pruned_0_66 Y Y
49 yolov2_voc_pruned_0_71 Y Y
50 yolov2_voc_pruned_0_77 Y Y
  1. No1-No19 neural network models are trained based on the Tensorflow framework.
  2. No20-No50 neural network models are trained based on the Caffe framework.

Device Support

The following platforms and EVBs are supported by the Vitis AI Library1.1.

Table 6. Edge Device Support
Platform EVB Version
Zynq UltraScale+ MPSoC ZU9EG Xilinx ZCU102 V1.1
Zynq® UltraScale+™ MPSoC ZU7EV Xilinx ZCU104 V1.0
Table 7. Cloud Device Support
Accelerator Cards
Xilinx Alveo U50

Limitations

Some neural networks with mobilenet as the backbone are not supported on U50.

Vitis AI Library 1.0 Release Notes

This section contains information regarding the features and updates of the Vitis™ AI Library 1.0 release. This release is the successor of last Xilinx® AI SDK v2.0 release.

The Vitis AI Library is a set of high-level libraries and APIs built for efficient AI inference with Deep-Learning Processor Unit (DPU). It provides an easy-to-use and unified interface by encapsulating many efficient and high-quality neural networks.

Key Features And Enhancements

This Vitis AI Library release includes the following key features and enhancements.

Support for new Vitis AI Runtime
The Vitis AI Library is updated to be based on the new Vitis AI Runtime with unified APIs. It also fully supports XRT 2019.2.
New DPU support
Besides DPU for edge devices, new AI Library will support new cloud based DPU IPs using same codes (runtime and models for cloud DPU will not be included in this release).
New Tensorflow model support
There are up to 21 tensorflow models supported, which are from official Tensorflow repository. The pre-compiled models for edge devices are included, while original models are released by updated Model Zoo.
New Libraries and Demos
There are two new libraries libdpmultitask and libdptfssd which supports multi-task models and SSD models from official tensorflow repository.

There is an updated classification demo that shows the usage of the unified APIs in Vitis AI runtime.

New Open Source Library
The libdpbase library is open source in this release, which shows how to use unified APIs in Vitis AI runtime to construct high-level libraries.
New Installation Method
The host side environment adopts docker image installation, which simplifies and unifies the installation process.

Compatibility

  • Vitis AI Library 1.0 has been tested with the following images.
    • xilinx-zcu102-dpu-v2019.2.img
    • xilinx-zcu104-dpu-v2019.2.img
  • For existing Xilinx AI SDK v2.0 users, the library interface remains consistent and the application can be directly ported to the new Vitis AI Library.

Model Support

The following models are supported by this version of the Vitis AI Library.

Table 8. Model Supported by the AI Library
No. Neural Network Application
1 inception_resnet_v2_tf Image Classification
2 inception_v1_tf
3 inception_v3_tf
4 inception_v4_2016_09_09_tf
5 mobilenet_v1_0_25_128_tf
6 mobilenet_v1_0_5_160_tf
7 mobilenet_v1_1_0_224_tf
8 mobilenet_v2_1_0_224_tf
9 mobilenet_v2_1_4_224_tf
10 resnet_v1_101_tf
11 resnet_v1_152_tf
12 resnet_v1_50_tf
13 vgg_16_tf
14 vgg_19_tf
15 ssd_mobilenet_v1_coco_tf Object Detection
16 ssd_mobilenet_v2_coco_tf
17 ssd_resnet_50_fpn_coco_tf
18 yolov3_voc_tf
19 mlperf_ssd_resnet34_tf
20 resnet50 Image Classification
21 resnet18
22 inception_v1
23 inception_v2
24 inception_v3
25 inception_v4
26 mobilenet_v2
27 squeezenet
28 ssd_pedestrain_pruned_0_97 ADAS Pedestrian Detection
29 ssd_traffic_pruned_0_9 Traffic Detection
30 ssd_adas_pruned_0_95 ADAS Vehicle Detection
31 ssd_mobilenet_v2 Object Detection
32 refinedet_pruned_0_8
33 refinedet_pruned_0_92
34 refinedet_pruned_0_96
35 vpgnet_pruned_0_99 ADAS Lane Detection
36 fpn ADAS Segmentation
37 sp_net Pose Estimation
38 openpose_pruned_0_3
39 densebox_320_320 Face Detection
40 densebox_640_360
41 face_landmark Face Detection and Recognition
42 reid Object tracking
43 multi_task ADAS
44 yolov3_adas_pruned_0_9 Object Detection
45 yolov3_voc
46 yolov3_bdd
47 yolov2_voc
48 yolov2_voc_pruned_0_66
49 yolov2_voc_pruned_0_71
50 yolov2_voc_pruned_0_77
  1. No1-No19 neural network models are trained based on the Tensorflow framework.
  2. No20-No50 neural network models are trained based on the Caffe framework.

Device Support

The following platforms and EVBs are supported by the Vitis AI Library1.0.

Table 9. Device Support
Platform EVB Version
Zynq UltraScale+ MPSoC ZU9EG Xilinx ZCU102 V1.1
Zynq® UltraScale+™ MPSoC ZU7EV Xilinx ZCU104 V1.0

Limitations

Because of the complicated configuration for SSD models from the official Tensorflow repository, there is a new libdptfssd library that is different from the original libdpssd library for caffe models. These two libraries may be merged in future releases.

Deprecated Features

The following features are deprecated in Vitis AI Library 1.0.

  • Removed demos.

    The squeezenet and SSD demos have been removed. Because we highly encourage customers to use high-level APIs from AI Library for applications and solutions, we only provide one classification demo in this release to show how to use low-level unified APIs in Vitis AI runtime.

  • Removed pre-compiled models.
    We removed six Tensorflow models in this release but provided in the previous Xilinx AI v2.0 release, to keep sync with updated Model Zoo. Models that are removed can be replaced by similar models in updated Model Zoo which come from Tensorflow slim models. The models are:
    • resnet_50_tf
    • inception_v1_tf
    • resnet_18_tf
    • mobilenet_v1_tf
    • mobilenet_v2_tf
    • ssd_voc_tf