MLPerf v1.0 Benchmark Demo on VCK5000

Xilinx
MLPref

MLPerf v1.0 Benchmark Demo on VCK5000

MLPerf.org issued its third round of inferencing result (MLPerf Inference v1.0). In this benchmark, Xilinx has achieved a result of ResNet-50 inference throughput at 5,921 FPS (Available catalog, server mode), using the latest Versal ACAP PCI-e card VCK5000 for Data Center. It showed the industry-leading performance acceleration with the Domain-specific Architecture optimized for AI inference on Xilinx adaptive computing platform. Now, the demo is available for free evaluation on the App Store. 

Vendor: Xilinx

Last Update: June 11, 2021

Size: 370MB

Container Version: xilinx/vitis-ai-cpu:1.3.598


Deployment Options

This application is containerized and can be easily run in a few minutes in the cloud, or on-premises.

On Premises
Alveo VCK5000
View & Buy Product
  • Xilinx Runtime: 2020.2_PU1
  • Target Platform: xilinx_vck5000-es1_gen3x16_base_2-3102046

 


Start Evaluation

Follow the instructions based on your deployment method.

Alveo VCK5000

1.

Host Setup

1.1 Vitis AI 1.3.1 Docker Installation

    docker pull xilinx/vitis-ai-cpu:1.3.598

1.2 Install VCK5000 Shell/XRT packages

After the installation, cold reboot machine. For CentOS7.4/REHL7.4:

    cd shell_xrt/CentOS7.4       
sudo yum install xrt_202110.2.9.204_7.4.1708-x86_64-xrt.rpm
sudo yum install xilinx-sc-fw-vck5000-4.4.6-2.e1f5e26.noarch.rpm
sudo yum install xilinx-vck5000-es1-gen3x16-validate-2-3105556.noarch.rpm
sudo yum install xilinx-vck5000-es1-gen3x16-base-2-3102046.noarch.rpm
sudo /opt/xilinx/xrt/bin/xbmgmt flash --update --shell xilinx_vck5000-es1_gen3x16_base_2

1.3. Install XBulter on AIE5000 host machine

Please refer to XBulter Installation (https://github.com/Xilinx/Vitis-AI/tree/master/setup/alveo/u200_u250/packages) to install XBulter. After the installation of XBulter, reboot xbulter service.

    sudo service xbutler restart 

2.

Running MLPerf

2.1 After downloaded Vitis-AI docker, enter Vitis-AI directory, and then start the Docker.

Update Vitis AI runtime libraries in docker

    cd vitis-ai-libs 
sudo dpkg -i librt-engine_1.3.0_amd64.deb
sudo dkpg -i libtarget-factory_1.3.0_amd64.deb

2.2 Dataset Downloading

Please download validation dataset from http://image-net.org/challenges/LSVRC/2012/


2.3 Build and install the MLPerf benchmark:

    cd xilinx-mlperf-v1.0/mlperf-vitis-benchmark-app
make clean&& make -j

2.4 Running MLPerf benchmark

Note that MLPerf ResNet50 benchmarking results may slightly fluctuate due to different configurations of hardware resource. For example, advanced CPU processor and higher PCIe bandwidth will bring better performance result.

  • MLPerf server performance test
    • cd mlperf-vitis-benchmark-app
    • ./run.sh -r [TARGET_QPS] --dir [IMG_DIR]

NOTE: default TARGET_QPS is 4000. To achieve better performance result, to change the value of TARGET_QPS.

--dir is to specify the location of imageNet validation dataset.

e.g. ./run.sh -r 5200 --dir  /workspace/CK-TOOLS/dataset-imagenet-ilsvrc2012-val
The result is shown in mlperf_log_summary.txt

  • MLPerf server accuracy test
    • conda activate vitis-ai-tensorflow
    • ./run.sh -m AccuracyOnly -n 50000 --dir [IMG_DIR]

  • MLPerf server accuracy test
    • cd mlperf-vitis-benchmark-app
    • ./run.sh -r [TARGET_QPS] -s Offline --dir [IMG_DIR]
       
  • MLPerf offline accuracy test
    • conda activate vitis-ai-tensorflow
    • ./run.sh -m AccuracyOnly -n 50000 -s Offline --dir [IMG_DIR]


3.

Result

Check the benchmark performance result in mlperf_log_summary.txt;
Check the accuracy result in accuracy.txt

** If you have any questions or technical inquiries, please contact with bingqing@xilinx.com