Vitis FPGA Development
Getting started with Vitis FPGA development.
Last updated
Getting started with Vitis FPGA development.
Last updated
ExCl → User Documentation → Vitis FPGA Development
| FPGA | State |
|---|---|
U250 | Attached to |
u55C | Attached to |
u280 | Aaron’s office |
This page covers how to access the Vitis development tools available in ExCL. The available FPGAs are listed in the FPGAs section. All Ubuntu 22.04 systems can load the Vitis/Vivado development tools as a module. See Quickstart to get started. The virtual systems have ThinLinc installed, which makes it easier to run graphical applications. See section Accessing ThinLinc to get started.
Vitis is now primarily deployed as a module for Ubuntu 22.04 systems. You can view available modules and versions with module avail and load the most recent version with module load Vitis. These modules should be able to work on any Ubuntu 22.04 system in ExCL.
| FPGA | Host System | Slurm GRES Name |
|---|---|---|
Spike | U250 | |
Spike | U55C | |
U280 |
Suggested machines to use for Vitis development are also setup with Slurm. Slurm is used as a resource manager to allocate compute resources as well as hardware resources. The use of Slurm is required to allocate FPGA hardware and reserve build resources on Triple Crown. It is also recommended to reserve resources when running test builds on Zenith. The best practice is to launch builds on fpgabuild with Slurm, then launch bitfile tests with Slurm. The use of Slurm is required to effectively share the FPGAs, and to share build resources with automated CI Runs, and other automated build and test scripts. As part of the Slurm interactive use or batch script, use modules to load the desired version of the tools. The rest of this section details how to use Slurm. See the Cheat Sheet for commonly used Slurm commands. See the Slurm Quick Start User Guide to learn the basics of using Slurm.
Allocate a build instance for one Vitis Build. Each Vitis build uses 8 threads by default. If you plan to use more threads, please adjust -c accordingly.
Where: -J, --job-name=<jobname> -p, --partition=<partition names> -c, --cpus-per-task=<ncpus>
Recommended: bash can be replaced with the build or execution command to run the command and get the results back to your terminal. Otherwise, you have to exit the bash shell launched by srun to release the resources.
Recommended: sbatch can be used with a script to queue the job and store the resulting output to a file. sbatch is better than srun for long-running builds.
Allocate the U250 FPGA to run hardware jobs. Please release the FPGA when you are done so that other jobs can use the FPGA.
Where: -J, --job-name=<jobname> -p, --partition=<partition names> --
gres="fpga:U250:1"specifies that you want to use 1 U250 FPGA.
Recommended: bash can be replaced with the build or execution command to run the command and get the results back to your terminal. Otherwise, you have to exit the bash shell launched by srun to release the resources.
Resources: "fpga:U250:1" can be replaced with the FPGA resource that you want to use. Multiple resources can also be reserved at a time. See FPGAs for a list of available FPGAs.
Where: -J, --job-name=<jobname> -p, --partition=<partition names> -c, --cpus-per-task=<ncpus> build.sh is a script to launch the build.
Recommended: The Slurm parameters can be stored in build.sh with #SBATCH <parameter>.
Template: See Slurm Templates · code.ornl.gov for Slurm sbatch script templates.
Where: -J, --job-name=<jobname> -p, --partition=<partition names> --
gres="fpga:U250:1"specifies that you want to use 1 U250 FPGA. run.sh is a script to launch the run.
Recommended: The Slurm parameters can be stored in build.sh with #SBATCH <parameter>.
Template: See Slurm Templates · code.ornl.gov for Slurm sbatch script templates.
From the login node run srun -J interactive_build -p fpgabuild -c 8 --pty bash to start a bash shell.
Use module load vitis to load the latest version of the vitis toolchain.
Use source /opt/xilinx/xrt/setup.sh to load the Xilinx Runtime (XRT).
Follow the quickstart to set up the Vitis Environment.
Go through the Vitis Getting Started Tutorials.
Go through the Vitis Hardware Accelerators Tutorials.
Go through the Vitis Accel Examples.
Use platforminfo to query additional information about an FPGA platform. See the example command below.
See ThinLinc Quickstart.
Note: Fish is not backwards compatible with Bash. See Fish for bash users (fishshell.com). So in order to load modules and source bash scripts, I have included the bass function. Prepend bass before the source or module commands to use bash features in fish.
Fish is installed system-wide with a default configuration based on Aaron's fish configuration that includes helpful functions to launch the Xilinx development tools. The next sections goes over the functions that this fish config provides.
sfpgabuild is a shortcut to calling srun -J interactive_build -p fpgabuild -c 8 --mem 8G --mail-type=END,FAIL --mail-user $user_email --pty $argv . Essentially it setups a FPGA build environment using slurm using resonable defaults. Each of the defaults can be overriden by spacifying the new parameter when calling sfpgabuild . sfpgabuild also modifies the prompt to remind you that you are in the fpga build environment.
sfpgarun is a shortcut to calling srun -J fpgarun-u250 -p fpgarun -c 8 --mem 8G --mail-type=END,FAIL --mail-user $user_email --gres="fpga:U250:1" --pty $argv . sfpgarun-u250 setups up an FPGA run environment complete with requesting the FPGA resource.
sfpgarun is a shortcut to calling srun -J fpgarun-u55c -p fpgarun -c 8 --mem 8G --mail-type=END,FAIL --mail-user $user_email --gres="fpga:U55C:1" --pty $argv . sfpgarun-u55c setups up an FPGA run environment complete with requesting the FPGA resource.
sfpgarun is a shortcut to calling XCL_EMULATION_MODE=hw_emu srun -J fpgarun -p fpgarun -c 8 --mem 8G --mail-type=END,FAIL --mail-user $user_email --pty $argv . sfpgarun-hw-emu setups up an FPGA run environment complete with specifying XCL_EMULATION_MODE.
sfpgarun is a shortcut to calling XCL_EMULATION_MODE=sw_emu srun -J fpgarun -p fpgarun -c 8 --mem 8G --mail-type=END,FAIL --mail-user $user_email --pty $argv . sfpgarun-sw-emu setups up an FPGA run environment complete with specifying XCL_EMULATION_MODE.
After running bass module load vitis, sfpgabuild, or sfpgarun, viv can be used to launch Vivado in the background and is a shortcut to calling vivado -nolog -nojournal.
In order to manually set up the the Xilinx license, set the environment variable XILINXD_LICENSE_FILE to 2100@license.ftpn.ornl.gov.
The FlexLM server uses ports 2100 and 2101.
Note: This step is done automatically by the module load command and manually setting up the license should not be needed.
Xilinx FPGA projects can be built using the Vitis Compiler, the Vitis GUI, Vitis HLS, or Vivado.
In general, I recommend using the Vitis compiler via the command line and scripts, because the workflow is easy to document, store in git, and run with GitLab CI. I recommend using Vitis HLS when trying to optimize kernel since it provides many profiling tools. See Vitis HLS Tutorial.
Tutorials are available to learn how to use Vitis. In particular, this Getting started with Vitis Tutorial goes over the building and running of an example application.
See the Vitis Documentation for more details on building and running FPGA applications.
The Vitis environment and tools are setup via the module files. To load the latest version of the Vitis environment use the following command. In bash:
In fish:
To see available versions use module avail. Then a specific version can be loaded by specifying the version, for example module load vitis/2020.2.
See the Vitis Documentation for more details on setting up the Vitis Environment.
Note: Because of issues with XRT and with OpenCL including the xilinx.icd by default, on many system we moved the xilinx.icd to /etc/OpenCL/vendors/xilinx/xilinx.icd. Now to load the FPGA as an OpenCL device, you must change the environment variable OPENCL_VENDOR_PATH to point to /etc/OpenCL/vendors/xilinx or /etc/OpenCL/vendors/all.
There are three build targets available when building an FPGA kernel with Vitis tools.
See the Vitis Documentation for more information.
| Software Emulation | Hardware Emulation | Hardware Execution |
|---|---|---|
Host application runs with a C/C++ or OpenCL™ model of the kernels. | Host application runs with a simulated RTL model of the kernels. | Host application runs with actual hardware implementation of the kernels. |
Used to confirm functional correctness of the system. | Test the host / kernel integration, get performance estimates. | Confirm that the system runs correctly and with desired performance. |
Fastest build time supports quick design iterations. | Best debug capabilities, moderate compilation time with increased visibility of the kernels. | Final FPGA implementation, long build time with accurate (actual) performance results. |
The designed build target is specified with the -t flag with v++.
The host program can be written using either the native XRT API or OpenCL API calls, and it is compiled using the GNU C++ compiler (g++). Each source file is compiled to an object file (.o) and linked with the Xilinx runtime (XRT) shared library to create the executable which runs on the host CPU.
See the Vitis Documentation for more information.
Important: Set up the command shell or window as described in Setting Up the Vitis Environment prior to running the tools.
Each source file of the host application is compiled into an object file (.o) using the g++ compiler.
The generated object files (.o) are linked with the Xilinx Runtime (XRT) shared library to create the executable host program. Linking is performed using the -l option.
Compiling and linking for x86 follows the standard g++ flow. The only requirement is to include the XRT header files and link the XRT shared libraries.
When compiling the source code, the following g++ options are required:
-I$XILINX_XRT/include/: XRT include directory.
-I$XILINX_VIVADO/include: Vivado tools include directory.
-std=c++11: Define the C++ language standard.
When linking the executable, the following g++ options are required:
-L$XILINX_XRT/lib/: Look in XRT library.
-lOpenCL: Search the named library during linking.
-lpthread: Search the named library during linking.
-lrt: Search the named library during linking.
-lstdc++: Search the named library during linking.
Note: In the Vitis Examples you may see the addition of xcl2.cpp source file, and the -I../libs/xcl2 include statement. These additions to the host program and g++ command provide access to helper utilities used by the example code, but are generally not required for your own code.
The kernel code is written in C, C++, OpenCL™ C, or RTL, and is built by compiling the kernel code into a Xilinx® object (XO) file, and linking the XO files into a device binary (XCLBIN) file, as shown in the following figure.
The process, as outlined above, has two steps:
Build the Xilinx object files from the kernel source code.
For C, C++, or OpenCL kernels, the v++ -c command compiles the source code into Xilinx object (XO) files. Multiple kernels are compiled into separate XO files.
For RTL kernels, the package_xo command produces the XO file to be used for linking. Refer to RTL Kernels for more information.
You can also create kernel object (XO) files working directly in the Vitis™ HLS tool. Refer to Compiling Kernels with the Vitis HLS for more information.
After compilation, the v++ -l command links one or multiple kernel objects (XO), together with the hardware platform XSA file, to produce the device binary XCLBIN file.
TIP: The v++ command can be used from the command line, in scripts, or a build system like make, and can also be used through the Vitis IDE as discussed in Using the Vitis IDE.
TIP: The output directories of v++ can be changed. See Vitis Documentation. This is particularly helpful when you want to build multiple versions of the kernel in the same file structure. The makefile example shows an example of how to do this.
See the Vitis Documentation for more information.
Important: Set up the command shell or window as described in Setting Up the Vitis Environment prior to running the tools.
The first stage in building the xclbin file is to compile the kernel code using the Xilinx Vitis compiler. There are multiple v++ options that need to be used to correctly compile your kernel. The following is an example command line to compile the vadd kernel:
The various arguments used are described below. Note that some of the arguments are required.
-t <arg>: Specifies the build target, as discussed in Build Targets. Software emulation (sw_emu) is used as an example. Optional. The default is hw.
--platform <arg>: Specifies the accelerator platform for the build. This is required because runtime features, and the target platform are linked as part of the FPGA binary. To compile a kernel for an embedded processor application, specify an embedded processor platform: --platform $PLATFORM_REPO_PATHS/zcu102_base/zcu102_base.xpfm.
-c: Compile the kernel. Required. The kernel must be compiled (-c) and linked (-l) in two separate steps.
-k <arg>: Name of the kernel associated with the source files.
-o'<output>.xo': Specify the shared object file output by the compiler. Optional.
<source_file>: Specify source files for the kernel. Multiple source files can be specified. Required.
The above list is a sample of the extensive options available. Refer to Vitis Compiler Command for details of the various command-line options. Refer to Output Directories of the v++ Command to get an understanding of the location of various output files.
Important: Set up the command shell or window as described in Setting Up the Vitis Environment prior to running the tools.
The kernel compilation process results in a Xilinx object (XO) file whether the kernel is written in C/C++, OpenCL C, or RTL. During the linking stage, XO files from different kernels are linked with the platform to create the FPGA binary container file (.xclbin) used by the host program.
Similar to compiling, linking requires several options. The following is an example command line to link the vadd kernel binary:
This command contains the following arguments:
-t <arg>: Specifies the build target. Software emulation (sw_emu) is used as an example. When linking, you must use the same -t and --platform arguments as specified when the input (XO) file was compiled.
--platform <arg>: Specifies the platform to link the kernels with. To link the kernels for an embedded processor application, you simply specify an embedded processor platform: --platform $PLATFORM_REPO_PATHS/zcu102_base/zcu102_base.xpfm
--link: Link the kernels and platform into an FPGA binary file (xclbin).
<input>.xo: Input object file. Multiple object files can be specified to build into the .xclbin.
-o'<output>.xclbin': Specify the output file name. The output file in the link stage will be an .xclbin file. The default output name is a.xclbin
--config ./connectivity.cfg: Specify a configuration file that is used to provide v++ command options for a variety of uses. Refer to Vitis Compiler Command for more information on the --config option.
TIP: Refer to Output Directories of the v++ Command to get an understanding of the location of various output files.
Beyond simply linking the Xilinx object (XO) files, the linking process is also where important architectural details are determined. In particular, this is where the number of compute unit (CUs) to instantiate into hardware is specified, connections from kernel ports to global memory are assigned, and CUs are assigned to SLRs. The following sections discuss some of these build options.
The Vitis™ analyzer is a graphical utility that allows you to view and analyze the reports generated while building and running the application. It is intended to let you review reports generated by both the Vitis compiler when the application is built, and the Xilinx® Runtime (XRT) library when the application is run. The Vitis analyzer can be used to view reports from both the v++ command line flow, and the Vitis integrated design environment (IDE). You will launch the tool using the vitis_analyzer command (see Setting Up the Vitis Environment).
See the Vitis Documentation for more information.
TLDR: Create an emconfig.json file using emconfigutil and set XCL_EMULATION_MODE to sw_emu or hw_emu before executing the host program. The device binary also has to be built for the corresponding target.
See the Vitis Documentation for more information.
Important: Set up the command shell or window as described in Setting Up the Vitis Environment prior to running the tools.
Set the desired runtime settings in the xrt.ini file. This step is optional.\
As described in xrt.ini File, the file specifies various parameters to control debugging, profiling, and message logging in XRT when running the host application and kernel execution. This enables the runtime to capture debugging and profile data as the application is running. The Emulation group in the xrt.ini provides features that affect your emulation run.
TIP: Be sure to use the v++ -g option when compiling your kernel code for emulation mode.\
Create an emconfig.json file from the target platform as described in emconfigutil Utility. This is required for running hardware or software emulation.\
The emulation configuration file, emconfig.json, is generated from the specified platform using the emconfigutil command, and provides information used by the XRT library during emulation. The following example creates the emconfig.json file for the specified target platform:
In emulation mode, the runtime looks for the emconfig.json file in the same directory as the host executable, and reads in the target configuration for the emulation runs. TIP: It is mandatory to have an up-to-date JSON file for running emulation on your target platform.\
Set the XCL_EMULATION_MODE environment variable to sw_emu (software emulation) or hw_emu (hardware emulation) as appropriate. This changes the application execution to emulation mode.\
Use the following syntax to set the environment variable for C shell (csh):
Bash shell:
IMPORTANT: The emulation targets will not run if the XCL_EMULATION_MODE environment variable is not properly set.\
Run the application.\
With the runtime initialization file (xrt.ini), emulation configuration file (emconfig.json), and the XCL_EMULATION_MODE environment set, run the host executable with the desired command line argument.
IMPORTANT: The INI and JSON files must be in the same directory as the executable.\
For example:
TIP: This command line assumes that the host program is written to take the name of the xclbin file as an argument, as most Vitis examples and tutorials do. However, your application may have the name of the xclbin file hard-coded into the host program, or may require a different approach to running the application.
TLDR: Make sure XCL_EMULATION_MODE is unset. Use a node with the FPGA hardware attached.
See the Vitis Documentation for more information.
TIP: To use the accelerator card, you must have it installed as described in Getting Started with Alveo Data Center Accelerator Cards (UG1301).
Edit the xrt.ini file as described in xrt.ini File.\
This is optional, but recommended when running on hardware for evaluation purposes. You can configure XRT with the xrt.ini file to capture debugging and profile data as the application is running. To capture event trace data when running the hardware, refer to Enabling Profiling in Your Application. To debug the running hardware, refer to Debugging During Hardware Execution.
TIP: Ensure to use the v++ -g option when compiling your kernel code for debugging.\
Unset the XCL_EMULATION_MODE environment variable.
IMPORTANT: The hardware build will not run if the XCL_EMULATION_MODE environment variable is set to an emulation target.\
For embedded platforms, boot the SD card. TIP: This step is only required for platforms using Xilinx embedded devices such as Versal ACAP or Zynq UltraScale+ MPSoC.\
For an embedded processor platform, copy the contents of the ./sd_card folder produced by the v++ --package command to an SD card as the boot device for your system. Boot your system from the SD card.\
Run your application.\
The specific command line to run the application will depend on your host code. A common implementation used in Xilinx tutorials and examples is as follows:
TIP: This command line assumes that the host program is written to take the name of the xclbin file as an argument, as most Vitis examples and tutorials do. However, your application can have the name of the xclbin file hard-coded into the host program, or can require a different approach to running the application.
A simple example Vitis project is available at https://code.ornl.gov/7ry/add_test. This project can be used to test the Vitis compile chain and Vitis HLS
The makefile used by this project is an example of how to create a makefile to build an FPGA accelerated application.
Vitis and Vivado will use 8 threads by default on Linux. Many of the Vivado tools can only utilize 8 threads for a given task. See the Multithreading in the Vivado Tools section from Vivado Design Suite User Guide Implementation (UG904). I found from experimenting that the block level synthesis task can leverage more than 8 threads, but will not do so unless you set the vivado.synth.jobs and vivado.impl.jobs flags.
Here is an example snippet from the Xilinx Buttom-Up RTL Tutorial which shows one way to query and set the number of CPUs to use.
