1 Introduction

This document briefly explains the steps to run ADS CFD Code Leo application on a Virtual Machine deployed in Azure Cloud Platform and presents the performance results.

This recipe document covers the following topics: -

• Deploy & connect a required virtual machine on Azure platform.
• Install required NVIDIA drivers in virtual machine.
• Install Application on virtual machine.
• Install license manager for current application
• Performance results of current application on azure virtual machine
• Azure cost
• Summary

2 Deploy Virtual Machine on Azure Cloud Platform

2.1 Azure Cloud Architecture for Application

The below Architecture explains the virtual machine running on an Azure Platform

2.2 Azure Virtual Machine (VM)

A VM is a virtualized instance of a computer that can perform almost all the same functions as a computer, including running applications and operating systems.

An Azure VM gives the flexibility of virtualization without having to buy and maintain the physical hardware that runs it. However, the user still needs to maintain the VM by performing tasks, such as configuring, patching, and installing the software that runs on it.

Things to be considered before deploying a VM,

• The names of the application resources
• The location where the resources are stored
• The size of the VM
• The operating system that the VM runs
• The configuration of the VM
• The related resources that the VM needs

There are different sizes and options available for the Azure virtual machines and user can use to run apps and workloads. Depending on the workload user must choose the appropriate VM size. For complete list, check this https://docs.microsoft.com/en-us/azure/virtual-machines/sizes/.

To test the performance of ADS CFD Code Leo on Azure Platform, NDv4 A100 andNCv3-series virtual machines are deployed.

1. Standard_ND96asr_v4: This VMs is powered by NVIDIA Ampere A100 Tensor Core GPUs and 96 physical 2nd-generation AMD Epyc™ CPU cores (2.44GHz). The “ND96asr_v4” VM has 8 GPUs with 40 GB of memory each and supported by 96 AMD processor cores with a total memory of 921GB. Each GPU features NVLINK 3.0 connectivity for communication within the VM.

2. Standard NC24s_v3: This VM belongs to Ncv3-series. This VMs are powered by NVIDIA Tesla V100 GPUs. These GPUs can provide 1.5x the computational performance of the NCv2-series.

2.3 Create a Virtual Machine on Azure Platform

Sign into Azure

Sign into the Azure portal by using https://portal.azure.com/

Free Trial subscriptionsare not eligible for limit or quota increases.

After the successful sign in or sign up, one has to upgrade the Azure subscription to Pay-As-You-Go to deploy a Virtual machine. For deployment, the user must have regional vCPU quota, which can be obtained by raising a request.

The step-by-step procedure to increase the vCPU quota is given below,

https://docs.microsoft.com/en-us/azure/azure-portal/supportability/per-vm-quota-requests

The step-by-step procedure to deploy Virtual machine is given below,

1. Type virtual machines in the marketplace search.
2. Under Services, select Virtual machines.
3. In the Virtual machines page, select Create Then Virtual machine.
4. In the Basics tab, under Project details, make sure the correct subscription is selected and then choose to Create new resource group. Type Azure-PerformanceTest(user choice) for the name.

5. Under Instance details, type Azure-VM (user choice) for the Virtual machine name and choose West Europe for Region. Choose CentOS, Version 8.1 HPC-Gen2 for the Image and Standard_ND96asr_v4 for the Size (user choice). Leave the other defaults.

Note:

• Region must be decided based on where the Virtual machine is going to be deployed. To avoid the network latency, the region should be near to the location where the VM is to be deployed.
• Image selection is of user choice based on the application user can choose the image (Windows 10, Linux based OS and Windows Server)

6. Under Administrator account, provide a username, such as Azureuser and a password. The password must be at least 12 characters long and meet the defined complexity requirements.

7. Under Inbound port rules, choose Allow selected ports and then select SSH (22) from the drop-down.

8. In Advanced tab, under the Extensions click Select an extension to install, select NVIDIA GPU Driver Extension (user choice)

Note: This Step 8, will installs the older version of Nvidia and Cuda driver hence the user can skip this step and proceed the VM deployment without extension. Follow manual installation as per chapter 2.5.

9. Leave the remaining defaults and then select the Review + create button at the bottom of the page.

After validation runs, select theCreatebutton at the bottom of the page.

10. After deployment is complete, select Go to resource.

2.4 Connect to virtual machine

Create a SSH connection to the virtual machine.

1. On the overview page of a virtual machine, select the Connect button then select SSH and copy the public IP address.

2. Open any SSH client tool of user choice, in this case PUTTY is used.

Enter the IP address and port as 22 and click Open

3. Enter username in Login as, enter the password, and click Enter.
4. Now user connected to the VM in Console

2.5 Install Nvidia Drivers

To take advantage of the GPU capabilities of Azure N-series VMs backed by NVIDIA GPUs, the user must install NVIDIA GPU drivers.

If the machine has a NVLink or NVSwitch connecting the GPUs like in the case of NDv4 virtual machine, one should also install the associated version of the Nvidia fabric manager.

There are two ways to install the drivers,

• TheNVIDIA GPU Driver Extensioninstalls appropriate NVIDIA CUDA or GRID drivers on an N-series VM. Install or manage the extension using the Azure portal (Ref 8)
• Install NVIDIA GPU drivers manually. (For the manual installation follow the procedure given below)

Nvidia Data Center Driver installation guide for CentOS 8

The step-by-step instruction is given below,

1. In some cases, user may need to install some additional dependencies that are required

For installing the NVIDIA drivers.

# sudo dnf install -y tar bzip2 make automake gcc gcc-c++ pciutils elfutils-libelf-devel libglvnd-devel

2. Enable the PowerTools repo and set up the external dependency on EPEL for DKMS.

# sudo dnf config-manager --set-enabled PowerTools

# sudo dnf install -y epel-release

3. Install the CUDA repository public GPG key

# Distribution=rhel8

4. Setup the CUDA network repository

# ARCH=$( /bin/arch)

# sudo dnf config-manager --add-repo http://developer.download.nvidia.com/compute/cuda/repos/$distribution/${ARCH}/cuda-$distribution.repo

5. For CentOS 8, ensure that the system has the correct Linux kernel sources from the CentOS repositories:

# sudo dnf install -y kernel-devel-$(uname -r) kernel-headers-$(uname -r)

6. Note that in some cases, user may have to update the running kernel as a pre-requisite:

# sudo dnf install -y kernel kernel-core kernel-modules

7. Update the repository cache and install the driver using the nvidia-driver meta-package

# sudo dnf clean all

# sudo dnf -y module install nvidia-driver:latest-dkms

8. Install fabric manager to activate the NVswitch (For NDv4 instance)

# sudo dnf module install nvidia-driver:latest-dkms/fm

Note: User ensure that fabric manager is in active state before start the runs/simulation.

9. Enable FM Service to Auto Start at Boot

# sudo systemctl enable nvidia-fabricmanager

Reference: https://docs.nvidia.com/datacenter/tesla/tesla-installation-notes/index.html#centos8/

3 Install ADS CFD Code Leo Application on Virtual Machine

3.1 Download product

Instructions on how to download Code Leo products from ADS CFD Portal:

Before beginning, open ADS CFD Portal in a web browser and log on.

1. Click on Support tab on home page.
2. Click on Download button.
3. Click the latest version Download icon for Linux.
4. That package contains one run file.

3.2 Install Code Leo application on Linux

Installation instructions for Code Leo can be found in below path:

https://new.aerodynamic-solutions.com/support

4 Performance results of Code Leo on Azure virtual machine

4.1 ADS CFD Code Leo Overview

The ADS CFD flow solver, Code Leo, was used to run both the steady-state and unsteady simulations. Code Leo is a density-based, compressible flow code, with explicit timemarching, and convergence acceleration. Code Leo handles both structured multi-block meshes and unstructured meshes with mixed tetrahedrons, pyramids, prisms and hex elements. Code Leo allows use of various turbomachinery rotor/stator interaction models including the sliding mesh, mixing plane and frozen rotor models. Code Leo is HPC-aware and uses MPI+MP for parallel computing.

Code Leo can be executed on both CPUs and GPUs. Originally designed for CPUs, ADS CFD has extended it to take advantage of the advanced GPU architecture following the advent of cost-effective GPUs. Code Leo allows users to switch easily between CPU and GPU solvers.

4.2 ADS CFD Code Leo v8.21.08 Performance Results

Model Details:

For performance evaluation CC3 wheel model is considered. CC3 wheel model has two parts as Impeller and Diffuser.

CC3 wheelImpeller and Diffuser

Full wheel time accurate analysis results are carried out on the following Virtual machine configurations. Azure Virtual machines tested: ND96asr_v4 and NC24s_v3.

The simulations are carried out to see the GPU acceleration on the chosen Azure Virtual Machine. ADS CFD provided CPU results which is used as baseline to compare GPU runs on both VM instance.

The elapsed time for CPU simulation is 3600 minutes which was carried out on server with Xeon 23 CPUs, clock speed of 2.4 GHz. Refer chapter 2.2 for the VM specifications.

4.2.1 Performance Results of NDv4 A100

Elapsed time - (“Minutes”)

Simulation Speed -up

Runs performed on Xeon 23 CPU with 2.4 GHz and A100 which has 8 GPUs, each GPU has 40 GB memory.

4.2.2 Performance Results of NCv3 V100

Elapsed time - (“Minutes”)

Note: The Model requires high GPU memory and hence the 1 GPU case could not perform the run.

Simulation Speed-up

Runs performed on Xeon 23 CPU with 2.4 GHz and V100 which has 4 GPUs, each GPU has 16 GB memory.

5 Azure Cost

For the below cost reports, the application installation time is not considered and only wall clock time is considered for the cost calculation. The Hourly rates reported are subject to change. For the current rate please refer the link “https://azure.microsoft.com/en-in/pricing/calculator/”

6 Summary

1. ADS CFD Code Leo is successfully tested on NDv4 and NCv3 Virtual Machines on Azure Cloud Platform.
2. For NDv4 A100 Virtual Machine, demonstrated good GPU acceleration. Every incremental GPU shows good speed up in all 8 GPUs and the peak performance of 80x is achieved with 8 GPUs.
3. For NCv3 V100 Virtual Machine, demonstrated good GPU acceleration. Every incremental GPU shows good speed up in all 4 GPUs and the peak performance of 32x is achieved with 4 GPUs. For the complex problems the 1 GPU memory of 16GB may not be sufficient hence 2 GPUs are recommended for this instance type.
4. GPU technology in ADS CFD Code Leo on Azure platform has brought unprecedented processing power.