# ascend-for-volcano **Repository Path**: TDlining/ascend-for-volcano ## Basic Information - **Project Name**: ascend-for-volcano - **Description**: 基于开源Volcano，增强华为NPU亲和性调度功能。 - **Primary Language**: Go - **License**: Apache-2.0 - **Default Branch**: ascend-volcano-plugin - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 0 - **Forks**: 46 - **Created**: 2023-05-09 - **Last Updated**: 2023-05-09 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README # NPU Affinity Scheduling Algorithm Design and Development Guide.en - [Introduction to Ascend-volcano-plugin](#introduction-to-ascend-volcano-plugin) - [Affinity Policies](#affinity-policies) - [Scheduling Algorithm Design](#scheduling-algorithm-design) - [Scheduling Algorithm Implementation](#scheduling-algorithm-implementation) - [Compilation Description](#compilation-description) - [Version Updates](#version-updates)

Introduction to Ascend-volcano-plugin

Based on the open source Volcano scheduling plugin mechanism, Ascend-volcano-plugin adds features such as affinity scheduling and virtual device re-scheduling to maximize the computing performance of Ascend AI Processors.

Affinity Policies

## Ascend 910 AI Processor Affinity Principles The Ascend 910 AI Processor is a high-performance AI processor developed by Huawei. The internal processors are connected in HCCS mode. Each physical device has eight processors and two HCCS rings. Each HCCS ring has four processors. The processors in the same HCCS can perform gradient switching, and processors in different HCCS cannot communicate with each other. That is, the Ascend 910 AI Processors allocated to the same pod $if the number of processors is less than or equal to 4$ must be in the same HCCS ring. Otherwise, the job fails to be executed. [Figure 1](#fig997414281914) shows the Ascend 910 AI Processor interconnection topology. **Figure 1** Ascend 910 AI Processor interconnection topology ![](figures/ascend-910-ai-processor-interconnection-topology.png "ascend-910-ai-processor-interconnection-topology") >![](figures/icon-note.gif) **NOTE:** >A0 to A7 in the figure are Ascend 910 AI Processors. ## Affinity Policies [Table 1](#table644103781119) lists the Ascend 910 AI Processor affinity policies formulated based on the characteristics and resource utilization rules of the Ascend 910 AI Processor. **Table 1** Ascend 910 AI Processor affinity policies

Priority	Policy Name	Details
1	HCCS affinity scheduling	If the number of applied Ascend 910 AI Processors is 1, the optimal choice is the remaining one Ascend 910 AI Processor in the same HCCS, and the less optimal choices are three Ascend 910 AI Processors, two Ascend 910 AI Processors, and then four Ascend 910 AI Processors. If the number of applied Ascend 910 AI Processors is 2, the optimal choice is the remaining two Ascend 910 AI Processors in the same HCCS, and the less optimal choices are four Ascend 910 AI Processors and then three Ascend 910 AI Processors. If the number of applied Ascend 910 AI Processors is 4, the remaining four Ascend 910 AI Processors in the same HCCS must be selected. If the number of applied Ascend 910 AI Processors is 8, all 8 Ascend 910 AI Processors of the node are selected.
2	Full priority scheduling	AI servers that have been allocated with Ascend 910 AI Processors are preferentially scheduled to reduce fragments. If the number of applied Ascend 910 AI Processors is 1, the optimal choices are nodes whose capacity is 8 and the number of remaining available processors in the HCCS is 1, and the less optimal choices are nodes whose number of remaining available processors is 3, 2, or 4. If the number of applied Ascend 910 AI Processors is 2, the optimal choices are nodes whose capacity is 8 and the number of remaining available processors in the HCCS is 2, and the less optimal choices are nodes whose number of remaining available processors is 4 or 3. If the number of applied Ascend 910 AI Processors is 4, the optimal choices are nodes whose capacity is 8 and the number of remaining available processors in the HCCS is 4. If the number of applied Ascend 910 AI Processors is 8, select a node whose capacity is 8 and that has no processor used.
3	Even number priority scheduling	The HCCS that meets policies 1 to 3 is preferentially selected, and then the HCCS whose number of remaining processors is an even number is selected.
--	Multi-node support	Only the 8 x N allocation mode is supported for a training job.

## Resource Application Constraints Based on the service model design, the training job requirements are as follows: 1. If the number of Ascend 910 AI Processors applied for by a training job is less than or equal to 4, the required Ascend 910 AI Processors need to be scheduled in the same HCCS. 2. If the number of Ascend 910 AI Processors applied for by a training job is 8, the Ascend 910 AI Processors of the node need to be allocated to the job. 3. If the number of NPUs to be applied for is greater than 8, the number of NPUs to be applied for must be 8 x N $N ≥ 1$. 4. If the number of Ascend 910 AI Processors applied for by a training job is less than or equal to 8, only one pod can be applied for. If the number is greater than 8, each pod has eight Ascend 910 AI Processors. 5. Only one vNPU can be applied for by a training job. 6. Resource application must comply with other constraints of the open source Volcano.

Scheduling Algorithm Design

## Scenarios Based on the affinity policies and service model design, [Table 1](#table34241172175) lists the scenarios. >![](figures/icon-note.gif) **NOTE:** >- The four groups from A to D indicate the four HCCS scenarios that meet the processor selection requirements. The priorities are in descending order. That is, B, C, or D is selected only when A does not meet the requirements. >- Node status in the group when the HCCS is met. The left part of \~ indicates the HCCS that meets the requirements, and the right part indicates the number of remaining processors in another HCCS. For example, for group A that applies for one processor, the other HCCS may have 0, 1, 2, 3, or 4 remaining processors. The priority of the node decreases accordingly. >- To apply for eight or more processors, refer to the scenarios when four or fewer processors are applied for. Additionally, all the processors are in group A and need to be used. **Table 1** Affinity policy scenario list

Scenario ID	Number of Processors Applied for the Job	A (Number of Remaining Processors on a Node)	B	C	D	Note
1	1	1~[0, 1, 2, 3, 4]	3~[0, 2, 3, 4]	2~[0, 2, 4]	4~[0, 4]	Set a node whose capacity to 7. A bad processor is regarded as used. Repeat from A to D.
2	2	2~[0, 1, 2, 3, 4]	4~[0, 1, 3, 4]	3~[0, 1]	-
3	4	4~[0, 1, 2, 3, 4]	-	-	-
4	8	8	-	-	-	-
5	8 x N	0 (All the eight processors are used.)	-	-	-	-

## Algorithm Design **Figure 1** Affinity algorithm design process ![](figures/affinity-algorithm-design-process.png "affinity-algorithm-design-process") Key processes in the figure are described as follows: 1. Obtains the number of Ascend 910 AI Processors applied for processing tasks. 2. Based on the number of applied Ascend 910 AI Processors, select the optimal node according to [Resource Application Constraints](#affinity-policies). 3. Select Ascend 910 AI Processors that meet the requirements from the selected node. 4. Save the selection result. 5. Weight the selected node. >![](figures/icon-note.gif) **NOTE:** >Steps [1](#li2081354582012) to [5](#li205713218818) are implemented in the registration function **batchNodeOrderFn** provided by Volcano. 6. Allocate and manage resources of the selected node. >![](figures/icon-note.gif) **NOTE:** >This step is implemented in the **AddEventHandler** function of Volcano. This function contains the **allocate** function for pre-allocating node resources. 7. After the preceding operations are complete, the Volcano framework submits the allocation result to Kubelet of Kubernetes for confirmation, and the allocation completes. ## Multi-Node Processing Principles This feature is designed based on the Volcano cluster. You only need to use the framework to weight each node and select the optimal node. Two-phase selection: Four preferred arrays are selected first, and then the final nodes are selected within the arrays. In this way, multiple nodes are considered as a whole. ## Concurrent Processing Principles Volcano does not provide the function callback method in the add method in the allocation phase. Therefore, processors are selected in the node filtering phase. In this way, when jobs are concurrently executed, the chips may be repeatedly allocated. Repeated allocation may occur in the following two scenarios: - Between jobs in a session: When multiple jobs need to be assigned at the same time and multiple jobs can be assigned to one node, the native Volcano only allocates the quantity but does not allocate processor IDs. As a result, processors of the specified quantity are allocated, but a processor may be allocated for multiple times. This program uses the **AddEventHandler** function provided by the Volcano framework to resolve the problem. The **allocate** method of the function is used to manage the allocation of node processors. In this way, repeated allocation is avoided. - Between different sessions: In this session, the processors are allocated in the weighting phase. If the resource is waiting to be released, that is, the resource cannot be allocated temporarily, the allocation fails. However, Volcano is not aware of the failure in this session. In the next session, the processor becomes available and will be allocated to another task. In this case, two tasks are allocated to the same processor, and one of the tasks fails. One of the solutions to this problem is to determine whether the resource is to be released when processor allocation is performed in the weighting phase. If yes, no allocation is performed in this session.

Scheduling Algorithm Implementation

## Process Design **Figure 1** Affinity program process $Volcano part$ ![](figures/affinity-program-process-(volcano-part).png "affinity-program-process-(volcano-part)") Affinity scheduling of Huawei Ascend AI Processors is based on the plugin mechanism of the open source Volcano, simplifying plugin development. Several plugin functions are implemented in the volcano-schedule framework. When Volcano runs each session, the implemented function runs according to the compiled rule, implementing affinity scheduling of processors. The affinity scheduling plugin implements the following functions: - validJobFn: This function is used to intercept jobs that apply for NPU resources. However, the number of applications must meet the affinity policies. For details about the storage space requirements of data stores of different types, see [Affinity Policies](#affinity-policies). - AddPredicateFn: This function is used to filter out nodes that do not meet affinity requirements. For example, if the task applies for two processors, but each of the two HCCSs on the node has one processor, the node meets the quantity requirement but does not meet the affinity requirement. Therefore, the node needs to be excluded. - AddBatchNodeOrderFn: This function is used to select nodes whose processors meet the affinity requirement, and put the result into a pod. - AddEventHandler: This function is used to manage the available Ascend 910 AI Processors of a node in a unified manner. This prevents incorrect allocation in the case of concurrent operations.

Compilation Description

## Preparations - Ensure that the PC can access the Internet and Git and Docker have been installed. For details, see [Installing Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) and [Install Docker Engine on Ubuntu](https://docs.docker.com/engine/install/ubuntu/). - The Go language environment $version 1.13 or later$ has been installed. You are advised to use the latest bugfix version. For details, visit [https://golang.org/](https://golang.org/). - The musl of version 1.2.0 or later has been installed. For details, see [http://musl.libc.org/](http://musl.libc.org/). - Configure the Go proxy address based on the network environment. In China, you can use **Goproxy China**, for example: ``` go env -w GOPROXY=https://goproxy.cn,direct ``` ## Compiling and Installing Volcano 1. Save the official open source code of Volcano v1.3.0 to **$GOPATH/src/volcano.sh/volcano**. 2. Copy the code directory **ascend-volcano-plugin** to the plugin path **$GOPATH/src/volcano.sh/volcano/pkg/scheduler/plugins/** of official open source code of Volcano. 3. Run the following commands to compile the Volcano binary and .so files: **cd $GOPATH/src/volcano.sh/volcano/pkg/scheduler/plugins/ascend-volcano-plugin/build** **chmod +x build.sh** **./build.sh** The compiled binary files and dynamic link library $DLL$ files are stored in the **ascend-volcano-plugin/output** directory. See [Table 1](#table922124765019). **Table 1** List of files in the **output** directory

File Name	Description
volcano-npu-{version}.so	DDL of the Huawei NPU scheduling plugin Volcano
Dockerfile-scheduler	Image building text file for Volcano scheduler
Dockerfile-controller	Image building text file for Volcano controller
volcano-{version}.yaml	Volcano startup configuration file
vc-scheduler	Volcano scheduler binary file
vc-controller-manager	Volcano controller binary file

>![](figures/icon-note.gif) **NOTE:** >_\{version\}_ indicates the version number. ## Prerequisites Perform operations described in all sections except "Preparing Software Packages" in section "Preparing for Installation" in the [_MindX DL User Guide_](https://www.hiascend.com/software/mindx-dl). For details, see "Installation and Deployment \> Preparations Before Installation" in the [_MindX DL User Guide_](https://www.hiascend.com/software/mindx-dl). ## Installing Volcano For details, see "Installation and Deployment" \> "Installing MindX DL" \> "Installing Volcano" in the [_MindX DL User Guide_](https://www.hiascend.com/software/mindx-dl).

Version Updates

Version	Date	Description
v2.0.2	2021-07-15	Added the support for virtual device scheduling. Implemented the NPU scheduling plugin, which corresponds to Volcano v1.3.0.
v1.0.1-r50	2021-04-15	Fixed the issue that the process breaks down after an open source community task enters the pending state. Optimized the code structure.
v1.0.1-r40	2021-01-11	This issue is the first official release.