• Elastic Cloud Server

ecs
  1. Help Center
  2. Elastic Cloud Server
  3. User Guide
  4. NICs
  5. Enabling NIC Multi-Queue

Enabling NIC Multi-Queue

Single-core CPU performance cannot meet the requirement of processing NIC interruptions incurred with the increase of network I/O bandwidth. NIC multi-queue enables multiple CPUs to process ECS NIC interruptions, thereby improving network PPS and I/O performance.

Support of NIC Multi-Queue

XEN ECSs do not support NIC multi-queue.

KVM ECSs created using the images listed in Table 1 support NIC multi-queue.

Table 1 Support of NIC multi-queue for KVM ECSs

OS

Image

Status

Windows

Windows Server 2008 WEB R2 64bit

Supported using private images

Windows Server 2008 R2 Standard/Datacenter/Enterprise 64bit

Supported using private images

Windows Server 2012 R2 Standard/Datacenter 64bit

Supported using private images

Windows Server 2016 Standard/Datacenter 64bit

Supported using private images

Linux

Ubuntu 14.04 server 32bit

Supported

Ubuntu 14.04/16.04 server 64bit

Supported

OpenSUSE 42.2 64bit

Supported

SUSE Enterprise 12 SP1/SP2 64bit

Supported

CentOS 6.8/6.9/7.0/7.1/7.2/7.3/7.4 64bit

Supported

Red Hat 7.2/7.3/7.4 64bit

Supported

Debian 8.0.0/9.0.0 64bit

Supported

Fedora 24 64bit

Supported

Fedora 25 64bit

Supported

EulerOS 2.2 64bit

Supported

NOTE:
  • To obtain the virtualization type and number of NIC queues of an ECS, see section ECS Types and Specifications.
  • Windows ECSs use private images to support NIC multi-queue. For details, see section "How Do I Set NIC Multi-queue Feature of an Image?" in Image Management Service User Guide.

Procedure

KVM Windows ECSs use private images to support NIC multi-queue. For details, see section "How Do I Set NIC Multi-queue Feature of an Image?" in Image Management Service User Guide.

This section uses a Linux ECS running CentOS 7.4 as an example to describe how to enable NIC multi-queue.

  1. Enable NIC multi-queue.

    1. Log in to the ECS.
    2. Run the following command to obtain the number of queues supported by the NIC and the number of queues with NIC multi-queue enabled:

      ethtool -l NIC

    3. Run the following command to configure the number of queues used by the NIC:

      ethtool -L NIC combined Number of queues

    An example is provided as follows:

    [root@localhost ~]# ethtool -l eth0  #View the number of queues used by NIC eth0.
    Channel parameters for eth0:
    Pre-set maximums:
    RX:               0
    TX:               0
    Other:                  0
    Combined: 4  #Indicates that a maximum of four queues can be enabled for the NIC.
    Current hardware settings:
    RX:               0
    TX:               0
    Other:                  0
    Combined: 1 #Indicates that one queue has been enabled.
    
    [root@localhost ~]# ethtool -L eth0 combined 4 #Enable four queues on NIC eth0.

  2. (Optional) Enable irqbalance so that the system automatically allocates NIC interrupts on multiple vCPUs.

    1. Run the following command to enable irqbalance:

      service irqbalance start

    2. Run the following command to view the irqbalan status:

      service irqbalance status

      If the Active value in the command output contains active(running), irqbalance has been enabled.

      Figure 1 Enabled irqbalance

  3. (Optional) Enable interrupt binding.

    Enabling irqbalance allows the system to automatically allocate NIC interrupts, improving network performance. If the improved network performance still fails to meet your requirements, manually configure interrupt affinity on the ECS.

    To do so, perform the following operations:

    Configure the following script so that one ECS vCPU serves the interrupt requests initialized by one queue. One queue corresponds to one interrupt, and one interrupt binds to one vCPU.

    #!/bin/bash
    service irqbalance stop
    
    eth_dirs=$(ls -d /sys/class/net/eth*)
    if [ $? -ne 0 ];then
        echo "Failed to find eth*  , sleep 30" >> $ecs_network_log
        sleep 30
        eth_dirs=$(ls -d /sys/class/net/eth*)
    fi
    
    for eth in $eth_dirs
    do
        cur_eth=$(basename $eth)
        cpu_count=`cat /proc/cpuinfo| grep "processor"| wc -l`
        virtio_name=$(ls -l /sys/class/net/"$cur_eth"/device/driver/ | grep pci |awk {'print $9'})
    
        affinity_cpu=0
        virtio_input="$virtio_name""-input"
        irqs_in=$(grep "$virtio_input" /proc/interrupts | awk -F ":" '{print $1}')
        for irq in ${irqs_in[*]}
        do
            echo $((affinity_cpu%cpu_count)) > /proc/irq/"$irq"/smp_affinity_list
            affinity_cpu=$[affinity_cpu+2]
        done
    
        affinity_cpu=1
        virtio_output="$virtio_name""-output"
        irqs_out=$(grep "$virtio_output" /proc/interrupts | awk -F ":" '{print $1}')
        for irq in ${irqs_out[*]}
        do
            echo $((affinity_cpu%cpu_count)) > /proc/irq/"$irq"/smp_affinity_list
            affinity_cpu=$[affinity_cpu+2]
        done
    done

  4. (Optional) Enable XPS and RPS.

    XPS allows the system with NIC multi-queue enabled to select a queue by vCPU when sending a data packet.

    #!/bin/bash
    # enable XPS feature
    cpu_count=$(grep -c processor /proc/cpuinfo)
    dec2hex(){
      echo $(printf "%x" $1)
    }
    eth_dirs=$(ls -d /sys/class/net/eth*)
    if [ $? -ne 0 ];then
        echo "Failed to find eth* , sleep 30" >> $ecs_network_log
        sleep 30
        eth_dirs=$(ls -d /sys/class/net/eth*)
    fi
    for eth in $eth_dirs
    do
        cpu_id=1
        cur_eth=$(basename $eth)
        cur_q_num=$(ethtool -l $cur_eth | grep -iA5 current | grep -i combined | awk {'print $2'})
        for((i=0;i<cur_q_num;i++))
        do
            if [ $i -eq $ cpu_count ];then
                cpu_id=1
            fi
            xps_file="/sys/class/net/${cur_eth}/queues/tx-$i/xps_cpus"
            rps_file="/sys/class/net/${cur_eth}/queues/rx-$i/rps_cpus"
            cpuset=$(dec2hex "$cpu_id")
            echo $cpuset > $xps_file
            echo $cpuset > $rps_file
            let cpu_id=cpu_id*2
        done
    done