Published: June 01, 2019
Updated: June 16, 2019
Tags: SR-IOV, Networking
5 min read

SR-IOV NIC Partitioning

The content of the post is still work in progress

Dell PowerEdge R730 with Intel X710 10G using RHEL7 & KVM is showcased in this post

SR-IOV

Single-root input/output virtualization [1], refers to the capability of splitting a single PCI hardware resource into multiple virtual PCI resources. Devicese like nVME, Network Interfaces and GPUs may use this capability in order to use the physical hardware according to the required use case.

A physical PCI device is referred to as a PF(Physical Function) and virtual PCI device are referred to as a VF(Virtual Function).

A great overview of SR-IOV from can be found on Scott Lowe's blog post [2].

SR-IOV Network Interfaces

General

In networking, a capable SR-IOV NIC is used in order to split a single physical NIC into multiple vNICs which can be used on the bare metal host or as part of virtual guest instances.

Possible Use Cases

It's important to remember that SR-IOV VFs reside on a physical NIC which may be a single point of failure if your network topology is not designed properly

Due to SR-IOV robustness, there are many network topologies that can be achieved with minimal amount of NICs, which will require less cabling and maintenance.

With a single 10GB/40GB/100GB SR-IOV NIC, we could build a setup which offers both redundancy and performance while taking minimal physical space.

Baremetal host with single NIC two ports split to multiple vNICs connected to two switches

In the diagram above, we have a bare metal host containing a NIC with 2 ports connected to different switches.

Each port, which is represented as an NIC inside of operating system, is split into several vNICs(VFs) which also are represented as an NIC.

VFs can leveraged to configure networking.
For example, in a cloud environment, where there are multiple networks which represent different components of the cloud can reside on VFs instead of separate physical NICs. If some networks require more bandwitdh than the rest, a QoS setting can be applied.

Enabling SR-IOV

Bios Configuration

Bios settings may differ, depends on Bios and NIC vendors, refer to the vendor's documentation

On supported NICs, a Bios option must be set:
Intel X710 NIC on Dell PowerEdge R730

Operating System

Operating System settings may differ based on NIC vendor and operating system distribution, refer to the vendor's/distribution's documentation

Once SR-IOV has been enabled in Bios, verify this:

  • Kernel drivers are loaded for your NIC:

    lsmod | grep i40e
    i40evf                103843  0
    i40e                  354807  0
    

    If drivers are not loaded:

    modprobe i40e i40evf
    
  • Discover the PCI slot of NIC

    lshw -c network -businfo 
    Bus info          Device      Class          Description
    ========================================================
    ...output omitted...
    pci@0000:82:00.3  p4p4        network        Ethernet Controller X710 for 10GbE SFP+
    
  • View total amount of SR-IOV VFs exposed by PF:

    cat /sys/class/net/p4p4/device/sriov_totalvfs
    32
    
  • View the amount of SR-IOV VFs active:

    cat /sys/class/net/p4p4/device/sriov_numvfs
    0
    
  • Set X amount of SR-IOV VFs (VFs start from 0, must be sriov_totalvfs-1):

    echo "4" > /sys/class/net/p4p4/device/sriov_numvfs
    

    In order to reset the SR-IOV VFs:

    echo "0" > /sys/class/net/p4p4/device/sriov_numvfs
    
  • Verify that SR-IOV VFs were created:

    cat /sys/class/net/p4p4/device/sriov_numvfs
    4
    
    ip link show p4p4
    13: p4p4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
        link/ether 3c:fd:fe:31:1f:c6 brd ff:ff:ff:ff:ff:ff
        vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
        vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
        vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
        vf 3 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
    
    ll /sys/class/net/p4p4*
    lrwxrwxrwx. 1 root root 0 May 24 22:30 /sys/class/net/p4p4 -> ../../devices/pci0000:80/0000:80:02.0/0000:82:00.3/net/p4p4
    lrwxrwxrwx. 1 root root 0 May 27 18:58 /sys/class/net/p4p4_0 -> ../../devices/pci0000:80/0000:80:02.0/0000:82:0e.0/net/p4p4_0
    lrwxrwxrwx. 1 root root 0 May 27 18:58 /sys/class/net/p4p4_1 -> ../../devices/pci0000:80/0000:80:02.0/0000:82:0e.1/net/p4p4_1
    lrwxrwxrwx. 1 root root 0 May 27 18:58 /sys/class/net/p4p4_2 -> ../../devices/pci0000:80/0000:80:02.0/0000:82:0e.2/net/p4p4_2
    lrwxrwxrwx. 1 root root 0 May 27 18:58 /sys/class/net/p4p4_3 -> ../../devices/pci0000:80/0000:80:02.0/0000:82:0e.3/net/p4p4_3
    

Leveraging SR-IOV VFs

Once SR-IOV has been configured, we can use them as NICs or pass them to VMs for increased performance.

VFs as NICs

Since VFs are represented as NICs, we can use native linux tools such as ip, ifconfig, nmcli, network-scripts and others in other to configure network.

Make sure your SR-IOV PF device settings are empty before using VFs:

ifconfig p4p4
p4p4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::3efd:feff:fe33:a8a6  prefixlen 64  scopeid 0x20<link>
        ether 3c:fd:fe:33:a8:a6  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Now you can configure VF, example of network script config file:

cat /etc/sysconfig/network-scripts/ifcfg-p4p4_1
DEVICE=p4p4_1
ONBOOT=yes
HOTPLUG=no
NM_CONTROLLED=no
PEERDNS=no
BOOTPROTO=static
IPADDR=10.20.151.125
NETMASK=255.255.255.0

Start NIC and verify that settings were applied:

if up p4p4_1
ifconfig p4p4_1
p4p4_1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.20.151.125  netmask 255.255.255.0  broadcast 10.20.151.255
        inet6 fe80::c8d9:10ff:fe01:8488  prefixlen 64  scopeid 0x20<link>
        ether ca:d9:10:01:84:88  txqueuelen 1000  (Ethernet)
        RX packets 2430103  bytes 320710952 (305.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2369103  bytes 433952302 (413.8 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

VFs for VMs

PCI Passthrough and more in-depth topics are out of scope of this blog post

Requires AMD-Vi or Intel VT-d to be enabled hypervisor host

Refer to your hypervisor's documentation regarding SR-IOV

On supported hypervisors, SR-IOV allows VMs to access the PCI directly which increases performance compared to using generic vNICs provided by your hypervisor.

Refer to RHEL's guide [3] to boot up an instance with SR-IOV VFs using KVM.


  1. SR-IOV Intel network spec ↩︎

  2. Scott's Weblog - What is SR-IOV ↩︎

  3. Red Hat Enterprise Linux 7 - Virtualization Deployment and Administration Guide - PCI Device Assignment with SR-IOV Devices ↩︎