linux - 在虚拟化环境中使用 DPDK 内核 NIC 接口(interface)

Question

我要开发一个DPDK笔记本电脑上的 Linux 应用程序，但 DPDK 不支持笔记本电脑的硬件。幸运的是，DPDK supports半虚拟化设备，包括 QEMU的virtio-net .

所以我正在尝试配置一个 QEMU guest 来运行 Kernel NIC Interface(KNI)在 virtio-net-pci设备。问题是 KNI 示例应用程序不接受 virtio-net-pci司机。

QEMU 命令

exec qemu-system-x86_64 -enable-kvm \
  -cpu host -smp 2 \
  -vga std \
  -mem-prealloc -mem-path /dev/hugepages \
  -drive file=GentooVM.img,if=virtio \
  -netdev user,id=vmnic,hostname=gentoo \
  -device virtio-net-pci,netdev=vmnic \
  -m 1024M \
  -monitor stdio \
  -name "Gentoo VM"

在 guest 中运行 KNI 示例应用程序

sudo ./examples/kni/build/app/kni -c 0x3 -n 4 -- \
-P -p 0x1 --config="(0,0,1)"

EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 0 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 2 lcore(s)
EAL: Probing VFIO support...
EAL:   IOMMU type 1 (Type 1) is supported
EAL:   IOMMU type 8 (No-IOMMU) is not supported
EAL: VFIO support initialized
EAL: Setting up physically contiguous memory...
...
EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
unreliable clock cycles !
EAL: Master lcore 0 is ready (tid=657d58c0;cpuset=[0])
PMD: rte_igbvf_pmd_init():  >>
EAL: lcore 1 is ready (tid=305ff700;cpuset=[1])
EAL: PCI device 0000:00:03.0 on NUMA socket -1
EAL:   probe driver: 1af4:1000 rte_virtio_pmd
EAL:   Not managed by a supported kernel driver(0), skipped
PMD: virtio_read_caps(): failed to map pci device!
PMD: vtpci_init(): trying with legacy virtio pci.
Segmentation fault

lspci 的输出 guest 中的命令

...
00:03.0 Ethernet controller: Red Hat, Inc Virtio network device

我注意到 pci_scan_one() 功能集 dev->kdrv = RTE_KDRV_NONE ，而驱动程序被检测为 virtio-pci (来自 /sys/bus/pci/devices/0000:00:03.0/driver)。

TAP网络

TAP 网络仍然存在相同的问题。在主机上，我从 Wi-Fi 接口(interface)配置了一个网桥，并将其连接到 TAP 接口(interface):

wifi_iface=wlp3s0
br_iface=br0
br_network='172.20.0.1/16'
br_dhcp_range='172.20.0.2,172.20.255.254'
tap_iface=tap0
user=ruslan

ip link add name $br_iface type bridge
ip addr add "$br_network" dev $br_iface
ip link set $br_iface up
dnsmasq --interface=$br_iface --bind-interfaces \
  --dhcp-range=$br_dhcp_range

modprobe tun
chmod 0666 /dev/net/tun

ip tuntap add dev $tap_iface mode tap user "$user"
ip link set $tap_iface up promisc on
ip link set $tap_iface master $br_iface

sysctl net.ipv4.ip_forward=1
sysctl net.ipv6.conf.default.forwarding=1
sysctl net.ipv6.conf.all.forwarding=1

iptables -t nat -A POSTROUTING -o $wifi_iface -j MASQUERADE
iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i $tap_iface -o $wifi_iface -j ACCEPT

QEMU 命令:

sudo qemu-system-x86_64 -enable-kvm \
  -cpu host -smp 2 \
  -vga std \
  -mem-prealloc -mem-path /dev/hugepages \
  -drive file=GentooVM.img,if=virtio \
  -netdev tap,id=vm1_p1,ifname=tap0,script=no,downscript=no,vhost=on \
  -device virtio-net-pci,netdev=vm1_p1,bus=pci.0,addr=0x3,ioeventfd=on \
  -m 1024M \
  -monitor stdio \
  -name "Gentoo VM" \
  $@

ifconfig guest 中的输出:

enp0s3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.20.196.253  netmask 255.255.0.0  broadcast 172.20.255.255
        inet6 fe80::59c1:f175:aeb3:433  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:12:34:56  txqueuelen 1000  (Ethernet)
        RX packets 9  bytes 1039 (1.0 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 21  bytes 1802 (1.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

以下命令的失败方式与上述“用户”网络的情况相同:

sudo ./examples/kni/build/app/kni -c 0x3 -n 4 -- \
-P -p 0x1 --config="(0,0,1)"
...
EAL: PCI device 0000:00:03.0 on NUMA socket -1
EAL:   probe driver: 1af4:1000 rte_virtio_pmd
EAL:   Not managed by a supported kernel driver(0), skipped
PMD: virtio_read_caps(): failed to map pci device!
PMD: vtpci_init(): trying with legacy virtio pci.

问题

甚至可以使用 virtio-net-pci 运行 KNI设备？

如果不可能，是否有其他选择可以在虚拟化环境中开发 DPDK KNI 应用程序？

Answer 1

virtio_read_caps()无法映射 PCI 设备，因为 DPDK 需要 binding network ports到以下内核模块之一:

uio_pci_generic ,

igb_uio ,

vfio-pci :

As of release 1.4, DPDK applications no longer automatically unbind all supported network ports from the kernel driver in use. Instead, all ports that are to be used by an DPDK application must be bound to the uio_pci_generic, igb_uio or vfio-pci module before the application is run. Any network ports under Linux* control will be ignored by the DPDK poll-mode drivers and cannot be used by the application.

因此，在运行 DPDK KNI 应用程序之前，我们应该

加载至少一个受支持的 I/O 内核模块(uio_pci_generic、igb_uio 或 vfio-pci)

加载$RTE_SDK/$RTE_TARGET/kmod/rte_kni.ko

通过 $RTE_SDK/tools/dpdk_nic_bind.py 将网络端口绑定(bind)到内核模块脚本。

例如:

$ sudo insmod $RTE_SDK/$RTE_TARGET/kmod/igb_uio.ko
$ sudo insmod $RTE_SDK/$RTE_TARGET/kmod/rte_kni.ko
# Obtain ID of the network port
$ python2 $RTE_SDK/tools/dpdk_nic_bind.py --status
Network devices using DPDK-compatible driver
============================================
0000:00:03.0 'Virtio network device' drv=igb_uio unused=vfio-pci,uio_pci_generic
$ sudo python2 $RTE_SDK/tools/dpdk_nic_bind.py --bind=igb_uio 0000:00:03.0

在哪里

$RTE_SDK - 指向 DPDK 安装目录。

$RTE_TARGET - 指向 DPDK 目标环境目录。