How to compile ixgbe driver on CentOS, RHEL or Fedora

Question: I want to build and install the latest ixgbe 10G NIC driver. How can I compile ixgbe driver on CentOS, Fedora or RHEL?

To use Intel's PCI Express 10G NICs (e.g., 82598, 82599, x540) on your Linux system, you need to install ixgbe driver. While modern Linux distributions come with ixgbe driver pre-installed as a loadable module, the pre-built ixgbe driver is not fully customizable with limited parameters. If you want to enable and customize all available features of the 10G NICs (e.g., RSS, multi-queue, virtual functions, hardware offload), you will need to build the driver from the source.

Here is how to compile ixgbe driver from the source on Red Hat based platforms (e.g., CentOS, RHEL or Fedora). For Debian based systems, refer to this guideline instead.

Step One: Install Prerequisites

First, set up necessary development environment and install matching kernel headers.

$ sudo yum install gcc make
$ sudo yum install kernel kernel-devel

If the above commands install a newer kernel, reboot the machine and proceed with the next step.

Step Two: Compile Ixgbe Driver

Download the latest ixgbe source code from the official site.

$ wget http://downloads.sourceforge.net/project/e1000/ixgbe%20stable/3.23.2/ixgbe-3.23.2.tar.gz

Be sure to check supported kernel versions. For example, the ixgbe driver 3.23.2 supports Linux kernel versions 2.6.18 up to 3.18.1.

Extract the tarball and compile it.

$ tar -xvf ixgbe-3.23.2.tar.gz
$ cd ixgbe-3.23.2/src
$ make

If successful, the compiled driver (ixgbe.ko) will be found in the current directory.

You can check the information of the compiled driver by running:

$ modinfo ./ixgbe.ko

The output will show a list of available parameters of the ixgbe driver.

Step Three: Load Ixgbe Driver

Now you are ready to load the compiled ixgbe driver.

If the stock ixgbe driver is already loaded on your system, you need to unload it first. Otherwise, you won't be able to load the new ixgbe driver.

$ sudo rmmod ixgbe.ko

Also, the latest ixgbe module may depend on other modules (e.g., ptp,dca,vxlan), so make sure to load all the modules listed in "depends:" section in the above modinfo output. For example:

$ sudo modprobe ptp
$ sudo modprobe dca
$ sudo modprobe vxlan

Then insert the compiled driver in the kernel by running:

$ sudo insmod ./ixgbe.ko

Optionally, you can supply any parameters while loading the driver.

$ sudo insmod ./ixgbe.ko FdirPballoc=3 RSS=16

To verify that the driver is loaded successfully, check the output of dmesg command.

$ dmesg
Intel(R) 10 Gigabit PCI Express Network Driver - version 3.23.2
Copyright (c) 1999-2014 Intel Corporation.
ixgbe 0000:21:00.0: PCI INT A -> GSI 64 (level, low) -> IRQ 64
ixgbe 0000:21:00.0: setting latency timer to 64
ixgbe: Receive-Side Scaling (RSS) set to 16
ixgbe: Flow Director packet buffer allocation set to 3
ixgbe: 0000:21:00.0: ixgbe_check_options: Flow Director will be allocated 256kB of packet buffer
ixgbe: 0000:21:00.0: ixgbe_check_options: FCoE Offload feature enabled
ixgbe 0000:21:00.0: irq 87 for MSI/MSI-X
ixgbe 0000:21:00.0: irq 88 for MSI/MSI-X
ixgbe 0000:21:00.0: irq 89 for MSI/MSI-X
ixgbe 0000:21:00.0: irq 90 for MSI/MSI-X
ixgbe 0000:21:00.0: irq 91 for MSI/MSI-X
ixgbe 0000:21:00.0: irq 92 for MSI/MSI-X
ixgbe 0000:21:00.0: irq 93 for MSI/MSI-X
ixgbe 0000:21:00.0: irq 94 for MSI/MSI-X
ixgbe 0000:21:00.0: irq 95 for MSI/MSI-X
ixgbe 0000:21:00.0: irq 96 for MSI/MSI-X
ixgbe 0000:21:00.0: irq 97 for MSI/MSI-X
ixgbe 0000:21:00.0: irq 98 for MSI/MSI-X
ixgbe 0000:21:00.0: irq 99 for MSI/MSI-X
ixgbe 0000:21:00.0: irq 100 for MSI/MSI-X
ixgbe 0000:21:00.0: irq 101 for MSI/MSI-X
ixgbe 0000:21:00.0: irq 102 for MSI/MSI-X
ixgbe 0000:21:00.0: irq 103 for MSI/MSI-X
ixgbe 0000:21:00.0: eth3: DCA registration failed: -1
ixgbe 0000:21:00.0: PCI Express bandwidth of 32GT/s available
ixgbe 0000:21:00.0: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
ixgbe 0000:21:00.0: eth3: MAC: 2, PHY: 9, SFP+: 3, PBA No: E68793-006
ixgbe 0000:21:00.0: 90:e2:ba:5b:e9:1c
ixgbe 0000:21:00.0: eth3: Enabled Features: RxQ: 16 TxQ: 16 FdirHash RSC 
ixgbe 0000:21:00.0: eth3: Intel(R) 10 Gigabit Network Connection
ixgbe 0000:21:00.1: PCI INT B -> GSI 68 (level, low) -> IRQ 68
ixgbe 0000:21:00.1: setting latency timer to 64
ixgbe: 0000:21:00.1: ixgbe_check_options: FCoE Offload feature enabled
ixgbe 0000:21:00.0: registered PHC device on eth3
ixgbe 0000:21:00.1: irq 104 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 105 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 106 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 107 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 108 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 109 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 110 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 111 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 112 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 113 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 114 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 115 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 116 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 117 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 118 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 119 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 120 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 121 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 122 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 123 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 124 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 125 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 126 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 127 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 128 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 129 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 130 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 131 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 132 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 133 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 134 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 135 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 136 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 137 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 138 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 139 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 140 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 141 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 142 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 143 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 144 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 145 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 146 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 147 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 148 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 149 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 150 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 151 for MSI/MSI-X
ixgbe 0000:21:00.1: irq 152 for MSI/MSI-X
ADDRCONF(NETDEV_UP): eth3: link is not ready
8021q: adding VLAN 0 to HW filter on device eth3
ixgbe 0000:21:00.1: eth4: DCA registration failed: -1
ixgbe 0000:21:00.1: PCI Express bandwidth of 32GT/s available
ixgbe 0000:21:00.1: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
ixgbe 0000:21:00.1: eth4: MAC: 2, PHY: 9, SFP+: 4, PBA No: E68793-006
ixgbe 0000:21:00.1: 90:e2:ba:5b:e9:1e
ixgbe 0000:21:00.1: eth4: Enabled Features: RxQ: 48 TxQ: 48 FdirHash RSC 
ixgbe 0000:21:00.1: eth4: Intel(R) 10 Gigabit Network Connection
ixgbe 0000:21:00.0: eth3: detected SFP+: 3
ixgbe 0000:21:00.1: registered PHC device on eth4
ADDRCONF(NETDEV_UP): eth4: link is not ready
8021q: adding VLAN 0 to HW filter on device eth4
ixgbe 0000:21:00.1: eth4: detected SFP+: 4
ixgbe 0000:21:00.0: eth3: NIC Link is Up 10 Gbps, Flow Control: RX/TX
ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
ixgbe 0000:21:00.1: eth4: NIC Link is Up 10 Gbps, Flow Control: RX/TX
ADDRCONF(NETDEV_CHANGE): eth4: link becomes ready
eth3: no IPv6 routers present
eth4: no IPv6 routers present

Step Four: Install Ixgbe Driver

Once you have checked that the driver is loaded successfully, go ahead and install the driver on your system.

$ sudo make install

ixgbe.ko will be installed in the following location.

/lib/modules/<kernel-version>/kernel/drivers/net/ixgbe

At this point, the compiled driver will be loaded automatically upon boot, or you can load it by running:

$ sudo modprobe ixgbe

Note that if you upgrade to a new kernel, you need to re-build the installed ixgbe driver against the new kernel. You can use DKMS to re-build ixgbe driver automatically. Refer to the tutorial on DKMS for more detail.

Download this article as ad-free PDF (made possible by your kind donation): 
Download PDF

Subscribe to Ask Xmodulo

Do you want to receive Linux related questions & answers published at Ask Xmodulo? Enter your email address below, and we will deliver our Linux Q&A straight to your email box, for free. Delivery powered by Google Feedburner.


Support Xmodulo

Did you find this tutorial helpful? Then please be generous and support Xmodulo!

12 thoughts on “How to compile ixgbe driver on CentOS, RHEL or Fedora

  1. Excellent tutorial! Many thanks Dan.

    After:
    insmod ./ixgbe.ko
    we needed to do:
    systemctl restart network

  2. I am having issues installing the 4.3.13 ixgbe driver on centos 5, could you please help. I get the following errors

    [root@VHDX-DIST src]# make
    common.mk:83: *** Kernel header files not in any of the expected locations.
    common.mk:84: *** Install the appropriate kernel development package, e.g.
    common.mk:85: *** kernel-devel, for building kernel modules and try again. Stop

    • Looks like the kernel headers you installed (kernel-devel) are not matched with the kernel you have. Running the following command will ensure that you install both the kernel and matching kernel headers on your system. Don't forget to reboot after installing a newer kernel.

      $ sudo yum install kernel kernel-devel
  3. @Dan : Thanks for responding, I'll try your suggestion but as per now below is my system configuration. Please let me know If I would need to do anything else.

    [root@VHDX-DIST src]# uname -a
    Linux VHDX-DIST 3.10.0+2 #1 SMP Tue Dec 9 12:45:36 EST 2014 x86_64 x86_64 x86_64 GNU/Linux

    [root@VHDX-DIST src]# rpm -qa | grep kernel
    kernel-3.10.41-323.380416
    kernel-devel-2.6.18-398.el5
    kernel-headers-2.6.18-398.el5

    • The installed kernel headers (2.6.18-398.el5) are not matched with the kernel you are using (3.10.0+2). You need to either boot into kernel 2.6.18-398.el5, or install kernel headers matched with 3.10.0+2.

      • Thanks again for your prompt response, I did try to update the kernel-headers but it seems 2.6.18-398.e15 are the latest one as yum install keep updating the same version. Seems like I would need to boot into kernel 2.6.18-398.e15

  4. What to do after yum update installs the new kernel?

    E.g. we have 3.10.0-327.10.1.el7.x86_64 kernel and yum just installed 3.10.0-327.13.1.el7.x86_64. Will the existing driver /lib/modules/3.10.0-327.10.1.el7.x86_64/updates/drivers/net/ethernet/intel/ixgbe/ixgbe.ko automatically work with the new kernel?

    Tried to compile it again, but it still uses old kernel header. Is it possible to somehow build ixgbe driver for the new kernel prior to booting it?

    • The existing driver will not work with a newer kernel. You need to re-compile it against the new kernel's header files. When you run make, it will automatically look for header files of the currently loaded kernel.

      Obviously it's quite cumbersome to have to re-compile the driver every time the kernel is updated. This is when dynamic kernel module support (DKMS) comes in handy. DKMS takes care of automatic compilation of any module built outside the kernel source tree. For more info on DKMS: http://xmodulo.com/build-kernel-module-dkms-linux.html

      • Many thanks for so prompt reply Dan! I'll look into DKMS.

        Just had server crash and I am suspecting that KernelCare had something to do with it. What do you think, could it be related to some clash with ixgbe driver compiled like this?

          • It looks like KernelCare is guilty for crash, don't know what else. Switched back to Ksplice.

            Tried using DKMS, but it looks like there is some clash since I've already have up and running module following your instructions on this page.

            [root@main ~]# dkms status
            ixgbe, 4.3.15, 3.10.0-327.10.1.el7.x86_64, x86_64: built

            [root@main ~]# dkms install -m ixgbe -v 4.3.15
            ixgbe:
            Running module version sanity check.
            - Original module
            - Multiple original modules exist but DKMS does not know which to pick
            - Due to the confusion, none will be considered during a later uninstall
            - Multiple same named modules!
            - 2 named ixgbe.ko in /lib/modules/3.10.0-327.10.1.el7.x86_64/
            - All instances of this module will now be stored for reference purposes ONLY
            - Storing in /var/lib/dkms/ixgbe/original_module/3.10.0-327.10.1.el7.x86_64/x86_64/collisions/
            - Stored /lib/modules/3.10.0-327.10.1.el7.x86_64/updates/drivers/net/ethernet/intel/ixgbe/ixgbe.ko
            - Stored /lib/modules/3.10.0-327.10.1.el7.x86_64/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko
            - Installation
            - Installing to /lib/modules/3.10.0-327.10.1.el7.x86_64/extra/
            Adding any weak-modules
            depmod: WARNING: could not open /lib/modules/3.10.0-327.10.1.el7.x86_64.ksplice-updates/modules.order: No such file or directory
            depmod: WARNING: could not open /lib/modules/3.10.0-327.10.1.el7.x86_64.ksplice-updates/modules.builtin: No such file or directory

            depmod...

            DKMS: install completed.

            [root@main ~]# dkms status | grep ixgbe
            ixgbe, 4.3.15, 3.10.0-327.10.1.el7.x86_64, x86_64: installed (original_module exists)
            ixgbe, 4.3.15, 3.10.0-327.10.1.el7.x86_64.ksplice-updates, x86_64: installed-weak from 3.10.0-327.10.1.el7.x86_64
            ixgbe, 4.3.15, 3.10.0-327.10.1.el7.x86_64, x86_64: installed-weak from 3.10.0-327.10.1.el7.x86_64

Leave a comment

Your email address will not be published. Required fields are marked *