Linux Ethernet Bonding Driver HOWTO
- Latest update: 21 June 2005
+ Latest update: 12 November 2007
Initial release : Thomas Davis <tadavis at lbl.gov>
Corrections, HA extensions : 2000/10/03-15 :
- Jay Vosburgh <fubar at us dot ibm dot com>
Reorganized and updated Feb 2005 by Jay Vosburgh
+Added Sysfs information: 2006/04/24
+ - Mitch Williams <mitch.a.williams at intel.com>
Introduction
============
2. Bonding Driver Options
3. Configuring Bonding Devices
-3.1 Configuration with sysconfig support
-3.1.1 Using DHCP with sysconfig
-3.1.2 Configuring Multiple Bonds with sysconfig
-3.2 Configuration with initscripts support
-3.2.1 Using DHCP with initscripts
-3.2.2 Configuring Multiple Bonds with initscripts
-3.3 Configuring Bonding Manually
+3.1 Configuration with Sysconfig Support
+3.1.1 Using DHCP with Sysconfig
+3.1.2 Configuring Multiple Bonds with Sysconfig
+3.2 Configuration with Initscripts Support
+3.2.1 Using DHCP with Initscripts
+3.2.2 Configuring Multiple Bonds with Initscripts
+3.3 Configuring Bonding Manually with Ifenslave
3.3.1 Configuring Multiple Bonds Manually
+3.4 Configuring Bonding Manually via Sysfs
-5. Querying Bonding Configuration
-5.1 Bonding Configuration
-5.2 Network Configuration
+4. Querying Bonding Configuration
+4.1 Bonding Configuration
+4.2 Network Configuration
-6. Switch Configuration
+5. Switch Configuration
-7. 802.1q VLAN Support
+6. 802.1q VLAN Support
-8. Link Monitoring
-8.1 ARP Monitor Operation
-8.2 Configuring Multiple ARP Targets
-8.3 MII Monitor Operation
+7. Link Monitoring
+7.1 ARP Monitor Operation
+7.2 Configuring Multiple ARP Targets
+7.3 MII Monitor Operation
-9. Potential Trouble Sources
-9.1 Adventures in Routing
-9.2 Ethernet Device Renaming
-9.3 Painfully Slow Or No Failed Link Detection By Miimon
+8. Potential Trouble Sources
+8.1 Adventures in Routing
+8.2 Ethernet Device Renaming
+8.3 Painfully Slow Or No Failed Link Detection By Miimon
-10. SNMP agents
+9. SNMP agents
-11. Promiscuous mode
+10. Promiscuous mode
-12. Configuring Bonding for High Availability
-12.1 High Availability in a Single Switch Topology
-12.2 High Availability in a Multiple Switch Topology
-12.2.1 HA Bonding Mode Selection for Multiple Switch Topology
-12.2.2 HA Link Monitoring for Multiple Switch Topology
+11. Configuring Bonding for High Availability
+11.1 High Availability in a Single Switch Topology
+11.2 High Availability in a Multiple Switch Topology
+11.2.1 HA Bonding Mode Selection for Multiple Switch Topology
+11.2.2 HA Link Monitoring for Multiple Switch Topology
-13. Configuring Bonding for Maximum Throughput
-13.1 Maximum Throughput in a Single Switch Topology
-13.1.1 MT Bonding Mode Selection for Single Switch Topology
-13.1.2 MT Link Monitoring for Single Switch Topology
-13.2 Maximum Throughput in a Multiple Switch Topology
-13.2.1 MT Bonding Mode Selection for Multiple Switch Topology
-13.2.2 MT Link Monitoring for Multiple Switch Topology
+12. Configuring Bonding for Maximum Throughput
+12.1 Maximum Throughput in a Single Switch Topology
+12.1.1 MT Bonding Mode Selection for Single Switch Topology
+12.1.2 MT Link Monitoring for Single Switch Topology
+12.2 Maximum Throughput in a Multiple Switch Topology
+12.2.1 MT Bonding Mode Selection for Multiple Switch Topology
+12.2.2 MT Link Monitoring for Multiple Switch Topology
-14. Switch Behavior Issues
-14.1 Link Establishment and Failover Delays
-14.2 Duplicated Incoming Packets
+13. Switch Behavior Issues
+13.1 Link Establishment and Failover Delays
+13.2 Duplicated Incoming Packets
-15. Hardware Specific Considerations
-15.1 IBM BladeCenter
+14. Hardware Specific Considerations
+14.1 IBM BladeCenter
-16. Frequently Asked Questions
+15. Frequently Asked Questions
-17. Resources and Links
+16. Resources and Links
1. Bonding Driver Installation
onwards) do not have /usr/include/linux symbolically linked to the
default kernel source include directory.
+SECOND IMPORTANT NOTE:
+ If you plan to configure bonding using sysfs, you do not need
+to use ifenslave.
2. Bonding Driver Options
=========================
- Options for the bonding driver are supplied as parameters to
-the bonding module at load time. They may be given as command line
-arguments to the insmod or modprobe command, but are usually specified
-in either the /etc/modules.conf or /etc/modprobe.conf configuration
-file, or in a distro-specific configuration file (some of which are
-detailed in the next section).
+ Options for the bonding driver are supplied as parameters to the
+bonding module at load time, or are specified via sysfs.
+
+ Module options may be given as command line arguments to the
+insmod or modprobe command, but are usually specified in either the
+/etc/modules.conf or /etc/modprobe.conf configuration file, or in a
+distro-specific configuration file (some of which are detailed in the next
+section).
+
+ Details on bonding support for sysfs is provided in the
+"Configuring Bonding Manually via Sysfs" section, below.
The available bonding driver parameters are listed below. If a
parameter is not specified the default value is used. When initially
arp_interval
Specifies the ARP link monitoring frequency in milliseconds.
+
+ The ARP monitor works by periodically checking the slave
+ devices to determine whether they have sent or received
+ traffic recently (the precise criteria depends upon the
+ bonding mode, and the state of the slave). Regular traffic is
+ generated via ARP probes issued for the addresses specified by
+ the arp_ip_target option.
+
+ This behavior can be modified by the arp_validate option,
+ below.
+
If ARP monitoring is used in an etherchannel compatible mode
(modes 0 and 2), the switch should be configured in a mode
that evenly distributes packets across all links. If the
maximum number of targets that can be specified is 16. The
default value is no IP addresses.
+arp_validate
+
+ Specifies whether or not ARP probes and replies should be
+ validated in the active-backup mode. This causes the ARP
+ monitor to examine the incoming ARP requests and replies, and
+ only consider a slave to be up if it is receiving the
+ appropriate ARP traffic.
+
+ Possible values are:
+
+ none or 0
+
+ No validation is performed. This is the default.
+
+ active or 1
+
+ Validation is performed only for the active slave.
+
+ backup or 2
+
+ Validation is performed only for backup slaves.
+
+ all or 3
+
+ Validation is performed for all slaves.
+
+ For the active slave, the validation checks ARP replies to
+ confirm that they were generated by an arp_ip_target. Since
+ backup slaves do not typically receive these replies, the
+ validation performed for backup slaves is on the ARP request
+ sent out via the active slave. It is possible that some
+ switch or network configurations may result in situations
+ wherein the backup slaves do not receive the ARP requests; in
+ such a situation, validation of backup slaves must be
+ disabled.
+
+ This option is useful in network configurations in which
+ multiple bonding hosts are concurrently issuing ARPs to one or
+ more targets beyond a common switch. Should the link between
+ the switch and target fail (but not the switch itself), the
+ probe traffic generated by the multiple bonding instances will
+ fool the standard ARP monitor into considering the links as
+ still up. Use of the arp_validate option can resolve this, as
+ the ARP monitor will only consider ARP requests and replies
+ associated with its own instance of bonding.
+
+ This option was added in bonding version 3.1.0.
+
downdelay
Specifies the time, in milliseconds, to wait before disabling
will be rounded down to the nearest multiple. The default
value is 0.
+fail_over_mac
+
+ Specifies whether active-backup mode should set all slaves to
+ the same MAC address (the traditional behavior), or, when
+ enabled, change the bond's MAC address when changing the
+ active interface (i.e., fail over the MAC address itself).
+
+ Fail over MAC is useful for devices that cannot ever alter
+ their MAC address, or for devices that refuse incoming
+ broadcasts with their own source MAC (which interferes with
+ the ARP monitor).
+
+ The down side of fail over MAC is that every device on the
+ network must be updated via gratuitous ARP, vs. just updating
+ a switch or set of switches (which often takes place for any
+ traffic, not just ARP traffic, if the switch snoops incoming
+ traffic to update its tables) for the traditional method. If
+ the gratuitous ARP is lost, communication may be disrupted.
+
+ When fail over MAC is used in conjuction with the mii monitor,
+ devices which assert link up prior to being able to actually
+ transmit and receive are particularly susecptible to loss of
+ the gratuitous ARP, and an appropriate updelay setting may be
+ required.
+
+ A value of 0 disables fail over MAC, and is the default. A
+ value of 1 enables fail over MAC. This option is enabled
+ automatically if the first slave added cannot change its MAC
+ address. This option may be modified via sysfs only when no
+ slaves are present in the bond.
+
+ This option was added in bonding version 3.2.0.
+
lacp_rate
Option specifying the rate in which we'll ask our link partner
In bonding version 2.6.2 or later, when a failover
occurs in active-backup mode, bonding will issue one
or more gratuitous ARPs on the newly active slave.
- One gratutious ARP is issued for the bonding master
+ One gratuitous ARP is issued for the bonding master
interface and each VLAN interfaces configured above
it, provided that the interface has at least one IP
address configured. Gratuitous ARPs issued for VLAN
When a link is reconnected or a new slave joins the
bond the receive traffic is redistributed among all
active slaves in the bond by initiating ARP Replies
- with the selected mac address to each of the
+ with the selected MAC address to each of the
clients. The updelay parameter (detailed below) must
be set to a value equal or greater than the switch's
forwarding delay so that the ARP Replies sent to the
This algorithm is 802.3ad compliant.
+ layer2+3
+
+ This policy uses a combination of layer2 and layer3
+ protocol information to generate the hash.
+
+ Uses XOR of hardware MAC addresses and IP addresses to
+ generate the hash. The formula is
+
+ (((source IP XOR dest IP) AND 0xffff) XOR
+ ( source MAC XOR destination MAC ))
+ modulo slave count
+
+ This algorithm will place all traffic to a particular
+ network peer on the same slave. For non-IP traffic,
+ the formula is the same as for the layer2 transmit
+ hash policy.
+
+ This policy is intended to provide a more balanced
+ distribution of traffic than layer2 alone, especially
+ in environments where a layer3 gateway device is
+ required to reach most destinations.
+
+ This algorithm is 802.3ad complient.
+
layer3+4
This policy uses upper layer protocol information,
or may not tolerate this noncompliance.
The default value is layer2. This option was added in bonding
-version 2.6.3. In earlier versions of bonding, this parameter does
-not exist, and the layer2 policy is the only policy.
+ version 2.6.3. In earlier versions of bonding, this parameter
+ does not exist, and the layer2 policy is the only policy. The
+ layer2+3 value was added for bonding version 3.2.2.
3. Configuring Bonding Devices
==============================
- There are, essentially, two methods for configuring bonding:
-with support from the distro's network initialization scripts, and
-without. Distros generally use one of two packages for the network
-initialization scripts: initscripts or sysconfig. Recent versions of
-these packages have support for bonding, while older versions do not.
+ You can configure bonding using either your distro's network
+initialization scripts, or manually using either ifenslave or the
+sysfs interface. Distros generally use one of two packages for the
+network initialization scripts: initscripts or sysconfig. Recent
+versions of these packages have support for bonding, while older
+versions do not.
We will first describe the options for configuring bonding for
distros using versions of initscripts and sysconfig with full or
If this returns any matches, then your initscripts or
sysconfig has support for bonding.
-3.1 Configuration with sysconfig support
+3.1 Configuration with Sysconfig Support
----------------------------------------
This section applies to distros using a version of sysconfig
SuSE SLES 9's networking configuration system does support
bonding, however, at this writing, the YaST system configuration
-frontend does not provide any means to work with bonding devices.
+front end does not provide any means to work with bonding devices.
Bonding devices can be managed by hand, however, as follows.
First, if they have not already been configured, configure the
Note that the template does not document the various BONDING_
settings described above, but does describe many of the other options.
-3.1.1 Using DHCP with sysconfig
+3.1.1 Using DHCP with Sysconfig
-------------------------------
Under sysconfig, configuring a device with BOOTPROTO='dhcp'
the slave devices. Without active slaves, the DHCP requests are not
sent to the network.
-3.1.2 Configuring Multiple Bonds with sysconfig
+3.1.2 Configuring Multiple Bonds with Sysconfig
-----------------------------------------------
The sysconfig network initialization system is capable of
options in the ifcfg-bondX file, it is not necessary to add them to
the system /etc/modules.conf or /etc/modprobe.conf configuration file.
-3.2 Configuration with initscripts support
+3.2 Configuration with Initscripts Support
------------------------------------------
- This section applies to distros using a version of initscripts
-with bonding support, for example, Red Hat Linux 9 or Red Hat
-Enterprise Linux version 3 or 4. On these systems, the network
-initialization scripts have some knowledge of bonding, and can be
-configured to control bonding devices.
+ This section applies to distros using a recent version of
+initscripts with bonding support, for example, Red Hat Enterprise Linux
+version 3 or later, Fedora, etc. On these systems, the network
+initialization scripts have knowledge of bonding, and can be configured to
+control bonding devices. Note that older versions of the initscripts
+package have lower levels of support for bonding; this will be noted where
+applicable.
These distros will not automatically load the network adapter
driver unless the ethX device is configured with an IP address.
Be sure to change the networking specific lines (IPADDR,
NETMASK, NETWORK and BROADCAST) to match your network configuration.
- Finally, it is necessary to edit /etc/modules.conf (or
-/etc/modprobe.conf, depending upon your distro) to load the bonding
-module with your desired options when the bond0 interface is brought
-up. The following lines in /etc/modules.conf (or modprobe.conf) will
-load the bonding module, and select its options:
+ For later versions of initscripts, such as that found with Fedora
+7 and Red Hat Enterprise Linux version 5 (or later), it is possible, and,
+indeed, preferable, to specify the bonding options in the ifcfg-bond0
+file, e.g. a line of the format:
+
+BONDING_OPTS="mode=active-backup arp_interval=60 arp_ip_target=+192.168.1.254"
+
+ will configure the bond with the specified options. The options
+specified in BONDING_OPTS are identical to the bonding module parameters
+except for the arp_ip_target field. Each target should be included as a
+separate option and should be preceded by a '+' to indicate it should be
+added to the list of queried targets, e.g.,
+
+ arp_ip_target=+192.168.1.1 arp_ip_target=+192.168.1.2
+
+ is the proper syntax to specify multiple targets. When specifying
+options via BONDING_OPTS, it is not necessary to edit /etc/modules.conf or
+/etc/modprobe.conf.
+
+ For older versions of initscripts that do not support
+BONDING_OPTS, it is necessary to edit /etc/modules.conf (or
+/etc/modprobe.conf, depending upon your distro) to load the bonding module
+with your desired options when the bond0 interface is brought up. The
+following lines in /etc/modules.conf (or modprobe.conf) will load the
+bonding module, and select its options:
alias bond0 bonding
options bond0 mode=balance-alb miimon=100
will restart the networking subsystem and your bond link should be now
up and running.
-3.2.1 Using DHCP with initscripts
+3.2.1 Using DHCP with Initscripts
---------------------------------
- Recent versions of initscripts (the version supplied with
-Fedora Core 3 and Red Hat Enterprise Linux 4 is reported to work) do
-have support for assigning IP information to bonding devices via DHCP.
+ Recent versions of initscripts (the versions supplied with Fedora
+Core 3 and Red Hat Enterprise Linux 4, or later versions, are reported to
+work) have support for assigning IP information to bonding devices via
+DHCP.
To configure bonding for DHCP, configure it as described
above, except replace the line "BOOTPROTO=none" with "BOOTPROTO=dhcp"
and add a line consisting of "TYPE=Bonding". Note that the TYPE value
is case sensitive.
-3.2.2 Configuring Multiple Bonds with initscripts
+3.2.2 Configuring Multiple Bonds with Initscripts
-------------------------------------------------
- At this writing, the initscripts package does not directly
-support loading the bonding driver multiple times, so the process for
-doing so is the same as described in the "Configuring Multiple Bonds
-Manually" section, below.
-
- NOTE: It has been observed that some Red Hat supplied kernels
-are apparently unable to rename modules at load time (the "-obonding1"
-part). Attempts to pass that option to modprobe will produce an
-"Operation not permitted" error. This has been reported on some
-Fedora Core kernels, and has been seen on RHEL 4 as well. On kernels
-exhibiting this problem, it will be impossible to configure multiple
-bonds with differing parameters.
+ Initscripts packages that are included with Fedora 7 and Red Hat
+Enterprise Linux 5 support multiple bonding interfaces by simply
+specifying the appropriate BONDING_OPTS= in ifcfg-bondX where X is the
+number of the bond. This support requires sysfs support in the kernel,
+and a bonding driver of version 3.0.0 or later. Other configurations may
+not support this method for specifying multiple bonding interfaces; for
+those instances, see the "Configuring Multiple Bonds Manually" section,
+below.
-3.3 Configuring Bonding Manually
---------------------------------
+3.3 Configuring Bonding Manually with Ifenslave
+-----------------------------------------------
This section applies to distros whose network initialization
scripts (the sysconfig or initscripts package) do not have specific
options, you may wish to use the "max_bonds" module parameter,
documented above.
- To create multiple bonding devices with differing options, it
-is necessary to load the bonding driver multiple times. Note that
-current versions of the sysconfig network initialization scripts
-handle this automatically; if your distro uses these scripts, no
-special action is needed. See the section Configuring Bonding
-Devices, above, if you're not sure about your network initialization
-scripts.
+ To create multiple bonding devices with differing options, it is
+preferrable to use bonding parameters exported by sysfs, documented in the
+section below.
+
+ For versions of bonding without sysfs support, the only means to
+provide multiple instances of bonding with differing options is to load
+the bonding driver multiple times. Note that current versions of the
+sysconfig network initialization scripts handle this automatically; if
+your distro uses these scripts, no special action is needed. See the
+section Configuring Bonding Devices, above, if you're not sure about your
+network initialization scripts.
To load multiple instances of the module, it is necessary to
specify a different name for each instance (the module loading system
requires that every loaded module, even multiple instances of the same
-module, have a unique name). This is accomplished by supplying
-multiple sets of bonding options in /etc/modprobe.conf, for example:
-
+module, have a unique name). This is accomplished by supplying multiple
+sets of bonding options in /etc/modprobe.conf, for example:
+
alias bond0 bonding
options bond0 -o bond0 mode=balance-rr miimon=100
its options. In that case, the second options line can be substituted
as follows:
-install bonding1 /sbin/modprobe bonding -obond1 mode=balance-alb miimon=50
+install bond1 /sbin/modprobe --ignore-install bonding -o bond1 \
+ mode=balance-alb miimon=50
This may be repeated any number of times, specifying a new and
unique name in place of bond1 for each subsequent instance.
+ It has been observed that some Red Hat supplied kernels are unable
+to rename modules at load time (the "-o bond1" part). Attempts to pass
+that option to modprobe will produce an "Operation not permitted" error.
+This has been reported on some Fedora Core kernels, and has been seen on
+RHEL 4 as well. On kernels exhibiting this problem, it will be impossible
+to configure multiple bonds with differing parameters (as they are older
+kernels, and also lack sysfs support).
+
+3.4 Configuring Bonding Manually via Sysfs
+------------------------------------------
+
+ Starting with version 3.0.0, Channel Bonding may be configured
+via the sysfs interface. This interface allows dynamic configuration
+of all bonds in the system without unloading the module. It also
+allows for adding and removing bonds at runtime. Ifenslave is no
+longer required, though it is still supported.
+
+ Use of the sysfs interface allows you to use multiple bonds
+with different configurations without having to reload the module.
+It also allows you to use multiple, differently configured bonds when
+bonding is compiled into the kernel.
+
+ You must have the sysfs filesystem mounted to configure
+bonding this way. The examples in this document assume that you
+are using the standard mount point for sysfs, e.g. /sys. If your
+sysfs filesystem is mounted elsewhere, you will need to adjust the
+example paths accordingly.
+
+Creating and Destroying Bonds
+-----------------------------
+To add a new bond foo:
+# echo +foo > /sys/class/net/bonding_masters
+
+To remove an existing bond bar:
+# echo -bar > /sys/class/net/bonding_masters
+
+To show all existing bonds:
+# cat /sys/class/net/bonding_masters
+
+NOTE: due to 4K size limitation of sysfs files, this list may be
+truncated if you have more than a few hundred bonds. This is unlikely
+to occur under normal operating conditions.
+
+Adding and Removing Slaves
+--------------------------
+ Interfaces may be enslaved to a bond using the file
+/sys/class/net/<bond>/bonding/slaves. The semantics for this file
+are the same as for the bonding_masters file.
+
+To enslave interface eth0 to bond bond0:
+# ifconfig bond0 up
+# echo +eth0 > /sys/class/net/bond0/bonding/slaves
+
+To free slave eth0 from bond bond0:
+# echo -eth0 > /sys/class/net/bond0/bonding/slaves
+
+ When an interface is enslaved to a bond, symlinks between the
+two are created in the sysfs filesystem. In this case, you would get
+/sys/class/net/bond0/slave_eth0 pointing to /sys/class/net/eth0, and
+/sys/class/net/eth0/master pointing to /sys/class/net/bond0.
+
+ This means that you can tell quickly whether or not an
+interface is enslaved by looking for the master symlink. Thus:
+# echo -eth0 > /sys/class/net/eth0/master/bonding/slaves
+will free eth0 from whatever bond it is enslaved to, regardless of
+the name of the bond interface.
+
+Changing a Bond's Configuration
+-------------------------------
+ Each bond may be configured individually by manipulating the
+files located in /sys/class/net/<bond name>/bonding
+
+ The names of these files correspond directly with the command-
+line parameters described elsewhere in this file, and, with the
+exception of arp_ip_target, they accept the same values. To see the
+current setting, simply cat the appropriate file.
-5. Querying Bonding Configuration
+ A few examples will be given here; for specific usage
+guidelines for each parameter, see the appropriate section in this
+document.
+
+To configure bond0 for balance-alb mode:
+# ifconfig bond0 down
+# echo 6 > /sys/class/net/bond0/bonding/mode
+ - or -
+# echo balance-alb > /sys/class/net/bond0/bonding/mode
+ NOTE: The bond interface must be down before the mode can be
+changed.
+
+To enable MII monitoring on bond0 with a 1 second interval:
+# echo 1000 > /sys/class/net/bond0/bonding/miimon
+ NOTE: If ARP monitoring is enabled, it will disabled when MII
+monitoring is enabled, and vice-versa.
+
+To add ARP targets:
+# echo +192.168.0.100 > /sys/class/net/bond0/bonding/arp_ip_target
+# echo +192.168.0.101 > /sys/class/net/bond0/bonding/arp_ip_target
+ NOTE: up to 10 target addresses may be specified.
+
+To remove an ARP target:
+# echo -192.168.0.100 > /sys/class/net/bond0/bonding/arp_ip_target
+
+Example Configuration
+---------------------
+ We begin with the same example that is shown in section 3.3,
+executed with sysfs, and without using ifenslave.
+
+ To make a simple bond of two e100 devices (presumed to be eth0
+and eth1), and have it persist across reboots, edit the appropriate
+file (/etc/init.d/boot.local or /etc/rc.d/rc.local), and add the
+following:
+
+modprobe bonding
+modprobe e100
+echo balance-alb > /sys/class/net/bond0/bonding/mode
+ifconfig bond0 192.168.1.1 netmask 255.255.255.0 up
+echo 100 > /sys/class/net/bond0/bonding/miimon
+echo +eth0 > /sys/class/net/bond0/bonding/slaves
+echo +eth1 > /sys/class/net/bond0/bonding/slaves
+
+ To add a second bond, with two e1000 interfaces in
+active-backup mode, using ARP monitoring, add the following lines to
+your init script:
+
+modprobe e1000
+echo +bond1 > /sys/class/net/bonding_masters
+echo active-backup > /sys/class/net/bond1/bonding/mode
+ifconfig bond1 192.168.2.1 netmask 255.255.255.0 up
+echo +192.168.2.100 /sys/class/net/bond1/bonding/arp_ip_target
+echo 2000 > /sys/class/net/bond1/bonding/arp_interval
+echo +eth2 > /sys/class/net/bond1/bonding/slaves
+echo +eth3 > /sys/class/net/bond1/bonding/slaves
+
+
+4. Querying Bonding Configuration
=================================
-5.1 Bonding Configuration
+4.1 Bonding Configuration
-------------------------
Each bonding device has a read-only file residing in the
The precise format and contents will change depending upon the
bonding configuration, state, and version of the bonding driver.
-5.2 Network configuration
+4.2 Network configuration
-------------------------
The network configuration can be inspected using the ifconfig
collisions:0 txqueuelen:0
eth0 Link encap:Ethernet HWaddr 00:C0:F0:1F:37:B4
- inet addr:XXX.XXX.XXX.YYY Bcast:XXX.XXX.XXX.255 Mask:255.255.252.0
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:3573025 errors:0 dropped:0 overruns:0 frame:0
TX packets:1643167 errors:1 dropped:0 overruns:1 carrier:0
Interrupt:10 Base address:0x1080
eth1 Link encap:Ethernet HWaddr 00:C0:F0:1F:37:B4
- inet addr:XXX.XXX.XXX.YYY Bcast:XXX.XXX.XXX.255 Mask:255.255.252.0
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:3651769 errors:0 dropped:0 overruns:0 frame:0
TX packets:1643480 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:100
Interrupt:9 Base address:0x1400
-6. Switch Configuration
+5. Switch Configuration
=======================
For this section, "switch" refers to whatever system the
with another EtherChannel group.
-7. 802.1q VLAN Support
+6. 802.1q VLAN Support
======================
It is possible to configure VLAN devices over a bond interface
mode, which might not be what you want.
-8. Link Monitoring
+7. Link Monitoring
==================
The bonding driver at present supports two schemes for
bonding driver itself, it is not possible to enable both ARP and MII
monitoring simultaneously.
-8.1 ARP Monitor Operation
+7.1 ARP Monitor Operation
-------------------------
The ARP monitor operates as its name suggests: it sends ARP
shows the ARP requests and replies on the network, then it may be that
your device driver is not updating last_rx and trans_start.
-8.2 Configuring Multiple ARP Targets
+7.2 Configuring Multiple ARP Targets
------------------------------------
While ARP monitoring can be done with just one target, it can
options bond0 arp_interval=60 arp_ip_target=192.168.0.100
-8.3 MII Monitor Operation
+7.3 MII Monitor Operation
-------------------------
The MII monitor monitors only the carrier state of the local
and ethtool requests), then the MII monitor will assume the link is
up.
-9. Potential Sources of Trouble
+8. Potential Sources of Trouble
===============================
-9.1 Adventures in Routing
+8.1 Adventures in Routing
-------------------------
When bonding is configured, it is important that the slave
-devices not have routes that supercede routes of the master (or,
+devices not have routes that supersede routes of the master (or,
generally, not have routes at all). For example, suppose the bonding
device bond0 has two slaves, eth0 and eth1, and the routing table is
as follows:
The solution here is simply to insure that slaves do not have
routes of their own, and if for some reason they must, those routes do
-not supercede routes of their master. This should generally be the
+not supersede routes of their master. This should generally be the
case, but unusual configurations or errant manual or automatic static
route additions may cause trouble.
-9.2 Ethernet Device Renaming
+8.2 Ethernet Device Renaming
----------------------------
On systems with network configuration scripts that do not
place. Full documentation on this can be found in the modprobe.conf
and modprobe manual pages.
-9.3. Painfully Slow Or No Failed Link Detection By Miimon
+8.3. Painfully Slow Or No Failed Link Detection By Miimon
---------------------------------------------------------
By default, bonding enables the use_carrier option, which
beyond other ports of a switch, or if a switch is refusing to pass
traffic while still maintaining carrier on.
-10. SNMP agents
+9. SNMP agents
===============
If running SNMP agents, the bonding driver should be loaded
and SNMP functions such as Interface_Scan_Next will report that
association.
-11. Promiscuous mode
+10. Promiscuous mode
====================
When running network monitoring tools, e.g., tcpdump, it is
the active slave changes (e.g., due to a link failure), the
promiscuous setting will be propagated to the new active slave.
-12. Configuring Bonding for High Availability
+11. Configuring Bonding for High Availability
=============================================
High Availability refers to configurations that provide
(i.e., the network always works), even though other configurations
could provide higher throughput.
-12.1 High Availability in a Single Switch Topology
+11.1 High Availability in a Single Switch Topology
--------------------------------------------------
If two hosts (or a host and a single switch) are directly
See Section 13, "Configuring Bonding for Maximum Throughput"
for information on configuring bonding with one peer device.
-12.2 High Availability in a Multiple Switch Topology
+11.2 High Availability in a Multiple Switch Topology
----------------------------------------------------
With multiple switches, the configuration of bonding and the
the outside world ("port3" on each switch). There is no technical
reason that this could not be extended to a third switch.
-12.2.1 HA Bonding Mode Selection for Multiple Switch Topology
+11.2.1 HA Bonding Mode Selection for Multiple Switch Topology
-------------------------------------------------------------
In a topology such as the example above, the active-backup and
necessary for some specific one-way traffic to reach both
independent networks, then the broadcast mode may be suitable.
-12.2.2 HA Link Monitoring Selection for Multiple Switch Topology
+11.2.2 HA Link Monitoring Selection for Multiple Switch Topology
----------------------------------------------------------------
The choice of link monitoring ultimately depends upon your
regardless of which switch is active, the ARP monitor has a suitable
target to query.
-
-13. Configuring Bonding for Maximum Throughput
+ Note, also, that of late many switches now support a functionality
+generally referred to as "trunk failover." This is a feature of the
+switch that causes the link state of a particular switch port to be set
+down (or up) when the state of another switch port goes down (or up).
+It's purpose is to propogate link failures from logically "exterior" ports
+to the logically "interior" ports that bonding is able to monitor via
+miimon. Availability and configuration for trunk failover varies by
+switch, but this can be a viable alternative to the ARP monitor when using
+suitable switches.
+
+12. Configuring Bonding for Maximum Throughput
==============================================
-13.1 Maximizing Throughput in a Single Switch Topology
+12.1 Maximizing Throughput in a Single Switch Topology
------------------------------------------------------
In a single switch configuration, the best method to maximize
mode is described below.
-13.1.1 MT Bonding Mode Selection for Single Switch Topology
+12.1.1 MT Bonding Mode Selection for Single Switch Topology
-----------------------------------------------------------
This configuration is the easiest to set up and to understand,
interfaces. It is therefore the only mode that will allow a
single TCP/IP stream to utilize more than one interface's
worth of throughput. This comes at a cost, however: the
- striping often results in peer systems receiving packets out
+ striping generally results in peer systems receiving packets out
of order, causing TCP/IP's congestion control system to kick
in, often by retransmitting segments.
interface's worth of throughput, even after adjusting
tcp_reordering.
- Note that this out of order delivery occurs when both the
- sending and receiving systems are utilizing a multiple
- interface bond. Consider a configuration in which a
- balance-rr bond feeds into a single higher capacity network
- channel (e.g., multiple 100Mb/sec ethernets feeding a single
- gigabit ethernet via an etherchannel capable switch). In this
- configuration, traffic sent from the multiple 100Mb devices to
- a destination connected to the gigabit device will not see
- packets out of order. However, traffic sent from the gigabit
- device to the multiple 100Mb devices may or may not see
- traffic out of order, depending upon the balance policy of the
- switch. Many switches do not support any modes that stripe
- traffic (instead choosing a port based upon IP or MAC level
- addresses); for those devices, traffic flowing from the
- gigabit device to the many 100Mb devices will only utilize one
- interface.
+ Note that the fraction of packets that will be delivered out of
+ order is highly variable, and is unlikely to be zero. The level
+ of reordering depends upon a variety of factors, including the
+ networking interfaces, the switch, and the topology of the
+ configuration. Speaking in general terms, higher speed network
+ cards produce more reordering (due to factors such as packet
+ coalescing), and a "many to many" topology will reorder at a
+ higher rate than a "many slow to one fast" configuration.
+
+ Many switches do not support any modes that stripe traffic
+ (instead choosing a port based upon IP or MAC level addresses);
+ for those devices, traffic for a particular connection flowing
+ through the switch to a balance-rr bond will not utilize greater
+ than one interface's worth of bandwidth.
If you are utilizing protocols other than TCP/IP, UDP for
example, and your application can tolerate out of order
device driver must support changing the hardware address while
the device is open.
-13.1.2 MT Link Monitoring for Single Switch Topology
+12.1.2 MT Link Monitoring for Single Switch Topology
----------------------------------------------------
The choice of link monitoring may largely depend upon which
the MII monitor (which does not provide as high a level of end to end
assurance as the ARP monitor).
-13.2 Maximum Throughput in a Multiple Switch Topology
+12.2 Maximum Throughput in a Multiple Switch Topology
-----------------------------------------------------
Multiple switches may be utilized to optimize for throughput
can be equipped with an additional network device connected to an
external network; this host then additionally acts as a gateway.
-13.2.1 MT Bonding Mode Selection for Multiple Switch Topology
+12.2.1 MT Bonding Mode Selection for Multiple Switch Topology
-------------------------------------------------------------
In actual practice, the bonding mode typically employed in
mode allows individual connections between two hosts to effectively
utilize greater than one interface's bandwidth.
-13.2.2 MT Link Monitoring for Multiple Switch Topology
+12.2.2 MT Link Monitoring for Multiple Switch Topology
------------------------------------------------------
Again, in actual practice, the MII monitor is most often used
needed as the number of systems involved grows (remember that each
host in the network is configured with bonding).
-14. Switch Behavior Issues
+13. Switch Behavior Issues
==========================
-14.1 Link Establishment and Failover Delays
+13.1 Link Establishment and Failover Delays
-------------------------------------------
Some switches exhibit undesirable behavior with regard to the
to not activate a backup interface immediately after a link goes down.
Failover may be delayed via the downdelay bonding module option.
-14.2 Duplicated Incoming Packets
+13.2 Duplicated Incoming Packets
--------------------------------
+ NOTE: Starting with version 3.0.2, the bonding driver has logic to
+suppress duplicate packets, which should largely eliminate this problem.
+The following description is kept for reference.
+
It is not uncommon to observe a short burst of duplicated
traffic when the bonding device is first used, or after it has been
idle for some period of time. This is most easily observed by issuing
most Cisco switches, the privileged command "clear mac address-table
dynamic" will accomplish this).
-15. Hardware Specific Considerations
+14. Hardware Specific Considerations
====================================
This section contains additional information for configuring
bonding on specific hardware platforms, or for interfacing bonding
with particular switches or other devices.
-15.1 IBM BladeCenter
+14.1 IBM BladeCenter
--------------------
This applies to the JS20 and similar systems.
avoid fail-over delay issues when using bonding.
-16. Frequently Asked Questions
+15. Frequently Asked Questions
==============================
1. Is it SMP safe?
EtherExpress PRO/100 and a 3com 3c905b, for example). For most modes,
devices need not be of the same speed.
+ Starting with version 3.2.1, bonding also supports Infiniband
+slaves in active-backup mode.
+
3. How many bonding devices can I have?
There is no limit.
support specific features (described in the appropriate section under
module parameters, above).
- In 802.3ad mode, it works with with systems that support IEEE
+ In 802.3ad mode, it works with systems that support IEEE
802.3ad Dynamic Link Aggregation. Most managed and many unmanaged
switches currently available support 802.3ad.
8. Where does a bonding device get its MAC address from?
- If not explicitly configured (with ifconfig or ip link), the
-MAC address of the bonding device is taken from its first slave
-device. This MAC address is then passed to all following slaves and
-remains persistent (even if the first slave is removed) until the
-bonding device is brought down or reconfigured.
+ When using slave devices that have fixed MAC addresses, or when
+the fail_over_mac option is enabled, the bonding device's MAC address is
+the MAC address of the active slave.
+
+ For other configurations, if not explicitly configured (with
+ifconfig or ip link), the MAC address of the bonding device is taken from
+its first slave device. This MAC address is then passed to all following
+slaves and remains persistent (even if the first slave is removed) until
+the bonding device is brought down or reconfigured.
If you wish to change the MAC address, you can set it with
ifconfig or ip link: