System Utilities

As we covered in “History of FreeBSD Part 4: BSD and TCP/IP”, TCP/IP was brought to Unix with BSD first. Many common network utilities like ifconfig, route and netstat date back to the 4.2BSD release and spread from there to other Unix systems. Linux started out with similar utilities modelled on the BSD utilities; these packages continue to be widely available, but most modern Linux distributions have deprecated them in favor of alternate utilities such as ip(8).

The change on Linux came about alongside a redesign of the kernel interfaces where use of the ioctl(2) system call was replaced by a more flexible socket based interface called netlink. Interestingly, netlink support was recently added to FreeBSD and should be available in FreeBSD 14. This is mainly relevant to the use of the linuxulator, for running Linux applications and containers, but should also make it easier to port Linux software to FreeBSD.

The Linux ip tool loosely corresponds to ifconfig on FreeBSD, but the arguments take a different form. ip takes a list of sub-commands in a strict and structured order. ifconfig takes an interface device as its first argument and the arguments that follow specify options or actions where the ordering is more flexible. 

The following table illustrates this with some corresponding commands for FreeBSD vs. Linux. (Don’t worry if it isn’t yet clear what the commands do.)

	Linux	FreeBSD
List all network interfaces and their properties.	ip a	ifconfig -a
Add an initial IP address to an interface	ip addr add 10.0.1.2 dev eno1	ifconfig em0 10.0.1.2/24
Add additional IP address to an interface	ip addr add 10.0.2.2 dev eno1	ifconfig em0 alias 10.0.2.2/24
Add a static route to the routing table	ip route add 10.0.128.0/24 via 10.0.1.1	route add 10.0.128.0/24 10.0.1.1
Create a bridge	ip link add br0 type bridge	ifconfig br0 create
Add an interface to a bridge	ip link set eth0 master br0	ifconfig br0 addm em0
Create an aggregated link using LACP	ip link add bond0 type 802.3ad	ifconfig lagg0 create laggproto lacp
Create a connected pair of interfaces	ip link add veth1 type veth peer name veth2	ifconfig epair0 create
Bring an interface up	ip link set eth0 up	ifconfig em0 up
Remove an interface	ip link delete bond0	ifconfig lagg0 destroy

The ip and ifconfig commands are only used to change the current, live network settings. Linux distributions take varying approaches to configuring network settings to be applied on boot. The FreeBSD initialization scripts use variables from /etc/rc.conf. These map closely to their associated ifconfig invocations. The value of a variable named ifconfig_interfaceis passed directly to ifconfig. The following shows both a simple and more complex example, one to just bring up a physical interface and a second to configure a lagg0 interface.

  ifconfig_bge0="up"
  ifconfig_lagg0="up laggproto lacp laggport bge0 laggport em0"

There is a dedicated variable for adding IP addresses to an interface, for example.

  ipv4_addrs_bridge0="10.0.1.2/24"

Interfaces that first need to be created can be listed in the variable cloned_interfaces. For example, the following creates a bridge, aggregated link and a tagged VLAN:

  cloned_interfaces="lagg0 bridge0 lagg0.2"

For a bridge, there are two variables that can be used to add devices to the bridge, For example:

autobridge_interfaces="bridge0"
  autobridge_bridge0="lagg0 em1 em2 em3"

Container Networking

Containers on Linux are formed from separate namespaces for users, process IDs, filesystem mounts, network stack and other internal structures. Each of these are independent—so a container may be configured to share the host network namespace, or to use a separate namespace. 

This choice of whether to share the host network stack also exists for jails on FreeBSD.

Host Networking

On Linux, running a container in the host network namespace means that the same IP addresses and routing tables are shared. On FreeBSD, IP addresses are assigned to jails. These are created as additional aliases on existing network interfaces. Jailed processes are restricted to only using addresses assigned to their jail. The end result is somewhat similar to using IPVLAN on Linux.

For communication between jails, it can be useful to assign them IP addresses in the loopback subnet, which means they are addressable only from the same physical system. 

So in /etc/jail.conf, you might define:

   ip4.addr = "lo0|127.0.0.6/8";

This configuration gives the jail the address 127.0.0.6, inside the localhost subnet (which contains all addresses from 127.0.0.0 to 127.255.255.255). Consider a host system with one jail which runs a database, and a separate jail which runs an application which connects to the database. Having the database listen on a public IP address—or even a LAN-addressable address—may not be desirable. Running on the localhost subnet ensures that the database will only be accessible to the physical machine it’s running on.

Jails on the same host may also communicate via a Unix domain socket which are special files. As jails are isolated in their own directories, a nullfs mount is needed to make a Unix domain socket accessible in two different jails.

Because the host system has visibility of all the IP addresses associated with jails, it is tidier to take care that processes on the host aren’t listening on the jail addresses. The default configuration for a lot of server software is to listen on all addresses on all interfaces. If both the host and jail listen on the same port, the jail does get precedence—but ensuring there’s no accidental overlap can avoid confusion (and potential edge-case bugs!) so it’s still a good idea.

Virtualized Networking

Having a separate, virtualized network stack is only useful if there is a network interface that can be used for communicating with other systems. On Linux, network interfaces can be assigned to a network namespace. Using ip, this involves a command such as the following.

  ip link set dev ipvl0 netns ns0

With VNET jails on FreeBSD, the same basic approach is taken but interfaces to be passed to the jail are configured in /etc/jail.conf with the vnet.interface parameter. 

In the article “Virtualise your network on FreeBSD with VNET”, we demonstrated this using an epair(4) interface. Epair is analogous to VETH on Linux – you get two interfaces connected back-to-back with the intention being that one end is assigned to a jail. 

In the earlier article this was used in conjunction with a bridge on the host system. This is a fairly simple and common setup, but VNET jails can have any number or type of network interfaces. Other common choices for VNET jails would be to pass through an entire physical interface from the host, an interface corresponding to a particular tagged VLAN, or use ng_eiface which provides a netgraph node.

Bridges, Routers and Firewalls

Given connections to more than one network, it can be useful to allow packets to flow between the networks. In the example of a VNET jail with an epair interface, it was necessary to use a bridge to combine the epair interface with the host’s physical interface. On FreeBSD, there is no separate brctl or bridge command and, as the earlier article demonstrates, bridges are created and configured directly using ifconfig.

With a bridge, packets flow freely from one interface to another. This occurs at what is known as layer 2 so the bridge pays no attention to the source and destination IP addresses of packets. Another approach is to configure the system to act as a router and perform IP forwarding. In this case, each of the connected networks will need to use a different IP subnet, and the machine doing routing will need an address in each subnet. 

Enabling routing involves setting kernel parameters with sysctl such as net.ipv4.conf.all.forwarding on Linux (or setting the same parameters persistently, using /etc/sysctl.conf). This corresponds to setting net.inet.ip.forwarding on FreeBSD, or persistently setting gateway_enable=”YES” in /etc/rc.conf. 

With gateway_enable (or net.ipv4.conf.all.forwarding) enabled, the network stack will then accept packets on the local network from other systems, check the IP addressing in the packet header and forward it over a different local connection if necessary.

If you have two separate networks, one is often considered more “protected” than the other. Inspecting and blocking certain traffic can help to protect the more trusted network. Because it operates at layer 3, routing already acts as a barrier to lower level protocols such as DHCP, NetBIOS, and any non-routable services that rely on broadcasts. But if the operator also configures a firewall, network traffic can be categorized and processed in much greater detail. 

Linux has both iptables and nftables firewalls available, where nftables is a successor that maintains a degree of backward compatibility. 

FreeBSD also offers a choice of firewall utilities. The most popular FreeBSD firewalls are pf(4) (which was forked from the OpenBSD firewall of the same name) and ipfw(8) which originated within FreeBSD. 

Both ipfw and pf use simple rules for matching packets, and can apply actions such as logging, blocking or accepting packets. Unlike Linux, there are no separate tables for INPUT, OUTPUT and FORWARD. 

For ipfw, rules have an associated number to determine the order in which they are processed. It is common to use a shell script to build up the firewall configuration by adding rules. With pf, the entire ruleset is defined in a single config file.

Once an outgoing or forwarded packet has been accepted by the firewall, the routing table is consulted to determine where it should be sent next. Linux allows for multiple routing tables and uses a routing policy to determine how a packet is routed. Packets are matched using firewall-like rules using ip-rule(8). 

On FreeBSD, the firewall itself can tag packets to assign them to a specific routing table. By default only one table is available and net.fibs needs to be set in /boot/loader.conf to allocate more. FreeBSD also has a setfib command which allows a command to be run with a different routing table. As an example, this could be used to make a command use a VPN. (FIB stands for forwarding information base, and is the term used for a routing table.) 

Link Aggregation

Servers frequently use multiple redundant components to ensure stability in the case of failures. For example, data may be duplicated across multiple storage devices to mitigate individual drive failure. 

For networking, the equivalent is to use link aggregation, which combines multiple physical network interfaces into a single logical interface. 

In addition to providing resiliency in the case of a failure, link aggregation can also be used to provide a performance benefit by balancing packets across the interfaces.

On Linux, there are two implementations for link aggregation – channel bonding and teaming. Teaming is newer and has its own tool for configuration while bonding is configured using ip. FreeBSD also has a choice of two implementations: lagg(4) and ng_one2many(4), both included in the base system. Of these, the latter is a Netgraph node and is best used in combination with other netgraph nodes. (Our earlier article, “Using Netgraph for FreeBSD’s bhyve Networking”, serves as an introduction to netgraph. The example it gives is of a bridge but configuring ng_one2many(4) is broadly similar to ng_bridge(4).)

Configuring lagg(4) is very similar to configuring if_bridge(4). Similarly to the bridge, the lagg is first created, then the lower level interfaces are associated with it. The example from the man page is:

  ifconfig lagg0 create
  ifconfig lagg0 laggproto lacp laggport bge0 laggport bge1

The LACP protocol here highlights one advantage that lagg(4) has over ng_one2many(4). LACP needs support from the network switch to which the two physical ports are connected, but it allows the switch itself to load-balance incoming packets, just as the server balances outgoing packets. With a capable enough managed switch stack, LACP can span multiple switch chassis, preventing the switch from becoming a single point of failure as well. 

Other protocols are also available such as a pure failover setup or load balancing only outgoing packets. There is currently no equivalent of the Linux bonding balance-alb mode which achieves load balancing of incoming packets by intercepting ARP requests to vary the hardware MAC address that external hosts see.

Virtual IPs

When you have an important service associated with a particular IP address on your network, the system serving that address becomes a single point of failure. Link aggregation allows for redundant network interfaces within a system but does not protect against failures of other components. 

Unlike link aggregation, virtual IP addresses are not permanently tied to any specific system. Moving an IP address thus provides for a way to move a service. Although this can be done manually, we can also configure a backup system to be brought into service automatically as soon as something happens to its primary. 

Correctly managing such a fail-over requires a well-designed protocol which both minimizes the downtime while a service fails over, and—even more importantly—ensures that it isn’t possible to get a split brain scenario where two systems try to behave as primaries at the same time.

The Virtual Router Redundancy Protocol (VRRP) is an open standard that serves this purpose. It is closely related to Cisco’s proprietary HSRP. As the names indicate, the focus of these was for use with routers but the use of a virtual IP can be applicable more broadly. 

On Linux, keepalived is commonly used for this purpose. There is a freevrrpd in FreeBSD ports, but there is also a solution in the base system in the form of carp(4). The CARP (Common Address Redundancy Protocol) protocol was developed by the OpenBSD community in response to patent issues with VRRP.

A virtual IP works especially well for stateless services—such as a simple router, which does not need to maintain state between successive packets or connections. Where state data is needed, the failover protocol must synchronise it to the backup server. 

When a router does connection tracking—such as for NAT—it is no longer stateless. pfsync(4) was built to work with carp and the pf firewall for this case. Another example of a service that only holds minimal state data that can be easily synchronised would be a DHCP server that only needs to keep track of the leases on IP addresses that it has handed out. 

Although these are relatively simple examples, CARP may also be used as a fail-over mechanism for more substantial services.

Traffic Shaping

Traffic shaping is used to control the priorities of different packets on a network. It can be used to throttle the bandwidth for certain types of traffic or to ensure some minimum throughput is available for more important traffic. 

On Linux, traffic shaping is controlled via the tc tool. On FreeBSD, the equivalent is dummynet(4) with ipfw and altq(4)with pf. Since FreeBSD 13, dummynet has been separated out from ipfw to also work with the pf firewall and now has it’s own dnctl(8) command. (We cover dummynet in more detail in “Dummynet: The Better Way of Building FreeBSD Networks”.)

The marketing from Internet Service Providers tends to stress bandwidth figures, but the delays due to poor latencies can have a significant impact on your experience of the actual speed of a network link. In particular you may have come across the term bufferbloat. 

Bufferbloat occurs when what should be a quick burst of network traffic gets queued up in a big buffer full of packets corresponding to large bulk transfers, resulting in increased network latency. With dummynet, bufferbloat problems can be alleviated by selecting a different packet scheduling algorithm such as fq_codel.

Conclusion

FreeBSD has a mature and reliable network stack with support for many advanced technologies, and a great breadth of practical functionality. Although FreeBSD and Linux differ significantly in both naming conventions and some underlying technologies, each offers mature, high-performance, and reliable networking to the system administrator who learns to use them properly.

Additional Resources

Here are links to further resources that you may find useful:

FreeBSD Handbook, Chapter 32. Advanced Networking

FreeBSD man page – ifconfig(8)

FreeBSD man page – if_bridge(4)

FreeBSD man page – ipfw, dnctl(8)

FreeBSD man page – pfctl(8)

FreeBSD man page – setfib(1)

FreeBSD man page – lagg(4)

FreeBSD man page – carp(4)