Using eBPF XDP At The Chaturbate Edge

Achieve high performance, low latency packet processing and filtering without traditional nftables/iptables firewalling or hacks like tc ingress hooks at streaming servers to drop typical DoS attack flows without any upstream mitigation support.

Berkeley Packet Filter

Enhancements to Berkeley Packet Filter (BPF) were added to Linux 4.x series kernels, allowing BPF to do much more than packet filtering. Custom analysis programs were developed in the form of eBPF Tracing Tools

There are already over 100 tools that can be used to generate histograms of IO operations, analyzing packet issues like TCP retransmits, and for doing security scrubbing of packets.  Combined with Linux Express Data Path (XDP) merged in kernel 4.8, there is now an available early hook in the receive RX path of the kernel, and subsequently a user-supplied eBPF program can decide the fate of the packet before any SKBs (sk_buff or socket buffers) are allocated.

skb = alloc_skb(len, GFP_KERNEL);

There is no waiting for memory allocation. XDP can drop 26 Mpps per core with commodity hardware . There are various scales of increasing packet drop throughput levels going from several thousand pps all the way up to 1.8M pps with traffic control (tc) ingress hooks, but the markedly fast option uses XDP.

Express Data Path

XDP is the Linux programmable packet processor. We can define an eBPF program to process a packet before it enters the in-kernel stack.  The goal is to decrease the number of instructions executed per packet. XDP does not require busy polling for packets, enabling it to be "always on". If a packet drop is destined to occur, it should happen as quickly as possible and with the shortest logical path.

Arthur Fabre has written a paper about the use of XDP eBPF for DDoS filtering, and has blogged about it several times. Chaturbate does not currently use upstream filtering/mitigation for client video server traffic because the distribution of the Chaturbate video servers are already globally distributed over a network capacity of several terabits per second with throughput consistently over 1 Tbps.  We want to achieve the same packet filtering throughput as Cloudflare but on our own hardware fanned out to every data center. Can it be done?

Program & Load The XDP Packet Filter

We can use a reference C module that Cloudflare provides at  https://raw.githubusercontent.com/cloudflare/cloudflare-blog/master/2018-07-dropping-packets/xdp-drop-ebpf.c (GPL) and then tweak the XDP_PASS and XDP_DROP logic to fit our own criteria.  The program will match and drop any IP packet that uses UDP as well as match destination subnet.  As previously stated, the XDP flow runs early on the RX hook and will process before any iptables PREROUTING.

At a very basic level, an eBPF XDP program will map the entrypoint function of the module to a named code segment in the object file that must be emitted for the function even if the function is not referenced, receive the start and end pointers of a packet to process, check the ethernet packet, and then return either an XDP_DROP  or an XDP_PASS result:

#include <linux/bpf.h>
#include <linux/if_ether.h>

#define SEC(NAME) __attribute__((section(NAME), used))

SEC("xdp_entrypoint")
int xdp_entrypoint(struct xdp_md *ctx) {
    void *data_end = (void *)(long)ctx->data_end;
    void *data = (void *)(long)ctx->data;
    struct ethhdr *eth = data;
    if (match_ethernet_packet(eth)) {
        action = XDP_PASS;
    }
    return XDP_DROP;  /* DEFAULT DROP */
}

To load the eBPF module, we will use the Linux iproute2 command to inject the bytecode into the appropriate network device (note you may need to upgrade iproute2  at https://mirrors.edge.kernel.org/pub/linux/utils/net/iproute2/) :

ip link set dev ext0 xdp obj xdp-ebpf-drop.o

We do some simple filtering and compile using clang > v3.7 to emit the BPF bytecode and then load it:

clang -Wall -O2 -emit-llvm -c xdp-drop-ebpf.c -S -o - | llc -march=bpf -filetype=obj -o xdp-drop-ebpf.o

The full module:

#include <linux/bpf.h>
#include <linux/if_ether.h>
#include <linux/in.h>
#include <linux/ip.h>
#include <linux/ipv6.h>
#include <linux/udp.h>

#include <stdint.h>

#define SEC(NAME) __attribute__((section(NAME), used))

#define htons(x) ((__be16)___constant_swab16((x)))
#define htonl(x) ((__be32)___constant_swab32((x)))

struct vlan_hdr {
	__be16 h_vlan_TCI;
	__be16 h_vlan_encapsulated_proto;
};

SEC("prog")
int xdp_drop_bad_cb_traffic(struct xdp_md *ctx)
{
	void *data_end = (void *)(long)ctx->data_end;
	void *data = (void *)(long)ctx->data;
	struct ethhdr *eth = data;

	uint64_t nh_off = sizeof(*eth);
	if (data + nh_off > data_end) {
		return XDP_PASS;
	}

	uint16_t h_proto = eth->h_proto;
	int i;

	/* Handle double VLAN tagged packet. See https://en.wikipedia.org/wiki/IEEE_802.1ad */
	for (i = 0; i < 2; i++) {
		if (h_proto == htons(ETH_P_8021Q) || h_proto == htons(ETH_P_8021AD)) {
			struct vlan_hdr *vhdr;

			vhdr = data + nh_off;
			nh_off += sizeof(struct vlan_hdr);
			if (data + nh_off > data_end) {
				return XDP_PASS;
			}
			h_proto = vhdr->h_vlan_encapsulated_proto;
		}
	}

	if (h_proto == htons(ETH_P_IP)) {
		struct iphdr *iph = data + nh_off;
		struct udphdr *udph = data + nh_off + sizeof(struct iphdr);
		if (udph + 1 > (struct udphdr *)data_end) {
			return XDP_PASS;
		}
		if (iph->protocol == IPPROTO_UDP) {
			return XDP_DROP;
		}
		if ((htonl(iph->daddr) & 0xFFFFFF00) != 0x83992900) {
			// not destined to 131.153.46.0/24
			return XDP_DROP;
		}
	} else if (h_proto == htons(ETH_P_IPV6)) {
		struct ipv6hdr *ip6h = data + nh_off;
		struct udphdr *udph = data + nh_off + sizeof(struct ipv6hdr);
		if (udph + 1 > (struct udphdr *)data_end) {
			return XDP_PASS;
		}
		if (ip6h->nexthdr == IPPROTO_UDP) {
			return XDP_DROP;
		}
	}

	return XDP_PASS; /* send everything else to nftables */
}

char _license[] SEC("license") = "GPL";

After compilation, the symbol table looks like

$ objdump -t xdp-drop-ebpf.o 

xdp-drop-ebpf.o:     file format elf64-little

SYMBOL TABLE:
0000000000000130 l       prog	0000000000000000 LBB0_10
0000000000000170 l       prog	0000000000000000 LBB0_12
00000000000001d8 l       prog	0000000000000000 LBB0_13
00000000000000c0 l       prog	0000000000000000 LBB0_14
0000000000000110 l       prog	0000000000000000 LBB0_16
0000000000000068 l       prog	0000000000000000 LBB0_3
0000000000000098 l       prog	0000000000000000 LBB0_5
0000000000000180 l       prog	0000000000000000 LBB0_7
0000000000000000 l    d  prog	0000000000000000 prog
0000000000000000 g       license	0000000000000000 _license
0000000000000000 g       prog	0000000000000000 xdp_drop_bad_cb_traffic

After attaching the code to our external interface, ip link shows xdp :

Now the ixgbe intel ethernet driver does not have XDP counters, but it does update the ifconfig counters appropriately. Turning off iptables/nftables and checking for dropped packets, we see the counter incrementing. There is also a way to update statistics yourself a la https://github.com/xdp-project/xdp-tutorial/blob/master/common/xdp_stats_kern.h

Chaturbate is looking for software engineers! If you are interested please drop a line at  programmingjobs@chaturbate.com.