Packet loopback code#

Packet loopback code#

This turns out to be somewhat complicated and, as we'll see, the Linux routing table code doesn't have high test coverage for doing this when dealing with the loopback interface and virtual ethernet pairs.Īnother piece of Mesos background is that when we introduced the port range based isolation, we were using an old kernel which had a few bugs that required us to turn off the reverse path filter. Mesos uses Linux Traffic Control (TC) filters and actions to move packets between containers and the host (per above, this is based on the port). In particular, we have to explicitly bring back the local network communications between different containers on the same host after isolation. The port range based network isolation is conceptually similar to the transport layer and the IP layer communication works as a whole stack, as normal. Packets go into one end of the pair and come out from the other end of the pair. To communicate between containers on the same host, we create a virtual ethernet pair for each container. Our container configuration isolates the entire TCP/IP stack with network namespaces, including routing tables, IP addresses, as well as the loopback interface, which is used for all local network communications. Instead, each host gets an IPv4 address and each container gets a port range. We have too many datacenter machines to assign an IPv4 address per container.

Setting this to 2 turns off 'strict' mode and enables 'loose' mode, which relaxes the interface matching rule, allowing packets that don't match the arrival interface to be received. This means that the sysctl .rp_filter is set to 1. The reverse path filter is enabled by default on many Linux distros. For example, loopback IP addresses in the subnet 127.0.0.0/8 should never appear on any hardware interface attached to the wire and will therefore be identified as martians and dropped. If not, the packet is considered a 'martian' address and dropped before it enters the IP stack. If the lookup succeeds and the next hop device returned by routing table matches the arrival interface, the packet is accepted. The reverse path filter is a mitigation against DDoS attacks that does a reverse routing table lookup for each incoming IP packet on the arrival interface by source IP address. To understand what happened, we'll need some background about a few different parts of our stack: Let's take a look at what this means and what happened. Bringing in our previously good firewall configuration didn't fix this because the default sysctl settings ran into a corner case kernel bug. Totally resetting the firewall gave us the default CentOS configuration, which blocks all traffic except for ssh. We expected resetting the firewall configuration to fix the issue, but resetting the firewall configuration exposed a kernel bug! Earlier last year, we identified a firewall misconfiguration which accidentally dropped most network traffic.