Category: network

Open Compute Project 2019

OCP has the mission to “design and enable the delivery of the most efficient server, storage and data center hardware designs for scalable computing”.

OCP had its global 2019 summit recently. Some interesting trends on hyperscale networks are discussed here and here with the use of F16 fabric network with its a focus on higher bandwidth but also performance at the right cost instead of at any cost. The heart of this new F16 fabric is the Minipack switch, with contribution from Arista which Facebook says will consume 50 percent less power and space than the Backpack switch it replaces in the network.  It is a 128x100Gb switch and uses a Broadcom Tomahawk-3 Asic. Quote: “a path from a rack in one building to a rack in another building over Fabric Aggregator was as many as 24 hops long before. With F16, same-fabric network paths are always the best case of six hops, and building-to-building flows always take eight hops. This results in half the number of intrafabric network hops and one-third the number of interfabric network hops between servers.”

Intel announced an industry collaboration around Platform Root of Trust at the Open Compute Project 2019 summit.

There’s a talk on Stratum and the use of P4 and Switch Abstraction Interface (SAI) for SDN, by Open Networking Foundation (ONF) and Google. Tencent has a use case for disaggregating their monolithic network into a modular switch with a network of controllers instead of a single controller.

Smaller data centers at the edge is another trend.

VPNFilter IoT Router Malware

Over 500k routers and gateways are estimated to be infected with malware dubbed VPNFilter, first reported in .

It has 3 stages. In stage 1 it adds itself to crontab to remain after a reboot. In stage 2 it adds a plugin architecture. In stage 3 it adds modules which instruct it to do specific things.  A factory reset and router restart in protected network was recommended to remove it. Disabling remote administration and changing passwords is recommended to prevent reinfection.

The 3rd stage module modifies IPtables rules, enabling mitm attacks and javascript injection.

The first action taken by the ssler module is to configure the device’s iptables to redirect all traffic destined for port 80 to its local service listening on port 8888. It starts by using the insmod command to insert three iptables modules into the kernel (ip_tables.ko, iptable_filter.ko, iptable_nat.ko) and then executes the following shell commands:

  • iptables -I INPUT -p tcp –dport 8888 -j ACCEPT
  • iptables -t nat -I PREROUTING -p tcp –dport 80 -j REDIRECT –to-port 8888
  • Example: ./ssler logs src: dst:

-A PREROUTING -s -d -p tcp -m tcp –dport 80 -j REDIRECT –to-ports 8888

To ensure that these rules do not get removed, ssler deletes them and then adds them back approximately every four minutes.

More behaviors of the malware are described at including photobucket request, fake CA certs claiming Microsoft issued them and ipify lookups.

YARA rules for detection –

YARA (yet another recursive acronym) is a format to specify rules match malware based on string patterns, regular expressions and their frequency of occurrence. A guide to writing effective ones is here.

User-Agent rule –

Ipify self-ip address querying service, with json output.


Traffic limits with HAProxy stick-table

A traffic rate limiting feature is required to keep an HTTP website backend safe from abusive or malfunctioning clients.  This requires the ability to track user sessions of a particular type and/or from a given IP address. HAProxy is an HTTP proxy which (when configured as reverse proxy to protect a website), receives client requests in its frontend and sends those requests to servers in its backend.   The config file has corresponding frontend and backend sections. Haproxy also has an in-memory table to store state related to incoming HTTP connections, indexed by a key such as client IP address.  This table is called a stick-table – it is enabled using the ‘stick-table’ directive in the haproxy config file.

The stick-table directive allows  specifying the key, the size of the table, the duration an entry (key) is kept in seconds and various counts such as currently active connections, connection rate, http request rate, http error rate etc.

Stick tables are very useful for rate-limiting traffic and tagging traffic that meets certain criteria such as a high connection or error rate with a header which can be used by the backend to log the traffic.

The origin of this rate-limiting feature request along with an example is at . Serverfault is a high traffic website so it is a good indication if the feature works for them.

frontend http
    bind *:2550

stick-table type ip size 200k expire 10m store gpc0

# check the source before tracking counters, that will allow it to
# expire the entry even if there is still activity.
acl whitelist src
acl source_is_abuser src_get_gpc0(http) gt 0
use_backend ease-up-y0 if source_is_abuser
tcp-request connection track-sc1 src if ! source_is_abuser

acl is_test1 hdr_sub(host) -i
acl is_test2 hdr_sub(host) -i

use_backend test1  if is_test1
use_backend test2  if is_test2

backend test1 
stick-table type ip size 200k expire 30s store conn_rate(100s),bytes_out_rate(60s) 
acl whitelist src

# values below are specific to the backend
tcp-request content  track-sc2 src
acl conn_rate_abuse  sc2_conn_rate gt 3
acl data_rate_abuse  sc2_bytes_out_rate  gt 20000000

# abuse is marked in the frontend so that it's shared between all sites
acl mark_as_abuser   sc1_inc_gpc0 gt 0
tcp-request content  reject if conn_rate_abuse !whitelist mark_as_abuser
tcp-request content  reject if data_rate_abuse mark_as_abuser

server local_apache localhost:80

Note that the frontend and backend sections have their own stick-table sections.

A general strategy would be to allow enough buffer for legitimate traffic to pass in, drop abnormally high traffic and flag intermediate risk traffic to the backend so it can either drop it or log the request for appropriate action, including potentially adding the IP to an abusers list for correlation, reverse lookup and other analysis. These objectives are achievable with stick-tables.

An overview of the HAProxy config file with the sections global, defaults, frontend, backend is here.

Stick tables use elastic binary trees-

Related, for analysis of packet captures in DDoS context, a useful tool is python dpkt – .



Linux Bridges and Firewalls for Containers

Hubs and Repeaters work on bits on Layer1. Bridges and Switches work on frames, the protocol data unit for Layer2. A Hub supports a star configuration – like a splitter it simply forwards traffic received on one segment to all the others. A Repeater amplifies signal for longer distances. A Bridge is a smarter type of Hub, usually implemented in software, which looks at the destination MAC address of the packet before forwarding to the segment with MAC matching that destination, thereby avoiding unnecessary network traffic on all the other segments. It does not look at IP addresses. A Switch is a reincarnation of a Bridge in hardware with several ports.  The basic operation of a Bridge/Switch is L2 forwarding, which includes ARL (Address Resolution Logic) table learning and ageing.  An L3 Router switches packets based on the IP address instead of the frame address.

A linux bridge from a VM to the Host, does not need an IP address, but it is a good idea if you want to talk to the host for management or set firewall rules on it.

To create a bridge br0 between two interfaces eth0 and eth1, one runs commands: brctl addbr br0, brctl stp br0 on, brctl addif br0 eth0, brctl addif br0 eth1.

Since a bridge is an L2 construct, how can one filter IP addresses with it. A quote: An Ethernet interface cannot usefully have an IP address if it is also attached to a bridge. However it is possible to achieve the same effect by binding an address to the bridge itself:

ifconfig br0

An example of firewall configuration on a bridge.

Troubleshooting bridges:

brctl showmacs br0

Docker creates a few bridges by default. A primer on Docker Container networking and the bridges it creates are here and  here. Here are three common ones.

bridge: docker0 bridge . This is the default, allowing all containers to talk to each other

host: container listens to host, without any isolation.

Some problems with host iptables rules reveal themselves in working with docker.

A look at firewalld, ebtables, ipset, nftables at . The command line tools talk to the kernel over a netlink socket.

A more extensible approach is with Berkeley Packet Filter (BPF), the classical one now called cBPF, the extended successor called eBPF.   A very basic problem is filtering HTTP or DNS packets based on the destination hostname, rather than the IP address.  This is discussed in an example by Cloudflare here.