Kernel Bypass: When System Calls Became the Enemy

I still remember the exact moment I realized we were in serious trouble. It was a Tuesday morning, the European markets had just opened, and our brand new colocation server was performing worse than the old box sitting in our office. Not marginally worse - catastrophically worse. We were seeing 8 microsecond round-trip times for orders that should have taken 2. The kicker? We were literally sitting in the same data center as the exchange, connected via cross-connect cables that cost more per month than most people’s rent.

The problem wasn’t the network. It wasn’t the CPU. It wasn’t even our code, really. The problem was that between us and the network card sat this thing called the Linux kernel, and every time we wanted to send a packet, we had to ask its permission. That permission slip - the system call - was costing us 500 nanoseconds each way. Multiply that by two system calls per packet (send and receive), add in the context switches, the copies, the interrupt handling, and suddenly you understand why we were getting destroyed by firms that had figured out how to bypass the kernel entirely.

Let me tell you about the six-month journey that took us from standard sockets to kernel bypass, why we ended up writing our own network stack, and how we learned that sometimes the operating system is just overhead you can’t afford.

The Anatomy of a System Call Disaster

First, let me show you what actually happens when you call send() on a standard TCP socket. This is the “simple” path that everyone learns in networking 101: