Shamsuddin AhmedSenior Software Engineer

Security From the Ground Up06 / 07

Binary Exploitation: When Data Overwrites Code's Own Bookkeeping

This is the master bug at its most fundamental. In SQL injection data became database code; in XSS it became browser code. Here, in a program written in C or C++, data written past the end of a buffer overwrites the program’s own control information, and the program starts doing what the attacker’s data says. This post is conceptual and defensive: how the class of bug works, why it was so devastating, and the four layers that turned it from routine to hard. Everything here is for understanding and prevention, on systems you own.

#1. The ground: the stack, and what lives next to your data

When a function runs, its local variables live on the stack, the region of memory from the pointers post. The stack also holds, right next to those locals, the bookkeeping the program needs to return: crucially, the return address, the location in the code to jump back to when the function finishes.

That adjacency is the whole problem. Your buffer and the return address are neighbours in memory, and C does not check that you stay inside your buffer.

   higher addresses
   +-----------------------------+
   |  return address             |  <- where the function jumps when done
   +-----------------------------+
   |  saved frame pointer        |
   +-----------------------------+
   |  char buffer[16]            |  <- your local array
   |                             |
   +-----------------------------+
   lower addresses

   writing past the end of buffer climbs UP toward the return address.

#2. The bug: writing past the end

Recall from the pointers post that an array in C has no memory of its own length, and nothing stops you writing past it. Here is the canonical unsafe pattern:

#include <stdio.h>

void greet(void) {
    char name[16];
    gets(name);           // reads input with NO limit on length
    printf("Hello, %s\n", name);
}

gets reads input until a newline, with no idea that name is only 16 bytes. Give it 16 characters and it fits. Give it 100 characters and it writes all 100, starting at name and climbing up the stack, straight over the saved frame pointer and the return address. This is a buffer overflow, and gets is so irredeemable that it was removed from the C standard library entirely.

At first this just crashes: the return address is now garbage, the function tries to jump to garbage, the program dies with a segmentation fault. A crash from too much input is already a denial-of-service bug. But the deeper danger is that the overwriting bytes are chosen by the attacker.

#3. The exploit, at the level you need to understand it

If the attacker controls the bytes that land on the return address, they control where the program jumps when the function returns. Instead of returning to the caller, it “returns” to an address the attacker picked.

In the earliest and simplest form of the attack, the attacker put their own machine-code instructions into the very buffer they were overflowing, and set the return address to point back at that buffer. The function returns into attacker-supplied code, which now runs with the program’s privileges. Data, the input, had become code, executed directly.

  attacker's input, longer than the buffer:

  [ attacker's code .... ][ padding ][ address of the buffer ]
    ^                                  ^
    lands in buffer[]                  overwrites return address,
                                       pointing back at the code

I am describing this at the level of the idea, not handing you a working exploit, because the point here is not to build one. The point is to understand why a memory-safety bug is not a mere crash but a full compromise, and therefore why the defences below, and memory-safe languages, matter so much. A single unchecked write can hand an attacker the program.

Related members of the same family, worth knowing by name: use-after-free (using a pointer after the memory was freed, so an attacker who controls the reused memory controls the program), double free, and integer overflow that produces a too-small buffer which is then overflowed. All are the pointer bugs from part 5, turned into weapons.

#4. The four defences that made it hard

Exploitation used to be routine. It is now difficult, not because the bugs vanished, but because four layers were added, each one breaking a step of the attack. Modern systems run all four by default, and understanding them tells you what still has to hold.

Non-executable memory (NX, or DEP). The processor marks the stack as data, not code, and refuses to execute instructions from it. This directly kills the classic attack of section 3: the code the attacker placed in the buffer will not run, because the stack is no longer executable. Attackers responded with “return-oriented programming”, stitching together existing snippets of the program’s own legitimate code, which is why NX is necessary but not sufficient.

Stack canaries. The compiler places a random value, a “canary”, between the local buffers and the return address, and checks it just before the function returns. An overflow that reaches the return address must first overwrite the canary; the check sees the canary changed and aborts the program before it can jump anywhere. It converts many overflows back into safe crashes.

   +---------------------+
   |  return address     |
   +---------------------+
   |  canary (random)    |  <- checked before return; if changed, abort
   +---------------------+
   |  buffer[16]         |
   +---------------------+

Address space layout randomisation (ASLR). The operating system loads the program’s pieces at random addresses each run. The attacker in section 3 needed to know the address of their buffer to set the return address; ASLR means that address is different every time and cannot be predicted, so a hard-coded address fails. Combined with position-independent executables (PIE), it randomises the program’s own code too.

Memory-safe languages. The deepest fix is to not have the bug. Languages that check bounds on every array access (which is nearly everything outside C and C++), and Rust’s ownership system which prevents use-after-free at compile time, remove this entire class of vulnerability by construction. The cost is either a runtime check or a stricter compiler, and for the vast majority of software it is a cost worth paying. This is why so much new systems software is being written in Rust: not fashion, but the elimination of the bug class this whole post is about.

Together these are defence in depth in its purest form: NX stops one technique, ASLR another, canaries a third, and a safe language stops all of them. An attacker now typically needs to defeat several at once, often by first finding a separate bug that leaks an address to beat ASLR, which is far harder than the original single overflow.

#5. What this means if you write C or C++

You will not add NX or ASLR yourself; the toolchain and OS do. Your job is to not write the overflow in the first place, and the rules are concrete:

  • Never use the unbounded functions. gets, strcpy, strcat, sprintf, scanf("%s") all write without a length limit. Use the bounded versions (fgets, strncpy, snprintf) and pass the buffer size, or better, use std::string and std::vector in C++ which manage their own length.
  • Carry the length with the buffer, always. The pointers post’s lesson: a raw array has forgotten its size, so you must track it and check every index against it.
  • Prefer the standard library’s containers over raw memory. std::vector::at() bounds-checks. std::string grows itself. The whole reason to use them, beyond convenience, is that they close this hole.
  • Turn on the compiler’s help. Warnings (-Wall -Wextra), the sanitizers (-fsanitize=address,undefined) during testing, and the hardening flags. AddressSanitizer in particular catches overflows and use-after-free at the moment they happen, in testing, before they ship.
  • Where you can choose the language, and safety matters, choose a memory-safe one. The single most effective thing you can do about this entire bug class is to write the code in a language that does not have it.

#The short version

  • On the stack, your local buffers sit right next to the return address, the bookkeeping that says where the function jumps when it finishes. C does not check that you stay inside your buffer.
  • A buffer overflow writes past the end of a buffer, climbing over that bookkeeping. At first it just crashes, which is already a denial-of-service; worse, the overwriting bytes are attacker-chosen.
  • If the attacker controls the bytes on the return address, they control where the program jumps. Classically they made it jump into code they placed in the buffer, so their data ran as code with the program’s privileges.
  • The same family includes use-after-free, double free, and integer overflows that make an undersized buffer. All are the pointer bugs from part 5 turned into weapons.
  • Four layers made it hard: non-executable memory (the buffer’s code will not run), stack canaries (an overflow trips a guard value and aborts), ASLR (addresses are randomised so they cannot be predicted), and memory-safe languages (the bug cannot exist).
  • If you write C or C++: never use unbounded string functions, carry the length with every buffer, prefer std::string and std::vector, and run the sanitizers in testing. Where you can choose the language and safety matters, choose one without the bug.

Next: network security, where the confusion is not data and code but who you are actually talking to, and cryptography is the answer.