Reverse Engineering Basics

intermediate45 minWriteup

Introduction to reversing for CTFs

Learning Objectives

  • Use disassemblers
  • Understand assembly basics
  • Find flags in binaries
  • Patch executables

Reverse engineering is the art of understanding how software works without source code. In CTFs, you'll analyze binaries to find hidden flags, bypass protections, or understand malware. It's like solving a puzzle where the instructions are in machine code!

Reverse engineering has a steep learning curve. Don't be discouraged! Start with simple challenges and gradually tackle harder ones. Every expert started by being confused by assembly.

Basic Concepts

1606070;"># What you're dealing with:
2 
3606070;"># Source Code (human-readable)
4if (password == 606070;">#a5d6ff;">"secret") { success(); }
5 
6606070;"># Assembly (low-level, human-readable-ish)
7cmp     eax, 0x736563726574
8je      success
9 
10606070;"># Machine Code (binary, CPU-readable)
113D 74 65 73 74 74 0A
12 
13606070;"># Your job: Go from machine code back to understanding
14 
15606070;"># File types:
16606070;"># - ELF: Linux executables
17606070;"># - PE: Windows executables (.exe, .dll)
18606070;"># - Mach-O: macOS executables

First Steps

bash
1606070;"># 1. Identify the file
2file binary
3606070;"># ELF 64-bit LSB executable, x86-64, dynamically linked
4 
5606070;"># 2. Check for strings
6strings binary | head -50
7strings binary | grep -i flag
8strings binary | grep -i pass
9 
10606070;"># 3. Run it (carefully! Use a VM)
11chmod +x binary
12./binary
13 
14606070;"># 4. Check linked libraries
15ldd binary
16 
17606070;"># 5. Quick analysis
18objdump -d binary | head -100
19nm binary  606070;"># List symbols

Strings First!

Many CTF reverse challenges have the flag or password in plaintext strings. Always run strings before diving into disassembly!

Essential Tools

bash
1606070;"># Ghidra - Free, powerful, NSA-developed
2606070;"># Download from ghidra-sre.org
3606070;"># GUI disassembler + decompiler
4 
5606070;"># Start Ghidra
6ghidraRun
7606070;"># Create project → Import binary → Analyze
8 
9606070;"># IDA Free - Industry standard (free version limited)
10606070;"># Download from hex-rays.com
11 
12606070;"># Cutter - Radare2 GUI (beginner-friendly)
13606070;"># rizin + cutter
14 
15606070;"># radare2 - Powerful CLI tool
16r2 binary
17[0x00401000]> aaa      606070;"># Analyze all
18[0x00401000]> afl      606070;"># List functions
19[0x00401000]> pdf @main 606070;"># Disassemble main
20 
21606070;"># Binary Ninja - Commercial but good (free cloud version)
22 
23606070;"># GDB - GNU Debugger
24gdb ./binary
25(gdb) run
26(gdb) break main
27(gdb) disassemble main

Assembly Basics

nasm
1; x86-64 Assembly Crash Course
2 
3; Registers (64-bit / 32-bit / 16-bit / 8-bit)
4; RAX / EAX / AX / AL - Accumulator
5; RBX / EBX / BX / BL - Base
6; RCX / ECX / CX / CL - Counter
7; RDX / EDX / DX / DL - Data
8; RSI / ESI / SI / SIL - Source Index
9; RDI / EDI / DI / DIL - Destination Index
10; RSP / ESP / SP / SPL - Stack Pointer
11; RBP / EBP / BP / BPL - Base Pointer
12; RIP / EIP / IP - Instruction Pointer
13 
14; Common instructions:
15mov rax, rbx      ; Copy rbx to rax
16push rax          ; Push rax onto stack
17pop rbx           ; Pop from stack into rbx
18add rax, 5        ; rax = rax + 5
19sub rax, 3        ; rax = rax - 3
20xor rax, rax      ; rax = 0 (clear register)
21cmp rax, rbx      ; Compare rax and rbx (sets flags)
22jmp label         ; Unconditional jump
23je label          ; Jump if equal (ZF=1)
24jne label         ; Jump if not equal (ZF=0)
25call function     ; Call function
26ret               ; Return from function
27 
28; Function calling (x64 Linux):
29; Arguments: RDI, RSI, RDX, RCX, R8, R9
30; Return value: RAX
You don't need to memorize all instructions. Learn the common ones (mov, cmp, jmp, call, ret) and look up others as needed!

Using Ghidra

Ghidra Workflow

1
Import BinaryFile → Import, select binary, accept defaults
2
AnalyzeWhen prompted, run auto-analysis (accept defaults)
3
Find MainSymbol Tree → Functions → main (or entry)
4
Read DecompilerRight panel shows C-like decompiled code
5
Follow CallsDouble-click function calls to navigate
1606070;"># Ghidra tips:
2 
3606070;"># Rename variables (make code readable)
4Right-click variable → Rename Variable
5 
6606070;"># Change type
7Right-click → Retype Variable
8 
9606070;"># Add comments
10; at end of line for end-of-line comment
11Right-click → Comments
12 
13606070;"># Search
14Search → Memory → String → 606070;">#a5d6ff;">"flag"
15Search → For Strings
16 
17606070;"># Cross-references (where is this used?)
18Right-click → References → Show References to

Dynamic Analysis with GDB

bash
1606070;"># Enhanced GDB with pwndbg or GEF
2606070;"># Install pwndbg: https://github.com/pwndbg/pwndbg
3 
4gdb ./binary
5 
6606070;"># Basic commands
7run                    606070;"># Run program
8break main             606070;"># Set breakpoint at main
9break *0x401234        606070;"># Break at address
10continue               606070;"># Continue execution
11step                   606070;"># Step one instruction (into)
12next                   606070;"># Step over function calls
13finish                 606070;"># Run until function returns
14 
15606070;"># Examine memory/registers
16info registers         606070;"># Show all registers
17print $rax             606070;"># Print register value
18x/10x $rsp             606070;"># Examine 10 hex values at stack
19x/s 0x401234           606070;"># Examine as string
20 
21606070;"># Modify execution
22set $rax = 1           606070;"># Change register
23set {int}0x601040 = 0  606070;"># Change memory
24 
25606070;"># With pwndbg:
26context                606070;"># Show registers, stack, disasm
27vmmap                  606070;"># Memory map
28search 606070;">#a5d6ff;">"flag"          # Search memory

CTF Reverse Tricks

bash
1606070;"># Bypass password checks:
2 
3606070;"># 1. Find the comparison in Ghidra
4606070;"># Look for strcmp, strncmp, or byte comparisons
5 
6606070;"># 2. Patch the binary
7606070;"># In Ghidra: Right-click instruction → Patch Instruction
8606070;"># Change JE (jump if equal) to JMP (always jump)
9 
10606070;"># 3. Or use GDB to set comparison result
11(gdb) break *0x401234  606070;"># Before comparison
12(gdb) run
13(gdb) set $eax = 0     606070;"># Make comparison succeed
14(gdb) continue
15 
16606070;"># Common patterns:
17 
18606070;"># XOR'd strings (hide flag from strings)
19for i in range(len(data)):
20    result += chr(data[i] ^ key[i % len(key)])
21 
22606070;"># Flag built character by character
23flag[0] = 606070;">#a5d6ff;">'f'
24flag[1] = 606070;">#a5d6ff;">'l'
25flag[2] = 606070;">#a5d6ff;">'a'
26flag[3] = 606070;">#a5d6ff;">'g'
27 
28606070;"># Flag computed at runtime
29606070;"># Set breakpoint after computation, examine memory

Knowledge Check

Quick Quiz
Question 1 of 2

What should you try first when reversing a CTF binary?

Key Takeaways

  • strings first - many CTF flags are in plaintext
  • Ghidra's decompiler shows C-like code (much easier than assembly)
  • Learn basic x86 instructions: mov, cmp, jmp, call, ret
  • GDB lets you modify execution in real-time
  • Patching binaries can bypass password checks
  • Start simple - reverse engineering has a learning curve