CTF Reverse Engineering for Beginners: file, strings, First Flag

Short answer: most beginner CTF reversing challenges fall to the same four moves, in this order. First, identify the file: is it ELF, PE, Mach-O, Python bytecode, an APK or WebAssembly? Second, run strings, because easy challenges sometimes leave the flag in plain text. Third, read the imports and sections to guess how the input gets checked. Fourth, read the check itself in a disassembler or decompiler and write a small script that reverses it. This guide walks through all four on two practice binaries you build yourself. The first one falls to strings. The second one doesn’t, and it shows why.

Who this is for: people new to CTFs who opened a “rev” challenge, saw a binary file and didn’t know where to start. You need a terminal and a basic idea of what a C program is. No assembly knowledge is assumed, and you’ll pick up the bits you need along the way.

What you will learn:

• How to recognize a file type and pick the right tool for it
• What strings can and can’t find, and why
• How imports and section sizes hint at the checking logic
• How to read a short checking loop and turn it into a solver
• How to prove your flag is right by running the original program

Where beginner challenges come from

You can practice legally on platforms built for it. picoCTF, from Carnegie Mellon University, describes its picoGym as a noncompetitive practice space with challenges from past competitions. As of 2026-10-14, picoctf.org announces that “picoCTF.org is now CyLab Security Academy”, so the branding may differ when you visit. CTFlearn describes itself as “the most beginner-friendly way to learn cyber security”. crackmes.one calls itself “a simple place where you can download crackmes to improve your reverse engineering skills”. Its files are user-submitted, so run them only inside a virtual machine.

For this guide we’ll build our own challenges instead. That way you can check every step against the source, and there’s no question about what you’re allowed to do with the file.

Build the practice binaries

Save these two files. Try not to read level2.c closely yet, since figuring it out is the exercise.

// level1.c: the flag is compared as a plain string.
#include <stdio.h>
#include <string.h>

int main(void) {
    char buf[64];
    printf("Enter the flag: ");
    if (!fgets(buf, sizeof buf, stdin)) return 1;
    buf[strcspn(buf, "\n")] = '\0';
    if (strcmp(buf, "practice{strings_finds_plain_text}") == 0) {
        puts("Correct!");
        return 0;
    }
    puts("Nope.");
    return 1;
}

// level2.c: the flag is stored XOR-encoded, so strings cannot see it.
#include <stdio.h>
#include <string.h>

static const unsigned char enc[] = {
    0x1a, 0x18, 0x0b, 0x09, 0x1e, 0x03, 0x09, 0x0f, 0x11, 0x12, 0x05, 0x18,
    0x35, 0x02, 0x03, 0x0e, 0x0f, 0x19, 0x35, 0x1e, 0x02, 0x0f, 0x35, 0x08,
    0x13, 0x1e, 0x0f, 0x19, 0x17
};
#define KEY 0x6a

static int check(const char *s) {
    if (strlen(s) != sizeof enc) return 0;
    for (size_t i = 0; i < sizeof enc; i++)
        if ((unsigned char)(s[i] ^ KEY) != enc[i]) return 0;
    return 1;
}

int main(void) {
    char buf[64];
    printf("Enter the flag: ");
    if (!fgets(buf, sizeof buf, stdin)) return 1;
    buf[strcspn(buf, "\n")] = '\0';
    puts(check(buf) ? "Correct!" : "Nope.");
    return 0;
}

clang -O2 -o level1 level1.c      # or: gcc -O2 -o level1 level1.c on Linux
clang -O2 -o level2 level2.c

I built and ran everything below on an Apple Silicon Mac with Apple clang 21, so my files are Mach-O arm64. On Linux with gcc you’ll get ELF x86-64 files instead. The steps are the same, but addresses, register names and some tool output will differ.

Step 1: Identify the file

file level1 level2

level1: Mach-O 64-bit executable arm64
level2: Mach-O 64-bit executable arm64

The file type decides your next tool, so don’t skip this step, even when the extension looks obvious. CTF files often have no extension at all, or a misleading one. The first bytes decide: 7F 45 4C 46 is ELF, MZ is a Windows PE, and CF FA ED FE is 64-bit Mach-O. Here is what file 5.41 printed for a few other files I had at hand:

The last row is a good reminder that data only means file didn’t recognize the format. It doesn’t mean the file is random.

If you don’t want to install anything, drop the file into this site’s native binary overview. It runs in your browser and doesn’t upload the file. For level2 it reported format Mach-O 64-bit, machine arm64, file type MH_EXECUTE, and one library, /usr/lib/libSystem.B.dylib.

Step 2: Try strings

strings level1

Enter the flag: 
practice{strings_finds_plain_text}
Correct!
Nope.

That’s the first flag. strings prints every run of printable characters it finds, four or more by default, and the comparison string sits in the binary’s read-only string section (__TEXT,__cstring here, .rodata on Linux). The browser tool’s strings tab showed the same flag at file offset 1511 in __TEXT,__cstring. Its default minimum length is 6, so it hides Nope., but you can change that on the page. In a real challenge you’d filter: strings level1 | grep -i 'flag\|ctf\|{'.

Now the second binary:

strings level2

+ih8Lih8
Enter the flag: 
Correct!
Nope.

No flag. +ih8Lih8 is noise: bytes from machine code that happen to be printable (the browser tool placed them in __TEXT,__text). This is the normal case. Challenge authors know about strings, so they encode, compute or split the flag. strings is still worth the five seconds. When it fails, it tells you to read code, and the strings it does show (Correct!, Nope.) point you to the code that matters.

Step 3: Read the imports and sections

Imports show which library functions the program calls:

nm -u level2

___stack_chk_fail
___stack_chk_guard
___stdinp
_fgets
_printf
_puts
_strcspn
_strlen

fgets reads your input, and strlen suggests the length is checked. There’s no strcmp or memcmp, so the comparison is done by hand somewhere. That’s a hint, but not proof. level1 doesn’t import strcmp either, even though its source calls it. At -O2, clang replaced the call with inline 8-byte compares. Treat imports as clues.

Sections are often more telling. The browser tool listed six sections for level2, and one of them stands out:

level1 has no __const section. level2 has a 29-byte block of constant data that isn’t text. A small constant table next to a length check is a classic sign of an encoded flag. Look at the raw bytes:

xxd -s 0x612 -l 29 level2

00000612: 1a18 0b09 1e03 090f 1112 0518 3502 030e  ............5...
00000622: 0f19 351e 020f 3508 131e 0f19 17         ..5...5......

Step 4: Read the check, then solve it

Now you need the code that uses those bytes. Ghidra (free) is the usual choice: open the binary, let it analyze, find main and read the decompiled loop. The guide to native binary analysis with REA and Ghidra shows how to let an AI agent drive Ghidra through REA. Ghidra isn’t installed on the machine I wrote this on, so here is the plain disassembly from macOS’s otool -tV level2, trimmed to the check (objdump -d gives the same on Linux):

0000000100000514	bl	0x1000005e4 ; symbol stub for: _strlen
0000000100000518	cmp	x0, #0x1d
000000010000051c	b.ne	0x100000564
0000000100000520	mov	x8, #0x0
0000000100000524	add	x9, sp, #0x8
0000000100000528	adrp	x10, 0 ; 0x100000000
000000010000052c	add	x10, x10, #0x612
0000000100000530	ldrb	w11, [x9, x8]
0000000100000534	ldrb	w12, [x10, x8]
0000000100000538	eor	w11, w12, w11
000000010000053c	and	w11, w11, #0xff
0000000100000540	cmp	w11, #0x6a
0000000100000544	ccmp	x8, #0x1c, #0x4, eq
0000000100000548	add	x8, x8, #0x1
000000010000054c	b.ne	0x100000530

You don’t need to know every ARM64 instruction to read this. strlen returns the input length in x0, and cmp x0, #0x1d requires 29. Otherwise the code jumps to 0x100000564, which prints Nope.. x10 is set to 0x100000612, the 29-byte table, and x8 is the index i. Each pass loads your character and the table byte (ldrb), XORs them (eor) and requires the result to be 0x6a. Then it increments i and loops until index 28 (0x1c). The compiler rearranged the source’s (s[i] ^ KEY) == enc[i] into (s[i] ^ enc[i]) == KEY, which is the same condition. XOR undoes itself, so each flag character is table[i] ^ 0x6a:

import sys

TABLE_OFFSET = 0x612   # file offset of the encoded bytes (vmaddr 0x100000612 - 0x100000000)
LENGTH = 0x1d          # cmp x0, #0x1d  -> the flag is 29 bytes
KEY = 0x6a             # cmp w11, #0x6a after eor

data = open(sys.argv[1], "rb").read()
enc = data[TABLE_OFFSET:TABLE_OFFSET + LENGTH]
print(bytes(b ^ KEY for b in enc).decode())

python3 solve.py level2

practice{xor_hides_the_bytes}

On an ELF file the table offset will differ. Take it from the section table, or from the address in the disassembly minus the segment’s load address.

Step 5: Prove it on the original program

A flag you computed isn’t a flag until the program accepts it. Pipe it in:

python3 solve.py level2 | ./level2

Enter the flag: Correct!

For a record you can show later, REA, the open-source project by morluto, can run the program in a pseudo-terminal and capture what it printed. rea capture-process worked on macOS for this. My scenario file:

{
  "executable": "/absolute/path/to/level2",
  "working_directory": ".",
  "events": [
    { "type": "input", "at_ms": 250, "data": "practice{xor_hides_the_bytes}\r" }
  ],
  "timeout_ms": 5000
}

rea capture-process ./scenario-pass.json --json > capture-pass.json

The capture’s last frame read Enter the flag: practice{xor_hides_the_bytes} / Correct!, with exit code 0. Then I ran a negative control with the last letter changed (bytez), which printed Nope.. A control matters here: level2 exits 0 either way, so only the printed text shows the difference. Each capture got its own Evidence ID. The AI agent CTF guide builds on this with REA’s static tools.

Common beginner mistakes

• Running unknown challenge files on your main machine. Your own builds are fine. Downloaded crackmes belong in a VM without your personal data.
• Stopping at strings. An empty result is information, not a dead end.
• Trusting a flag you never tested. Off-by-one table offsets and wrong keys produce flag-shaped garbage. Always run the original.
• Fighting the disassembly line by line. Find the strings that matter (Correct!), follow their references to the code, and read only the loop around the comparison.

FAQ

Do I need to learn assembly first?

No. Learn it as you go. Most beginner checks are a length test plus a loop with one arithmetic operation, as here. A decompiler like Ghidra turns that into C-like code, and the disassembly becomes readable once you know what to look for.

Why do my addresses differ from this guide?

A different compiler, optimization level or OS produces different code and layout. The method stays the same: find the length check, the table and the operation.

Is it legal to reverse these files?

Your own builds and challenges published for practice, yes. Follow each platform’s rules, and don’t apply these techniques to software you aren’t authorized to analyze.

Next steps

• Solve CTF reversing challenges with REA and an AI agent: the same workflow, with an agent doing the reading and Evidence IDs backing every claim.
• Free native binary analysis with REA and Ghidra: set up the decompiler step you skipped here.
• ELF, PE and Mach-O viewer online: more on reading sections, symbols and strings in the browser.
• Reverse a WebAssembly file with WASM to WAT: when file says WebAssembly.

Practice on binaries you built, on CTF challenges published for practice, or on files you’re authorized to analyze. Before running anything you downloaded, read REA’s DISCLAIMER.md.