Reversing a Broken Rust Binary

This post describes reverse engineering a Rust binary from the rev section of TBTL Capture The Flag 2024. The challenge is named Safe Rust and provides a compiled Rust program.

The challenge description indicates that the program “isn’t functioning properly”, and indeed when we run it we get the message

thread 'main' has overflowed its stack
fatal runtime error: stack overflow

Interesting, let’s dig into the binary. The entry point, _start, has a typical libc entry which points to the main function.

main function

It calls std::rt::lang_start_internal::h983d5a44e7093a3b with a reference to another function, along with the command line arguments provided and some other data. The name suggests this is an internal function provided by the Rust compiler, possibly to start the program.

The code for this function doesn’t reveal anything particularly interesting: it mostly just calls a lot of other standard library functions. However it does eventually call the function provided as the first argument. Thus we can conclude it is just boilerplate code to setup the Rust runtime and call the real main function, rust::main::h1eef4902b3075149. This function looks like:

Rust main function

Looks like it contains a lot of unused lines, possibly Binary Ninja’s HLIL has made some mistakes. We could examine the disassembly to investigate this, but there are other more interesting parts of the function.

It calls rust::generate_key::hb8d203faab29ab07 with 0xbabadeda, then performs a loop to fetch a quadword from the data section and XOR it with this key. It also updates the key via calling rust::generate_key::hb8d203faab29ab07 at the end of each iteration. Smells like decrypting the flag one character at a time.

The generate_key function is too long to show, but the important part is:

generate key function

arg1 is the first parameter of the function (so 0xbabadeda in the first call). It performs an arithmetic operation on arg1 and stores it as r15_6, then has a loop which allocates and drops some memory. At the end of the loop it updates r15_6 via another similar arithmetic expression. Finally it conditionally drops some more memory, then returns the generated key r15_6.

The memory allocation and dropping logic isn’t entirely clear, so I decided to debug the program to see what was going on. Although the program does error, luckily it does reach this function once before crashing. Specifically it crashes during the deallocation in the branch immediately before returning the key.

I decided to try preventing the program from entering this branch. The disassembly for this part looks like:


The temp4 == 1 conditional jump is at address 0x87ea. The jne instruction will jump when the zero flag is not set, to 0x8811 in this case, which performs cleanup and returns the key. Otherwise it continues onto the deallocation part which is where the error occurs.

Therefore we can breakpoint 0x87ea and ensure ZF is set to avoid entering the branch and hitting the error. I did this and this function returned normally. Interestingly, when control was transferred back to rust::main::h1eef4902b3075149, it continued as normal and used the generated key to decrypt the value 0x54 from the data section.

This is the character code for T, which is what we’d expect the flag to start with. Doing the same for a few more iterations indicates that the program is now decrypting the flag character by character, without any errors.

I wrote a script using the GDB Python API to automate this process, and obtain the flag.

import gdb

flag_chars = []

def handle_stop(e):
    if isinstance(e, gdb.BreakpointEvent):
        # the breakpoint just before error branch is taken
        if e.breakpoint.number == 1:
            gdb.execute('set $eflags &= ~(1 << 6)')
        # the breakpoint after each flag char is decrypted
            char = gdb.parse_and_eval('$rax')

base = 0x555555554000

# breakpoint just before error occurs so we can patch ZF
gdb.Breakpoint(f'*{base + 0x87ea}')

# breakpoint to where we can read flag chars out
gdb.Breakpoint(f'*{base + 0x890e}')


flag = ''.join([chr(x) for x in flag_chars])

It sets two breakpoints: the first to ensure ZF is set to avoid hitting the error, and the second to obtain each decrypted character. It takes a little while (probably because generating each key involves 999,999 loop iterations), but eventually prints out the flag


A static solution

After solving the challenge via debugging, I decided I wanted to do it statically too. When debugging I noticed that the memory allocated and later deallocated within generate_key function was not actually used in the calculation of the key; each XOR key is actually calculated via repeated arithmetic operations on itself.

Here is a script to mimic this behaviour statically:

valdiation_data = bytes.fromhex('2ca8bc140000000059830218000000000604402b00000000f8f9135200000000c8d5c16900000000e927c8000000000013b896560000000052023665000000004107fb07000000000f03634e000000008ad5c34100000000f835232100000000a47be06f00000000cf96120b000000004e54f115000000000673f912000000006e704b46000000002c44190700000000a6ed1c5a00000000d3e6c71a0000000051622421000000007ca6797d0000000040f3954200000000c6f8bf61000000006607c9360000000061164c02000000009fd345250000000099c8c208')

def generate_key(operand):
    key = (operand * 0xbc8f) % 0x7fffffff

    for _ in range(0, 0xf423f):
        key = (key * 0xbc8f) % 0x7fffffff

    return key

def main():
    key = generate_key(0xbabadeda)
    flag = ''

    for i in range(0, 224, 8):
        data_bytes = valdiation_data[i:i+4]
        data = int.from_bytes(data_bytes, byteorder='little')

        char = data ^ key
        flag += chr(char)

        key = generate_key(key)



The validation_data is just a dump of the data section found at data_45050. This script also takes a few seconds, but prints out the flag as desired.


The source code for the challenge can be found here.