Reversing VMCrack
Today we will be taking a look at one of, if not the hardest, reversing challenges from Hack The Box. The challenge is called vmcrack and is well known among the HTB community for being difficult. This is backed up by the difficulty ratings for the challenge: the majority of users rated it 10/10, known as brainfuck difficulty.
Preface: This was one of my personal favourite challenges and if you are an obfuscation enjoyer I recommend checking it out for yourself before reading the rest of this article. The challenge is now retired so publishing solutions is fine; doing so for active challenges is frowned upon.
Once again I won’t be covering the basics of how VM obfuscation works; if you would like to read about some simpler VMs see my previous articles here and here, and there are plenty of good online resources describing VM obfuscation such as this.
Let’s dive in. The file provided, vmcrack.exe, is a 32-bit Windows PE executable. The entry point calls an function which appears to be a fairly standard WinMainCRTStartup function. Following through this leads us to the main function:
It initialises some global variables to some functions and pointers to seemingly empty locations in memory. The calls to __acrt_iob_func (which is an internal CRT function used to access IO streams) with parameters 1 and 0 resolve to stdout and stdin respectively. The last four functions, sub_401d30 and so on, appear to be crypto related functions upon inspection; we will analyse them later.
Jumping to sub_402390, we see a fairly messy function with lots of unused variables. First it seems to do something related to exception handling, and has an int1 instruction. The decompilation doesn’t display this well, so let’s check the disassembly:
This is still a bit confusing, but it seems to be similar to a common anti-debugging mechanism. The way it works is it creates an exception handler, has an int1 instruction which triggers a single step exception. If the program is running as normal the exception handler will be triggered and the program will do some sensitive logic (such as storing a specific value in a register) then resumes execution. However if a debugger is present then the debugger will handle the exception itself and the exception handler won’t be triggered. The program can then check if the value was set to see if it is being debugged and react accordingly. A good example of this anti-debug mechanism can be found on page 11 of this resource.
Indeed the program does later skip over the rest of the function if eax is zero. However I was unable to get this specific anti-debugging mechanism to trigger correctly within this binary when using x32dbg, perhaps it is targeted at older debuggers or the author made a mistake here. If anyone knows the answer please let me know.
The rest of the function looks like:
It sets some global variables, uses memset to clear some memory at data_409000, sets data_409028 to 0x10000 then calls sub_405350 with a lot of arguments. It then repeats this pattern again with some different values (minus setting the global variables). Following the final call to sub_402150 reveals a lot more occurrences of this pattern, and nothing much else.
Clearly sub_405350 is important so let’s check it out:
Seems like it’s backing up some values in an unknown structure. sub_405340 looks like:
Something funny is going on here, a lot of non-standard calling conventions and jumps to arbitrary locations. The program seems to calculate the jump location based on edi and esi, however looking at the calls to sub_405350 indicate that nothing special is being stored in these registers.
In fact this is the first case of our decompiler messing us up (will become a trend). If we look at the disassembly of sub_405350 instead:
We can see that all of the 8 general-purpose x86 registers are being stored in memory at the location of the 7th argument. If we look back at the previous calls, this is the memory we cleared using memset, data_409000. This seems like backing up the registers before entering a virtualised function.
The 3rd and 2nd last lines, which were missing from the decompilation, explain how the jump location is calculated. The 8th argument is stored in esi, and data_408018 is stored in edi. If we look at the disassembly for sub_405340:
A byte is loaded from esi and we jump to edi + (eax * 4). Intuition tells us that we are entering a virtualised function: edi/data_40818 is the base of the VM handler table and the byte loaded from esi is the first opcode, therefore esi/8th argument to sub_405350 is the bytecode. We will denote sub_405350 as the VM entry function.
To double check this we can look at data_40818: we see that it is the start of a series of pointers to functions (each separated by 4 bytes since the binary is 32-bit), exactly what we’d expect for a VM handler table. Similarly we can confirm that the 8th argument to the VM entry function is always a pointer to memory which could plausibly be bytecode.
Taking a look at the handlers, the decompilation doesn’t make much sense. The handlers appear to be reusing different sections of code, and chaining them via pushing the next function to the stack (similar to ROP). In some cases a function is pushed to the stack and not accessed until 3 functions later, which can make it hard to follow. Combined with using the VM’s registers in the structure we identified before rather than normal registers, this makes the decompilation very messy. For example the first handler I checked looks like:
and sub_40444f looks like:
However if we use the disassembly in graph mode:
We can see that this is most likely a dec handler. Going forward I will be using the disassembly over the decompilation when dealing with the handlers for this reason. Note that I’ve only shown the function which performs the core dec logic; each handler is made up of many different chained functions which are less interesting.
The value in edx appears to indicate the size of the operand being decremented. 1 corresponds to a byte, 2 to a word and 3 to a dword. If it is not one of these values then the program deliberately triggers an error.
Presumably the value is read from the bytecode and stored in edx by the functions called earlier. Similarly the target of the decrement, eax, must also be setup earlier. sub_404440 demonstrates how the handlers push the address of a function to be executed later to the stack, and jump to the next function. Later functions will pop this address off the stack and jump to it.
Although this knowledge was useful, when I was first statically analysing the handlers, the overall flow and purpose of each function was still a little unclear. I could tell the VM was most likely stack based, and appeared to have a number of different addressing modes, but it was easy to waste time tracing the exact execution of each handler. As a result I decided to move to dynamic analysis to see if I could observe the VM’s execution.
Dynamic Analysis
Running the program prompts us with the message
[Human.IO]::Translate("Credentials por favor"):
Providing a random input causes the program to output
You entered a credential and the device lighted up, but nothing happened... maybe it's broken?
Very cryptic. None of these strings were present in the binary, they must be created by the VM or some other obfuscated logic. Let’s try to debug the binary with x32dbg.
When we load the binary in x32dbg the initial breakpoint is where ntdll.dll loads the application. I added a breakpoint at the main function we identified before, and let the program execute, however I ran into a problem: the main function is never reached. Instead the program enters some TLS callback functions (which are functions a program can register to execute before the main program) and eventually termintates. If we check the console we can see that You notice a weird sound coming out of the device, so you throw it away in fear! has been logged.
This didn’t happen when we run the program normally, so it is definitely some anti-debugging behaviour; we will have to analyse these TLS callbacks and evade the detections before we can progress.
Going back to static analysis, the first TLS callback is uninteresting, just ensures that the program is being run in 32-bit mode and if not prints an error message and exits. There are three more TLS callbacks:
They all follow a similar pattern: they call sub_4027f0 and if the result meets some condition then they call our VM entry function.
sub_4027f0 is fairly long and messy so I will skip over the static analysis and say that it uses the PEB (Process Environment Block) to iterate over all loaded modules and find a function by name. The function name is derived by XORing the bytes given by the second parameter with the key given by the third parameter.
So for TLS_Entry_1, the leading non-zero bytes at data_406464 are 82a9a4a2aa93a4acaeb5a485a4a3b4a6a6a4b391b3a4b2a4afb5 and the XOR key is 0xc1. This decodes to CheckRemoteDebuggerPresent. Doing the same for the other two gives us NtQueryInformationProcess and NtSetInformationThread respectively.
This tells us that each of these callback functions is calling a function to detect a debugger, and if one is found then it calls some virtualised functions to print a cryptic message and exit.
The first function, CheckRemoteDebugger present is pretty self-explanatory: it returns 1 if the process is being debugged. Note that the first argument is supplied as -1, which indicates we want information about the current process.
The second function, NtQueryInformationProcess, is also used to query information about a process. However the value supplied for the ProcessInformationClass being queried (second argument) is 0x1e / 30, which doesn’t appear to be a valid option according to the Microsoft documentation. However some more in depth documentation reveals that this value corresponds to ProcessDebugObjectHandle. Seems to be another straightforward anti-debug check, more information can be found here.
The last function, NtSetInformationThread, is used to change thread priority. However it can also be used to hide a thread from the debugger, and prevent it from being able to control the process anymore. A good explanation of this can be found here.
Evading these checks is fairly easy: I simply patched the start of each function to jump to the ret instruction at the end. Using an anti-anti-debug plugin such as ScyllaHide would have been the easiest solution, however I wanted to understand each anti-debug check before evading it.
Now that I was able to debug the rest of the program, I followed the execution of the VM. As we determined from static analysis the VM stores its state starting at ebx, and has a register for each of the general purpose registers, from offset 0x4 to 0x20. It also has a custom register at offset 0x0 which appeared only to be used for intermediate operations. There was also another register at offset 0x24 whose purpose was not entirely clear. Taking a look at one of the handlers which uses it:
It performs a bitwise and with the register and 0x800, and if the result is non-zero then it moves our position in the bytecode (i.e. the instruction pointer). This seems like a conditional jump, which would suggest that 0x24 is the flags register. 0x800 = 2^11 and the 11th bit of the flags register is the overflow flag, thus this is a jo or jump if overflow handler.
The last slot in the context, at offset 0x28, is a form of stack pointer. We previously saw that before running the VM, the program allocates 0x1002c bytes of space and sets offset 0x28 to 0x10000. The VM’s state is 0x2c bytes, and the remaining space is the virtual stack. The top of the stack is calculated by adding the value at offset 0x28 to the address at the end of the VM’s state struct.
For example the following function pops from the stack:
I manually traced the VM’s execution for a few instructions and obtained the instructions:
push ebp
mov ebp, esp
sub esp, 0x8
A typical function prologue; clearly the source program was a full x86 program rather than some basic assembly program written for the purpose of being VM obfuscated. Obtaining these instructions via manual debugging was pretty inefficient, so writing a disassembler is the next logical step.
Disassembler
The only part I didn’t fully understand yet was the different addressing modes of the VM. I performed some more static analysis and, similar to the operand size flag, the first byte is a flag indicating the type of addressing. Modes 1 and 3 were simple: 1 corresponded to a register (so it was followed by a size flag and the register offset), and 3 was an immediate dword loaded from the 4 following bytes of the bytecode.
Addressing mode 2 first loaded a size flag, but then loaded 3 bytes followed by a dword. These are then combined to give the memory address to load. The logic for this looks like:
The first two bytes loaded represent registers, although in some cases only the first is used. These are stored in edx and ecx in the above function. The third byte is a bit flag that tells you how to combine these values: 0 is just the first register, 1 is edx+ecx, 2 is edx+(ecx*2), 4 is edx+(ecx*4) and 8 is edx+(ecx*8). Finally the last dword is an offset added to this expression to give the final memory address to load from, although it is often 0. We can deduce that addressing mode 2 provides a more complex way of accessing heap memory, compared to just a simple register or immediate dword access like the other two modes.
We can now build a disassembler. My implementation can be found here. There were a couple of handlers whose purpose was still unclear, such as the very last handler which appeared to check the Thread Information Block to compare the stack base to the stack limit and decrease the stack base if necessary. However it appeared as though it would always result in an error, and it was never used in the bytecode so I couldn’t investigate it any further and chose to represent it with a ud2 instruction. If anyone knows what the purpose of this handler I’d love to hear it.
There were 5 separate virtualised functions called from various points in the program: I have included the disassembly for each in the repo. In some cases the program actually changed the value of pointers within the bytecode, before executing the virtualised function. I deduced these were parameters of the virtualised functions.
The first disassembled function was:
0x41a508: push dword ebp
0x41a50c: mov dword ebp, dword esp
0x41a513: sub esp, 0x8
0x41a516: mov dword [esp], param1
0x41a525: mov dword [esp+0x4], param2
0x41a534: push dword edi
0x41a538: push dword esi
0x41a53c: push dword ebx
0x41a540: xor dword eax, dword eax
0x41a547: mov dword ecx, 0xffff
0x41a550: mov dword edi, dword [ebp-0x8]
0x41a55d: repne scasb byte [edi]
0x41a55f: sub dword ecx, 0xffff
0x41a568: not dword ecx
0x41a56c: xor dword edx, dword edx
0x41a573: mov dword esi, dword ebp
0x41a57a: sub dword esi, 0x4
0x41a583: mov dword edi, dword [ebp-0x8]
0x41a590: xor dword ebx, dword ebx
0x41a597: cmp dword ebx, 0x4
0x41a5a0: jz 0x41a590
0x41a5a6: mov byte al, byte [esi+ebx]
0x41a5b3: inc dword ebx
0x41a5b7: xor byte [edi+edx], byte al
0x41a5c4: add byte [edi+edx], 0x7c
0x41a5d3: inc dword edx
0x41a5d7: cmp dword edx, dword ecx
0x41a5de: jnz 0x41a597
0x41a5e4: push dword edi
0x41a5e8: call param3
0x41a5ee: add esp, 0x4
0x41a5f1: pop dword ebx
0x41a5f5: pop dword esi
0x41a5f9: pop dword edi
0x41a5fd: xor dword eax, dword eax
0x41a604: add esp, 0x8
0x41a607: pop dword ebp
0x41a60b: ret
It starts with a typical function prologue, then pushes the parameters onto the stack and backs up some registers. The important part is:
0x41a540: xor dword eax, dword eax
0x41a547: mov dword ecx, 0xffff
0x41a550: mov dword edi, dword [ebp-0x8]
0x41a55d: repne scasb byte [edi]
0x41a55f: sub dword ecx, 0xffff
0x41a568: not dword ecx
It stores 0 in eax and ebp-0x8 = esp = param1 in edi. It then uses the repne scasb to scan through characters in param1 until it is equal to the value in eax (0). So essentially it just scans through a null-terminated string from param1, and uses ecx to keep track of the number of characters.
If we analyse the concrete values used for param1, we see it is always a garbled string pointer. param2 is always a 4-byte value. param3 is always the puts function.
The remaining important part is
0x41a56c: xor dword edx, dword edx
0x41a573: mov dword esi, dword ebp
0x41a57a: sub dword esi, 0x4
0x41a583: mov dword edi, dword [ebp-0x8]
0x41a590: xor dword ebx, dword ebx
0x41a597: cmp dword ebx, 0x4
0x41a5a0: jz 0x41a590
0x41a5a6: mov byte al, byte [esi+ebx]
0x41a5b3: inc dword ebx
0x41a5b7: xor byte [edi+edx], byte al
0x41a5c4: add byte [edi+edx], 0x7c
0x41a5d3: inc dword edx
0x41a5d7: cmp dword edx, dword ecx
0x41a5de: jnz 0x41a597
0x41a5e4: push dword edi
0x41a5e8: call param3
0x41a5ee: add esp, 0x4
The same analysis can be performed, and it is equivalent to the following pseudocode:
i = 0
j = 0
while i < length of param1:
j %= 4
*(param1)[i] ^= *(param2)[j]
*(param1)[i] -= 0x7c
i += 1
j += 1
param3(param1)
This appears to be a string decoding algorithm, which aligns with the values for the parameters we observed. param1 is the encoded string, param2 is the decoding key and param3 is the puts function to print out the decoded string. So the overall function’s purpose is to decode and print a provided string.
I used this knowledge to decode the strings being printed for each call of the virtualised function. The results were:
TLS_Entry_1 (CheckRemoteDebugger check)
You notice a weird sound coming out of the device, so you throw it away in fear!
(recall we saw this message when we first tried to debug the binary)
TLS_Entry_2 (NtQueryInformationProcess check)
A bright red light emerges from the device… it is as if it is scanning us… IT MUST HAVE DETECTED US…. RUN!!!
TLS_Entry_3 (NtSetInformationThread / hide from debugger trick)
You noticed the device trying to drill itself into the ground and stopped it - the device self-destructed in defense…
Remaining calls
The device released a fury of permafrost, and everything within a 10 metre radius froze solid… including you…
The device straight up refuses to respond to you.. THE DEBUGGING IS WEAK IN THIS ONE!
You entered a credential and the device lighted up, but nothing happened… maybe it’s broken?
The messages from the TLS callbacks make sense: we already know they perform anti-debug checks and when triggered they print out some message and exit. The next 2 calls suggest there are some further anti-debug checks, and the last one might correspond to some flag validation logic.
There were also 2 further calls of this virtualised function which the encoded string decoded to meaningless values. Perhaps these strings are modified further at runtime before being decoded and printed.
The second virtualised function was a very simple one which immediately called exit to terminate the process. It was used in the TLS callbacks immediately after the calls to the first virtualised function to decode and print the messages.
The third function was the most interesting one: I won’t detail the full analysis process, but this is my annotated version of the disassembly:
; this function returns 0 for debugging detected, 1 for wrong credential and 2 for correct credential
0x41a000: jmp 0x41a010
0x41a006: xor dword eax, dword eax ; sub to exit early and return 0
0x41a00d: ret
0x41a010: mov dword ecx, [fs:0x30] ; PEB
0x41a019: mov dword eax, byte [ecx+0x2] ; PEB->BeingDebugged
0x41a026: test dword eax, dword eax
0x41a02d: jnz 0x41a006 ; exit if being debugged
0x41a033: mov dword eax, dword [ecx+0x68] ; PEB->NtGlobalFlag
0x41a040: and dword eax, 0x70
0x41a049: cmp dword eax, 0x70 ; check NtGlobalFlag for debugger flags https://anti-debug.checkpoint.com/techniques/debug-flags.html
0x41a052: jz 0x41a006 ; exit if debugger flags present
0x41a058: mov dword edx, dword [ecx+0x18] ; PEB->ProcessHeap
0x41a065: mov dword eax, dword [edx+0x40] ; ProcessHeap->Flags
0x41a072: test dword eax, 0x2 ; check Flags for debugger https://anti-debug.checkpoint.com/techniques/debug-flags.html#manual-checks-heap-flags
0x41a07b: jz 0x41a006 ; exit if debugger flags present
0x41a081: mov dword eax, dword [edx+0x44] ; ProcessHeap->ForceFlags
0x41a08e: test dword eax, dword eax ; check ForceFlags for debugger https://anti-debug.checkpoint.com/techniques/debug-flags.html#manual-checks-heap-flags
0x41a095: jnz 0x41a006 ; exit if debugger flags present
; section below accesses the x64 PEB via [gs:0x60] and performs the same checks
0x41a09b: mov dword ecx, [gs:0x60] ; PEB (x64 for some reason)
0x41a0a4: mov dword eax, byte [ecx+0x2] ; PEB->BeingDebugged
0x41a0b1: test dword eax, dword eax
0x41a0b8: jnz 0x41a006 ; exit if debugger flags present
0x41a0be: mov dword eax, dword [ecx+0xbc] ; PEB->NtGlobalFlag
0x41a0cb: and dword eax, 0x70
0x41a0d4: cmp dword eax, 0x70
0x41a0dd: jz 0x41a006 ; exit if debugger flags present
0x41a0e3: mov dword edx, dword [ecx+0x30] ; PEB->ProcessHeap
0x41a0f0: mov dword eax, dword [edx+0x70] ; ProcessHeap->Flags
0x41a0fd: test dword eax, 0x2
0x41a106: jz 0x41a006 ; exit if debugger flags present
0x41a10c: mov dword eax, dword [edx+0x74] ; ProcessHeap->ForceFlags
0x41a119: test dword eax, dword eax
0x41a120: jnz 0x41a006 ; exit if debugger present
0x41a126: push dword esi
0x41a12a: push dword edi
0x41a12e: push 0x0
0x41a134: mov dword eax, dword esp ; store stack pointer before pushing encoded string bytes
0x41a13b: push 0x6f75666d ; push encoded string bytes
0x41a141: push 0x3d20392e
0x41a147: push 0x296f3d20
0x41a14d: push 0x3f6f3c23
0x41a153: push 0x2e263b21
0x41a159: push 0x2a2b2a3d
0x41a15f: push 0xc6d672a
0x41a165: push 0x3b2e233c
0x41a16b: push 0x212e3d1b
0x41a171: push 0x75751200
0x41a177: push 0x661212e
0x41a17d: push 0x223a0714
0x41a183: mov dword ecx, dword eax
0x41a18a: sub dword ecx, dword esp ; number of bytes pushed
0x41a191: xor byte [esp+ecx-0x1], 0x4f ; xor each byte with 0x4f
0x41a1a0: dec dword ecx
0x41a1a4: jnz 0x41a191 ; loop for each byte in string
0x41a1aa: mov dword eax, dword esp
0x41a1b1: push stdout
0x41a1b7: push dword eax
0x41a1bb: call fputs ; print "[Human.IO]::Translate("Credentials por favor"):"
0x41a1c1: add esp, 0x3c
0x41a1c4: sub esp, 0x100
0x41a1c7: mov dword eax, dword esp
0x41a1ce: push 0x0
0x41a1d4: push 0x100
0x41a1da: push dword eax
0x41a1de: call fgets ; read user input from stdin
0x41a1e4: add esp, 0xc
0x41a1e7: xor dword eax, dword eax
0x41a1ee: mov dword ecx, 0xffff ; read max 65535 chars
0x41a1f7: mov dword edi, dword esp
0x41a1fe: repne scasb byte [edi] ; read until end of input string
0x41a200: sub dword ecx, 0xffff
0x41a209: not dword ecx
0x41a20d: dec dword ecx ; index of last char in string - newline char
0x41a211: mov byte [esp+ecx], 0x0 ; trim newline char
0x41a220: mov dword edx, dword ecx
0x41a227: mov dword esi, dword esp
0x41a22e: mov dword edi, credential_dest_ptr
0x41a237: rep movsb byte [edi], byte [esi] ; copy input string to *credential_dest_ptr
0x41a239: mov dword ecx, dword edx ; put string length back in ecx
0x41a240: jmp 0x41a25d
0x41a246: mov dword eax, 0x1 ; sub to exit early and return 1
0x41a24f: add esp, 0x100
0x41a252: pop dword edi
0x41a256: pop dword esi
0x41a25a: ret
0x41a25d: cmp dword edx, 0x3b ; check length is 59
0x41a266: jnz 0x41a246 ; exit if not
0x41a26c: mov dword eax, byte [esp+ecx] ; routine to encode input string
0x41a279: inc byte [esp+ecx-0x1]
0x41a283: rol byte [esp+ecx-0x1], 0x9f
0x41a292: sub byte [esp+ecx-0x1], 0xe
0x41a2a1: not byte [esp+ecx-0x1]
0x41a2ab: xor byte [esp+ecx-0x1], 0xc3
0x41a2ba: neg byte [esp+ecx-0x1]
0x41a2c4: add byte [esp+ecx-0x1], 0x3e
0x41a2d3: ror byte [esp+ecx-0x1], 0x1d
0x41a2e2: dec byte [esp+ecx-0x1]
0x41a2ec: xor byte [esp+ecx-0x1], byte al
0x41a2f9: dec dword ecx
0x41a2fd: jnz 0x41a26c ; loop for each input string char
0x41a303: mov dword edi, dword esp
0x41a30a: push 0x17edf5 ; push bytes from another encoded string
0x41a310: push -0x120a120b
0x41a316: push -0x5a2a2902
0x41a31c: push -0x29012902
0x41a322: push -0x3d3e6e9a
0x41a328: push 0x5a32066e
0x41a32e: push 0x6d29692d
0x41a334: push 0x2e6a7236
0x41a33a: push 0x35093565
0x41a340: push -0x758fac85
0x41a346: push 0xff8acdc
0x41a34c: push -0x6f98a458
0x41a352: push -0xc575fe1
0x41a358: push -0x773c7059
0x41a35e: push -0x40146374
0x41a364: mov dword esi, dword esp
0x41a36b: mov dword ecx, dword edx
0x41a372: rep cmpsb byte [esi], byte [edi] ; compare encoded input string against bytes on stack
0x41a374: add esp, 0x3c
0x41a377: jnz 0x41a246 ; return 1 if they don't match
0x41a37d: mov dword eax, 0x2
0x41a386: jmp 0x41a24f ; return 2
As you can see it performs some additional anti-debug checks via the PEB, and exits early with return value 0 if they were triggered. The section starting at 0x41a126 pushes an encoded string to the stack, then decodes it by XORing each byte with 0x4f. It then prints the decoded string and takes some user input via gets. Implementing this string decoding ourselves reveals that the prompt is the one we saw when we previously ran the binary without the debugger attached, [Human.IO]::Translate("Credentials por favor"):.
Next it transforms each character of the provided input with a sequence of different operations, such as negating, xoring and left and right rotations. Finally it compares the transformed string against a hardcoded one, and returns 2 if they match, 1 otherwise.
The string transformation logic for each character looks like:
0x41a279: inc byte [esp+ecx-0x1]
0x41a283: rol byte [esp+ecx-0x1], 0x9f
0x41a292: sub byte [esp+ecx-0x1], 0xe
0x41a2a1: not byte [esp+ecx-0x1]
0x41a2ab: xor byte [esp+ecx-0x1], 0xc3
0x41a2ba: neg byte [esp+ecx-0x1]
0x41a2c4: add byte [esp+ecx-0x1], 0x3e
0x41a2d3: ror byte [esp+ecx-0x1], 0x1d
0x41a2e2: dec byte [esp+ecx-0x1]
0x41a2ec: xor byte [esp+ecx-0x1], byte al
I wrote a small script to reverse this logic and decode the expected string that the transformed input was compared against and, after spending a while debugging several stupid mistakes, obtained the string [Alien.IO]::Translate("Skrr pip pop udurak reeeee skiiiii").
Although there was still a final virtualised function to analyse left, I ran the vmcrack binary and provided this string when prompted for input.
Success! It printed a picture of an ET-like alien holding a sign containing the flag:
HTB{V1RTU4L_M4CH1N35_G035_BRRRRRR!11!1!1!}
I won’t go into depth of the final virtualised function, but my analysis can be found in the annotated disassembly file. Essentially it takes the credential we provided to the previous function, obtains MD5 and SHA256 hashes of it and uses them as well as some other complicated decryption logic to decrypt the message we saw above.
It takes a CRC checksum of the decrypted message to check if the provided credential was correct and prints out the message if so. This explains why we failed to decode one of the messages previously; the message needed to be decrypted first.
Epilogue
The challenge can be found on HackTheBox. It is now in the retired section of the reversing category.
The repo containing the disassembler and annotated disassembly is here.
If you enjoyed this writeup or solved vmcrack yourself, you might also want to check out the binary VM challenge I wrote for HackTheBox, vvm. It can be found in the active reversing category.
If you would like to provide feedback on this article, or discuss any part of it please email me at bensb1@protonmail.com.