This challenge asks for a password. We see that it calls getenv to check for LD_PRELOAD, ptrace which are used to detect if a processes is debugged.
Since this is dynamic binary we can bypass these without patching via LD_PRELOAD, we let getenv return 0, and ptrace return 0 to bypass the protection.
After asking for the password we reach this.
rsi point to a block of code bytes, rcx is a counter in that block.
our input is xored with dword in the rsi[rcx]
increment counter, then compare to see if we reached the end which is 0xf00d, and jump to rsi if we reached the end. rsi is pointing to this block : 0x6010D8-0x601129 which is supposed to be valid instructions after xoring. We dump the block for analysis to find the key.
We assume the function contains at least int 0x80, and the key is printable. We write a script that searches for such thing
These are the possible beginning of the key, we use them and it seems '4b' is the one that makes sense here. Applying the key and looking at the disassembly, trying to make sense of the rest of it reveals that it's 'f6', we can also use the above script to find the last two characters by looking at 0x9090.
password : 4bf6.