Building A MacOS Debugger

Notes on what I'm learning and hacking on and sometimes bigger deep dive articles.

I've had this idea for a while. I want to create a debugger that lets you visualize what your program is doing. Most debuggers will let you stop execution at points and take a look at variables and maybe a raw view of memory, see the call stack, and stuff like that. I want one that shows me where all my data is in memory and how the processor is accessing that memory. Memory access patterns are one of the main things you have to get right on modern processors to make code fast and I've found it really difficult to see what's going on. On my old blog, I wrote a post about some memory access stuff but was bummed there was no good way to visualize it for a real program. I wonder if I could build the tool I needed.

I mostly work on macOS. As far as debuggers go lldb is really the only one you can use. It's what all the "codes" use (Xcode and VScode) and any other debuggers use. It's actually almost impossible to use anything else because the way ptrace works on macOS (the syscall that lets you do debugger stuff like pause execution) doesn't let you read or write registers. You would also have to use private APIs so you couldn't distribute the debugger without dealing with some weird codesigning stuff and some system preferences stuff people would have to enable and it's all just kinda annoying. This is probably why everybody just builds on top of lldb.

I did discover something in my research though. lldb has two parts. It has a server, the actual process that makes the ptrace calls and inspects registers and whatever else, and it has a client, the lldb cli that you type commands into. The server process is called debugserver and it ships with macOS as part of the Xcode command line tools. It's possible to write your own client that talks to debugserver directly, which means you can build a debugger that can do all the same stuff lldb can do, but without having to use any system APIs yourself. This seems promising and fun.

debugserver and lldb talk over the gdb remote protocol. It's a low-level way for lldb to say stuff like "set this breakpoint" or "continue execution" and debugserver to reply. You can see all the packets sent between lldb and debugserver by starting an lldb session and turning on gdb-remote logging with log enable gdb-remote all -f logfile.

I compiled a simple program, hello_world.c.

#include <stdio.h>

int main()
{
    printf("Hello, World!");
}

Then I ran it with lldb and turned on logging. I just ran the program, no breakpoints or anything.

❯ /usr/bin/lldb test/a.out
(lldb) target create "test/a.out"
Current executable set to '/Users/steve/dev/debugger/test/a.out' (arm64).
(lldb) log enable gdb-remote all -f lldb.log
(lldb) r
Process 48876 launched: '/Users/steve/dev/debugger/test/a.out' (arm64)
Hello, World!Process 48876 exited with status = 0 (0x00000000) 
(lldb) 

The log file for this is HUGE. lldb does a lot of work before even starting the process. It sends commands to look up information about the processor like its registers. It sets a few different breakpoints in startup code and runs to them so it can look up things like what dynamic libraries are loaded and other things I can't even really tell yet. Finally, at the end, it sends a continue command and gets back a process exited reply.

<   5> send packet: $c#63
<   7> read packet: $W00#00

Cool, can we just do that without all the extra stuff lldb does? Time to write a debugger. First, launch the program under debugserver. It's put in a very specific place when you install the Xcode command line tools so it'll probably be here for everyone, if it isn't try xcode-select --install.

❯ /Library/Developer/CommandLineTools/Library/PrivateFrameworks/LLDB.framework/Resources/debugserver 0.0.0.0:1234 test/a.out
debugserver-@(#)PROGRAM:LLDB  PROJECT:lldb-1400.0.38.13
 for arm64.
Listening to port 1234 for a connection from 0.0.0.0...

Then we can connect to that, I'm using telnet (you can get it with brew install telnet).

❯ telnet 0.0.0.0 1234
Trying 0.0.0.0...
Connected to 0.0.0.0.
Escape character is '^]'.

Then we can try sending just the continue command. Commands are a $ followed by the command, followed by a checksum. The gdb server protocol works over TCP but it also works over serial and there's a lot of stuff in it about acking responses and requesting re-sends when things mess up.

$c#63
+$O48656c6c6f2c20576f726c6421#c2

That's not the $W00#00 I expected but it's even better. $O is console output and the rest of the message is hex-encoded ASCII. Decoding it gives Hello, World!.

We can send + to say we received the message and then it gives us what we were looking for. The whole session looks like this.

Trying 0.0.0.0...
Connected to 0.0.0.0.
Escape character is '^]'.
$c#63  
+$O48656c6c6f2c20576f726c6421#c2
+
$W00#b7
+
Connection closed by foreign host.

And there you have it, you don't have to use lldb anymore, now you can debug your programs with debugserver and telnet!