Scrawl: Building a Tool for Humans and Agents

🎬 Also available as a blog walkthrough video with a narrated walkthrough.

The Itch

I was watching a colleague present. They moved across slides, a browser, a terminal, and a diagram, and the whole time they were drawing on the screen to make a point. A circle here, an arrow there, an underline on the line that mattered. It made the talk easy to follow, and I wanted to be able to do the same thing.

So I went looking for a tool. I found a few. Then I hit the familiar wall: I would have to install several, learn each one, and decide whether any of them actually fit how I present, before I knew if the time was worth it. That evaluation is its own afternoon. I did not want to spend it.

So I Built It (and I’m Not a macOS Developer)

Here is the part that would have stopped me a year ago. I am not a macOS developer. I have never shipped a native AppKit app. On my own I would not have known where to start, and the cost of learning would have been far higher than the cost of trialing the tools I had just dismissed.

With an AI agent, that math flipped. For me, building exactly what I wanted became cheaper than evaluating what already existed. I described the behavior I needed, the agent and I worked through the AppKit details together, and a couple of sessions later I had a working overlay. I learned a lot about transparent NSPanel windows along the way, which is the real reason I build things.

That alone would be a small story: another person used an AI tool to make a thing. The part worth writing about is what I did next.

The Thesis: Two Front Doors, One Core

When the tool worked for me, I asked an obvious question for 2026: who else uses my tools now? The honest answer is that it is not only people anymore. I have AI agents doing real work on my screen every day. If I am building a tool that draws on the screen, an agent that wants to point at something is just as much a user as I am.

So I gave Scrawl two front doors over one shared core.

The first door is for humans. A floating toolbar, global hotkeys, freehand drawing, a ghost mode that lets clicks pass through, a fade mode for quick emphasis, color swatches, and pen widths. The things you would expect.

The second door is for agents. A small loopback HTTP server and a thin wrapper for the Model Context Protocol [2] expose the core’s drawing primitives as structured commands. An agent can switch modes, draw shapes, place text, and clear the canvas, using the same engine the toolbar drives.

Neither door is an afterthought. The MCP server is not a bonus feature bolted on at the end. It is a design stance: a tool built in 2026 should assume both kinds of users from the start, and serve them through one core instead of two parallel implementations. That is the whole idea, and the rest of this post is what it looks like in practice.

What Scrawl Does (the Human Door)

Scrawl is a tiny native macOS menu-bar app. There is no Dock icon and no Xcode project required, the Swift toolchain that comes with the Xcode Command Line Tools is enough. You build and run it from the terminal with swift build and swift run. On launch you get a pencil icon in the menu bar and a dark floating toolbar at the bottom of the screen.

The controls are deliberately small:

• Draw and Ghost. Click Draw (or press ⌃⌥D) and a red frame appears around the screen to show that draw mode is on. Drag to draw. Ghost mode (or Esc) lets your clicks pass through to the app underneath while your ink stays visible, so you can advance slides and then keep annotating.
• Color and width. Pick a color swatch and a pen width of S, M, or L.
• Fade mode. Toggle Fade on and each finished stroke holds for a couple of seconds, then fades out on its own. Good for a quick “circle this, move on” without clearing.
• Undo and Clear. Undo removes the last stroke (⌃⌥Z); Clear wipes everything (⌃⌥C).
• Global hotkeys. With optional Accessibility permission, ⌃⌥D, ⌃⌥C, ⌃⌥Z, and Esc work even while another app is focused. The toolbar buttons always work without any permission.

It draws on top of anything: slides, a browser, a video, a terminal. It needs no Screen Recording permission, because it overlays the screen rather than capturing it. In a whole-display screen share (Teams, Zoom, Chime), your ink shows up for everyone, because the share captures the whole screen including the overlay, a window-scoped share won’t include it.

The Twist: An Agent Can Drive It

Here is the demo. Watch closely, because nobody is touching the keyboard or mouse. This is an AI agent operating Scrawl, and it is the same assistant that helped me build the tool.

The agent opens by using the human door. It moves the real cursor to the floating toolbar, clicks Draw, picks red, hand-draws an underline under the title, and flips on Fade mode. Nothing is special-cased. It is clicking the same buttons a person clicks. The page it is marking up is one of my own posts [3].

Then it switches to the agent door. Through Scrawl’s control API it places precise annotations on the page: an orange arrow at a heading, a yellow underline under the line that matters, a blue box around a section title, a green underline and arrow on a key sentence, a red box around a paragraph. Each one lands exactly where it should, because the API takes normalized screen coordinates and the agent converts each element’s on-page box into them before it draws.

That is the thesis made concrete. The same core, reached two ways, by two kinds of user, in one short clip.

The agent door is a loopback-only HTTP server on 127.0.0.1. You POST a single command or a batch of them. Coordinates are normalized from 0 to 1 with a top-left origin over the whole screen, so a caller working from a browser element box is one small conversion away: divide the viewport CSS pixels by the screen dimensions, then correct for window position, browser chrome, scroll offset, and display scaling. A batch looks like this:

{"commands": [
  {"op": "arrow",  "from": [0.2, 0.5], "to": [0.55, 0.42], "color": "orange"},
  {"op": "rect",   "from": [0.1, 0.3], "to": [0.9, 0.4],   "color": "blue", "width": 4},
  {"op": "text",   "at": [0.3, 0.7],   "text": "look here", "color": "white"}
]}

The MCP wrapper is a thin layer over that server. It turns each drawing primitive into an MCP tool, so an agent whose MCP host is configured to run the wrapper can annotate the screen without knowing the HTTP details. The agent in the demo is doing exactly this.

A word on security, because an unauthenticated local server deserves one. The control server binds to 127.0.0.1 only and is unauthenticated, and it can do exactly one category of thing: draw, clear, or switch mode. It has no file access and no system access. Any local process can draw on your screen while Scrawl is running, which is the trade for zero-friction local automation. If you do not want it, start Scrawl with --no-serve and the door is closed.

How It Works, Briefly

The core is small. A transparent, borderless panel sits at screen-saver window level and floats above normal windows and full-screen presentations. A canvas view records freehand strokes and renders them. The toolbar sits one level above the overlay so its buttons stay clickable even while you are drawing. Ghost mode is just a flag that tells the window to ignore mouse events so clicks pass through.

Both front doors call into that one canvas. The toolbar and hotkeys drive it for humans; the loopback server and MCP wrapper drive it for agents. There is no second drawing engine.

If you have built overlays before, none of this is exotic. The point is not the cleverness of the core. The point is that one modest core, exposed through two first-class doors, they do not surface identical capabilities yet, as the What’s Next list admits, serves both kinds of user without duplicating itself.

Where I take this next

Scrawl is a small tool and I am keeping it that way, but a few things are on my mind:

• Snapshot export. Save the current screen with its annotations as a PNG or PDF.
• Graphic-tablet control. Pen input for cleaner freehand, since a mouse is a blunt drawing instrument.
• Shapes in the toolbar. Arrows, boxes, and ellipses already exist as API commands; the human door should get buttons for them too.
• Multi-monitor overlays. Today it covers the main screen; presentations often span more.

Each of these has to keep the two-door rule: if humans get it through the toolbar, agents should get it through the API, over the same core.

Try It

Scrawl is open source under the MIT license. The repository is at github.com/stechr/scrawl [1]. It is a single Swift package: build it, run it, and start drawing. If you want the agent door, the control API and the MCP wrapper are documented in the README.

I did not set out to build a macOS app. I set out to draw on my screen during a talk, and an AI agent made building the thing cheaper than shopping for it. The lesson I am taking from this is smaller than “agents will build everything” and more useful. When you build a tool now, assume an agent will want to use it too, and give it a door. The core you already wrote is the only one you need.

Sources

• [1] Scrawl on GitHub, the open-source repository (MIT).
• [2] Model Context Protocol, the open standard the agent door wraps.
• [3] From Columbo to Coworker, the blog post the agent annotates in the demo.

About the Author

Stefan Christoph is a Principal Solutions Architect at AWS, focused on agentic AI, media & entertainment, and helping builders move from demo to production. He writes about AI architecture, developer productivity, and the future of software.

This is a personal blog. Opinions expressed here are my own and do not represent the views or positions of my employer.

Learn more →

• MCP
• AgenticAI
• DeveloperTools
• MacOS
• Swift