The Tools We're Testing Right Now: Agent Swarms and AI Product Docs

Why the team tests tools in the open

Night School regularly opens with the Gauntlet team showing what's at the top of their minds — the tools, workflows, and methodologies they're experimenting with before deciding whether to fold them into how they actually work. This session pairs an engineering-and-ops view from Ash with a product view from Derek, so the same question (“is this worth adopting?”) gets answered from two angles.

Grokbot and the chief-of-staff pattern

Ash's main demo is Grokbot, run through the Grok desktop app, using a single “chief of staff” agent to manage a swarm of sub-agents. The setup order matters: create the chief of staff first, give it a name and a description that spells out how it should lead the swarm and what to use the swarm for, then instruct it — explicitly using the words “swarm” and “bots” — to spawn workers. In the demo it fans out into an FAQ bot, a SERP bot, a community bot, and a Gauntlet-mentions bot, all combing the Gauntlet AI site and search results for missing questions, broken links, and 404s. The chief of staff then maintains a running report or dashboard so the human can verify the work rather than letting it run blind.

Managing a swarm without losing the thread

The practical ceiling Ash hits is context, not capability — things get messy past 15 to 20 agents. His answer is to manage the swarm the way you'd manage a team: have the chief of staff distill the important events and failure states, and back it with a queue in an outside store like Notion or SQLite. Once work is queued, the manager stops re-checking it and moves on, then comes back to the queue on a schedule. He also notes that evals and observability aren't solved on Grok yet; the interface is new, and rigorous tracing would mean piping the log database into an external tool.

When there's no MCP, build a CLI

A recurring wall is tools with a bad MCP or no MCP at all. Ash's workaround: you can make a CLI tool out of anything. Using Playwright and JavaScript to drive browser actions deterministically, he has Cowork walk through an app (his example is Facebook Marketplace), build a spec for a CLI, and hand that to Claude Code to generate it. The result is an agent that can act in any app or browser without waiting for an official integration.

Matching the harness to the work

Ash's daily workhorse model is Grok 4.6, and he escalates to Opus or Sonnet when Grok can't finish a task. For harnesses, Codex stays his primary coding engine, while Grokbot handles the surrounding work — QA'ing deployed sites via a dev or staging URL, writing specs, finding API docs, reviewing PRs, and research-heavy marketing tasks like combing the Facebook ads library or triaging resumes. Internal, Claude-entrenched team work runs through Cowork. His read on why Grokbot lands: it feels built for someone without a terminal background, a stepping stone to capability that Hermes- or Pi-style developer harnesses assume you already have.

ChatPRD: from blank page to first draft

Derek's tool is ChatPRD, built by a chief product officer, which he's been testing with the product team. It does two things well: it gets you to a draft fast, and it carries strong integrations — Slack, Google Docs, Notion, Teams, Confluence, Jira, and GitHub. Templates cover PRDs, one-pagers, go-to-market briefs, competitive analysis, API documentation, and coding rules, and the enterprise tier can scrape competitor information into the draft as you write. Because the tool is opinionated toward a product-first mindset, it asks the clarifying questions engineers often skip — users, outcomes, personas — which is exactly where engineers tend to struggle when writing a vision doc.

Feeding tools clean inputs

Both presenters converge on the same lesson: don't dump raw material into a model. Derek runs rough meeting transcripts through a dedicated notes skill first, then uses Cowork to file them, so his chats don't balloon and hallucinate. Ash goes further and stores text in SQL — modeling the data to match the information type — because agents navigate structured data far more reliably than a loose pile of tokens. As Derek puts it, SQL is a data structure, and LLMs work well with structured data.

Overnight and managed agents

Derek keeps most agent work in the daytime with checkpoints until he trusts a routine, but he does have Cowork draft social posts and morning email replies (around 6 AM) for him to approve on login. Ash points to three ways to run work unattended: a routine on Grok, a scheduled task on Cowork, or a managed agent running in the cloud — hosted by Anthropic, or a Cursor cloud agent built with the Cursor SDK — which he frames as a “dark factory” that can run around the clock. He references an earlier Night School on managed agents led by Matt Wood for the deeper version.