Adding Factory's Droid to learnmcp.ai for code review

I run a number of production apps, and I'm usually making dozens of changes a week across them, bouncing between different subagents as I go. I miss things. That's fine when it's a typo. It's not fine when it's a credential or an auth check that ends up on the internet.

I wanted an agent keeping my code quality in check so I could build and ship more with confidence. I was making some changes to learnmcp.ai, my open source learning framework that runs right within your favorite agentic coding tool, and the idea was to add a pre-commit hook that has an agent review every staged change before it lands. The commit is the cheapest place to catch a problem, as long as the check is quick, can't leak what it's protecting, and doesn't lock me out of my own repo when it breaks. I spent more time on that last requirement than on the review itself.

Why review the staged diff with an LLM

Linters and tests only catch what you wrote rules for. A model reading a diff will point out the things you never thought to write a rule for: a missing authorization check, sensitive data headed for a log line, a change that breaks a caller three files away.

I picked Factory's Droid because of its headless mode. droid exec "<prompt>" runs one prompt, prints the answer and exits, which is exactly what a git hook needs. The hook also doesn't care who ran git commit. If Claude Code commits, it gets the same review I do.

The point of all this is automation. I can't read every diff across that many apps, so I want an agent doing the review work and, over time, more of the follow-up, which is how I scale without the quality slipping. Tests and a secret scanner still run beside it, because an agent needs something deterministic keeping it honest.

The pre-commit hook

Here's what I built, and how it works. I ended up with eight rules, and they matter more than the script.

1. I review the staged diff, not the whole codebase. It's quick, and it's the only part that changed. Full sweeps can wait for CI.
2. Sensitive files never get sent. I exclude them by path, every time.
3. I don't trust the agent to behave, so it runs from an empty temp directory that holds only the diff.
4. gitleaks scans first. If it finds a secret, the AI review is skipped and the secret never leaves my machine.
5. If the reviewer itself breaks, the commit goes through. If it finds a problem, the commit stops.
6. The reply has to start with VERDICT: PASS or VERDICT: FAIL, so a script can read it.
7. Nothing sensitive gets written to my own logs.
8. It reports. It never fixes. Fixing is a separate step I review.

Only a FAIL verdict stops a commit. Everything else either passes or fails open.

Setting it up

Install Droid and sign in first. On a Mac:

brew install --cask droid
droid                         # first run opens the browser to sign in
droid exec "say hello"        # confirms headless mode works

Install steps and flags change, so check Factory's docs before you trust my commands. For CI you'd use a FACTORY_API_KEY environment variable from your secrets store, never a file in the repo.

.git/hooks isn't version controlled, so nobody else on the project would get the hook. I keep it in a tracked folder and point git at it:

mkdir .githooks && touch .githooks/pre-commit && chmod +x .githooks/pre-commit
git config core.hooksPath .githooks     # once per clone

Here's the core of the hook, trimmed so you can see where each rule lives. The full script is in this gist.

#!/usr/bin/env bash
set -uo pipefail
command -v droid >/dev/null 2>&1 || { echo "droid not installed, skipping"; exit 0; }
 
# Rule 2: sensitive paths never make it into the diff.
EXCLUDES=(
  ':!.env' ':!.env.*' ':!**/.env' ':!**/.env.*'
  ':!*.pem' ':!*.key' ':!*.p12' ':!*.pfx'
  ':!**/credentials*' ':!**/secret*' ':!**/*token*'
  ':!.npmrc' ':!*.tfstate' ':!*.tfvars' ':!package-lock.json'
)
DIFF=$(git diff --cached --diff-filter=ACM -- . "${EXCLUDES[@]}" | head -c 200000)
[ -z "$DIFF" ] && exit 0
 
# Rule 4: a deterministic scan runs first. A hit means the diff is never sent.
if command -v gitleaks >/dev/null 2>&1; then
  gitleaks protect --staged --redact >/dev/null 2>&1 || {
    echo "gitleaks flagged the staged changes, skipping AI review"; exit 0; }
fi
 
# Rule 3: droid runs from an empty temp dir that holds only the diff.
SANDBOX=$(mktemp -d); trap 'rm -rf "$SANDBOX"' EXIT
printf '%s' "$DIFF" > "$SANDBOX/staged.diff"
 
# Rules 6 and 8: report only, and the first line is a verdict.
OUT=$(cd "$SANDBOX" && droid exec \
  "Review the git diff in ./staged.diff. It is the only file available; do not look elsewhere. \
Report only; do not modify anything. Flag: hardcoded secrets or credentials, \
injection/auth/authorization flaws, unsafe deserialization, sensitive data in logs, \
and likely regressions. The first line of your reply must be exactly 'VERDICT: FAIL' \
if you have any findings, or 'VERDICT: PASS' if you have none. \
After that, list findings as file:line - severity - issue." 2>&1)
STATUS=$?
echo "$OUT"
 
# Rule 5: fail open if the reviewer broke, fail closed on a FAIL verdict.
[ "$STATUS" -ne 0 ] && { echo "review failed to run (commit not blocked)"; exit 0; }
printf '%s' "$OUT" | grep -q '^VERDICT: FAIL' && {
  echo "BLOCKED. Fix and re-commit, or bypass with: git commit --no-verify"; exit 1; }
exit 0

The head -c 200000 caps how much diff gets sent, so a giant refactor gets partly reviewed instead of not at all. I exclude package-lock.json because a model reading a lockfile diff tells you nothing.

Findings can quote the very secret they found, so the log on disk keeps only the verdict and the locations:

umask 077; mkdir -p .droid-review; chmod 700 .droid-review
# log "file:line - severity" only; the issue text stays on the terminal
printf '%s\n' "$OUT" | sed -E -n 's/^([^ ]+:[0-9]+ - [^-]+) - .*/\1 (details: terminal only)/p' \
  >> .droid-review/review.log

Add .droid-review/ to .gitignore so the log never gets committed.

What it caught on learnmcp.ai

In one of its early runs the hook blocked a commit and flagged my own logging code. The log saved the whole VERDICT: line, so any trailing text on it could carry a credential onto disk. I had Claude Code fix it instead of bypassing the hook. The log now keeps only an exact PASS or FAIL, and the directory is owner only, which is the snippet above.

The log, the terminal and Claude Code's explanation of the block, in one window.

It also found a real bug in the plugin's track colour lookup in packages/plugin/hooks/register.tsx. A cartridge ID like constructor or toString resolved an inherited object property, so the lookup returned a function where it should have returned a colour string. Nobody is going to name a track constructor, but the fix was one line: Object.hasOwn.

Limits you should know about

LLM output is nondeterministic. The same diff can get different findings on different runs, and a false positive will block you sometimes. That's why --no-verify exists and why the hook fails open when the reviewer breaks.

The temp directory only limits where the agent looks by default. It isn't a hard sandbox, and an agent could still read an absolute path. If you need real isolation, use a container or a file-read deny list if your tool offers one.

Path exclusions can't catch a secret pasted into a normal source file, which is why the gitleaks pre-scan exists. It has a cost: when gitleaks flags something, the AI review is skipped, so that commit gets no review at all. A reply with no verdict line also fails open. The hook prints an "inconclusive" message and you have to notice it.

Your diff goes to a third party, so read their data retention and training policy before you point this at private code. The hook also reviews code without running it, so keep a deterministic test step next to it.

Don't auto-fix inside the hook. An edit mid-commit means the diff that was reviewed isn't the one that gets committed, it breaks partial staging with git add -p, and it ships changes nobody read.

Letting it fix things on a branch

The autonomy levels are the part of Factory I'm most excited about. I said not to auto-fix inside the hook and I still mean it. The hook stays report only. What I'm considering is a separate job that runs after a review fails and lets Droid fix its own findings on a branch, so the diff that got reviewed is never the one being edited.

Per the exec docs, --auto low allows file edits, --auto medium adds builds, tests, git commit and git checkout, and --auto high adds remote writes, deployments and database migrations. A run that asks for more than its level stops, exits non-zero and makes no partial changes. I'd start at medium: edit, run the tests and commit on a throwaway branch, then open a pull request. high is more authority than a job that fixes a log line needs.

For now I'm staying in the loop. Droid fixes on the branch, the pull request runs the usual required checks (tests, gitleaks, and a droid-action review), and I read the diff and merge. I don't want the same model writing a fix and approving it, and I'd like to see how often its fixes are wrong before I step out. If they hold up, I'd let GitHub's branch protection do the merging once checks pass, starting with low-severity findings and keeping a human on anything that touches auth or secrets. I haven't built any of this. What I want to know first is whether the tests catch a bad fix.

What I learned

The review itself was the easy part. Getting Droid to read a diff and tell me what's wrong took one prompt. Making sure it couldn't leak what it was reading, couldn't lock me out when it broke, and couldn't rewrite code I hadn't looked at took all the thinking. One of its early catches was a hole in my own logging, which says a lot about where the risk was. It wasn't the model's judgment. It was everything I wrapped around it.

I also stopped treating the model as the safety net. gitleaks and the path exclusions handle the parts that have to be deterministic, and Droid covers what rules can't. That split is what makes me comfortable handing it more. Today it reviews. Next it fixes on a branch while I read the diff, and if that holds up, it takes more of the follow-up off my plate. That's what I'm after: shipping more across my apps without the quality slipping.

If you want to try the hook, grab the script, edit the EXCLUDES list for the files you'd never want a third party to see, and tell me what it catches in your repo.

Try learnmcp.ai

learnmcp.ai is the project I was working on when I built this, and it's the repo Droid reviews every commit of. It's an open source learning framework that runs inside your agentic coding tool and recognizes when you've done something worth learning as you work. In Claude Code:

claude plugin marketplace add quintonwall/learnmcp
claude plugin install learnmcp@quintonwall

Restart Claude Code and that's it. No account, no API keys, nothing to configure. The longer story is in LearnMCP: Vibe Learning for the Agentic Era.

Subscribe to the site at quintonwall.com and follow me on YouTube at @seeqcode for more.