AI PR
Reviewer

The Problem

AI code review has an obvious pitch and an obvious failure mode: a wall of confident prose about your diff, with no locations, no evidence, and no way to check any of it short of re-reviewing the PR yourself.

I built this project to attack that from both ends. On the product end, a review is only worth reading if every finding carries an exact file and line range plus the code it's talking about — so verifying a claim costs one click, not a re-read. On the engineering end, I wanted a public, inspectable answer to what shipping an LLM feature actually involves once you leave the happy path. The model call is the easy part; everything around it is where this project lives:

• A provider dies at step 40 of a 60-step review — do you restart, or resume on the next one?
• Free-tier rate limits arrive mid-stream, shaped differently by every provider's API.
• The model reports an issue on a line that isn't in the diff, or “quotes” code it never read.
• A public endpoint with no signup spends real API quota, so it needs abuse control that won't take the site down with it.

Working demos that stop before these problems are everywhere. This project is about the rest — with the source public, so none of it has to be taken on faith.

The Approach

The naive build is a pipeline: fetch the diff, stuff it into one prompt, parse whatever comes back. It breaks on the first real PR — diffs overflow free-tier context windows, and a diff alone often can't tell you whether a change is wrong. The answer usually lives in the surrounding file.

So the reviewer is an agent, not a pipeline. The model gets six tools over an Octokit adapter — get_pr_metadata, get_pr_files_summary, get_diff, get_file_contents, list_directory, emit_issue — and decides what to read within a 60-step budget. The tool descriptions push back against over-fetching: full file contents are documented as a last resort for when the diff alone is insufficient, and PRs are capped at 15 changed files instead of pretending unlimited scope works on free-tier context.

The decision that shapes everything else: findings are tool calls, not prose. When the model confirms a problem, it calls emit_issue with a Zod-validated payload — file, 1-based line range, a severity from a fixed four-level rubric, title, explanation, optional fix. Malformed output is rejected at the boundary instead of leaking into the UI. Repeats are deduplicated by a content hash and answered with duplicate: true, so the model learns not to resend.

And the model never sends code. It sends locations; the backend slices the actual snippet out of the PR's unified diff, three context lines around the target range, and pins the GitHub link to the reviewed commit. A hallucinated quote has no path to the screen.

The system prompt is deliberately conservative: report only what this PR introduces, skip anything uncertain, and when there's nothing to say, say exactly “No issues found.” An AI reviewer that pads its output to look thorough is worse than none.

The Solution

What you see: paste a PR URL and the review streams in over SSE. An agent console narrates the run — each tool call as a row with a live outcome (running, ok, skipped, failed), interleaved with the model's commentary — while a step counter and a token counter tick as it works. Issue cards appear the moment emit_issue fires, not when the review ends: severity, title, markdown explanation, optional suggested fix, and the real diff slice with the target lines highlighted.

Every completed public review persists to a shareable page at /r/<slug>. The slug is derived, not random: a hash of owner/repo#pr@headSha encoded to 11 base62 characters, upserted on conflict — reviewing the same head twice updates the same URL instead of minting a new one. The share page re-renders the cards server-side with severity counts, the provider that produced the review, and the reviewed commit SHA.

Two opt-in paths extend the free flow. Premium mode: bring your own Anthropic key and the review runs on Claude Sonnet instead of the free chain — the key travels inside that one request and is never stored. Private PRs: bring a GitHub PAT with read access; the agent reviews as usual, but the result is never persisted — no share page, nothing written.

Architecture

The codebase enforces a one-way graph — app/ is routing and UI, lib/ is the domain, and lib/ never imports from app/. External boundaries speak their own language: GitHub responses and LLM tool schemas stay snake_case and are translated to camelCase domain types at the adapter, so the domain never learns the shape of an external payload. These aren't habits — the module boundaries and the naming contract are written down as rules the codebase is held to.

The centerpiece is the provider failover chain. The free path runs Cerebras (zai-glm-4.7), then Groq (gpt-oss-120b), then Gemini 2.5 Flash. Every failure passes through a taxonomy — eight classes, each with a verdict on whether to hop: rate limits, overloads, retryable 5xx and context overflow move to the next provider; invalid keys and user aborts stop the run. The part that took real iteration: a hop must resume, not restart. The transcript accumulated so far — tool calls and their results — is sanitized and handed to the next model: reasoning blocks stripped (they're provider-specific), empty text parts dropped, orphaned tool results removed, because no provider accepts a tool result whose call it can't see.

// Failover hands the accumulated transcript to the next provider —
// minus reasoning blocks and orphaned tool results.
function sanitizeForHandoff(messages: ModelMessage[]): ModelMessage[] {
  const keptToolCallIds = new Set<string>();
  const out: ModelMessage[] = [];
  for (const message of messages) {
    if (message.role === "assistant" && typeof message.content !== "string") {
      const content = message.content
        .filter((p) => p.type !== "reasoning")
        .filter((p) => !(p.type === "text" && p.text === ""));
      if (content.length === 0) continue;
      for (const p of content) {
        if (p.type === "tool-call") keptToolCallIds.add(p.toolCallId);
      }
      out.push({ ...message, content });
      continue;
    }
    if (message.role === "tool") {
      const content = message.content.filter(
        (p) => p.type === "tool-result" && keptToolCallIds.has(p.toolCallId)
      );
      if (content.length === 0) continue;
      out.push({ ...message, content });
      continue;
    }
    out.push(message);
  }
  return out;
}

Abuse control is a handwritten sliding-window log in Redis Lua. One script walks every window atomically — the review gate holds two (3 per hour and 10 per day, per IP), with a broader 25-per-hour request gate in front — and consumes only if all of them allow, returning the retry-after of the tightest violator. It fails open by design: the eval races a 500 ms timeout, and any Redis failure admits the request. Losing rate limiting for a few minutes is cheaper than turning a Redis hiccup into an outage. Client IPs come from the rightmost X-Forwarded-For entry, and only when TRUST_PROXY is explicitly enabled.

-- One atomic pass: check every sliding window, consume only if all allow.
for i = 1, #KEYS do
  local window = tonumber(ARGV[2 * i + 1])
  local limit = tonumber(ARGV[2 * i + 2])
  redis.call('ZREMRANGEBYSCORE', KEYS[i], '-inf', now - window)
  local count = redis.call('ZCARD', KEYS[i])
  if count >= limit then
    local idx = count - limit
    local entry = redis.call('ZRANGE', KEYS[i], idx, idx, 'WITHSCORES')
    local wait = tonumber(entry[2]) + window - now
    if wait > retryAfter then
      retryAfter = wait
      blocked = i
    end
  end
end
if blocked > 0 then
  return {0, blocked, retryAfter}
end
for i = 1, #KEYS do
  redis.call('ZADD', KEYS[i], now, ARGV[2])
  redis.call('PEXPIRE', KEYS[i], ARGV[2 * i + 1])
end

The ops story is small and strict. The environment is a Zod schema; the container entrypoint runs migrations, then a startup probe that asserts the env and pings Postgres and Redis with 5-second timeouts before the server boots — a misconfigured container exits instead of limping. CI enforces exactly that behavior: the pipeline docker-runs the freshly built image with no environment and fails the build if the container doesn't. Green builds push to GHCR, and Coolify auto-deploys behind Traefik on a VPS I run myself; a Docker HEALTHCHECK polls the liveness endpoint every 10 seconds, and the SSE stream passes through the proxy incrementally rather than buffering.

Key technical decisions

• Findings as tool calls, not prose — Zod at the boundary, content-hash dedup, and a UI that never parses free text.
• Server-side enrichment — the model sends locations; the backend slices real diff lines, so hallucinated code can't render.
• Failover carries the transcript — a sanitized tool-call history hands off mid-review instead of restarting from zero.
• Fail-open rate limiter — Redis being down costs limits, not availability; 500 ms budget on the Lua eval.
• Derived share slugs — hashed from the PR identity, so the same review maps to the same URL forever.
• Fixture mode for UI work — MOCK_REVIEW streams the same chunk types as production, with six injectable failure scenarios; front-end iteration burns no tokens.

Trade-offs

The project keeps an explicit list of what it deliberately does not do. The ones that matter:

1. 01No hexagonal/DDD layering. The domain is data-in/data-out with no invariants to defend; use-case classes would be ceremony. Ports exist only where a real external service does — GitHub sits behind one, nothing else earns one.
2. 02No libraries where a page of code suffices. The rate limiter is a Lua script and a factory function; “config management” is one Zod schema. The written rule: a dependency arrives when the simple version stops coping, not before.
3. 03A hard 15-file cap instead of chunking huge PRs. Free-tier context windows are the real constraint. An honest 400 up front beats a review that silently skipped half the diff — the failure is visible instead of buried in the output.
4. 04No accounts and no stored secrets. Anthropic keys and GitHub PATs live only in the request that carries them; private-PR reviews are never persisted. Less product surface, far less liability for an app strangers are invited to poke at.
5. 05No src/, no features/, no barrel files, no central types/ folder. One dominant feature lives in one home (lib/review/); structure follows the domain, not a template. For a single-feature app, scaffolding layers are cost without payoff.

Proof

Engineering facts, each checkable in the source:

Output you can trust

no invented code

every snippet is pulled straight from your PR and linked to the exact lines on GitHub — the reviewer can't make code up

Resilience

3-model auto-failover

if one AI provider fails or rate-limits mid-review, it resumes on the next instead of restarting from zero

Runs in public

no signup, abuse-proofed

a free endpoint spends real API budget, so it caps usage per person — and a glitch in that cap can't take the site down

Reviews in context

reads beyond the diff

a multi-step agent opens the surrounding files to judge a change in context, not just the diff

Tech Stack