Build a PR Review Bot That Comments on GitHub PRs with a Free LLM

What We're Building

An automated PR review bot that fires on every new or updated pull request. It grabs the diff, ships it to a free LLM inference endpoint, and posts inline review comments directly on the PR's changed lines. No SaaS subscriptions, no GPU rental, no token invoices.

Feature list:

• Triggers on pull_request open and synchronize events
• Parses unified diffs and maps hunks to file/line positions
• Sends code chunks to Cloudflare Workers AI (Llama 3.1 8B or similar free model)
• Posts inline review comments via GitHub's REST API with Octokit
• Skips files over a configurable size threshold to stay under rate limits
• Marks the review as a single coherent review (not a spray of individual comments)

This is the same pattern FDEs use to ship a working prototype in a day—no fluff, no over-engineering, just a tight loop from diff to actionable feedback.

Architecture: How the Pieces Fit

The GitHub Actions runner checks out the PR branch, computes the diff against the base, and ships chunks to a Cloudflare Worker. The Worker calls Workers AI (free tier, 10k neurons/day), formats the response into the shape Octokit expects, and posts back. The Action never touches an API key for a paid LLM provider.

If you've built multi-agent pipelines like the Multi-Agent Research Assistant, this is a simpler single-agent pattern—but the orchestration discipline is the same: keep the LLM call stateless, idempotent, and retryable.

Prerequisites (All Free Tier)

Free tier limits to know:

• Workers AI: 10,000 neurons per day (roughly 1,000-2,000 Llama 3.1 8B calls depending on input length)
• GitHub Actions: 2,000 minutes/month for private repos, unlimited for public
• Cloudflare Workers: 100,000 requests/day free

For a typical team's PR volume, you'll stay well within these limits. If you're handling higher throughput, the SQL Analyst Agent guide covers similar free-tier scaling patterns.

Step 1: Scaffold the Cloudflare Worker

Create a new Worker project:

npm create cloudflare@latest pr-review-bot -- --type=hello-world
cd pr-review-bot
npm install @octokit/rest

Edit wrangler.toml to bind Workers AI:

name = "pr-review-bot"
main = "src/index.js"
compatibility_date = "2024-12-01"

[ai]
binding = "AI"

The Worker will receive POST requests containing code diffs and a GitHub token. It calls Workers AI, parses the response, and posts a review. Keep the Worker thin—no diff parsing here, just inference and API calls.

Step 2: Wire Up the LLM Inference

In src/index.js, set up the core handler:

export default {
  async fetch(request, env) {
    if (request.method !== 'POST') {
      return new Response('POST only', { status: 405 });
    }

    const { codeChunk, filePath, lineStart, githubToken, owner, repo, pullNumber, commitId } = await request.json();

    const prompt = `You are a senior code reviewer. Review the following code diff chunk from ${filePath}.
Focus on: bugs, security vulnerabilities (SQL injection, XSS, auth bypass), logic errors, and style issues that could cause bugs.
Be concise. Return a JSON array of comments with fields: "line" (relative to the chunk start), "body" (the comment text).
If no issues found, return an empty array.

Code chunk (starting at line ${lineStart}):
\`\`\`
${codeChunk}
\`\`\``;

    const response = await env.AI.run('@cf/meta/llama-3.1-8b-instruct', {
      messages: [{ role: 'user', content: prompt }],
      max_tokens: 512,
    });

    const rawOutput = response.response || response;
    let comments;
    try {
      // Extract JSON from potential markdown fences
      const jsonMatch = rawOutput.match(/```json\s*([\s\S]*?)\s*```/) || rawOutput.match(/```\s*([\s\S]*?)\s*```/);
      const jsonStr = jsonMatch ? jsonMatch[1] : rawOutput;
      comments = JSON.parse(jsonStr);
    } catch {
      comments = [];
    }

    return new Response(JSON.stringify({ comments }), {
      headers: { 'Content-Type': 'application/json' },
    });
  }
};

Why Llama 3.1 8B? It's the most capable free model on Workers AI, handles code review prompts well, and fits within the 10k neuron daily cap for moderate PR volume. The prompt engineering here is intentionally narrow—we're asking for structured JSON output and giving explicit focus areas to reduce hallucinations.

Deploy with wrangler deploy. Note your Worker URL (e.g., https://pr-review-bot.your-subdomain.workers.dev).

Step 3: Build the GitHub Action Runner

Create .github/workflows/pr-review.yml in your target repo:

name: AI PR Review

on:
  pull_request:
    types: [opened, synchronize]

permissions:
  contents: read
  pull-requests: write

jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Run PR Review
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          WORKER_URL: ${{ secrets.WORKER_URL }}
        run: |
          node .github/scripts/pr-review.js

The pull-requests: write permission is critical—without it, Octokit can't post review comments. The fetch-depth: 0 ensures we have the full git history to compute a proper diff.

Create .github/scripts/pr-review.js:

const { execSync } = require('child_process');
const { Octokit } = require('@octokit/rest');

const octokit = new Octokit({ auth: process.env.GITHUB_TOKEN });
const workerUrl = process.env.WORKER_URL;

async function main() {
  const event = JSON.parse(process.env.GITHUB_EVENT_PATH || '{}');
  const { owner, repo } = event.repository;
  const pullNumber = event.pull_request.number;
  const baseRef = event.pull_request.base.sha;
  const headRef = event.pull_request.head.sha;

  // Get the diff
  const diff = execSync(`git diff ${baseRef}..${headRef}`).toString();
  if (!diff.trim()) {
    console.log('No diff found, skipping review.');
    return;
  }

  const chunks = parseDiffIntoChunks(diff, 1500); // 1500 chars per chunk
  const allComments = [];

  for (const chunk of chunks) {
    const response = await fetch(workerUrl, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({
        codeChunk: chunk.code,
        filePath: chunk.filePath,
        lineStart: chunk.lineStart,
        githubToken: process.env.GITHUB_TOKEN,
        owner,
        repo,
        pullNumber,
        commitId: headRef,
      }),
    });

    const { comments } = await response.json();
    for (const c of comments) {
      allComments.push({
        path: chunk.filePath,
        line: chunk.lineStart + c.line - 1,
        body: c.body,
      });
    }
  }

  if (allComments.length > 0) {
    await octokit.pulls.createReview({
      owner,
      repo,
      pull_number: pullNumber,
      commit_id: headRef,
      event: 'COMMENT',
      comments: allComments,
    });
    console.log(`Posted ${allComments.length} review comments.`);
  } else {
    console.log('No issues found. No review comments posted.');
  }
}

main().catch(err => {
  console.error(err);
  process.exit(1);
});

Step 4: Diff Parsing and Inline Comment Logic

Add the parseDiffIntoChunks function to pr-review.js. This is the trickiest part—incorrect line mapping means comments land on the wrong lines.

function parseDiffIntoChunks(diff, maxChunkSize) {
  const chunks = [];
  const lines = diff.split('\n');
  let currentFile = null;
  let currentLineStart = 0;
  let currentChunk = '';

  for (const line of lines) {
    // Detect file header
    const fileMatch = line.match(/^diff --git a\/(.*) b\/(.*)/);
    if (fileMatch) {
      // Flush previous file's remaining chunk
      if (currentChunk.trim()) {
        chunks.push({
          filePath: currentFile,
          lineStart: currentLineStart,
          code: currentChunk,
        });
      }
      currentFile = fileMatch[2];
      currentLineStart = 0;
      currentChunk = '';
      continue;
    }

    // Detect hunk header: @@ -oldStart,oldCount +newStart,newCount @@
    const hunkMatch = line.match(/^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@/);
    if (hunkMatch) {
      // Flush previous chunk if it exists
      if (currentChunk.trim()) {
        chunks.push({
          filePath: currentFile,
          lineStart: currentLineStart,
          code: currentChunk,
        });
      }
      currentLineStart = parseInt(hunkMatch[1], 10);
      currentChunk = line + '\n';
      continue;
    }

    // Skip binary files and index lines
    if (line.startsWith('index ') || line.startsWith('Binary files')) continue;

    // Accumulate lines, splitting when approaching maxChunkSize
    if (currentChunk.length + line.length > maxChunkSize && currentChunk.trim()) {
      chunks.push({
        filePath: currentFile,
        lineStart: currentLineStart,
        code: currentChunk,
      });
      // Start new chunk from current hunk position
      currentChunk = line + '\n';
    } else {
      currentChunk += line + '\n';
    }
  }

  // Flush final chunk
  if (currentChunk.trim()) {
    chunks.push({
      filePath: currentFile,
      lineStart: currentLineStart,
      code: currentChunk,
    });
  }

  return chunks;
}

Key detail: The lineStart tracks the new file line number (the + side of the hunk header). When the LLM returns a comment with line: 3, we compute the absolute line as chunk.lineStart + 3 - 1 because the hunk's first line at +startLine is index 0 in our chunk.

Step 5: Deploy and Test End-to-End

1. Add the Worker URL secret to your GitHub repo: Settings → Secrets and variables → Actions → New repository secret. Name it WORKER_URL, value is your deployed Worker URL.

2. Push the workflow and script to your default branch:

   git add .github/workflows/pr-review.yml .github/scripts/pr-review.js
   git commit -m "Add AI PR review bot"
   git push
   

3. Create a test PR with a deliberate bug:

   // In some file:
   const query = "SELECT * FROM users WHERE id = '" + userId + "'";
   

4. Watch the Action run. Within 30-60 seconds, you should see an inline comment flagging the SQL injection.

If nothing appears, check the Action logs. Common issues: Worker URL misconfigured, missing pull-requests: write permission, or the LLM returning malformed JSON (the try/catch handles this gracefully).

For a deeper dive on shipping prototypes under constraints, see How FDEs Turn a Messy Customer Problem into a Shipped Prototype in a Week.

Sensible Extensions

Add a review summary comment. Instead of only inline comments, post a top-level review body summarizing the findings. Modify the Worker to return a summary field alongside comments.

File-type filtering. Skip generated files, minified bundles, and lockfiles. Add a shouldReview(filePath) function that checks extensions and paths:

const SKIP_PATTERNS = [
  /\.lock$/,
  /\.min\.(js|css)$/,
  /package-lock\.json$/,
  /dist\//,
  /node_modules\//,
];

function shouldReview(filePath) {
  return !SKIP_PATTERNS.some(p => p.test(filePath));
}

Confidence scoring. Ask the LLM to include a confidence field (0-1) and only post comments above a threshold (e.g., 0.7). This reduces noise from low-confidence hallucinations.

Multi-model fallback. If Workers AI is rate-limited, fall back to a Groq free-tier endpoint. The pattern is identical to the Multi-Agent Research Assistant setup.

Review existing PR comments. Before posting, fetch existing review comments on the PR and deduplicate. If the bot already flagged line 42, don't post the same finding again.

Common Pitfalls

Line number off-by-one. The most frequent bug. Test with a single-file, single-hunk diff first. Verify that line numbers in the GitHub UI match what your parser computes.

Hunk header regex too greedy. The regex @@ -\d+(?:,\d+)? \+(\d+)(?:,\d+)? @@ handles standard unified diff headers. Some git configurations produce @@ -1 +1,3 @@ (no count on the old side). Test with your repo's actual diff output.

LLM returns non-JSON. Even with explicit prompting, Llama 3.1 sometimes wraps JSON in markdown fences or adds trailing text. The extraction logic in Step 2 handles this, but monitor your logs for parse failures.

Rate limiting. Workers AI free tier is 10k neurons/day. Each review call with 1500-char chunks uses roughly 5-10 neurons. That's 1,000-2,000 chunks per day. For repos with large PRs, increase maxChunkSize or add a daily cap.

GitHub token scope. The default GITHUB_TOKEN in Actions has the necessary permissions if you declare pull-requests: write. If you're using a personal access token, it needs repo scope.

FAQ

Q: Can I use a different free LLM provider? Yes. Swap the Worker's env.AI.run call for a fetch to Groq (free tier), Google Gemini (free tier), or any OpenAI-compatible endpoint. The Worker pattern stays identical—just change the fetch URL and auth header.

Q: What if the diff is huge? The chunker splits at 1500 characters by default. For PRs with 50+ files, the Action might hit the 6-hour timeout. Add a MAX_FILES limit (e.g., 20) and skip the rest with a comment noting the limit.

Q: Does this work with private repositories? Yes, but GitHub Actions minutes are limited to 2,000/month on the free plan. A typical review run takes 30-60 seconds, so you can handle roughly 2,000-4,000 PRs per month.

Q: How do I prevent the bot from reviewing its own PRs? Check the PR author in the Action script. If event.pull_request.user.login === 'github-actions[bot]', skip the review.

Q: Can I customize the review prompt per repository? Absolutely. Add a .github/pr-review-prompt.md file to your repo and read it in the Action script. Pass it as an additional field to the Worker, which prepends it to the system prompt. This lets teams define their own style guides and security policies.

Q: How does this compare to CodeRabbit or other paid tools? It's lighter weight and free, but you own the prompt engineering and maintenance. For teams that want full control over what gets flagged and how, this is the right tradeoff. If you're evaluating build-vs-buy decisions in an enterprise context, the Deploying an LLM Feature at an Enterprise Customer case study walks through the same calculus.