Build a GitHub Issue Triager With Free LLMs: Auto-Label, Route Bugs

What We're Building

A GitHub App that listens for new issues, runs a zero-shot classifier on Groq’s blazing-fast inference, then labels and routes the issue automatically. No GPU, no credit card, no monthly bill.

Feature list:

• Listens to issues.opened webhook events
• Classifies issues into bug, feature, question, documentation
• Applies corresponding GitHub labels
• Assigns bug issues to an on-call engineer (configurable)
• Runs entirely on free tiers: Groq Cloud, Cloudflare Workers AI, GitHub
• Responds in under 2 seconds end-to-end

Architecture Overview

GitHub Repo (new issue)
        │
        ▼
GitHub App Webhook ──► Cloudflare Worker (ingress)
                              │
                              ├── Validate payload & signature
                              ├── Extract issue title + body
                              ├── Call Groq API (mixtral-8x7b-32768)
                              ├── Parse classification result
                              ├── Call GitHub API (add labels, assign)
                              └── Return 200 OK

Why this stack:

• Cloudflare Workers – free 100k requests/day, global edge, no cold starts
• Groq – free tier gives ~30 requests/min on Mixtral, inference under 500ms
• GitHub Apps – fine-grained permissions, no personal access tokens

Prerequisites (All Free Tier)

1. Cloudflare account – workers.dev subdomain, free tier includes 100k req/day
2. Groq API key – sign up at console.groq.com, free tier: 30 req/min, 14,400 req/day
3. GitHub account – obviously
4. Node.js 18+ – for Wrangler CLI
5. Wrangler CLI – npm install -g wrangler

Step 1: Scaffold a Cloudflare Worker

mkdir gh-issue-triager
cd gh-issue-triager
npm create cloudflare@latest . -- --type simple

Accept defaults. This gives you a bare src/index.js and wrangler.toml.

wrangler.toml (replace with your values):

name = "gh-issue-triager"
main = "src/index.js"
compatibility_date = "2024-12-01"

[vars]
GROQ_API_KEY = "gsk_your_key_here"
GITHUB_APP_ID = "123456"
GITHUB_APP_PRIVATE_KEY = """-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----"""
GITHUB_WEBHOOK_SECRET = "your_webhook_secret"

Important: Never commit secrets. Use wrangler secret put in production, but for this guide we’ll use vars for simplicity.

Step 2: Register a GitHub App and Get Credentials

1. Go to Settings > Developer settings > GitHub Apps > New GitHub App
2. Fill in:
   • Name: Issue Triager (dev)
   • Homepage URL: your repo URL
   • Webhook URL: https://gh-issue-triager.your-subdomain.workers.dev/webhook
   • Webhook secret: generate a random string, save it
3. Permissions:
   • Issues: Read & Write
   • Metadata: Read-only (auto-selected)
4. Subscribe to events: Issues
5. Where can this app be installed? Only on this account (for now)
6. Create the app, then generate a private key (.pem file)
7. Note the App ID at the top
8. Install the app on your test repository

Step 3: Implement the Webhook Handler

src/index.js:

import { verify } from '@octokit/webhooks-methods';

export default {
  async fetch(request, env) {
    if (request.method !== 'POST') {
      return new Response('Method not allowed', { status: 405 });
    }

    const signature = request.headers.get('x-hub-signature-256');
    const payload = await request.text();

    // Verify webhook signature
    const isValid = await verify(env.GITHUB_WEBHOOK_SECRET, payload, signature);
    if (!isValid) {
      return new Response('Invalid signature', { status: 401 });
    }

    const event = request.headers.get('x-github-event');
    const body = JSON.parse(payload);

    // Only process newly opened issues
    if (event === 'issues' && body.action === 'opened') {
      // Fire and forget – we respond 200 immediately
      const ctx = new ExecutionContext();
      ctx.waitUntil(handleIssue(body, env));
    }

    return new Response('OK', { status: 200 });
  },
};

Why fire-and-forget: GitHub expects a response within 10 seconds. Groq is fast, but network + GitHub API calls can push it. We acknowledge the webhook immediately and process asynchronously via waitUntil.

Step 4: Call Groq for Issue Classification

Add this function to src/index.js:

async function classifyIssue(title, body, env) {
  const prompt = `Classify this GitHub issue into exactly one category: bug, feature, question, documentation.
Respond with ONLY the category name, nothing else.

Title: ${title}
Body: ${body.substring(0, 1000)}

Category:`;

  const response = await fetch('https://api.groq.com/openai/v1/chat/completions', {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${env.GROQ_API_KEY}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      model: 'mixtral-8x7b-32768',
      messages: [{ role: 'user', content: prompt }],
      temperature: 0.1,
      max_tokens: 10,
      stop: ['\n'],
    }),
  });

  const data = await response.json();
  const category = data.choices[0].message.content.trim().toLowerCase();

  // Validate output
  const validCategories = ['bug', 'feature', 'question', 'documentation'];
  return validCategories.includes(category) ? category : 'question';
}

Why Mixtral: It’s the best free model on Groq for instruction-following. Temperature 0.1 keeps it deterministic. We truncate body to 1000 chars to stay under token limits.

Step 5: Apply Labels and Assignees via GitHub API

Add the handleIssue function:

async function handleIssue(payload, env) {
  const { repository, issue } = payload;
  const [owner, repo] = repository.full_name.split('/');

  try {
    const category = await classifyIssue(issue.title, issue.body, env);

    // Get installation access token
    const token = await getInstallationToken(payload.installation.id, env);

    // Apply label
    await applyLabel(owner, repo, issue.number, category, token);

    // Assign if bug
    if (category === 'bug') {
      await assignIssue(owner, repo, issue.number, token);
    }

    console.log(`Issue #${issue.number} classified as ${category}`);
  } catch (error) {
    console.error('Error handling issue:', error);
  }
}

async function getInstallationToken(installationId, env) {
  const appToken = await getAppJwt(env);
  const response = await fetch(
    `https://api.github.com/app/installations/${installationId}/access_tokens`,
    {
      method: 'POST',
      headers: {
        'Authorization': `Bearer ${appToken}`,
        'Accept': 'application/vnd.github.v3+json',
      },
    }
  );
  const data = await response.json();
  return data.token;
}

async function getAppJwt(env) {
  // Generate JWT for GitHub App authentication
  const now = Math.floor(Date.now() / 1000);
  const payload = {
    iat: now - 60,
    exp: now + 600,
    iss: env.GITHUB_APP_ID,
  };

  const encoder = new TextEncoder();
  const keyData = encoder.encode(env.GITHUB_APP_PRIVATE_KEY);
  const key = await crypto.subtle.importKey(
    'pkcs8',
    pemToArrayBuffer(env.GITHUB_APP_PRIVATE_KEY),
    { name: 'RSASSA-PKCS1-v1_5', hash: 'SHA-256' },
    false,
    ['sign']
  );

  const header = { alg: 'RS256', typ: 'JWT' };
  const segments = [
    btoa(JSON.stringify(header)),
    btoa(JSON.stringify(payload)),
  ];
  const signingInput = segments.join('.');
  const signature = await crypto.subtle.sign(
    'RSASSA-PKCS1-v1_5',
    key,
    encoder.encode(signingInput)
  );
  segments.push(btoa(String.fromCharCode(...new Uint8Array(signature))));
  return segments.join('.');
}

function pemToArrayBuffer(pem) {
  const b64 = pem
    .replace('-----BEGIN RSA PRIVATE KEY-----', '')
    .replace('-----END RSA PRIVATE KEY-----', '')
    .replace(/\s/g, '');
  const binary = atob(b64);
  const bytes = new Uint8Array(binary.length);
  for (let i = 0; i < binary.length; i++) {
    bytes[i] = binary.charCodeAt(i);
  }
  return bytes.buffer;
}

async function applyLabel(owner, repo, issueNumber, label, token) {
  // First ensure the label exists
  const existingLabels = await fetch(
    `https://api.github.com/repos/${owner}/${repo}/labels`,
    { headers: { Authorization: `token ${token}` } }
  ).then(r => r.json());

  const labelExists = existingLabels.some(l => l.name === label);
  if (!labelExists) {
    // Create label with a default color
    const colors = { bug: 'd73a4a', feature: 'a2eeef', question: 'd876e3', documentation: '0075ca' };
    await fetch(`https://api.github.com/repos/${owner}/${repo}/labels`, {
      method: 'POST',
      headers: { Authorization: `token ${token}`, 'Content-Type': 'application/json' },
      body: JSON.stringify({ name: label, color: colors[label] || 'ededed' }),
    });
  }

  // Apply label to issue
  await fetch(
    `https://api.github.com/repos/${owner}/${repo}/issues/${issueNumber}/labels`,
    {
      method: 'POST',
      headers: { Authorization: `token ${token}`, 'Content-Type': 'application/json' },
      body: JSON.stringify({ labels: [label] }),
    }
  );
}

async function assignIssue(owner, repo, issueNumber, token) {
  // Assign to a default user – configure this in env vars
  const assignee = 'your-oncall-engineer'; // Replace or use env var
  await fetch(
    `https://api.github.com/repos/${owner}/${repo}/issues/${issueNumber}/assignees`,
    {
      method: 'POST',
      headers: { Authorization: `token ${token}`, 'Content-Type': 'application/json' },
      body: JSON.stringify({ assignees: [assignee] }),
    }
  );
}

Label colors: We auto-create labels with standard GitHub colors if they don’t exist. You can customize the palette.

Step 6: Deploy and Test End-to-End

# Install dependencies
npm install @octokit/webhooks-methods

# Deploy
wrangler deploy

# Tail logs to debug
wrangler tail

Testing:

1. Go to your test repo
2. Create a new issue: "App crashes when clicking Save button"
3. Within 2 seconds, the bug label should appear, and your on-call engineer gets assigned
4. Try a feature request: "Add dark mode toggle" → feature label

Verify in Groq Console: Check usage at console.groq.com – each classification uses ~50 tokens.

Sensible Extensions

• Route by team: Extend classification to frontend, backend, infra and assign to different teams
• Add priority: Use Groq to also output P0 through P3 based on keywords like "crash", "data loss"
• Auto-close spam: Classify as spam and close with a comment
• Sentiment analysis: Flag angry users for priority response
• Slack notification: Post to a channel when a bug is filed – use Cloudflare Workers AI free tier
• Dashboard: Log classifications to Cloudflare D1 (free tier) for analytics

Common Pitfalls

1. Webhook timeout: Always use waitUntil – GitHub retries if you don’t respond in 10s
2. Rate limits: Groq free tier is 30 req/min. If you get bursts of issues, implement a queue or exponential backoff
3. PEM formatting: The private key must include \n newlines in TOML. Use triple-quoted strings
4. Label already exists: GitHub returns 422 if you try to create a duplicate label. We check first, but add error handling for race conditions
5. Mixtral hallucination: Occasionally outputs "Category:" prefix. Our validation catches it and defaults to question
6. Installation token expiry: Tokens expire after 1 hour. Since we generate fresh per webhook, this isn’t an issue

FAQ

Q: Can I use this on a public repo with many issues? A: Yes. Groq’s free tier handles ~14k issues/day. For larger volumes, batch process or upgrade to Groq’s paid tier ($0.27/1M tokens).

Q: What if Groq is down? A: The worker will throw, and GitHub will retry the webhook up to 3 times. Consider adding a fallback to Cloudflare Workers AI with Llama 3.

Q: Can I customize the classification categories? A: Absolutely. Edit the prompt and validCategories array. The model handles arbitrary labels well.

Q: Is the private key secure in wrangler.toml? A: No. Use wrangler secret put GITHUB_APP_PRIVATE_KEY for production. We used vars for tutorial simplicity.

Q: How do I add more complex routing logic? A: Replace the simple classifyIssue prompt with a structured output approach using JSON mode – available on Groq’s paid tier, or parse the free response carefully.

Q: Can I run this on a schedule instead of webhooks? A: Yes, use Cloudflare Cron Triggers to poll GitHub issues API periodically – useful for repos where you can’t install apps.

---

Next steps: Grab the complete source from our GitHub template repo and have this running in 10 minutes. For production hardening, read our guide on securing Cloudflare Workers.