When LLMs Hallucinate CVEs: The SQLite Vulnerability Report That Wasn't

The Bug Report That Never Was

In early 2025, a security researcher submitted a vulnerability report claiming multiple critical CVEs in SQLite. The report was detailed, technically fluent, and referenced specific code paths, buffer overflow conditions, and exploitation scenarios. It looked like the work of a competent reverse engineer who had spent weeks in the SQLite source tree.

It was entirely fabricated.

The researcher had used a large language model to generate the report. Not to assist in analysis—to create the analysis from scratch. The LLM hallucinated function names that don't exist, memory corruption patterns that SQLite's architecture explicitly prevents, and exploitation chains that made no sense against the actual codebase.

JFrog's security research team tore the report apart and published their findings. Their conclusion was blunt: these weren't real vulnerabilities. They were "LLM slop"—plausible-sounding technical prose with zero grounding in reality.

This wasn't a junior engineer making a mistake. This was a deliberate experiment that exposed something uncomfortable: current LLMs can generate security reports convincing enough to trigger a CVE assignment process. And if a human reviewer hadn't dug into the actual source code, those CVEs might have sailed through.

Anatomy of a Hallucinated Vulnerability

Let's get specific about what the LLM generated and why it fooled people at first glance.

The report claimed a heap buffer overflow in SQLite's printf() implementation. Real vulnerability reports follow a pattern: they identify the vulnerable function, show the input that triggers the bug, trace the memory corruption, and demonstrate control of the instruction pointer. The LLM reproduced this pattern perfectly. It named a function (sqlite3_str_vappendf), described a format string edge case, and walked through a hypothetical heap layout showing how an attacker could overwrite adjacent allocations.

The problem? That function doesn't handle the code path described. SQLite's custom printf implementation uses a fundamentally different internal structure than what the LLM assumed. The memory layout described was architecturally impossible given SQLite's allocation strategy. Every technical detail was wrong, but wrong in a way that required deep familiarity with the codebase to spot.

This is the dangerous pattern: LLMs are excellent at generating text that matches the form of expert output without containing the substance. For vulnerability reports specifically, this is catastrophic. The entire CVE ecosystem relies on trust in the reporter's technical diligence. An LLM can bypass that trust mechanism instantly.

Why This Hits FDEs Harder Than Most

Forward Deployed Engineers sit at the intersection of customer problems, production code, and rapid prototyping. You're often the first person evaluating whether a reported vulnerability affects your customer's deployment. You're also the person building AI features that might themselves generate security-relevant output.

This incident exposes three specific risks for FDEs:

1. Triage contamination. If you're on-call and a customer forwards a scary-looking CVE report, your instinct is to assess impact fast. A well-formatted, technically fluent report creates authority bias. You might spend hours investigating a phantom vulnerability before realizing the report itself is synthetic.

2. AI-generated security claims in your own pipelines. If you're building features that use LLMs to analyze code, generate documentation, or triage issues, your system can produce the same class of hallucination. Imagine an internal tool that scans dependencies and generates security advisories. Without guardrails, it could flag nonexistent vulnerabilities and trigger expensive, unnecessary remediation work.

3. Trust erosion with customers. If you pass an AI-generated security claim to a customer without verification, and they discover it's hallucinated, you've damaged the engineering relationship. In the FDE world, trust is your primary currency.

This ties directly to the broader pattern explored in our piece on why LLMs amplify the gap between senior and junior engineering output. The SQLite incident is a perfect case study: the LLM generated output that looked expert-level, but only actual expertise could detect the fraud. The tool didn't close the gap—it widened the surface area for confident-sounding mistakes.

The Prompt Engineering Trap We're All In

The SQLite report didn't emerge from a single prompt like "find vulnerabilities in SQLite." The researcher used a multi-step chain: feed the LLM source code snippets, ask it to identify potential issues, then ask it to elaborate each issue into a full CVE report. Each step looked reasonable in isolation. The final output was a composite of plausible fragments stitched together by the model's pattern-matching.

This reveals the fundamental limitation of prompt engineering as a safety mechanism. You cannot prompt your way out of hallucination when the task requires factual grounding the model doesn't possess. The LLM doesn't know it doesn't know SQLite internals. It knows the shape of a vulnerability report and generates text that conforms to that shape.

For engineers building AI systems, the uncomfortable truth is that chain-of-thought, retrieval-augmented generation, and structured prompting all reduce hallucination rates but cannot eliminate them. When the output domain is security vulnerabilities—where a single false positive can trigger multi-team fire drills—"reduced" isn't good enough.

How to Build Defensive Pipelines Against AI Slop

The SQLite incident isn't an argument against using LLMs. It's an argument for treating LLM output the way you'd treat a junior engineer's first draft: useful as a starting point, dangerous as a final deliverable.

Here's a defensive architecture for any pipeline that generates security-relevant claims:

Static Verification means checking every claim against the actual codebase. If the LLM says "function X has a buffer overflow at line Y," your pipeline should verify that function X exists, that line Y contains what the LLM claims, and that the described data flow is possible. This is automatable for most claims.

Sandbox Execution takes this further: if the LLM provides a proof-of-concept, execute it in an isolated environment against the exact version of the software. A hallucinated exploit will fail. A real one will produce observable effects. This is the gold standard for filtering synthetic vulnerabilities.

Source Grounding Check is a specific technique: for every technical claim, require the LLM to cite the exact source file, function signature, and line number. Then programmatically verify those citations exist. LLMs are bad at precise citations when hallucinating—they'll generate plausible but nonexistent function names. This check catches them.

Confidence Scoring doesn't mean asking the LLM "how confident are you?" (it will lie). It means measuring internal consistency: does the exploit chain logically hold together? Do the described memory offsets align with the actual structure sizes? Inconsistency signals hallucination.

The key insight: none of these gates require human judgment for the initial filter. You can build a pipeline that catches 90%+ of hallucinated vulnerability reports automatically before a human ever looks at them.

A Practical Test: Reproducing the Hallucination Pattern

You can test this pattern yourself with any capable LLM. The exercise is instructive for understanding the failure mode.

Take an open-source codebase you know well. Feed the LLM a few hundred lines of C or Rust source—enough to give it surface-level context but not enough to actually understand the architecture. Ask it: "Identify potential memory safety issues in this code and generate a vulnerability report."

The model will likely produce something that looks credible. It will identify pointer arithmetic, format string usage, or array indexing and construct a plausible overflow scenario. Then do what the JFrog researchers did: check every claim against the actual source. You'll find that the specific code paths don't exist, the described data flows are architecturally impossible, or the exploitation primitives don't match the actual memory layout.

This exercise builds the muscle memory that distinguishes senior engineers from juniors in the AI era: the reflex to verify before trusting. It's the same deliberate practice approach we recommend for retyping LLM-generated code to prevent cognitive debt. The goal isn't to avoid LLMs—it's to build the verification instincts that make LLMs safe to use.

For FDEs specifically, this practice maps directly to customer work. When a customer asks "is this AI-generated vulnerability report real?" you need to have internalized the verification workflow. You can't learn it in the moment.

The Balanced Take: LLMs as Accelerators, Not Authorities

None of this means LLMs are useless for security work. They're extraordinary accelerators when used correctly.

An LLM can triage a thousand-line diff and flag the five most interesting changes for human review. It can generate fuzzing harnesses for newly added API endpoints. It can explain complex exploit chains in plain English. These are all legitimate, high-value use cases where the LLM is operating on real input and the output is explicitly flagged as "needs human verification."

The failure mode in the SQLite incident was treating the LLM as an authority rather than an accelerator. The researcher didn't use the model to help analyze SQLite; they used it to replace analysis entirely. That's the line you cannot cross.

For FDEs shipping AI features, this distinction should shape your product design. If your feature generates security claims, the UI should never present them as definitive. Every claim should carry a confidence indicator, a link to the evidence, and an explicit prompt for human verification. Your customers should never be in a position where they're making security decisions based on unverified AI output.

This design philosophy extends beyond security. The FDE toolkit approach—building integration wrappers and demo scaffolds—works because those artifacts are explicitly provisional. They're starting points, not finished products. Apply that same mindset to AI-generated output: it's a scaffold for human judgment, not a replacement for it.

FAQ: Hallucinated CVEs and AI-Generated Security Claims

Q: How do I verify if a CVE report was AI-generated?

Check for specific, verifiable claims: function names, line numbers, memory offsets. Cross-reference each against the actual source code. AI-generated reports typically fail on specifics—the function might not exist, or the described data flow doesn't match the architecture. Also look for overly polished prose with no signs of actual debugging (stack traces, register dumps, GDB output).

Q: Should I ban LLMs from my security pipeline?

No. Use them for triage, explanation, and fuzzing harness generation. Never use them as the final authority on whether a vulnerability exists. The pipeline architecture described above—static verification, sandbox execution, source grounding checks—lets you capture the acceleration benefits while filtering out hallucinations.

Q: Can prompt engineering fix this?

Prompt engineering can reduce hallucination rates but cannot eliminate them. Chain-of-thought, few-shot examples, and explicit grounding instructions all help, but the fundamental limitation remains: the model doesn't know what it doesn't know. For security-critical claims, you need verification gates beyond prompting.

Q: What's the risk to the CVE ecosystem?

If AI-generated vulnerability reports become common, CVE authorities will need to implement verification requirements before assignment. This could slow down legitimate vulnerability disclosure. There's also a risk of CVE fatigue—if enough hallucinated CVEs get assigned and later retracted, the ecosystem's credibility erodes. This is already happening in adjacent spaces like AI-generated DMCA takedown requests.

Q: How does this relate to FDE work specifically?

FDEs are often the bridge between security research and customer impact. If you're evaluating a CVE for a customer deployment, you need to be able to spot synthetic reports quickly. If you're building AI features that touch security data, you need defensive pipelines. The skills required—source code verification, sandbox testing, architectural reasoning—are core FDE competencies that the highest-leverage FDE skills in the AI era emphasize heavily.

Q: What should I do if I receive a suspicious vulnerability report?

Treat it as unverified until you've independently confirmed at least one specific technical claim. Don't escalate based on the report's format or fluency. If the report lacks reproducible steps or references code that doesn't exist, flag it. The JFrog team's methodology in their original analysis is a good template: try to reproduce, check every claim against source, and publish your findings transparently.