Claude Code Effort A/B Tests Signal LLM-as-a-Service Pricing Shift

What Happened: The A/B Test on Effort

On March 21, 2025, a user reported on X that Claude Code appeared to be serving reduced effort levels without explicit user configuration. The symptom was stark: a coding agent that normally fires off 15-20 tool calls per turn suddenly became far more conservative, executing only a handful of operations before pausing or asking for confirmation.

This isn't a bug. It's an A/B test on the effort parameter—a dial that controls how aggressively the model pursues multi-step tool execution before yielding control back to the human in the loop. Anthropic has been tuning this parameter server-side, and some users are landing in experimental buckets where effort is clamped lower than the default.

The observable difference is quantitative. A typical Claude Code session with default effort might chain together file reads, grep searches, edits, and test runs in a single burst. Under the reduced-effort variant, the model stops after one or two tool calls, effectively forcing more frequent human checkpoints. For engineers who've grown accustomed to the "fire-and-forget" flow of high-effort agentic coding, this feels like a regression. But it's more nuanced than that.

The Effort Dial, Explained

Anthropic introduced the effort parameter as a control surface for agentic behavior. It's not a temperature setting, a token limit, or a system prompt tweak. It directly influences the model's propensity to continue executing tool calls without re-prompting. Think of it as a throttle on autonomy:

The A/B test appears to be shifting some users from Medium to Low—or introducing a new intermediate tier—without changing the client-side API. This is classic server-side experimentation: change one variable, measure completion rates, error rates, and token consumption, then decide whether to roll out broadly.

Why This Matters for Engineers and FDEs

For individual developers, the immediate impact is on workflow rhythm. If you've built muscle memory around Claude Code executing a full task while you context-switch to Slack or email, the reduced-effort variant breaks that flow. You're pulled back to the terminal more frequently, which can feel like micromanaging a junior dev who suddenly needs approval for every sed command.

But the implications run deeper for Forward Deployed Engineers (FDEs) and anyone building AI-augmented workflows in production environments.

Latency and Cost Are Coupled

Every tool-call pause is a round-trip to the API. More pauses mean more individual requests, each with its own context window overhead. Paradoxically, reduced effort can increase total token consumption because the model re-ingests the full conversation history on each turn. A high-effort agent that chains 10 edits in one go might burn 50K tokens. The same task split across 5 low-effort turns could burn 80K+ tokens due to repeated context prefixes.

For FDEs operating inside customer environments—where API costs are often passed through or scrutinized—this isn't trivial. A 60% increase in token consumption per task completion directly impacts margins and customer perception. If you're embedding AI tooling into a client's CI/CD pipeline, as discussed in our Palantir Embed Model breakdown, cost predictability is non-negotiable.

Automation Reliability Takes a Hit

The most powerful Claude Code workflows are those where the engineer defines a goal and the agent executes autonomously. Reduced effort breaks the continuity of these chains. A multi-file refactor that previously completed in one pass now requires manual approval at each step, introducing failure modes: the engineer gets distracted, the context decays, or a partial state is left dangling.

This is exactly the problem space that tools like Autolith aim to solve—closing the loop with a live runtime so the agent can verify its own work. When effort is clamped, the agent loses the autonomy to run that verification loop, and the human becomes the bottleneck.

The Hidden Shift: From Per-Token to Per-Outcome Pricing

Here's the strategic layer most engineers are missing. This A/B test isn't really about effort levels. It's a pricing signal.

The LLM-as-a-Service market is currently dominated by per-token pricing: you pay for what you consume. But per-token pricing creates a misalignment. The provider's incentive is to maximize token consumption (more revenue), while the user's incentive is to minimize it (lower cost). Agentic workflows amplify this tension because a "helpful" agent that chains many tool calls is also a very expensive agent.

Anthropic's effort A/B test looks like the early stages of a shift toward outcome-based or tiered pricing. By controlling effort server-side, Anthropic can segment users into implicit pricing tiers:

• Low effort: Budget tier. Lower cost per task, but more human involvement required.
• High effort: Premium tier. Higher cost per task, but greater autonomy and throughput.

This mirrors how every SaaS product eventually segments: Slack limits message history, GitHub caps Actions minutes, Figma restricts editor seats. The LLM providers are discovering that raw token pricing doesn't capture the value of completed work. A high-effort agent that ships a feature in one shot delivers more value than a low-effort agent that requires 10 manual approvals—even if both consume the same tokens.

For FDEs, this shift has direct implications. When you're working with product and engineering teams post-sale to integrate AI tooling, you're now negotiating not just API access but effective capability tiers. The "same model" may behave very differently depending on the pricing tier your customer is on. This isn't theoretical—it's happening now, silently, in A/B tests.

The Enterprise Angle

Enterprise contracts often include throughput guarantees, latency SLAs, and support tiers. But they rarely specify agentic effort levels. If Anthropic formalizes effort as a pricing dimension, enterprise FDEs will need to audit what their customers are actually getting. A contract that says "Claude 3.5 Sonnet access" may mean very different things depending on the effort tier provisioned behind the scenes.

How to Use or Test Reduced Effort Today

If you want to understand how this A/B test affects your workflows—or if you've been bucketed into the low-effort variant and want to regain control—here's the practical playbook.

1. Check Your Current Effort Level

Claude Code doesn't expose the active effort level directly in the UI, but you can infer it from behavior. Run a multi-step task and count the tool calls per turn:

# In Claude Code, run a task that requires multiple operations
> Refactor the authentication module to use async/await instead of callbacks.
  Update all call sites and run the test suite.

If Claude stops after 1-2 file edits and asks "Shall I continue?", you're likely in a low-effort bucket. If it powers through 5+ edits and runs tests without pausing, you're on the default or high-effort path.

2. Explicitly Set Effort (If Available)

Anthropic's API supports an effort parameter in some configurations. In Claude Code, you can try setting it via configuration or system prompt injection:

{
  "model": "claude-sonnet-4-20250514",
  "max_tokens": 4096,
  "tools": [...],
  "effort": "high"
}

Note: Server-side A/B tests may override client-specified effort. If you set effort: "high" and still see conservative behavior, you're in an experimental bucket that's ignoring the parameter.

3. Work Around Low Effort with Prompt Engineering

If you're stuck in a low-effort bucket, you can partially compensate with explicit prompting:

You are operating in an autonomous coding mode. Execute ALL steps without
asking for confirmation unless you encounter an error or ambiguity.
After each tool call, immediately proceed to the next logical step.
Do not pause to summarize or ask permission.

This isn't a perfect substitute for the effort parameter—the model may still have internal guardrails that limit chaining—but it can push behavior toward higher autonomy.

4. Monitor Token Consumption

If you suspect you're in an A/B test, track your token usage per completed task. The low-effort variant often consumes more tokens overall due to repeated context prefixes. Use Anthropic's usage dashboard or log usage fields from API responses:

import json

# After each API call, log token counts
response = client.messages.create(...)
usage = response.usage
print(f"Input: {usage.input_tokens}, Output: {usage.output_tokens}")

# Aggregate per task completion to compare efficiency

A 20-40% increase in tokens-per-task is a strong signal you're in a reduced-effort bucket.

5. The FDE Play: Dual-Mode Workflows

For FDEs building production workflows, the smart move is to design for both modes. Structure your agentic pipelines so they can operate in high-autonomy mode when available, but degrade gracefully to step-by-step execution when effort is clamped. This is similar to how Huzzah skips the chat interface by defining clear task boundaries—your workflow should be effort-agnostic.

A Balanced Take: Not a Downgrade, a Rebalancing

The engineer's instinct is to treat reduced effort as a degradation—"they're making the model dumber to save compute." That's too simplistic.

The Case for Reduced Effort

High-effort agents are impressive but error-prone. When Claude Code chains 15 tool calls without human oversight, it can propagate mistakes: a bad edit in step 3 cascades through steps 4-15, and the human discovers the mess only at the end. Reduced effort forces checkpoints that catch errors earlier. For junior engineers or unfamiliar codebases, this is genuinely safer.

There's also a security dimension. An agent with high autonomy can theoretically execute destructive operations (rm -rf, destructive git operations, database migrations) before a human can intervene. Effort clamping is a safety rail.

The Case Against

For experienced engineers who know their codebase and have strong test suites, reduced effort is pure overhead. The agent's judgment about when to pause should be based on uncertainty, not an arbitrary server-side dial. If the model is confident and correct, forcing a checkpoint wastes time and breaks flow.

The deeper concern is the pricing implication. If effort becomes a paid tier, we're moving toward a world where "AI capability" is artificially segmented—not by model quality, but by a parameter that controls autonomy. This feels like airlines charging extra for legroom in seats that already exist.

The FDE Perspective

For FDEs, the balanced take is pragmatic: this is a signal to build effort-aware workflows. Don't assume a fixed autonomy level. Design systems that can operate across the effort spectrum, and monitor the actual behavior (not just the API contract) of the models you're embedding in customer environments. The sampling settings that make local LLMs feel dumber are a reminder that model behavior is shaped by many hidden parameters—effort is just the latest one to surface.

FAQ

Q: Is Anthropic permanently reducing Claude Code's effort level?

No evidence of a permanent change yet. This is an A/B test—Anthropic is collecting data on how different effort levels affect completion rates, error rates, and user satisfaction. The final decision will depend on those metrics.

Q: Can I opt out of the A/B test?

Not directly. Server-side experiments are assigned by Anthropic's infrastructure. You can try creating a new API key or using a different account, but there's no guarantee that will change your bucket.

Q: Does this affect the Claude API or just Claude Code?

The A/B test appears focused on Claude Code (the agentic coding tool), but the effort parameter exists in the underlying API. If Anthropic rolls this out broadly, it could affect all agentic use cases.

Q: How do I know if I'm in the test?

Run a multi-step coding task. If Claude stops after 1-2 tool calls and asks for confirmation on tasks it previously completed autonomously, you're likely in a reduced-effort bucket.

Q: Will this change how I should budget for AI tooling?

Possibly. If effort tiers become a pricing dimension, you'll need to factor in not just token costs but effective capability levels. For FDEs managing customer deployments, this means auditing what effort tier your customers are actually receiving under their contracts.

Q: Is there a way to force high effort regardless of the A/B test?

No guaranteed method exists while the server-side override is active. Explicit prompt engineering can help, but the model may still have internal guardrails that limit tool-call chaining. The most reliable approach is to design workflows that function correctly at any effort level.