Free vs paid LLMs: 76-task benchmark

Ficelle does not claim “unlimited free AI”. It claims routing intelligence: send the work that free models handle well to free, and keep paying only where a paid model is measurably better. This benchmark quantifies where that line sits.

TL;DR

On 76 real tasks across four categories, free / free-tier routing was at quality parity with gpt-5 (the premium baseline) on classification (100% quality-safe), JSON extraction (95%), and reasoning (94%). The only category where the paid model clearly won was polished content writing (50%).

Routing the three parity categories to free instead of gpt-5 saves roughly $61 per 10,000 requests with no measurable quality loss: about $610/mo at 100k req/mo and $6,100/mo at 1M. The surprise: reasoning, which we expected to be free’s weak spot, held up. Content is the real “keep paying” zone.

What we tested

76 tasks across 4 categories: content writing (20), JSON extraction (20), classification (20), reasoning/code (16).
Free arm: live Ficelle routing against its free / free-tier provider pool.
Paid arm: OpenAI direct: gpt-5-mini (the realistic cheap alternative) and gpt-5 (premium, the model people most often over-use for simple work).
Quality judge: gpt-5.4, neutral to both arms, scoring each pair against a per-task rubric with A/B position swapping.
Objective checks where possible: JSON field accuracy and classification exact-match.
Total cost to run the whole benchmark: $0.68.

Results vs `gpt-5` (premium baseline)

Quality-safe share = the fraction of tasks where free ties or beats paid. “Bankable” savings count only those quality-safe tasks.

Use case	Free win/tie/loss	Quality-safe	Bankable $/mo @10k
classification	1/19/0	100%	$2.72
json-extraction	0/19/1	95%	$16.62
reasoning	0/15/1	94%	$41.54
content	4/6/10	50%	$23.16

The honest read

Classification is solved for free models. 100% quality-safe. Paying a premium model to label or route text is pure waste.
JSON extraction is near-parity (95%). Free models extract structured fields as accurately as gpt-5 on all but the occasional edge case.
Reasoning held up, the surprise. We expected multi-step reasoning/code to be where free fell apart. It didn’t: 94% quality-safe. Because these tasks are token-heavy, this is also the largest bankable saving.
Content is the real weak spot. Polished writing under tight constraints is where gpt-5 genuinely wins. The free output is serviceable but less refined: the honest “keep the paid model, or have a human edit” zone.
The “bankable” column is the point. It strips out savings that come at a quality cost. Never quote raw savings; quote bankable.

The economics

Savings scale linearly with volume. Summing the three parity categories versus gpt-5:

Monthly volume (per category)	Bankable saving, quality intact
10,000	~$61/mo
100,000	~$610/mo
1,000,000	~$6,100/mo

The real lever is not “free beats mini by pennies”. It is finding the premium-model calls a team makes on work a free model does identically, and the volume they run it at.

Routing recommendation

Route to free, no quality loss: classification, JSON/structured extraction, and most reasoning/code.
Keep premium (or human-edit): customer-facing polished content.
Ficelle’s job is to make this split automatic and observable, and to fall back safely when a free provider rate-limits or fails, so the savings do not come with reliability surprises.

Caveats

Quality is judged by an LLM against rubrics: directional, not absolute. Human spot-check before quoting numbers.
Free availability is not guaranteed; one free call timed out during testing. This is exactly why fallback and observability matter.
Prices are published list prices as of 18/06/2026 (gpt-5-mini $0.25/$2.00, gpt-5 $1.25/$10.00 per 1M tokens). Re-verify before quoting.
Sample size is 20 tasks/category, enough to kill N=4 noise, but a real audit should use a larger, workload-specific set.