Ficelle
Provider breakage is the silent killer of AI agents
Free and free-tier LLM providers rate-limit, retire models, and misreport capabilities, weekly. If your agent points at one provider, it breaks when they do. Routing with cooldowns, quarantine, and fallback is the fix.
Point an agent at a single LLM provider and you have inherited that provider’s worst day. Free and free-tier endpoints are the most volatile of all: quota windows differ across providers, models get retired with little notice, and a few even misreport what they can do.
The three ways providers break
- Rate limits. Providers can limit requests, tokens, or shared compute. A 429 can indicate a short throughput window or an exhausted daily allowance; check the response and quota scope before retrying.
- Retirement. A model id you depend on returns 404/410 one morning because the provider pulled it.
- Capability drift. A model claims tool support in its catalog but fails the actual tool call, or silently drops structured-output mode.
Any one of these turns a working agent into a stream of failed runs, and if your framework has a silent paid fallback, into a surprise bill instead.
Diagnose the quota before retrying
Groq enforces organization-wide request and token limits. Its request-remaining headers refer to daily requests; token-remaining headers refer to tokens per minute. Gemini shares quota across keys in the same project. Requesty shares its free allowance across free models, so changing model IDs does not reset it.
OpenRouter distinguishes a negative-balance 402 from an upstream-capacity 429 in its limits documentation. Check the error body and account state before adding retries. With Cerebras, an oversized max_completion_tokens can reserve enough quota to reject a request before inference; its uncached and total token buckets are separate.
How Ficelle absorbs it
Ficelle treats every provider as unreliable by default and routes around failure:
- Cooldowns. A model that errors or rate-limits is put on a timed cooldown so traffic moves to the next candidate instead of hammering a dead endpoint.
- Quarantine. Persistent failures, such as a retired model returning 404/410, or a model the runtime reports as billed, are quarantined out of the eligible pool.
- Benchmarked capability. Capability checks measure tool-call, JSON, and vision compliance. A model that claims a capability but fails the check doesn’t get that work.
- Safe fallback. When the top model is unavailable, Ficelle can retry another configured, enabled candidate that meets the request requirements, within strict-zero rules, so fallback never silently means “paid”.
If no eligible candidate can serve the request, Ficelle returns an error. Quota-provider billing remains subject to the upstream account plan; a local guard cannot make paid overflow free.
Keeping the provider pool usable
Free models are a commodity; they exist on many providers at once. Keeping a pool usable takes repeated checks of model IDs, quotas, capabilities, and account access. Ficelle maintains those routing checks and provider integrations while keeping keys on your machine.
