Which model for which job: what we actually run, and what it costs

2,059 words, about 9 min read

On 4 August someone on the platform team pointed a frontier model at 2,800 PHP files to rename one interface and update its implementations. It worked. It cost about USD 496 in one pass, and we ran three passes because the first two prompts produced diffs that failed bin/phpunit. Rector did 2,640 of those files in under two minutes for nothing, on the fourth attempt, after I stopped being clever.

That is the whole argument of this post. We are a backend team in Manila running a Symfony 8 platform on PHP 8.4 and an internal ops and finance app on Laravel 13. We pay for model tokens out of a real budget that finance reviews monthly. What follows is how we route work to models and agent tooling, with the arithmetic, and the places where we got it wrong.

What we pay, as of 25 August 2026

Prices are per million tokens, taken from each vendor’s own pricing page on 25 August 2026. They move constantly: OpenAI cut GPT-5.6 Sol to $4 in and $20 out on 21 August, four days before I wrote this, and GPT-5.6 Luna dropped 80% on 30 July. Treat the table as a snapshot, not a fact.

Two footnotes that cost us money before we read them. Gemini 3.7 Flash pricing is introductory and doubles on 1 January 2027. And on the GPT-5.6 family, prompts over 272K input tokens are billed at 2x input and 1.5x output for the whole request, so a 300K-token prompt is not 10% more expensive than a 272K one.

I am not quoting leaderboard percentages. I check SWE-bench and Terminal-Bench like everyone else, but a resolution rate tells you nothing about which agent runner, cost ceiling, and step limit produced it, and none of those runs looked like our codebase. Our routing decisions come from an eval set of our own tickets and our own pull requests. Benchmarks are how we decide what to test, not what to buy.

The 2,800-file rename

Mechanical refactors across a large PHP codebase are the job most people reach for a frontier model on, and it is almost always the wrong call. The arithmetic for our rename: average 3,200 input tokens and 2,900 output tokens per file, times 2,800 files, is 8.96M in and 8.12M out. On Claude Fable 5 at $10 and $50 that is $89.60 plus $406.00, so $495.60 a pass. On Claude Haiku 4.5 at $1 and $5 it is $8.96 plus $40.60, so $49.56. Same edit, roughly a tenth of the price.

The real answer was neither. An AST rewrite is deterministic, reviewable as one diff, and free. We now run Rector first and hand the model only the residue: 160 files with dynamic calls and string class names that no rule could reach. That cost under $3 on Haiku 4.5. The rule I wrote on the whiteboard afterwards: if the transformation can be expressed as a rule, write the rule.

Review, where the useful answer is often “I don’t know”

This is the one place we pay for the expensive model on purpose. Pull requests touching src/Billing/ or the COD remittance ledger go to Claude Opus 5 at $5 and $25, and the prompt demands a three-way verdict: approve, block, or unknown with a list of what it would need to decide.

We measured why. On 120 labelled pull requests where we had already agreed the correct answer, 31 genuinely could not be decided from the diff alone. Claude Haiku 4.5 returned “unknown” on 3 of them and confidently approved most of the rest. Claude Opus 5 returned “unknown” on 27. A reviewer that never abstains is worse than no reviewer, because it launders a guess into a green check. A typical PR here is 40K input tokens and 3K output, which is $0.275 on Opus 5. At roughly 90 reviewed PRs a month that is under $25. Nobody in finance has ever asked about that line.

Ticket routing, where the unit cost is the whole argument

The ops app routes about 40,000 support tickets a month and normalises perhaps 300,000 delivery addresses. At 800 input and 20 output tokens per ticket, that is 32M in and 0.8M out monthly. GPT-5.6 Luna: $6.40 plus $0.96, so $7.36. GPT-5.6 Sol on the same volume: $128.00 plus $16.00, so $144.00. Twenty times the price for a decision a human overrides in one click. Address normalisation runs through the Batch API, which is half price on both input and output, because nobody is waiting for it.

And then the cheap model cost us real money. In May, tickets whose bodies contained the Tagalog phrase “hindi pa po na-remit” were being classified as delivery_delay with confidence 0.94 instead of cod_remittance. Delivery delays go to a queue with a 72-hour SLA. Remittance disputes have a 7-day window with the courier partner, after which we cannot claim. It ran for nine days across roughly 1,900 tickets before a finance analyst noticed the aging report. We wrote off PHP 240,000 we could not reclaim.

The fix was not a better model. It was a confidence floor plus a second opinion: anything under 0.85, and anything the regex catches as remittance language regardless of confidence, goes to GPT-5.6 Terra. That escalation touches about 4% of traffic, 1,600 tickets, and adds roughly $3 a month. The routing now lives in one class, so the model names are a config change rather than a hunt.

<?php

namespace AppAi;

enum Job
{
    case TicketRouting;
    case AddressNormalisation;
    case TariffLookup;
    case PrReview;
}

final readonly class ModelRouter
{
    public function pick(Job $job, bool $mayLeaveCountry): string
    {
        if (!$mayLeaveCountry) {
            return 'local:ministral-3-8b';
        }

        return match ($job) {
            Job::TicketRouting, Job::AddressNormalisation => 'gpt-5.6-luna',
            Job::TariffLookup => 'gemini-3.7-flash',
            Job::PrReview => 'claude-opus-5',
        };
    }
}

The 412-page tariff circular

Customs schedules arrive as scanned PDFs and we need to answer questions against them for months. Ours came to roughly 380,000 tokens. Forty questions, naively re-sending the whole document each time, is 15.2M input tokens. On Gemini 3.7 Flash at $0.75 that is $11.40. On Claude Opus 5 at $5 it is $76.00. On GPT-5.6 Sol it is not $60.80 but $121.60, because every one of those requests is over the 272K threshold and doubles.

With Gemini context caching at $0.075 per 1M through 31 December 2026, plus $0.50 per 1M tokens per hour of storage, the same forty questions are $0.29 for the first pass, about $1.11 for the rest, and roughly $0.19 an hour to keep the document warm. The lesson is dull and worth repeating: on long-context work the cache and the price tier matter more than the model. The first parse into structured rows runs on Gemini 3.5 Flash-Lite at $0.30 and $2.50, and only the ambiguous rows get escalated.

When the data cannot leave

Two enterprise shippers will not let consignee names and addresses cross a border. This is where the vendor marketing stops helping. OpenAI’s data residency list includes Singapore, Japan, India, and Australia for storage, but those regions carry no regional processing at all: only the US, the EU, and the UAE do inference in-region. Mistral offers EU and US endpoints at 1.1x list. Nobody offers the Philippines. So for those two accounts, on-prem is not a preference, it is the only option that passes review.

We run Ministral 3 8B, which is Apache 2.0 with a 256k context, for classification, and we keep gpt-oss-120b loaded for anything harder. That model is 117B parameters with 5.1B active and was post-trained with MXFP4 quantisation so it fits on a single 80GB card. That single fact is the entire hardware plan. One colocated box in Makati with one 80GB GPU, quoted to us at about USD 2,900 a month landed and racked, replaces a compliance conversation that was going nowhere. It is slower, it is worse at long reasoning chains, and the customers who demanded it are fine with both.

Agents in production, and the three things we will not automate

We run exactly two unattended agents: one that triages failed webhook deliveries and one that drafts refund proposals for dispatch to approve. Both are wrapped in the same four guardrails.

Spend caps at two levels. OpenAI hard project limits return a 429 with project_spend_limit_exceeded, and the docs are honest that enforcement is not instantaneous. Anthropic session budgets take max_list_cost as whole US cents in a string and pause the session with budget_reached, with overshoot bounded by one in-flight request per thread. We set both, and we set the project limit low enough that hitting it is a page, not a surprise invoice.

Idempotency, learned the hard way. A tool call timed out at our gateway while the model kept retrying, and the refund agent created three proposals for booking BK-2026-0714-8831. The fix was a unique index and a key derived from the booking plus the action plus the UTC date, so the second and third attempts die at the database instead of in review:

-$this->proposals->save(new RefundProposal($booking, $amount));
+// src/Ops/Agent/RefundProposalHandler.php
+$key = sprintf('%s-%s-%s', $booking->reference(), 'refund', $now->format('Y-m-d'));
+$this->proposals->saveIdempotent($key, new RefundProposal($booking, $amount));

It now fails with SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'BK-2026-0714-8831-refund-2026-07-14' for key 'uniq_agent_action', which is exactly what we want it to do.

Every prompt and every response is logged. Not sampled. We log the job, model ID, a SHA-256 of the prompt, token counts split by cached and uncached, the computed list cost, and the decision. Four reasons: we cannot reproduce a wrong classification six weeks later without it, courier partners dispute our remittance decisions and we have to show our working, we re-cost every job the week a vendor changes prices, and our eval sets are built from real traffic rather than invented cases. The vendors’ own retention is theirs, not ours: OpenAI keeps abuse-monitoring logs 30 days by default, and Anthropic requires 30-day retention on some models outright.

Tool permissions are the fourth. The MCP specification revision 2026-07-28 is blunt that tool annotations from an untrusted server should be treated as untrusted and that hosts must get explicit consent before invoking a tool. We read that as: allow-list the servers, and never let a tool description decide its own blast radius.

What we will not let an agent do unattended, and I do not expect this list to shrink:

• Move money. A refund proposal is a row in a table until a human in dispatch clicks it.
• Write to the shipment status machine. A wrong status propagates to the seller, the courier, and the consignee’s SMS within seconds.
• Touch production schema or infrastructure. Not because the model is bad at it, but because nobody is awake at 02:00 to notice.

If you want one thing to carry into next quarter, it is this: keep the model names in a single class and keep an eval set of 200 real items per job. Every name in that table above will be wrong by February, and the introductory Gemini price expires on 1 January. The thing that has not moved all year is the ratio. The cheapest model we trust and the most expensive one we buy are still about 40x apart on output tokens, and the shape of the job, not the leaderboard, tells you which end you belong on. We are re-running our own numbers the first week of September, and I expect at least two rows to change.

Sources

• OpenAI API pricing: per-million prices for GPT-5.6 Sol, Terra, and Luna, plus the 50% Batch discount.
• GPT-5.6 Sol model page: 1,050,000 context window, 922,000 max input, and the 2x input / 1.5x output surcharge above 272K tokens.
• OpenAI API changelog: GPT-5.6 released 9 July 2026, Luna price cut 30 July, Sol repriced to $4 / $20 on 21 August 2026.
• Anthropic pricing: Claude Opus 5 at $5 / $25, Sonnet 5 at $2 / $10, Haiku 4.5 at $1 / $5, Fable 5 at $10 / $50, and the Batch API discount.
• Claude models overview: context windows and max output per model, and Sonnet 5’s standard pricing after 10 August 2026.
• Gemini API pricing: Gemini 3.7 Flash at $0.75 / $3.75 through 31 December 2026, Flash-Lite at $0.30 / $2.50, and context caching rates.
• Gemini 3.7 Flash model page: 1,048,576 input and 65,536 output token limits.
• OpenAI data controls: which residency regions support regional processing and which only support storage.
• gpt-oss-120b model card: 117B parameters, 5.1B active, MXFP4 quantisation, single 80GB GPU.
• Ministral 3 8B model page: Apache 2.0, 256k context, $0.15 in and out, released 2 December 2025.