Build an AI SaaS platform: the architecture when the model is a dependency

On a Tuesday afternoon a model provider has a partial outage. Requests do not fail outright; they take forty seconds and then return. Your web workers fill up waiting, the queue backs up behind them, and customers who were not using the AI feature at all see the dashboard time out. Two weeks later the same provider updates the model behind the alias you call, and the summaries your product generates get shorter and start leaving out dates. Nobody deployed anything. A customer notices before you do.

The guides that rank for building an AI SaaS are mostly agency step-by-step lists: find the problem, pick a model, build an MVP, launch. They are about building a SaaS with AI. This post is about a SaaS whose product is AI, where a model call sits in the middle of the core workflow. In that product the model is a dependency, and it behaves like no other dependency you have. The architecture has to start from that.

Why the model is an unusual dependency

Every row on the right needs a decision in code, and they all point to the same structure: no part of your product calls a provider SDK directly. Every call goes through one internal gateway that you own.

One gateway for every model call

The feature asks for a named task, not a model. summarise_call is the contract; which model, which prompt, which provider and which fallback serve it is configuration inside the gateway. That one decision is what makes the next five sections possible, and it is cheap to make on day one and expensive to retrofit after forty call sites import the SDK directly.

1. Per-tenant cost, metered and capped

In a normal SaaS the cost of serving a customer is roughly flat, so a flat price works. In an AI SaaS the cost of a customer is however many tokens they cause, and the spread is wide: the heaviest tenant can cost many times the median on the same plan. Without metering, you find out which customers lose you money from the provider invoice, a month late and with no tenant on it.

final class ModelGateway
{
    public function run(Task $task, Tenant $tenant, array $input): TaskResult
    {
        $estimate = $this->pricing->estimate($task, $input);

        // Quota first: routing and caching reduce cost, only a quota bounds it.
        if (! $this->quotas->canSpend($tenant, $estimate)) {
            throw new QuotaExceeded($tenant, $task);
        }

        $result = $this->callWithFallback($task, $this->policy->redact($tenant, $input));

        $this->usage->record(
            tenant: $tenant,
            task: $task->name,
            model: $result->model,             // the exact snapshot that answered
            promptVersion: $task->promptVersion,
            inputTokens: $result->inputTokens,
            outputTokens: $result->outputTokens,
            costMicros: $this->pricing->cost($result),
            idempotencyKey: $task->idempotencyKey,
        );

        return $result;
    }
}

Three details matter more than the rest. The quota check happens before the call, against an estimate, so a tenant cannot overshoot by one enormous request. The usage record carries an idempotency key, so a retried job is not counted twice. And the cost is stored in your own price table at the time of the call, because provider prices change and you will want to know what last March actually cost. Once usage is recorded per tenant, the cost levers in not every call needs the expensive model (caching, routing simple tasks down a tier) become measurable instead of guessed.

Pricing follows from the meter, not the other way round. Plans with a usage allowance and a visible counter, or a platform fee plus metered usage, survive heavy users. Unlimited AI on a flat plan only survives until someone automates against it. Agentic SaaS in 2026 goes further into why per-seat pricing stops working once agents are the ones calling you.

2. Outages, slow responses and fallback

Model providers have outages, and a degraded provider is worse than a down one, because slow responses hold your workers instead of failing fast. The rules that keep the rest of the product standing:

• Never call a model inside a web request if you can avoid it. Put the call on a queue and stream or poll the result. A slow provider then fills a queue, not your web workers.
• Set timeouts per task, based on that task's observed latency, not the SDK default. A classification that normally takes two seconds should not wait sixty.
• Retry once with backoff, then fall back. The fallback is a second model, often from a second provider, configured per task.
• Only fall back to a model that passes the same evals. A fallback that produces worse output silently is a quality incident disguised as uptime.
• Have a degraded mode for each feature. Queue the work and tell the user it will finish later, or switch the feature off with a clear message. Never let it take the rest of the product down.
• Use a circuit breaker. After a run of failures, stop calling the primary for a few minutes and go straight to the fallback, rather than paying the timeout on every request.

3. Prompts and models, versioned with the release

If your prompts live in a database table that anyone can edit, or your code calls a floating model alias, then your product's behaviour can change without a deploy. That is the second half of the opening story, and it is entirely avoidable.

• Prompts are code. They live in the repository, go through review, and carry a version number that changes whenever the text does.
• Pin model snapshots, not aliases that move. Upgrading the model is then a pull request with an eval run attached, not something that happens to you.
• Record both on every output. When a customer reports a bad summary, you need to know exactly which prompt version and model snapshot produced it, and whether the current ones still would.
• Track deprecation dates. Providers retire snapshots on a schedule. Put each pinned model's retirement date somewhere a scheduled job checks, so the migration is planned rather than forced.

4. Evals in CI, before anything ships

Unit tests do not help when the output is different every run. What does help is a set of real inputs with a definition of a good answer for each: required facts present, forbidden content absent, valid structure, length within bounds, and for some tasks a rubric scored by a second model. Run that set in CI whenever a prompt, a model pin or the retrieval code changes, and block the merge when the score drops.

Start with fifty cases taken from real usage, especially the ones customers complained about, and add every new failure as a case. I have argued before that evals are the product: in an AI SaaS they are the only thing that lets you change models or providers as the market moves without betting the product on it. If retrieval is part of the product, RAG in production covers the failures your evals should be written to catch.

5. Data that must never reach a provider

The first enterprise prospect will send a security questionnaire, and at least a page of it will be about what you send to model providers. Answer it in the architecture, not in a sales call.

1. 1Classify fields, not tables. For each piece of customer data, decide whether it may be sent to a model, must be redacted first, or must never leave your systems. Card numbers, credentials and government identifiers are usually the third kind.
2. 2Redact in the gateway, so no feature can bypass the policy by calling a provider directly. Replace values with placeholders and restore them in the output where needed.
3. 3Use the provider terms that match your promises: a data processing agreement, no training on your data, and zero or minimal retention where the provider offers it. Write down which provider, model and region each task uses.
4. 4Make it a tenant setting where buyers need it. Some customers will require a specific region, a specific provider, or AI features switched off for one workspace. If the gateway already routes per task, routing per tenant is a small step.
5. 5Keep tenant isolation in retrieval. Vector search is a query like any other. Filter by tenant in the query itself, never by trusting the model to ignore another customer's documents.

What stays the same

An AI SaaS is still a SaaS. Tenancy, authentication, billing state, queues, backups and failed-job alerts matter as much as ever, and more, because model calls are slow and expensive and run in exactly the background jobs that fail silently. The architecture decisions that are hard to undo apply unchanged, and the launch checklist still decides how the first month goes.

The order I build it in

1. 1SaaS foundations: tenancy, authentication, roles, billing state, queues.
2. 2The gateway, with one task, one pinned model, usage recording and a quota, before there is a second call site.
3. 3Twenty to fifty eval cases for that task, running in CI.
4. 4The data policy: field classification and redaction in the gateway, provider terms confirmed.
5. 5Fallback, timeouts and a circuit breaker, tested with a drill.
6. 6Pricing with a usage allowance, using a few weeks of real per-tenant cost data.
7. 7Then more AI features, each one a new task in the gateway with its own evals.

Treat the model the way you treat a payment provider: behind your own interface, metered, versioned, tested and replaceable. The products that do this can follow the market. The ones that do not are rebuilt every time it moves.

If you are planning a SaaS where a model sits in the core workflow, or you have one live and the provider invoice, an outage or a quiet model update has already surprised you, I will go through the architecture with you: the gateway, quotas and pricing, fallback, evals and the data policy. You get a written plan of what to build or change first and what each step costs. Book a SaaS architecture review.