Resilient AI Routes

A same-shape fallback pattern for guarding AI calls against budget limits, timeouts, and bad output

Why AI Calls Need Guards

Model calls fail in ways a normal backend call usually doesn't: latency is highly variable (P50 and P99 can be 10x apart), providers rate-limit or have outages, structured-output requests can fail schema validation (Workers AI can return JSON Mode couldn't be met), and completions can come back empty or flagged by moderation. A route that just awaits env.AI.run() and returns whatever comes back turns every one of those into a 500 for the caller.

Treat these as expected operational states, not exceptions, and design the route to degrade instead of fail outright.

The Same-Shape Fallback Pattern

The core idea: whichever guard trips, the route returns the same response shape it would on a normal success. The client does not need a separate code path for "the model produced this" versus "a guard substituted this" -- it always gets a renderable payload, and can optionally read a fallback field to show a subtler UI hint.

type AiRouteResult<T> = {
  ok: true;
  data: T;
  fallback: null | {
    guard: "budget" | "call" | "output";
    reason: FallbackReason; // stable token -- see "Reason Taxonomy" below
    detail: string; // free-text or numeric context, never used as a metric label
  };
};

data always matches the same schema whether it came from the model or from a guard's fallback value. fallback is null on the happy path and populated, whenever a guard substitutes its own value, with which guard tripped, a stable reason token safe to use as a metric label, and a detail string carrying the free-text or numeric specifics that token deliberately excludes.

Three Guards, One Envelope

Wrap every model call with three guards, applied in this order:

1. Budget guard (pre-call). Reject before spending a model call at all: per-user or per-tenant rate limit, a daily spend cap, or an open circuit breaker from a recent run of failures.

2. Call guard (in-call). Race the model call against a hard deadline so a slow or hung provider request cannot hang the Worker response, and catch provider-side errors in the same place.

3. Output guard (post-call). Validate the model's response against the same schema the client expects -- schema failures, empty completions, or a moderation block -- before it ever reaches the caller.

Each guard, on trip, returns the identical envelope shape with a different fallback.guard value populated instead of throwing.

interface Env {
  AI: Ai;
  RATE_LIMIT: KVNamespace;
}

interface AiSummary {
  label: string;
  confidence: number;
}

type FallbackReason =
  | "rate_limited"
  | "budget_exceeded"
  | "circuit_open"
  | "timeout"
  | "provider_error"
  | "schema_invalid"
  | "empty_response"
  | "moderation_blocked";

type AiRouteResult<T> = {
  ok: true;
  data: T;
  fallback: null | { guard: "budget" | "call" | "output"; reason: FallbackReason; detail: string };
};

const MODEL_ID = "@cf/meta/llama-3.3-70b-instruct-fp8-fast";
const TIMEOUT_MS = 8_000;
const DAILY_CALL_LIMIT = 20;

const SUMMARY_SCHEMA = {
  type: "json_schema",
  json_schema: {
    type: "object",
    properties: {
      label: { type: "string" },
      confidence: { type: "number" },
    },
    required: ["label", "confidence"],
  },
} as const;

const DEFAULT_SUMMARY: AiSummary = { label: "unavailable", confidence: 0 };

function fallbackResult<T>(
  data: T,
  guard: "budget" | "call" | "output",
  reason: FallbackReason,
  detail: string,
): AiRouteResult<T> {
  return { ok: true, data, fallback: { guard, reason, detail } };
}

// A fixed TTL longer than one UTC day, so a write made just before midnight
// isn't evicted before the day it belongs to is over.
const BUDGET_KEY_TTL_SECONDS = 2 * 24 * 60 * 60;

function dailyBudgetKey(userId: string, now = new Date()): string {
  // Scoping the key to the UTC day, not just the user, is what makes
  // "daily" actually reset -- see the warning below for why a rolling TTL
  // on a user-only key does not.
  return `ai-budget:${userId}:${now.toISOString().slice(0, 10)}`; // YYYY-MM-DD, UTC
}

// Guard 1: budget -- reject before the model call happens at all.
async function checkBudgetGuard(
  env: Env,
  userId: string,
): Promise<{ reason: FallbackReason; detail: string } | null> {
  const key = dailyBudgetKey(userId);
  const count = Number((await env.RATE_LIMIT.get(key)) ?? "0");
  if (count >= DAILY_CALL_LIMIT) {
    return {
      reason: "budget_exceeded",
      detail: `daily call limit of ${DAILY_CALL_LIMIT} reached`,
    };
  }
  // Charged before the call happens, not after it succeeds -- a timeout or
  // provider error below still burns this increment. That's the tradeoff a
  // pre-call guard makes: charging after success instead would undercount
  // exactly the calls expensive enough to be worth guarding against, and
  // let a string of failures retry past the budget uncounted.
  await env.RATE_LIMIT.put(key, String(count + 1), { expirationTtl: BUDGET_KEY_TTL_SECONDS });
  return null;
}

class TimeoutError extends Error {}

function timeoutRejection(ms: number): { promise: Promise<never>; cancel: () => void } {
  let handle: ReturnType<typeof setTimeout>;
  const promise = new Promise<never>((_, reject) => {
    handle = setTimeout(() => reject(new TimeoutError()), ms);
  });
  return { promise, cancel: () => clearTimeout(handle) };
}

// Guard 2: call -- race against a deadline, catch provider errors here too.
async function callWithGuard(
  env: Env,
  prompt: string,
): Promise<{ ok: true; value: unknown } | { ok: false; reason: FallbackReason; detail: string }> {
  const timeout = timeoutRejection(TIMEOUT_MS);
  try {
    const result = await Promise.race([
      env.AI.run(MODEL_ID, {
        messages: [{ role: "user", content: prompt }],
        response_format: SUMMARY_SCHEMA,
      }),
      timeout.promise,
    ]);
    return { ok: true, value: result };
  } catch (err) {
    if (err instanceof TimeoutError) {
      return { ok: false, reason: "timeout", detail: `model call exceeded ${TIMEOUT_MS}ms` };
    }
    return { ok: false, reason: "provider_error", detail: "provider returned an error" };
  } finally {
    // Clears the timer whichever side of the race settles first -- left
    // armed, it would otherwise fire after the response has already gone
    // out, rejecting a promise nothing is still attached to.
    timeout.cancel();
  }
}

// Guard 3: output -- validate the shape before it reaches the caller.
function checkOutputGuard(candidate: unknown): AiSummary | null {
  if (
    candidate &&
    typeof candidate === "object" &&
    "label" in candidate &&
    "confidence" in candidate &&
    typeof (candidate as AiSummary).label === "string" &&
    typeof (candidate as AiSummary).confidence === "number"
  ) {
    return candidate as AiSummary;
  }
  return null;
}

async function handleSummaryRoute(
  env: Env,
  userId: string,
  prompt: string,
): Promise<AiRouteResult<AiSummary>> {
  const budgetFailure = await checkBudgetGuard(env, userId);
  if (budgetFailure) {
    return fallbackResult(DEFAULT_SUMMARY, "budget", budgetFailure.reason, budgetFailure.detail);
  }

  const called = await callWithGuard(env, prompt);
  if (!called.ok) {
    return fallbackResult(DEFAULT_SUMMARY, "call", called.reason, called.detail);
  }

  // Workers AI wraps every text-generation call in { response, usage,
  // tool_calls } -- JSON Mode nests the schema-validated result under
  // `response`, not at the top level. See "Unwrapping the Workers AI
  // Envelope" below.
  const raw = (called.value as { response?: unknown }).response;
  const validated = checkOutputGuard(raw);
  if (!validated) {
    return fallbackResult(
      DEFAULT_SUMMARY,
      "output",
      "schema_invalid",
      "response failed schema validation",
    );
  }

  return { ok: true, data: validated, fallback: null };
}

Unwrapping the Workers AI Envelope

env.AI.run() never returns the schema object directly. Workers AI wraps every text-generation call in { response, usage, tool_calls }, and JSON Mode nests the schema-validated result under that response key as a parsed JavaScript object, not a JSON string -- confirmed against Workers AI's JSON Mode docs for @cf/meta/llama-3.3-70b-instruct-fp8-fast, the model this page uses. checkOutputGuard expects the schema's own fields (label, confidence) at the top level, so it has to run against called.value.response, never against called.value itself. Validating the wrapper instead of the payload fails on every successful call, and because this pattern's whole thesis is "a fallback is not a failure," the route then silently and permanently serves the fallback -- the worst possible way for this particular bug to present.

Reason Taxonomy

fallback.reason is a closed set of stable tokens -- safe to use as a metric label because the set of distinct values can never grow. Free-text or numeric detail (a timeout's duration, a provider's raw error message) belongs in the sibling fallback.detail field instead, which is fine to log but never fine to key a metric on:

A route does not need every row on day one. Start with the guards that match real failures you have seen, and add reasons as new failure modes show up in the fallback logs below.

Fallback ≠ Broken

A guard tripping is not the same as the route being down. Keep that distinction in how you measure and alert on it:

• Emit fallbacks as a labeled counter (guard + reason -- never detail, which is free text and would give the metric unbounded cardinality), separate from the counter for uncaught exceptions and 5xx responses. A request that returns 200 with fallback populated succeeded from the caller's point of view.

• Alert on the rate of fallbacks over a window (for example, provider_error fallbacks above 20% of calls over 5 minutes), not on any single occurrence. A nonzero fallback rate is normal background noise from timeouts and quota edges.

• Log the guard and reason on every fallback so trends are diagnosable. A spike isolated to output.schema_invalid usually points at a prompt or schema regression you shipped, not a provider incident.

• Keep the fallback payload genuinely useful: a cached last-good answer, a simpler deterministic computation, or an honest "try again" message. A same-shape but empty stub just moves the failure to the client silently.

Related Docs

• Chat Memory and RAG -- the request boundary and storage guide this pattern sits underneath.

• Workers AI Models and Pricing -- JSON Mode behavior and the JSON Mode couldn't be met failure this pattern guards against.

• Workers AI Streaming SSE Proxy -- the matching pattern for a streaming route, including client-disconnect handling and a runtime kill switch.