Workers AI rejectIfBusy: Fail Fast or Wait?

Cloudflare Workers AI now lets an application decline to wait when inference capacity is busy. Set rejectIfBusy: true, and a synchronous request is rejected when capacity is unavailable instead of remaining in the capacity queue.

That sounds like a small request option. Operationally, it decides who owns the waiting time. Without it, the platform can queue the request. With it, your application receives a capacity signal and can retry, use another model, fall back to a non-AI path, or tell the user to try again.

The right choice depends on the deadline and the value of a late answer. An autocomplete suggestion that arrives after the user has typed the next sentence is waste. A background classification job may be perfectly useful thirty seconds later. Queueing treats those two requests alike unless the application supplies a policy.

This guide shows how rejectIfBusy works, how to handle its error correctly, and how to turn the option into a measured overload strategy rather than another boolean added to every request.

The short answer

Use rejectIfBusy when a queued inference is likely to become useless before capacity returns, the caller has a fallback, or accepting more waiting work would make an overloaded system harder to recover.

Leave it off when the request can wait, the user prefers a delayed result to no result, or your application has no safe way to handle a capacity rejection.

Cloudflare's official documentation says a rejected request returns HTTP 429, internal error code 3040, and the message Capacity temporarily exceeded, please try again. This is a capacity signal. It is not the same as an invalid model request or the daily free-allocation limit.

What changes when you fail fast

Assume an interactive request has a four-second product deadline. The model service becomes busy at the moment the request arrives.

If the request waits in a capacity queue, several outcomes are possible. Capacity may return quickly and the answer may still be useful. It may return at 3.8 seconds, leaving almost no time for inference and network delivery. It may return after the browser has disconnected. The model could then perform work that the user never sees.

With fail-fast behavior, the application receives a rejection close to admission time. It can spend the remaining deadline on a smaller model, cached answer, search result, deterministic rule, or clear error state.

This is admission control. It does not increase GPU capacity or make inference faster. It prevents a caller from pretending that waiting has no cost.

Queueing remains useful. It smooths short bursts, improves utilization, and can absorb temporary contention. Problems appear when the queue is invisible to the product deadline. At that point, waiting converts a capacity shortage into tail latency, timeouts, abandoned requests, and retry storms.

How to set rejectIfBusy

The parameter location differs by API surface. That detail is easy to miss.

For a Workers AI binding, pass the option as the third argument to env.AI.run():

const result = await env.AI.run(
  "@cf/google/gemma-4-26b-a4b-it",
  {
    messages: [
      { role: "user", content: "Summarize this incident report." },
    ],
  },
  { rejectIfBusy: true },
)

Do not put the option inside the model input object. Cloudflare states that the binding reads it from the third argument.

For the native REST API, put it in the request body's options object:

{
  "messages": [
    { "role": "user", "content": "Summarize this incident report." }
  ],
  "options": {
    "rejectIfBusy": true
  }
}

The OpenAI-compatible Chat Completions endpoint also accepts a top-level options object. Cloudflare warns that some OpenAI clients strip unknown fields. Inspect the actual HTTP request or run a controlled capacity test before assuming the option reaches the service.

Recognize the right 429

An HTTP status alone is not enough to choose a retry policy. Cloudflare's Workers AI error reference lists several errors, including more than one that uses 429.

Code 3040 means the service is out of capacity. The same code is returned when rejectIfBusy triggers. Code 3036 indicates that an account has used its daily free allocation. Retrying a quota-exhausted request every second will not make it succeed. Retrying a temporary capacity rejection may succeed, but only if the request still has time and the retry does not add to an overload loop.

Normalize provider errors at the boundary:

type InferenceFailure =
  | { kind: "capacity"; retryable: true }
  | { kind: "quota"; retryable: false }
  | { kind: "invalid_request"; retryable: false }
  | { kind: "upstream"; retryable: true }

function classifyWorkersAIError(error: unknown): InferenceFailure {
  const value = error as {
    status?: number
    code?: number
    message?: string
  }

  if (value.status === 429 && value.code === 3040) {
    return { kind: "capacity", retryable: true }
  }

  if (value.status === 429 && value.code === 3036) {
    return { kind: "quota", retryable: false }
  }

  if (value.status && value.status >= 500) {
    return { kind: "upstream", retryable: true }
  }

  return { kind: "invalid_request", retryable: false }
}

Adapt the parsing to the error shape produced by your binding or client. Log the provider status and internal code, but avoid logging prompts, credentials, or sensitive model output by default.

Decide from the request's deadline

The application should know how much time remains before it sends an inference request. A fixed retry count ignores work already spent on authentication, database queries, retrieval, prompt construction, and network hops.

Carry a deadline through the request:

const startedAt = Date.now()
const deadlineAt = startedAt + 4_000

function remainingMs() {
  return Math.max(0, deadlineAt - Date.now())
}

Before retrying, reserve time for the model and the response path. If 600 milliseconds remain and the normal inference takes two seconds, retrying only creates a later failure.

A useful policy is:

1. Attempt the preferred model with rejectIfBusy.
2. On code 3040, check the remaining deadline.
3. Retry once with jitter only when enough time remains.
4. Otherwise use a cheaper or more available model, a deterministic path, or a retryable response.
5. Stop when the caller disconnects or the deadline expires.

The exact numbers should come from observed latency, not a universal template.

Retry without creating a storm

Immediate retries synchronize clients. A burst that encounters no capacity returns together, retries together, and produces another burst.

Use bounded exponential backoff with jitter:

function retryDelay(attempt: number) {
  const base = Math.min(100 * 2 ** attempt, 800)
  return Math.floor(Math.random() * base)
}

For a four-second interactive deadline, one retry may be enough. For background work, a durable queue can retry over a longer period. Keep client retries and server retries coordinated. If the browser retries twice, the API retries twice, and an AI gateway retries twice, one user action can become many provider calls.

Attach an idempotency key or request identifier when downstream actions can be repeated. Text generation itself may produce different output on each attempt. Tool calls triggered by that output need their own deduplication and authorization controls.

Build a fallback ladder

A fallback should preserve the product's intent at lower quality or reduced scope. It should not silently change the meaning of the operation.

For customer-support drafting, the ladder might be:

1. preferred instruction model;
2. smaller approved model with a shorter prompt;
3. retrieval results presented without generated prose;
4. saved draft and a message that generation is temporarily unavailable.

For content moderation, a weaker fallback may be unsafe. The application might quarantine the content for later review instead of allowing it. For code generation, a cached generic answer could be more harmful than an honest failure.

Cloudflare AI Gateway offers controls such as retries and model fallback. If you combine gateway behavior with application behavior, document which layer makes each decision. The article on identity-based AI model routing explains why model access and fallback should also depend on the caller's authority and data boundary.

Match the policy to the workload

Interactive autocomplete has a short useful life. Enable fail-fast behavior, avoid multiple retries, and degrade to no suggestion.

A conversational assistant can tolerate more delay, but users need progress and a cancel path. A single bounded retry or model fallback may work. Do not leave the interface showing an indefinite typing indicator after the original request has timed out.

Document processing is usually background work. A durable queue is better than repeatedly asking the browser to retry. Leaving rejectIfBusy off may be appropriate if the platform queue fits the job's service objective. If job ownership, visibility, or scheduling matters, manage the queue in your application and use fail-fast admission at the provider boundary.

Safety checks and authorization decisions should fail closed. If the AI call is only advisory, use a deterministic safety baseline. If the model is required, return a pending or unavailable state rather than bypassing the control.

Batch enrichment can wait and is cost sensitive. Spread work, respect rate limits, and use backpressure before sending requests. rejectIfBusy is a final capacity signal, not the batch scheduler.

Protect the service with local concurrency limits

Fail-fast provider behavior does not stop your Worker from accepting unlimited upstream work. Apply concurrency limits before inference. A simple semaphore can cap active requests per isolate or process, but distributed traffic needs a shared strategy or platform-level control.

Limit by workload and tenant. One bulk customer should not consume the entire interactive pool. Separate queues for latency-sensitive and background work. Set per-tenant budgets and maximum prompt sizes before a request reaches the model.

Rate limits and capacity are different. Cloudflare publishes Workers AI limits by task and model. Staying below an account rate limit does not guarantee immediate GPU capacity. A capacity rejection does not necessarily mean the account exceeded a documented requests-per-minute limit.

Keep tenant policy intact during fallback

Capacity pressure is a bad time to loosen data controls. A fallback model or endpoint may use a different provider, region, retention policy, context window, safety profile, or commercial agreement. Route only to destinations already approved for that tenant and data class.

Attach policy information before model selection. The router should know whether the prompt contains personal data, source code, health information, payment details, or customer-confidential documents. It should also know whether the tenant permits cross-provider fallback. A model that is technically available can still be an invalid destination.

type InferencePolicy = {
  tenantId: string
  dataClass: "public" | "internal" | "restricted"
  allowedModels: string[]
  allowProviderFallback: boolean
  deadlineAt: number
}

When the preferred model returns 3040, filter candidates through policy before considering latency or price. If no approved fallback remains, fail honestly. Do not send the prompt to an unapproved model just to improve the apparent success rate.

Tool permissions need the same treatment. A smaller fallback model may have different tool-calling behavior or weaker instruction following. Re-run server-side authorization for every tool call and keep high-impact tools behind approval. Never inherit a model's earlier authorization decision from a failed attempt.

Cache use also needs boundaries. A cached answer can be an excellent overload response for public documentation. It is unsafe when the cache key omits tenant, permission, locale, source version, or other context that affects the answer. Record whether the user received a generated, cached, retrieved, or deterministic response so incidents and evaluations can reconstruct the path.

Finally, protect the error channel. Internal capacity codes, model identifiers, account limits, and routing policy are useful in logs but may reveal infrastructure details to an anonymous caller. Return a stable product error and keep the diagnostic fields in protected telemetry associated with the request ID.

Measure whether fail-fast helps

Track at least these events:

• inference attempts by model and workload;
• accepted requests;
• capacity rejections with code 3040;
• quota and rate-limit rejections;
• queue or gateway wait time when observable;
• retry attempts and eventual outcome;
• fallback selection and success;
• end-to-end latency, including p95 and p99;
• client cancellations;
• generated work discarded after disconnect;
• cost or neuron usage by outcome.

Do not report a lower latency percentile without reporting rejected requests. A service can look fast by failing every busy call immediately. Pair latency with success rate, fallback rate, and user-visible completion.

Create separate dashboards for interactive and background traffic. Their healthy shapes differ. Interactive workloads care about deadline success. Background workloads care about completion age, backlog, throughput, and cost.

A useful capacity test increases offered load gradually while recording completed, rejected, timed-out, and cancelled requests. Compare queueing with fail-fast behavior at the same traffic levels. The goal is to find where useful completions stop increasing and tail latency starts consuming the deadline.

Return an honest response to the user

Map provider errors to product language. Users do not need Cloudflare's internal code, but support and telemetry do.

For an interactive endpoint, a structured response can preserve both:

{
  "error": "AI_CAPACITY_TEMPORARY",
  "message": "Generation is temporarily busy. Try again in a moment.",
  "retryable": true,
  "requestId": "req_01..."
}

Use an appropriate HTTP status for your contract. Avoid claiming that a retry will succeed at a specific time unless you have a reliable signal. Do not turn a capacity response into a generic 500 that triggers broad incident alarms and hides the overload pattern.

If a fallback produced the answer, consider exposing that in telemetry and, when it changes quality materially, in the interface. Silent fallback can complicate debugging and evaluation.

Test overload before production finds it

A normal functional test will rarely exercise rejectIfBusy. The provider has capacity, the request succeeds, and the team learns nothing about its rejection path. Build a test seam around the inference client so code 3040 can be injected deterministically.

Test at least these cases:

• the first attempt receives 3040 and a retry succeeds within the deadline;
• the first attempt receives 3040 with too little time left to retry;
• the preferred model is busy and the fallback succeeds;
• both preferred and fallback models reject capacity;
• the caller disconnects before the retry timer fires;
• the account returns quota code 3036, which must not enter the capacity retry loop;
• a client library drops the custom options field;
• two application replicas receive the same idempotent job.

Then run a load test in a controlled environment. Increase concurrency rather than only total request count. Record the point where capacity rejections begin, how the fallback behaves, and whether local queues continue growing after useful throughput flattens. Stop the test if it threatens production quotas or shared tenants.

Fault injection at the client boundary is more repeatable than trying to exhaust a shared model. It also lets CI verify the response contract on every change. A staging load test is still valuable for timing and concurrency behavior, but it should not be the only way to reach the error branch.

Cancellation is part of capacity management

When a browser closes, an upstream request times out, or the user presses Stop, propagate cancellation as far as the client and platform allow. A retry scheduled after cancellation should never start. Retrieval, prompt construction, tool calls, and fallback selection should all check the same abort signal.

export async function generateWithDeadline(
  input: ModelInput,
  signal: AbortSignal,
) {
  if (signal.aborted) throw signal.reason

  const result = await env.AI.run(
    MODEL,
    input,
    { rejectIfBusy: true },
  )

  if (signal.aborted) throw signal.reason
  return result
}

The example prevents work before the call and discards a late result. Whether an in-flight inference can receive the signal depends on the exact client and interface in use, so verify cancellation support before promising it. The application should always stop retries and avoid launching downstream tools for an abandoned turn.

Cancellation metrics reveal hidden waste. Count provider completions that arrive after the client has gone, fallback work started after an upstream timeout, and tool calls whose result had no consumer. rejectIfBusy protects admission during capacity pressure. Cancellation protects capacity after the request has lost its value.

A production policy example

Write the policy before shipping the flag:

Workload: editor autocomplete
Deadline: 2,500 ms end to end
Admission: rejectIfBusy enabled
Retries: one, only when at least 1,800 ms remain
Fallback: smaller approved model
Final degradation: return no suggestion
Client retry: none automatically
Metrics: capacity rejection, fallback, deadline success, cancellation

For background summarization:

Workload: uploaded-document summary
Deadline: 15 minutes
Admission: application queue controls dispatch
Provider behavior: rejectIfBusy enabled at dispatch boundary
Retries: durable exponential backoff, capped attempts
Fallback: none unless document policy permits another model
Final degradation: job remains failed with a retry action
Metrics: queue age, attempts, completion age, capacity rejection, cost

Both examples use rejectIfBusy, but for different reasons. The interactive path protects a product deadline. The background path keeps queue ownership and retry visibility inside the application.

What to ship

Add rejectIfBusy only after defining the error parser, deadline, retry ceiling, fallback, and metrics. Verify that the client library preserves the option. Test code 3040 separately from quota exhaustion. Cancel work when the caller leaves, and keep retry behavior in one clearly owned layer.

Cloudflare's option gives the application a useful capacity signal. The engineering work is deciding what the signal should mean for each request. A short-lived suggestion should fail quickly. A durable job should wait somewhere you can observe and control it.

---