Gemini Live Through an AI Gateway: A Production Voice-Agent Architecture

A Live Voice Gateway Is A Runtime, Not A Model Proxy

A normal AI gateway answers request-level questions:

• Which model should receive this request?
• Is the caller allowed to use it?
• How much did it cost?
• Should it be retried, cached, or rejected?

A voice gateway answers those questions while a conversation is already running. Audio arrives continuously. The model emits audio, text, tool requests, and errors. The user interrupts. Tools finish late. Connections disappear halfway through a turn.

The unit of architecture is therefore not an HTTP request. It is a live session with several independent streams:

client microphone
      |
      | audio frames
      v
+-------------------+
| Edge WebSocket    |
| authentication    |
| rate limits       |
+-------------------+
      |
      v
+-------------------+
| Session manager   |
| routing policy    |
| state + timers    |
+-------------------+
      |
      +----------------------+
      |                      |
      v                      v
+-------------------+  +-------------------+
| Gemini Live       |  | Tool runner       |
| upstream session  |  | APIs, queues, DB  |
+-------------------+  +-------------------+
      |
      | audio events, text,
      | tool requests, errors
      v
client speaker

The gateway should not sit in the audio path merely because every component is called an “AI service.” Keep the hot path small. Authentication, session ownership, routing, backpressure, cancellation, and policy checks belong there. Slow business operations usually don't.

That distinction matters for Gemini 3.8 Live and Gemini 3.8 Live Extended Thinking. A voice session needs fast turn-taking even when some work requires deeper reasoning. If Extended Thinking blocks the conversational path, the assistant may be correct and still feel broken.

A Session Has More Than One Clock

A live session has at least four clocks:

1. Transport time: when audio frames arrive and leave.
2. Turn-detection time: when the system decides the user has started or stopped speaking.
3. Model time: when the upstream session emits audio, text, or a tool request.
4. Business-operation time: when a tool or downstream service finishes.

These clocks should not be collapsed into one request timeout.

A tool call may take longer than the acceptable delay for acknowledging the user. Depending on the supported interaction model, the assistant can keep the session responsive while the tool runs rather than holding the WebSocket in a vague waiting state. If the model or gateway can't provide useful interim behavior, track the tool operation separately and enforce its own deadline.

An internal state model can make those distinctions explicit:

type SessionState =
  | "connecting"
  | "active"
  | "user_speaking"
  | "assistant_speaking"
  | "tool_pending"
  | "interrupting"
  | "degraded"
  | "closing"
  | "closed";

type LiveSession = {
  id: string;
  userId: string;
  state: SessionState;
  route: {
    provider: string;
    model: string;
    mode: "standard" | "extended-thinking";
  };
  startedAt: number;
  lastClientAudioAt?: number;
  lastUpstreamEventAt?: number;
  activeToolCallId?: string;
  interruptionGeneration: number;
};

The exact upstream event names depend on the Gemini Live interface exposed by the provider or the implementation. The important part is separation of concerns. A session manager should know whether the session is active, interrupted, waiting on a tool, or degrading without pretending that every transition is an ordinary request and response.

Keep Control Events Separate From Audio

Audio frames are frequent and relatively boring. Control events are sparse and consequential.

Treat them differently in the internal protocol:

type ClientMessage =
  | {
      type: "audio";
      sessionId: string;
      sequence: number;
      payload: ArrayBuffer;
    }
  | {
      type: "control";
      sessionId: string;
      command:
        | "start"
        | "stop"
        | "cancel_response"
        | "mute"
        | "resume";
    }
  | {
      type: "tool_result";
      sessionId: string;
      callId: string;
      result: unknown;
    };

This gives policy a clear home. A client should not be able to smuggle a control action into an audio message, and an audio queue should not delay a cancellation command behind a large backlog of frames.

The same rule applies in the other direction. Model audio, model text, tool requests, usage events, and errors should not all arrive as one untyped stream.

type GatewayEvent =
  | { type: "audio_out"; payload: Uint8Array; sequence: number }
  | { type: "transcript"; role: "user" | "assistant"; text: string }
  | { type: "tool_request"; callId: string; name: string; args: unknown }
  | { type: "turn_started"; role: "user" | "assistant" }
  | { type: "turn_ended"; role: "user" | "assistant" }
  | { type: "interruption_applied"; generation: number }
  | { type: "session_error"; code: string; retryable: boolean };

A gateway that treats every event as a log line will eventually lose the difference between “the user stopped speaking,” “the model stopped speaking,” and “the upstream connection died.” Those are different operational events and need different responses.

The Core WebSocket Flow

The most reliable design is usually a pair of long-lived connections:

• one WebSocket between the client and the gateway
• one upstream live connection between the gateway and the model provider

The gateway owns the relationship between them. It shouldn't blindly pipe bytes in both directions.

async function handleLiveConnection(
  client: WebSocket,
  request: Request,
) {
  const identity = await authenticate(request);
  const session = await createSession(identity);

  const upstream = await connectToLiveModel({
    sessionId: session.id,
    model: session.route.model,
    thinkingMode: session.route.mode,
  });

  const abort = new AbortController();

  client.addEventListener("message", async (event) => {
    try {
      const message = decodeClientMessage(event.data);

      if (message.type === "audio") {
        await session.audioIn.push(message);
        return;
      }

      if (message.type === "control") {
        await session.control.handle(message.command);
        return;
      }

      if (message.type === "tool_result") {
        await session.tools.resolve(message);
      }
    } catch (error) {
      await reportProtocolError(client, error);
    }
  });

  upstream.onEvent(async (event) => {
    const gatewayEvents = await session.handleUpstreamEvent(event);

    for (const gatewayEvent of gatewayEvents) {
      sendToClient(client, gatewayEvent);
    }
  });

  client.addEventListener("close", async () => {
    abort.abort();
    await upstream.close();
    await session.close("client_disconnected");
  });
}

This sample deliberately leaves provider-specific connection and event names behind an adapter. That adapter is valuable even if there is only one model today.

interface LiveModelAdapter {
  connect(input: ConnectInput): Promise<LiveModelConnection>;
}

interface LiveModelConnection {
  sendAudio(frame: AudioFrame): Promise<void>;
  sendControl(command: UpstreamCommand): Promise<void>;
  sendToolResult(result: ToolResult): Promise<void>;
  onEvent(handler: (event: ModelEvent) => void): void;
  close(): Promise<void>;
}

The adapter isn't pretending all providers behave the same. It is a boundary where differences become explicit. One provider may support native interruption semantics. Another may require the gateway to stop forwarding audio and discard buffered output. One may expose detailed usage events. Another may provide aggregate usage later.

Represent those differences as capabilities rather than scattering conditionals through the session manager:

type LiveCapabilities = {
  nativeCancellation: boolean;
  serverTurnDetection: boolean;
  toolCallStreaming: boolean;
  usageEvents: boolean;
  resumableSessions: boolean;
};

Backpressure Is A Product Decision

Audio keeps arriving when the upstream model slows down. Without backpressure, the gateway has only bad choices:

• grow memory until the process is killed
• add latency by buffering indefinitely
• drop frames silently
• disconnect the user

There's no universal answer because dropping audio and dropping control events have very different consequences.

A practical buffer is bounded and observable:

class AudioBuffer {
  private frames: AudioFrame[] = [];
  private readonly maxFrames: number;

  constructor(maxFrames: number) {
    this.maxFrames = maxFrames;
  }

  push(frame: AudioFrame) {
    if (this.frames.length >= this.maxFrames) {
      throw new Error("audio_backpressure_limit");
    }

    this.frames.push(frame);
  }

  shift(): AudioFrame | undefined {
    return this.frames.shift();
  }

  clear() {
    this.frames = [];
  }

  get size() {
    return this.frames.length;
  }
}

In production, a full buffer should produce a metric and a session-level decision, not an unhandled exception.

Possible decisions include:

• ask the client to reduce capture rate, if the protocol supports it
• pause or reject new audio input
• enter a degraded mode
• terminate the session cleanly and offer reconnect
• switch to a fallback route if the session can be re-established safely

Be conservative about automatic retries. Retrying a stateless text request is one thing. Retrying a live audio stream can duplicate speech, reorder events, or cause the model to answer an old turn after the user has moved on.

Session Affinity Has To Be Explicit

A live session needs a clear owner. If the gateway runs on multiple instances, a later WebSocket event cannot depend on a random load-balancer decision.

Common approaches include:

• keeping the entire session on one gateway process
• using WebSocket connection affinity
• storing session metadata in a shared store while stream handling stays local
• using a dedicated session-worker model

The shared store should contain coordination data, not necessarily every audio frame. Persisting raw audio in a general-purpose session database creates cost, privacy, and throughput problems. If audio retention is required, treat it as a separate data pipeline with its own policy.

type SessionRecord = {
  id: string;
  ownerInstance: string;
  userId: string;
  createdAt: string;
  expiresAt: string;
  route: string;
  status: "active" | "closing" | "closed" | "failed";
  lastHeartbeatAt: string;
  interruptionGeneration: number;
};

The owner instance should renew the lease while the session is healthy. If the lease expires, another instance may clean up stale state, but it should not automatically take over the audio stream unless the protocol and provider support safe resumption.

Routing Standard Live And Extended Thinking

Multi-model routing is useful when it solves a real constraint. HydraFusion is a useful reference point: its stated goal is runtime orchestration across models for a task-specific balance of performance, cost, and latency, with pricing based on token consumption by model. That is the general shape of a gateway policy, not a reason to add model hopping to every turn.

Voice sessions make routing harder because the decision is usually made at session start while the user's needs change during the conversation.

A sensible first version uses session-level routing:

type RouteRequest = {
  tenantId: string;
  userId: string;
  requestedMode: "standard" | "extended-thinking";
  useCase: "support" | "assistant" | "workflow";
  region: string;
};

function chooseRoute(input: RouteRequest): Route {
  if (input.requestedMode === "extended-thinking") {
    return {
      provider: "configured-provider",
      model: "gemini-3.8-live-extended-thinking",
      reason: "requested_by_policy",
    };
  }

  return {
    provider: "configured-provider",
    model: "gemini-3.8-live",
    reason: "default_live_route",
  };
}

The model names represent the product distinction in this architecture. They should be configured rather than hard-coded because the available identifiers and provider endpoints must come from the actual service contract.

Don't Route Every Audio Turn Independently

Switching models in the middle of a live turn is risky. The new model may not have the same conversational state, tool context, audio assumptions, or interruption state.

If adaptive routing is necessary, prefer explicit boundaries:

• route a new session differently
• route a new user turn after a completed turn
• route a separate tool or reasoning job
• create a controlled handoff with a compact session summary

type HandoffContext = {
  sessionId: string;
  conversationSummary: string;
  activeTask?: string;
  userPreferences?: Record<string, string>;
  pendingToolState?: {
    callId: string;
    name: string;
    status: "pending";
  };
  reason: "latency_degradation" | "capability_required" | "provider_failure";
};

The summary is data with privacy implications. Don't copy an entire transcript into every route by default. Decide what context is necessary, how long it remains available, and whether the user or tenant has opted out of retention.

Extended Thinking Needs A Separate Budget

Extended Thinking should not be configured as a free upgrade for every voice interaction. It can change both latency and cost, and it may be unnecessary for simple turns.

One useful pattern is to keep the live conversational route responsive and delegate selected tasks to a reasoning route:

user asks a simple question
    |
    v
standard live route responds in the conversation

user asks for a complex comparison
    |
    +--> live route acknowledges and frames the task
    |
    +--> reasoning route evaluates the task
    |
    +--> gateway returns a bounded result to the live session

The exact behavior depends on the supported model and tool protocol. The architectural point is to make the expensive path deliberate.

function shouldUseExtendedThinking(input: {
  requested: boolean;
  taskClass: "simple" | "multi_step" | "high_risk";
  tenantAllowsIt: boolean;
}) {
  if (!input.tenantAllowsIt) return false;
  if (input.requested) return true;

  return (
    input.taskClass === "multi_step" ||
    input.taskClass === "high_risk"
  );
}

This policy doesn't claim that a classifier can reliably identify every difficult request. It gives the system an auditable place to make the decision and a place to add limits later.

Model Hopping Makes Latency Less Predictable

HydraFusion's multi-model approach highlights the upside of runtime orchestration, but the trade-off matters for voice: every additional hop adds another possible delay and failure domain.

client
  -> gateway
  -> router
  -> model A
  -> tool
  -> model B
  -> gateway
  -> client

That can be the right architecture for a complex workflow. It is a poor default for every spoken turn.

Measure the complete path:

type TurnTiming = {
  sessionId: string;
  turnId: string;
  captureStartedAt?: number;
  gatewayReceivedAt?: number;
  upstreamSentAt?: number;
  firstModelEventAt?: number;
  firstAudioOutAt?: number;
  userInterruptAt?: number;
  responseCompletedAt?: number;
  toolStartedAt?: number;
  toolCompletedAt?: number;
};

First audio and completed response are separate measurements. A session can have a good time to first audio and still feel poor if it pauses repeatedly, ignores interruptions, or waits too long for a tool result.

What The Gateway Should Own

The gateway should own decisions that need consistency across clients, models, and providers.

Identity And Tenant Policy

Authenticate the client before opening an upstream live session. Bind that session to the authenticated tenant and user rather than trusting identifiers supplied in arbitrary client messages.

Tenant policy can control:

• which live models are available
• whether Extended Thinking is allowed
• which tools may be called
• maximum concurrent sessions
• audio retention
• transcript retention
• regional routing
• budget limits
• fallback behavior

Evaluate these policies at session creation and, where relevant, at each tool request. A session that was valid at connection time should not gain access to a newly enabled tool because the client changed a field in a message.

Tool Authorization And Execution

The model can request a tool, but it should not decide whether the caller is authorized to perform the underlying action.

async function handleToolRequest(
  session: LiveSession,
  request: ToolRequest,
) {
  const policy = await loadToolPolicy(session.userId, request.name);

  if (!policy.allowed) {
    return {
      callId: request.callId,
      status: "denied",
      error: "tool_not_allowed",
    };
  }

  const args = validateToolArguments(request.name, request.args);

  const result = await runTool({
    tenantId: session.tenantId,
    userId: session.userId,
    name: request.name,
    args,
    deadlineMs: policy.deadlineMs,
  });

  return {
    callId: request.callId,
    status: "completed",
    result,
  };
}

The tool runner should enforce idempotency where an operation has side effects. A reconnect, duplicate model event, or client retry must not create two orders, send two messages, or execute the same administrative action twice.

Redaction And Data Handling

The gateway is one of the few places where audio, transcripts, identity, routing decisions, and tool activity meet. That makes it a useful privacy control point and a tempting place to log too much.

Default logs should contain metadata rather than raw content:

{
  "event": "live_turn_completed",
  "session_id": "session_opaque_id",
  "tenant_id": "tenant_opaque_id",
  "route": "standard_live",
  "duration_ms": 0,
  "interrupted": true,
  "tool_count": 1,
  "audio_retained": false
}

The zero duration is illustrative, not a measurement. In a real event, populate it from the timing record.

If transcript or audio capture is enabled, make it an explicit policy decision. Record:

• whether capture was enabled
• why it was enabled
• who can access it
• how long it is retained
• how deletion works
• whether sensitive fields are redacted
• whether provider-side retention is controlled separately

Turning off application logging does not necessarily turn off provider retention. Provider data handling is a separate contract and needs to be verified against the actual service documentation.

Health, Capacity, And Failover

A live gateway needs health signals at several levels:

• edge health
• WebSocket acceptance rate
• upstream connection success
• active sessions
• audio backlog
• model event delay
• tool latency
• clean versus abnormal session termination
• provider errors
• budget consumption

The August 2026 GitHub availability report is a useful reminder that capacity monitoring, retry policy, and resiliency planning are not abstract infrastructure concerns. A gateway that depends on one cloud path or one database primary inherits that dependency's failure modes. The report's discussion of Azure migration and database-primary considerations is a relevant example of why “we have a second application instance” is not the same as end-to-end failover.

For live sessions, failover should be stated precisely. It might mean:

• new sessions are routed to another provider
• existing sessions receive a clean reconnect option
• tool execution continues while the audio route is rebuilt
• the gateway stops accepting new sessions during capacity pressure
• the client gets a degraded text-only mode

It usually does not mean transparently moving an active audio stream to another model with no user-visible effect. That requires explicit support for state transfer and careful testing. A generic retry loop is not a session failover strategy.

Real-Time Audio Flow And Session Semantics

The user experience is determined by the complete path, not model latency alone:

microphone
  -> capture buffer
  -> encode
  -> client WebSocket
  -> edge
  -> session runtime
  -> model input
  -> model output
  -> gateway output buffer
  -> client decoder
  -> speaker

Every boundary can add delay. Some delays are unavoidable. Others are accidental, such as waiting for a database write before forwarding audio or buffering too much input while waiting for a turn to end.

The gateway should measure each segment separately. Otherwise, a provider improvement can be hidden by a client-side buffer, or a network problem can be misdiagnosed as slow generation.

Audio Capture, Encoding, And Transport

The client should capture audio in the format expected by the live model integration or convert it once at a clearly defined boundary. Sample rate, channel layout, sample format, and codec affect bandwidth, CPU usage, voice activity detection, and buffering.

Use binary WebSocket frames for audio where the integration supports them. Keep control events in a separate typed envelope.

type ControlEvent =
  | {
      type: "session.start";
      sessionId: string;
      clientVersion: string;
    }
  | {
      type: "turn.cancel";
      turnId: string;
      reason: "user_interrupt" | "timeout" | "policy";
    }
  | {
      type: "tool.confirm";
      requestId: string;
    };

type AudioFrame = {
  sequence: number;
  capturedAtMs: number;
  bytes: Uint8Array;
};

The client should attach a monotonically increasing sequence number to audio frames. The gateway can use it to detect gaps, estimate transport delay, and distinguish a late frame from a duplicate.

Don't assume every missing frame should be retransmitted. In conversational audio, waiting for an old frame can be worse than dropping it and continuing. The correct behavior depends on the model protocol and product tolerance for gaps. Make the policy explicit.

Session Lifecycle And WebSocket Semantics

A session should have a state machine rather than a collection of boolean flags:

NEW
  |
  v
AUTHENTICATING --> REJECTED
  |
  v
STARTING_MODEL --> FAILED
  |
  v
ACTIVE <------+
  |           |
  |           v
  |      RECONNECTING
  |           |
  v           +----> ACTIVE
INTERRUPTING
  |
  v
CLOSING --> CLOSED

A handshake should establish:

• authenticated principal
• tenant and policy context
• session identifier
• supported client capabilities
• selected model class or routing policy
• audio contract
• session expiry or maximum duration
• resume behavior, if supported

Keep-alives should prove that both the WebSocket and the session runtime are alive. A WebSocket can remain open while the model connection behind it has failed. Application-level health events can detect that distinction.

Reconnect handling is where many stateful designs become vague. Decide what the client sends after reconnect:

• last received server event sequence
• last played audio sequence
• session identifier and reconnect token
• client capability information
• whether the microphone was still active

The gateway can then choose among several strategies:

1. Resume the existing runtime.
2. Reattach the client to a still-running runtime.
3. Start a new model session with a compact summary.
4. Reject resume and require a new session.

Don't promise transparent resume unless the system can preserve the relevant state. Replaying all previous audio frames is an expensive and semantically risky substitute for recovery.

Latency Budgets And Measurement

The supplied material does not provide Gemini 3.8 Live-specific latency targets, so any production target must come from measurement rather than a provider guarantee.

A useful budget breaks the user-visible delay into components:

time to first playable response
  = capture and turn detection
  + client-to-gateway transport
  + gateway scheduling
  + model time to first audio
  + gateway-to-client transport
  + client playback buffer

For a tool-backed response:

tool path
  = tool authorization
  + tool execution
  + result transfer
  + model continuation

Measure at least:

• user speech end to gateway turn detection
• turn detection to provider request
• provider request to first model audio
• first model audio to first speaker playback
• user interruption to output cancellation
• tool request to tool result
• tool result to resumed model audio
• reconnect start to usable session

Use client and gateway clocks carefully. Clock skew can make distributed timestamps look precise while being wrong. Where possible, measure durations inside one process or use a synchronized tracing system.

Latency percentiles matter more than averages. A smooth median with a long tail can still produce a frustrating voice experience. Break down tail latency by model route, tool, region, network condition, and interruption status.

Avoid hiding queueing time inside “model latency.” If the gateway waits for a worker before opening the provider request, that is gateway scheduling delay.

Interruption Handling

Interruption is normal conversation behavior. It is not an exceptional error.

When the user begins speaking while the model is producing audio, the gateway should:

1. detect or receive the interruption signal
2. stop or mark current output as cancelled
3. cancel the active model turn where supported
4. cancel or quarantine tool work that is no longer relevant
5. advance the session to the new user turn
6. prevent late events from the old turn reaching the speaker

The last step is easy to miss. Cancellation is asynchronous. A provider may emit an output event after the gateway requests cancellation. Every output event should carry a turn or generation identifier.

type OutputEvent = {
  sessionId: string;
  turnId: string;
  generation: number;
  kind: "audio" | "transcript" | "tool_request";
  payload: unknown;
};

class TurnGate {
  private activeGeneration = 0;

  beginTurn(): number {
    this.activeGeneration += 1;
    return this.activeGeneration;
  }

  accepts(event: OutputEvent): boolean {
    return event.generation === this.activeGeneration;
  }
}

A tool call needs its own interruption policy. A read-only lookup may be allowed to finish and populate a short-lived cache. A side-effecting action should be cancelled if possible, or protected by an idempotency key if cancellation races with execution.

Fallback behavior should preserve the conversational contract:

• if the live model is temporarily unavailable, speak a short service message
• if a tool is slow, acknowledge the request and wait within a bounded time or explain that the information is unavailable
• if audio output fails but text is available, offer a text response
• if repeated interruptions make the stream unstable, reset the turn or ask the user to repeat
• if the session cannot be safely resumed, close it clearly and provide a way to start again

Graceful degradation is not pretending nothing failed. It is making the failure understandable without leaking internal details or speaking stale output.

Tool Calls And Selective Multi-Model Orchestration

The most useful multi-model design is selective.

HydraFusion's central idea is runtime orchestration across models for task-specific tradeoffs between quality, cost, and latency. That maps naturally to a voice gateway, but the voice version needs stricter boundaries. A user is waiting in real time, and a model switch can alter timing, context handling, tool semantics, and recovery behavior.

Treat routing as a state transition with a reason, not as a hidden optimization performed on every event.

Stable Routing Paths

A practical policy can start with a few stable paths:

ordinary conversation
  -> Gemini Live

conversation requiring an approved tool
  -> Gemini Live
  -> tool executor
  -> Gemini Live continuation

deep or complex request
  -> policy check
  -> Extended Thinking path
  -> concise spoken answer

provider degradation
  -> approved fallback route
  -> constrained session behavior

Use signals available before the decision:

• session mode
• user or tenant policy
• request category
• tool requirement
• provider health
• current capacity
• budget remaining
• whether the current turn can tolerate additional latency

Semantic routing can help, but an unconstrained classifier in the critical path may cost enough time to erase the benefit of the selected model. Classify at stable boundaries where possible.

type RouteContext = {
  sessionMode: "conversation" | "assisted";
  requestClass: "ordinary" | "tool" | "deep" | "unknown";
  providerHealthy: boolean;
  budgetAllowsDeepPath: boolean;
};

function chooseRoute(ctx: RouteContext) {
  if (!ctx.providerHealthy) return "fallback";
  if (ctx.requestClass === "deep" && ctx.budgetAllowsDeepPath) {
    return "extended-thinking";
  }
  return "live";
}

The fallback route should contain only integrations tested for audio contract, session semantics, privacy, and recovery. The supplied notes don't identify a specific alternate provider or model.

Parallel model execution deserves caution. Running several agents at once can help offline exploration, and isolated worktrees are useful for preventing concurrent coding tasks from interfering with one another. In a live voice session, however, parallelism multiplies token usage and produces competing outputs.

Use parallel calls for bounded, non-user-facing work where the result justifies the cost. Don't stream multiple speculative voices to the caller and hope the gateway chooses the right one later.

Tool Authorization, Execution, And Results

A tool call introduces a second runtime into the conversation. The model may request a search, summary, account lookup, or data fetch, but the gateway remains responsible for authorization, validation, timeout, and result handling.

model requests tool
  -> gateway validates tool name
  -> gateway validates arguments
  -> policy authorizes operation
  -> executor runs with timeout
  -> result is redacted and normalized
  -> model receives bounded result
  -> user receives response audio

The model should receive only fields required to complete the turn. A database row, internal error message, or full search-result payload should not pass through by default.

type ToolResult =
  | {
      ok: true;
      requestId: string;
      data: unknown;
      truncated: boolean;
    }
  | {
      ok: false;
      requestId: string;
      code: "timeout" | "denied" | "unavailable" | "invalid";
      retryable: boolean;
    };

async function runTool(
  request: ToolRequest,
  policy: ToolPolicy
): Promise<ToolResult> {
  const decision = authorizeTool(request, policy);

  if (decision === "deny") {
    return {
      ok: false,
      requestId: request.requestId,
      code: "denied",
      retryable: false
    };
  }

  if (decision === "confirm") {
    return {
      ok: false,
      requestId: request.requestId,
      code: "denied",
      retryable: false
    };
  }

  try {
    const data = await executeWithTimeout(
      request.toolName,
      request.arguments,
      policy.maxExecutionMs
    );

    return {
      ok: true,
      requestId: request.requestId,
      data: limitToolResult(data),
      truncated: wasTruncated(data)
    };
  } catch {
    return {
      ok: false,
      requestId: request.requestId,
      code: "unavailable",
      retryable: true
    };
  }
}

A confirmation state should be represented explicitly in the real protocol rather than collapsed into denied. Model intent, user approval, and execution authorization are separate decisions.

For side-effecting tools, use a request ID or idempotency key. If the WebSocket disconnects after the tool executes but before the result reaches the client, a retry must not create a second side effect.

Cost-Aware Routing

HydraFusion's research-preview pricing model is based on model token consumption, which makes the cost implication of orchestration straightforward: every additional model path can add to the bill. Voice sessions add duration, transcript context, tool results, and parallel work.

A cost policy should be attached to the session and checked at route boundaries:

session budget
  |
  +-- live conversation allocation
  +-- tool-result allocation
  +-- extended-thinking reserve
  +-- recovery/fallback reserve

Don't treat the budget as a single number that the model can consume freely. Reserve capacity for recovery. If a session spends everything on an early expensive operation, the system may have no room for a final response or fallback.

Useful controls include:

• cap Extended Thinking turns per session
• limit tool-call retries
• bound tool-result size
• avoid parallel model calls unless required
• summarize older context instead of growing it indefinitely
• stop or downgrade sessions that exceed policy limits
• apply tenant-level concurrency limits
• record route and usage metadata for cost attribution

Cost and latency often point in the same direction, but not always. A cheaper model may require more retries or tool calls. A more expensive model may resolve a request in one pass. Measure cost per successful user task, not only cost per invocation.

Observability That Follows The User Experience

A live voice gateway can be technically available and still feel broken.

The WebSocket may remain open. The model may return valid events. The tool may eventually succeed. Yet if the user waits too long after speaking, hears stale audio after an interruption, or gets silence while a downstream service retries, the system has failed from their point of view.

Measure interactions rather than HTTP requests. Every user turn needs a correlation ID that follows audio frames, model events, tool calls, and playback state.

Latency Components And Budgets

A practical interaction budget includes:

Component	What it measures	What to watch
Capture and encode	Speech becoming a transport frame	Client CPU and conversion
Uplink transport	Client to gateway	Network distance and packet loss
Gateway admission	Authentication, scheduling, forwarding	Queue depth and overloaded workers
Model first audio	Forwarded input to first model audio	Model load and route
Downlink transport	Model audio to client	Region placement and jitter
Playback start	Audio becoming audible	Client buffer and decoder
Turn completion	Final response or interruption	Turn detection and cancellation
Tool round trip	Tool request through result	External latency and retries

For voice, time to first audible response is usually more important than total turn duration. A long answer can be acceptable if it starts promptly and can be interrupted cleanly. A short answer after a long silence feels much worse.

Keep separate budgets for the normal path and degraded paths involving tools, model switches, retries, reconnects, or provider timeouts.

interaction_start
  ├── audio_capture_ms
  ├── uplink_ms
  ├── gateway_queue_ms
  ├── model_first_event_ms
  ├── model_first_audio_ms
  ├── downlink_ms
  ├── playback_start_ms
  ├── tool_wait_ms
  └── interruption_to_playback_stop_ms

Track at least median, p95, and p99 for user-visible components. Break down the tail by model route, tool, region, network condition, and interruption status.

Telemetry Schema

Use traces for individual interactions, metrics for population behavior, and logs for state transitions that need investigation.

A trace can show why one turn was slow. A metric can show that p95 first-audio latency increased across a region. A structured log can preserve the reason a session moved from active to degraded.

voice.session
  ├── client.audio_ingest
  ├── gateway.authenticate
  ├── gateway.route
  ├── provider.live_turn
  │     ├── provider.first_event
  │     ├── provider.first_audio
  │     └── provider.turn_complete
  ├── tool.invoke
  │     └── tool.result
  ├── gateway.interruption
  └── client.playback

At the session level, record:

• session ID in a controlled form
• tenant or account reference
• gateway region
• provider route
• client and gateway protocol versions
• session start and end reason
• reconnect count
• total duration
• whether retention was enabled

At the interaction level, record:

• interaction ID
• model route
• first event and first audio timestamps
• completion timestamp
• interruption state
• tool usage
• retry count
• terminal status

Avoid transcript text, raw audio, authentication tokens, and tool arguments in default traces.

A small metrics set is more useful than hundreds of unowned counters:

live_session_started_total
live_session_ended_total{reason}
live_session_active
live_session_reconnect_total{cause}
live_interaction_total{status}
live_first_audio_latency_ms{route,region}
live_turn_duration_ms{route}
live_interruption_stop_latency_ms{route}
live_tool_duration_ms{tool,status}
live_provider_error_total{provider,class}
live_gateway_queue_depth{region}
live_audio_buffer_underrun_total{client_version}

Don't use session IDs, user IDs, request IDs, or arbitrary tool arguments as metric labels. High-cardinality labels can turn an observability system into another availability problem.

Dashboards And Privacy-Aware Monitoring

Organize dashboards around user-visible outcomes.

Voice Experience

Show:

• active sessions
• unexpected session endings
• time to first audible response
• interruption stop latency
• audio underruns
• reconnect rate
• tool-related silence duration

Split charts by region, client version, model route, and gateway version.

Provider And Routing

Show:

• provider error classes
• first-event and first-audio latency
• route switches
• fallback success rate
• provider connection saturation
• usage consumption where available

Multi-model routing can balance performance, cost, and latency, as HydraFusion describes at a high level. It also adds variance. A route switch may improve availability while making the next turn slower or changing response behavior.

Capacity

Show:

• gateway worker utilization
• connection pressure
• queue depth
• active WebSocket count
• connection creation rate
• outbound provider connections
• memory pressure
• rate-limit responses

The August 2026 GitHub availability material reinforces that capacity monitoring and retry behavior are availability features, not housekeeping.

The default telemetry payload should contain metadata, not content. For most production debugging, engineers need to know that a response was interrupted or a tool timed out. They don't need the user's recording.

{
  "tool_name": "calendar_lookup",
  "tool_status": "timeout",
  "argument_policy": "redacted",
  "result_class": "upstream_timeout",
  "duration_ms": 1800
}

Redaction must happen before export. Once sensitive content enters a log sink, removing it from one dashboard does not remove it from replicas, exports, or backups.

Use temporary diagnostics carefully:

• require an incident or change identifier
• restrict the session or tenant scope
• set automatic expiration
• show who enabled capture
• encrypt captured data
• delete it when the investigation ends
• record the deletion event

Privacy, Security, And Data Lifecycles

Privacy decisions belong in the session design, not in a policy document added after launch.

A live voice system processes a continuous stream. Audio may be buffered in the client, gateway, provider connection, tool layer, monitoring system, and support workflow.

microphone
  → client buffer
  → encrypted gateway connection
  → gateway transient buffer
  → live model session
  → optional tool request
  → optional transcript or audit store
  → deletion and retention controls

For every boundary, answer:

• Is the data copied?
• Is it encrypted?
• How long does it exist?
• Who can access it?
• Can it be deleted?
• What happens when the session fails halfway through?

Data Minimization And Retention

The safest audio retention policy is not retaining audio.

In streaming-only mode, the client sends frames to the gateway, the gateway forwards them to the live session, and transient buffers are released after forwarding or playback. This does not eliminate provider-side handling or operational metadata, but it avoids accidentally creating an application-owned recording system.

An ephemeral session should have a clear lifecycle:

created
  → authenticated
  → active
  → draining
  → closed
  → expired

Buffers should be scoped to the session and released when it reaches closed or expired.

Retention needs separate policies for separate data classes:

Data class	Default purpose	Retention decision
Raw audio	Usually not required after streaming	Disable by default
Temporary audio buffers	Transport and playback	Release at session close
Transcript text	Continuity, support, or user history	Retain only when required
Tool audit record	Security and accountability	Apply audit policy
Aggregate metrics	Reliability and capacity trends	Retain without content
Error traces	Incident investigation	Limited operational window
Consent records	Proof of user choice	Apply legal and product requirements
Deletion records	Lifecycle enforcement	Keep a non-content audit record

The actual periods must come from the requirements that apply to the service. The architecture should support different policies rather than one hard-coded duration.

Deletion should be an idempotent workflow:

deletion_requested
  → records_located
  → application_copies_deleted
  → indexes_updated
  → downstream_deletions_requested
  → completion_verified
  → deletion_audited

Keep the audit entry about the action, not the deleted content:

{
  "event": "data.deletion.completed",
  "request_id": "del_204",
  "subject_ref": "user_hash_88",
  "data_classes": ["transcript", "session_metadata"],
  "completed_at": "2026-09-16T13:00:00Z",
  "actor": "privacy_worker",
  "verification": "store_and_index_checked"
}

If a provider or downstream store is unavailable, record the pending state and retry safely. Quietly marking the request complete because one store succeeded is not a lifecycle.

Auditability And Access Control

For every sensitive operation, record who or what performed it, what class of data it affected, why it was allowed, and whether it succeeded.

Relevant operations include:

• starting a session
• enabling a tool
• changing a routing policy
• accessing a support capture
• exporting a transcript
• changing retention settings
• processing a deletion request
• granting temporary operator access

A latency engineer may need route, region, and timing data without transcript access. A support operator may need a customer-visible transcript without provider credentials or raw audio. Define those paths explicitly.

Consent should be represented as state that the gateway can enforce. If the user has not permitted retention, the downstream transcript store should not receive the data.

A consent record should identify:

• user or account reference
• notice or policy version
• processing purpose
• time and channel of consent
• withdrawal state
• affected session or data classes

Support access should be:

• explicitly requested
• approved by an authorized role
• limited to a session or case
• time-bound
• logged
• revoked automatically

A dashboard that lets anyone search all voice sessions is not an audit system. It is a future incident.

Security At The Live Boundary

Authenticate the client before creating an upstream session. Don't treat possession of a WebSocket URL as sufficient authorization if it can be replayed or shared.

Session credentials should be:

• short-lived where possible
• scoped to the intended session
• excluded from logs
• rotated according to the key policy
• invalidated when the session ends or is revoked

Keep provider credentials on the server. A client should not receive a general-purpose provider key merely because it needs to send audio.

Network controls still matter inside the service:

• isolate gateway workers from unrelated workloads
• restrict outbound destinations
• use private connectivity where supported
• apply egress policies to tool workers
• separate production and test credentials
• rotate secrets without dropping active sessions where possible

Abuse controls should cover session creation, active session count, tool calls, and reconnect attempts. A reconnect storm can be more damaging than the original provider outage.

Finally, treat prompts, tool results, and model-generated instructions as untrusted input to the control plane. Model output must not decide its own authorization, retention policy, routing override, or audit suppression.

Failure Modes, Fallbacks, And Recovery

A live gateway has more failure modes than a conventional API because work is continuous and stateful.

The client can lose connectivity while the provider remains healthy. The provider can accept a session but stop producing audio. The model can return a tool request while the tool system is overloaded. The gateway can be healthy but unable to create upstream sessions because a quota or regional capacity limit has been reached.

Write these failures down before implementing fallback behavior. Otherwise each component will retry according to its own assumptions, and the system will turn a small outage into a retry storm.

Failure class	Example	User-visible result	Preferred response
Client transport	Wi-Fi change, mobile handoff	Silence or reconnect	Resume or restart with clear state
Gateway capacity	Worker or queue saturation	Delayed start	Shed load or reject early
Provider connection	Upstream close or timeout	Stalled response	Fail over if state permits
Model response	Invalid or incomplete event	Partial answer	Stop safely and recover
Tool dependency	Timeout or authorization failure	Missing capability	Explain limitation and continue
Security or policy	Expired session or denied tool	Refused action	Fail closed
Data path	Audit store unavailable	Core voice may still work	Degrade storage within policy
Regional outage	Gateway or provider region impaired	Session loss	Reconnect or move new sessions

Redundancy And Active Sessions

Use redundancy at session admission first. New sessions should avoid an unhealthy region or provider route. Existing live sessions are harder because moving them may require recreating state and interrupting playback.

A practical design separates:

• a session directory that knows where a session is active
• gateway workers that own WebSocket connections
• a provider connection manager
• routing policy that selects an alternate path
• a client reconnect protocol

The session directory should not carry audio:

{
  "session_id": "session_7f2",
  "gateway_region": "asia-south",
  "provider_route": "primary",
  "state": "active",
  "last_client_sequence": 1842,
  "last_server_sequence": 927,
  "expires_at": "2026-09-16T12:20:00Z"
}

Sequence numbers help detect gaps and duplicate delivery. They don't make a session resumable by themselves.

A recovery attempt should have explicit stages:

1. detect the broken connection
2. stop accepting audio for the dead path
3. flush or discard stale playback according to policy
4. attempt supported session recovery
5. otherwise create a replacement session
6. notify the client of the new state
7. resume only after authorization and routing checks

Don't retry every failure. An authentication error will not be fixed by reconnecting five times. A regional network failure may justify a bounded attempt on another route. Classify errors first.

Graceful Degradation And Hard Failure

Graceful degradation is useful when the core interaction remains truthful:

• continue voice conversation without an optional transcript store
• answer without an unavailable nonessential tool
• switch to a lower-capability route
• return text when audio playback cannot continue
• stop a long tool call and offer a retry
• preserve the current turn while disabling a feature

Hard fail when continuing would be misleading, unsafe, or impossible to authorize:

• expired or invalid credentials
• denied access to a requested tool
• uncertain identity for a sensitive action
• corrupted session state
• missing policy decision
• inability to protect or route audio securely
• provider output that cannot be associated with the active session

A client-facing state machine might look like this:

active
  → degraded
  → recovering
  → active

active
  → interrupted
  → recovering
  → active

active
  → failed
  → reconnect_required

Every transition needs a reason class and an allowed next action. A degraded session may continue with tools disabled. A failed session should not accept more audio just because the WebSocket remains open.

Recovery Runbooks

Provider Latency Or Timeout

1. Check first-audio latency by route and region.
2. Compare the affected route with the fallback.
3. Check gateway queue depth and outbound connection pressure.
4. Determine whether failures affect new sessions, existing sessions, or both.
5. Stop automatic retries if the error is systemic.
6. Shift new sessions according to policy.
7. Preserve a sample of error traces and provider response classes.
8. Communicate the degraded capability and next review time.

Don't start by restarting all workers. That can destroy healthy sessions and erase useful evidence.

Reconnect Storm

1. Chart reconnects by client version, region, and close reason.
2. Check whether the gateway or provider is closing connections.
3. Apply bounded backoff with jitter.
4. Reject excess session creation early.
5. Protect the provider with a connection-rate limit.
6. Verify that session ownership leases aren't duplicated.
7. Restore admission gradually.

Tool Dependency Failure

1. Check tool timeout and error rates.
2. Separate authorization failures from upstream unavailability.
3. Disable the affected tool if retries increase pressure.
4. Keep voice sessions alive when the tool is optional.
5. Return a truthful capability message.
6. Re-enable after a controlled health check.

Tool failures should not automatically restart the model session. Restarting loses context and can make a dependency problem look like a model problem.

Audio Playback Stall

1. Check whether model audio is being generated.
2. Compare server output timestamps with client playback acknowledgements.
3. Inspect downlink loss, buffer underruns, and decoder errors.
4. Determine whether the client, gateway, or provider is stuck.
5. Stop stale playback before recovery.
6. Reconnect or switch to text according to client capability.

Server traces alone cannot tell you whether bytes reached the speaker. Client-side telemetry is required.

Stale Output And Duplicate Speech

A late event from a failed route can arrive after the fallback has started. Use session, turn, and generation identifiers to reject stale output. The output sink should have one active producer at a time.

type ActiveRoute = {
  routeId: string;
  turnId: string;
  generation: number;
};

function canEmit(
  active: ActiveRoute,
  event: { routeId: string; turnId: string; generation: number }
): boolean {
  return (
    active.routeId === event.routeId &&
    active.turnId === event.turnId &&
    active.generation === event.generation
  );
}

This check belongs close to the audio output boundary. Earlier checks help, but the final sink must refuse stale audio because that is where duplicate speech becomes visible.

Rollout From Beta To Production

A live gateway should earn production traffic in stages.

The first milestone isn't “the model answered.” It is proving that the complete loop works under controlled conditions: audio enters, a session is authorized, model audio returns, interruptions stop stale playback, tools obey policy, and the system leaves no unexpected data behind.

Every phase needs a way to reduce traffic, disable optional tools, switch routes, or stop new sessions without taking down unrelated services.

Phase 0: Contract And Harness Validation

Before real users, validate:

• client and gateway event schemas
• session ownership rules
• sequence numbers and reconnect behavior
• interruption and cancellation semantics
• tool authorization
• telemetry fields and redaction
• retention and deletion workflows
• provider error classification

Use synthetic audio and controlled tool responses. A test harness should inject delays, disconnects, malformed events, duplicate events, and provider errors. Testing only the happy path gives a false sense of readiness because many live failures are ordering and lifecycle failures.

Phase 1: Internal Traffic

Run with tightly bounded concurrency and keep raw content capture disabled unless an approved test requires it. Focus on:

• session creation success
• first-audio timing
• interruption stop timing
• reconnect outcomes
• tool timeout behavior
• trace completeness
• resource usage per active session

This phase should establish what healthy looks like before alerts are defined.

Phase 2: Small User Pilot

Choose a narrow audience and limited capability set. Keep optional tools behind feature flags. Route a controlled percentage of eligible sessions to the new gateway while the existing path remains available.

Define exit criteria before starting:

• no unresolved security or privacy-critical findings
• session close behavior is understood
• fallback behavior is truthful
• latency tails fit the chosen budget
• on-call staff can execute recovery runbooks
• deletion and access workflows have been tested

Don't expand because the median looks good if interruption failures or reconnect loops remain unexplained.

Phase 3: Gradual Scale

Increase traffic in steps. After each step, observe long enough to expose connection churn, tool usage, and delayed cleanup.

Control the blast radius with:

• tenant or account allow-lists
• region-level rollout flags
• client-version gates
• maximum active sessions
• tool-specific enablement
• provider route caps
• automatic rollback thresholds

Parallel-agent work benefits from isolated worktrees because concurrent tasks don't interfere with one another. The same principle applies operationally: isolate rollout configuration and experiments so one change doesn't silently alter every active session.

Phase 4: Production Default

Make the new path the default only while the fallback remains operational. A fallback that hasn't been exercised is documentation, not resilience.

Keep:

• a tested rollback switch
• a provider route override
• a tool disable switch
• a session admission limit
• an incident owner for each dependency
• a change log tied to deployed versions

Production readiness is the ability to detect, contain, explain, and recover from instability.

Backward Compatibility And Change Control

The client should not understand every provider-specific event. Put the compatibility boundary in the gateway and expose a versioned internal protocol.

For each event, define:

• event type
• required and optional fields
• sequence behavior
• acknowledgement behavior
• retry and duplicate semantics
• behavior when the client does not understand a field

Additive changes are easier than semantic changes. A new optional field is usually safe. Changing “turn complete” into “response complete” without versioning can break playback and tool state.

Keep the gateway capable of serving older clients during migration. Measure use of old contracts, publish an end date, and remove them through a documented change process.

Provider changes need the same adapter discipline:

{
  "type": "audio.delta",
  "session_id": "session_7f2",
  "interaction_id": "turn_018",
  "sequence": 927,
  "audio": "<transient payload>",
  "is_final": false
}

The adapter can map provider-specific events into that form. The rest of the system should not need to know whether a provider calls the event a delta, chunk, part, or response segment.

When changing routes or models, compare behavior as well as latency. A fallback can be technically successful while changing tool-call frequency, response length, interruption behavior, or cost.

Incident Readiness

Set service objectives around experience:

• successful session establishment
• time to first audible response
• interruption stop latency
• reconnect recovery rate
• tool-call success
• unexpected session termination
• percentage of sessions with complete telemetry

Define rollback triggers before the incident:

rollback if:
  first_audio_p95 exceeds the approved threshold
  unexpected session endings rise above the approved rate
  reconnect attempts create sustained admission pressure
  privacy redaction checks fail
  tool authorization errors indicate a policy regression

The actual thresholds belong in the approved operating policy. The important part is deciding them before production traffic.

Keep the blast radius bounded. If a gateway version has a session-state bug, it should not own every region and client version at once. If a provider route is expensive, cap the sessions that can select it. If a tool integration is new, enable it for a narrow policy scope.

The on-call handoff should include:

• rollout percentage
• enabled routes and tools
• known degraded modes
• dashboard links
• rollback controls
• dependency contacts
• recent changes
• open incidents
• data-capture status

A live voice incident is a poor time to discover that the person who enabled debug audio capture has left for the day.

Implementation Boundaries And Open Questions

A gateway can make a live model easier to operate, but it can't remove the underlying uncertainty. It adds another distributed system between the user and the model. Every boundary gives you a policy point and another failure point.

The supplied material does not include an official Gemini 3.8 Live API specification, detailed session diagrams, concrete latency budgets, or a production rollout case study for a Gemini-like live audio gateway. Those gaps should remain visible in the design review.

The architecture assumes that:

• Gemini 3.8 Live and Gemini 3.8 Live Extended Thinking expose a contract that can be adapted behind a gateway interface
• the gateway can identify turn boundaries, interruptions, model output, and tool calls
• cloud-processing consent is established before audio is sent
• provider health and capacity can be measured well enough to support routing
• the organization accepts some implementation complexity in exchange for control over privacy, cost, and failover

Provider Contract Questions

Before implementation, verify:

• connection and authentication semantics
• audio encodings and framing rules
• session limits
• interruption behavior
• tool-call message format
• usage reporting
• Extended Thinking behavior in a live stream
• regional availability and data handling
• retry and resume semantics

Until those answers are confirmed, keep the provider adapter narrow. Don't spread assumed message names or undocumented fields across the gateway.

State And Fallback Questions

Create and test a formal state machine for:

• connecting
• active input
• model output
• interruption
• tool execution
• provider failure
• route change
• draining
• closed

Test every transition with delayed frames, duplicate events, client disconnects, provider errors, and late tool results.

A fallback route is not equivalent merely because both providers generate audio. Differences in interruption, tool calling, context handling, voice behavior, and safety policy can change the product experience. A fallback contract needs capability checks and explicit user-facing behavior.

Capacity And Cost Questions

There are no Gemini 3.8 Live production benchmarks, latency distributions, or cost curves in the supplied notes. Don't publish a gateway SLA based on assumptions. Measure connection setup, first output, interruption response, tool latency, session duration, and settled usage during staged rollout.

Token-based costs can escalate with long sessions, repeated tools, retries, duplicated context, and concurrent agents. Use budget controls at session, tenant, route, tool, and experiment levels.

A budget check should happen before an expensive operation:

type Usage = {
  inputTokens?: number;
  outputTokens?: number;
  toolCalls: number;
  estimatedCostUnits: number;
};

type Budget = {
  maxCostUnits: number;
  maxToolCalls: number;
};

function budgetDecision(
  usage: Usage,
  budget: Budget,
): "continue" | "degrade" | "stop" {
  if (usage.estimatedCostUnits >= budget.maxCostUnits) {
    return "stop";
  }

  if (usage.toolCalls >= budget.maxToolCalls) {
    return "degrade";
  }

  return "continue";
}

The gateway should distinguish estimated usage from settled provider usage. Estimates help admission control, but they are not invoices. Reconcile provider usage asynchronously and investigate divergence before widening traffic.

Multi-Agent Expansion

Parallel agent sessions can be isolated with separate Git worktrees in the Copilot example, which is useful as an isolation concept. It does not prove that parallel live voice agents should share a gateway session or provider connection.

If the product later adds parallel agents, give each agent:

• an explicit budget
• a trace context
• a cancellation path
• an isolation boundary
• a clear output-selection rule
• a tool authorization scope

The remaining work is clear in shape: validate the vendor contract, implement the provider adapter behind narrow interfaces, exercise the state machine under failure, and collect real latency and usage data before widening traffic. The architecture should make those unknowns cheap to test rather than pretending they have already been settled.

---