Treat a conversation as a state machine
A browser voice session has more states than connected or disconnected. Use explicit states such as idle, requesting permission, connecting, listening, responding, interrupted, stopping, and failed. The microphone indicator should reflect actual capture, not merely an open WebSocket.
Keep the provider protocol in a transport adapter. Your page should respond to application events rather than depending on every upstream event field.
type VoiceState =
| 'idle' | 'connecting' | 'listening'
| 'responding' | 'stopping' | 'failed';
function stopSession(socket: WebSocket, stream: MediaStream) {
stream.getTracks().forEach(track => track.stop());
socket.close(1000, 'User stopped the session');
}
This illustrates local cleanup. A complete implementation must also stop queued playback, release audio resources, and reconcile any server-side session.
Get credentials from your server
Follow xAI’s documented ephemeral-token flow for browser clients. The browser requests a short-lived session credential from your application after authentication and allowance checks. Never embed the permanent API key in a JavaScript bundle, source map, query string, or browser storage.
Issue a credential only when a user starts a session, not on every page view. If setup fails, show a clear retry action instead of silently opening repeated connections.
Separate audio capture and playback
Check the supported audio encoding, sample rate, and event format in the official Speech to Speech documentation. A MediaRecorder container is not automatically equivalent to the raw audio format an API expects.
When an interruption occurs, stop local queued output as well as updating the conversation state. Otherwise an assistant can appear to ignore the user even when the server accepted the interrupt.
Test the end of the session
Press stop during recording, while connecting, and halfway through playback. Navigate away, deny microphone permission, unplug an input device, and briefly lose the network. Confirm that no hidden recording track or abandoned billable connection remains.
The text-to-speech playground does not require microphone access. A future live-conversation UI should request it only when the user explicitly starts that workflow.