What is the AgentStartedSpeaking Message

The AgentStartedSpeakingmessage will signal the client that the agent has begun streaming audio.

Receiving AgentStartedSpeaking

The server will send an AgentStartedSpeaking message when it begins streaming an agent audio response to the client for playback.

{
  "type": "AgentStartedSpeaking",
  "total_latency": 0.0, // Seconds from receiving the user's utterance to producing the agent's reply
  "tts_latency": 0.0, // The portion of total latency attributable to text-to-speech
  "ttt_latency": 0.0 // The portion of total latency attributable to text-to-text (usually an LLM)
}

Conclusion

The AgentStartedSpeaking message notifies the client when the server begins streaming an agent's audio response for playback. This message includes key latency metrics, such as total latency, text-to-speech latency, and text-to-text latency, providing valuable insights into the response generation process. These details help developers monitor and optimize system performance, ensuring a smoother and more responsive user experience.