ext-apps/examples/say-server/server.py
Olivier Chafik e6983b36d0
feat(examples): add say-server - streaming Pocket TTS with karaoke highlighting (#304)
* Add say-server: streaming TTS MCP App with karaoke UI

- Streaming TTS MCP App with real-time word highlighting
- React widget with pause/resume controls
- Debounced model context updates
- Self-contained server.py for standalone uv run

Co-authored-by: Claude <noreply@anthropic.com>

* Update server.py

* fix(say-server): fix React/Zod version conflicts in widget

- Pin Zod to v3.25.1 via esm.sh deps param (fixes .custom() error)
- Add external=react,react-dom to prevent duplicate React instances
- Update React from 19.1.0 to 19.2.0
- Import React for Babel's classic JSX transform
- Consolidate imports: only use @modelcontextprotocol/ext-apps/react
- Use Babel from unpkg.com instead of esm.sh
- Simplify resource decorator with meta= parameter
- Update CSP to allow both esm.sh and unpkg.com

* fix(say-server): show play button when paused, restart when finished

* feat(say-server): add fullscreen button, remove inline scroll

- Add fullscreen toggle button (appears on hover when available)
- Remove max-height/scroll in inline mode for cleaner display
- Enable scrolling only in fullscreen mode
- Track displayMode and fullscreenAvailable state
- Handle onHostContextChanged to detect fullscreen support

* fix(say-server): fix fullscreen button visibility and prevent multiple audio streams

- Check initial host context for fullscreen availability (not just updates)
- Add audioOperationInProgressRef guard to prevent concurrent audio operations
- Prevents race conditions from rapid clicks/double-clicks

* fix(qr-server): use meta parameter instead of manual handler patching

Simplified CSP metadata handling to match say-server's approach:
- Use meta parameter on @mcp.resource decorator directly
- Removed custom _read_resource_with_meta handler and handler replacement

* fix(say-server): simplify controls to prevent multiple audio streams

- Remove onClick from text display (was conflicting with double-click)
- Always show overlay button (with pause icon when playing)
- Lower opacity (0.3) when playing, full on hover
- Button shows: ▶️ (play), ⏸️ (pause), 🔄 (restart)
- Double-click text to restart still works
- Eliminates click/dblclick race condition

* fix(say-server): only show pause button on hover

- Play/restart buttons visible normally
- Pause button hidden until hover (doesn't block karaoke text)

* fix(say-server): fix partial input handling and session management

- Add promise lock to initTTSQueue to prevent concurrent initialization
- Reset queueIdRef and lastTextRef in ontoolresult for next tool call
- Close existing AudioContext when starting new session
- Reset all playback state (chunks, timings, position) on new session
- Fixes: text deltas not sent when new tool call has shorter text

* fix(say-server): detect new sessions by text comparison, not queueIdRef reset

- Don't reset queueIdRef in ontoolresult - audio should keep playing
- Detect new sessions in ontoolinputpartial/ontoolinput by checking if
  new text starts with lastTextRef (continuation) or not (new session)
- Only reset and create new queue when genuinely new session detected
- Fixes: pause/play was creating new streams because queueIdRef was reset

* feat(say-server): add theming, floating button, and keyboard shortcuts

Theming:
- Use CSS variables (--font-sans, --color-text-primary/secondary) with fallbacks
- Apply host fonts via applyHostFonts() on connect and context changes
- Apply host style variables via applyHostStyleVariables()
- Apply document theme via applyDocumentTheme()

Floating button:
- Button follows cursor position (below current line being spoken)
- Uses invisible cursor marker element for accurate positioning
- Shows ⏸️ when playing, ▶️ when paused, 🔄 when finished
- Low opacity when playing, full on hover

Keyboard shortcuts:
- Space: toggle play/pause
- Enter: toggle fullscreen (when available)

UI changes:
- Large overlay only shown for initial play (idle state)
- Floating button used during playback for less intrusive controls

* docs(say-server): add TODO items for future improvements

* fix(say-server): simplify to fixed play button at top right

- Remove floating cursor-following button (was too complex)
- Fixed play button at top-right corner
- Low opacity when playing, full on hover
- Keep theming and keyboard shortcuts (Space/Enter)
- Add say-server to E2E tests

* test(e2e): add say-server to tests but skip in CI

Say-server requires Pocket TTS model (~500MB) and GPU support,
which isn't available in the Docker CI environment. Skip it for now.

* test(e2e): add say-server and qr-server to screenshot generation

- Remove say-server from SKIP_SERVERS (widget works without TTS model)
- Add say-server masks for play buttons
- Add both say-server and qr-server to grid screenshot generator

* fix(say-server): restore working play/pause overlay, keep theming and keyboard shortcuts

Reverted to working commit 413beca and carefully reapplied:
- CSS variables for theming (--font-sans, --color-text-primary/secondary)
- Dark mode support via @media (prefers-color-scheme)
- Host theming via applyHostStyleVariables/applyHostFonts/applyDocumentTheme
- Keyboard shortcuts: Space = play/pause, Enter = fullscreen

Preserved working behavior:
- Play overlay visible when not playing, hover to show when playing
- Pause on click works correctly
- Karaoke highlighting works

* fix(say-server): minimal CSS-only theming, no JS changes from working version

Only CSS changes from working commit 413beca:
- Added CSS variables for theming foundation
- No JavaScript changes (reverted theming calls and keyboard shortcuts)

This isolates whether the issue is CSS or JS related.

* feat(say-server): add autoPlay param and Reset button

- Add autoPlay parameter (default: true) with Pydantic Field description
  - Description visible in MCP tool schema: 'browsers may block autoplay'
- Add Reset button next to Fullscreen button (bottom right)
- Both buttons share controlBtn base styles, shown on hover
- Widget reads autoPlay from tool input params

* fix(say-server): update FastMCP API for new SDK version

Python SDK refactored transport params from constructor to run():
- Remove port, stateless_http from FastMCP() constructor
- Pass stateless_http=True to streamable_http_app() instead

* debug(say-server): add console logs to trace audio stream issue

Logging added to:
- togglePlayPause (all branches)
- initTTSQueue (entry and AudioContext creation)
- restartPlayback (entry)
- ontoolinputpartial/ontoolinput (entry and new session detection)

Open browser DevTools console and reproduce the issue - logs will show what's being called.

* feat(say-server): add play/pause buttons in top-left and bottom-right corners

- Top left: playPauseTopBtn for quick access
- Bottom right: playPauseBtn alongside reset and fullscreen
- Both show on hover with same styling as other control buttons
- Icons: ▶️ (play), ⏸️ (pause), 🔄 (restart when finished)

* fix(say-server): simplify UI and fix pause/resume audio duplication

- Remove central play overlay, use single toolbar in top-right
- Fix audio duplication on pause/resume (only defer to pendingChunks for initial autoplay)
- Remove double-click to restart (use toolbar restart button)
- Sync displayMode when host changes it externally
- Click on text to play/pause

* feat(say-server): add voice selection support

- Add list_voices tool to show available voices
- Add voice parameter to say tool (predefined names, HuggingFace URLs, or local paths)
- Predefined voices: alba, marius, javert, jean, cosette, eponine, azelma, fantine
- Update README with voice documentation

* docs(say-server): improve tool description for better LLM triggering

- Add English-only constraint prominently in first line and note
- List common trigger patterns (say, speak, read aloud, etc.)
- Update text parameter description to mention English
- Keep description concise for quick LLM scanning

* feat(say-server): add speak lock for multi-widget coordination

When multiple TTS widgets exist, only one plays at a time:
- Widget announces to localStorage when starting playback
- Playing widgets poll localStorage every 200ms
- If another widget starts, previous one pauses automatically
- Lock is cleared on pause/finish/teardown

Uses _meta.widgetUUID in tool result for widget identification.

* update package.json of say

* feat(say-server): add screenshots and update README gallery

- Add say-server to demo gallery with 300x300 grid-cell.png
- Fix qr-server grid-cell.png dimensions (was 672x668, now 300x300)
- Add golden snapshot for say-server E2E tests
- Add full screenshot and grid-cell thumbnail for say-server

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* style(say-server): apply prettier formatting

- Add blank lines in README for better readability
- Add trailing newline to package.json

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(say-server): add Escape key to exit fullscreen and update docs

- Add onKeyDown handler to main element for Escape key
- Update README with full list of MCP App features:
  - Model context updates
  - Native theming
  - Fullscreen mode
  - Multi-widget speak lock
- Update server.py docstring with feature documentation

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(say-server): add queue timeout to prevent infinite polling

The TTS queue could hang forever if:
- The LLM stops generating mid-stream (no end signal)
- User cancels before tool completes
- Network interruption prevents end_tts_queue call

Changes:
- Add last_activity timestamp to track queue activity
- Add 30-second timeout (QUEUE_TIMEOUT_SECONDS)
- Poll with 5-second timeout, checking for stale queues
- Mark stale queues as "error" so widget stops polling
- Return error message in poll_tts_audio response
- Add error logging in widget polling loop

This prevents tool calls from running forever when queues are abandoned.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* debug(say-server): add logging to track queue lifecycle

Add console logging to debug why end_tts_queue might not be called:

Widget-side:
- Log queueId in ontoolinputpartial, ontoolinput, ontoolresult
- Log when new session is detected (queue reset)
- Log when initTTSQueue fails
- Log when end_tts_queue is called/not called

Server-side:
- Log when end_tts_queue is called
- Log warnings for unknown queues
- Log info for already-ended queues

This will help identify if the issue is:
- User interruption (ontoolresult never called)
- Session reset (queueId becomes null)
- Host bug (callback dropped)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* perf(say-server): add initial delay before polling

Add 150ms delay before first poll to give TTS model time to generate
the first audio chunk. This reduces unnecessary polling and server load.

Current backoff after that:
- 30ms if chunks received (keep streaming fast)
- 80ms if no chunks (wait longer for generation)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(say-server): use adaptive polling instead of fixed initial delay

Previous approach: Always wait 150ms before first poll
Problem: Even when chunks are ready, we wait unnecessarily

New approach: Adaptive polling
- Start polling immediately (no initial delay)
- 20ms backoff when receiving chunks (fast streaming)
- Exponential backoff when no chunks: 50ms → 100ms → 150ms max
- Reset backoff when chunks start flowing

This reduces latency when TTS is fast while being polite when it's slow.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(say-server): add speaker icon to server info

Uses SVG data URI for a speaker with sound waves icon.

* docs(say-server): add comprehensive third-party licensing information

- Expand License section in README with component attribution table
- Add voice collection license matrix with commercial use indicators
- Add warning about non-commercial expresso/ears voice collections
- Clarify NON-COMMERCIAL restriction in list_voices() output
- Use consistent CC-BY formatting (hyphenated)

* Update last activity to prevent timeout during active polling

* feat(say-server): add info button, SVG icons, and UI fixes

- Replace emoji play/pause buttons with neutral white SVG icons
- Add (i) info button in bottom-right with attribution popup
- Use app.openLink() API for external links in popup
- Add dynamic padding to fit popup when shown
- Make create_tts_queue async to fix AnyIO event loop error
- Add better error logging for TTS queue initialization
- Update README with openLink feature documentation
- Remove unnecessary HuggingFace login prerequisites

* docs(say-server): document multi-widget speak lock mechanism

* prettier:fix

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-20 00:41:50 +00:00

1457 lines
57 KiB
Python
Executable file

#!/usr/bin/env uv run --default-index https://pypi.org/simple
# /// script
# requires-python = ">=3.10"
# dependencies = [
# "mcp @ git+https://github.com/modelcontextprotocol/python-sdk@main",
# "uvicorn>=0.34.0",
# "starlette>=0.46.0",
# "pocket-tts>=1.0.1",
# ]
# ///
"""
Say Demo - MCP App for streaming text-to-speech.
This MCP server provides a "say" tool that speaks text using TTS.
The widget receives streaming partial input and starts speaking immediately.
Architecture:
- The `say` tool itself is a no-op - it just triggers the widget
- The widget uses `ontoolinputpartial` to receive text as it streams
- Widget calls private tools to create TTS queue, add text, and poll audio
- Audio plays in the widget using Web Audio API
- Model context updates show playback progress to the LLM
- Native theming adapts to dark/light mode automatically
- Fullscreen mode with Escape key to exit
- Multi-widget speak lock coordinates playback across instances
Usage:
# Start the MCP server
python server.py
# Or with stdio transport (for Claude Desktop)
python server.py --stdio
"""
from __future__ import annotations
import asyncio
import base64
import logging
import os
import sys
import time
import uuid
from dataclasses import dataclass, field
from pathlib import Path
from typing import Annotated, Literal
from pydantic import Field
import torch
import uvicorn
from mcp.server.fastmcp import FastMCP
from mcp import types
from mcp.types import Icon
from starlette.middleware.cors import CORSMiddleware
from pocket_tts.models.tts_model import TTSModel, prepare_text_prompt
from pocket_tts.default_parameters import DEFAULT_AUDIO_PROMPT
logger = logging.getLogger(__name__)
WIDGET_URI = "ui://say-demo/widget.html"
HOST = os.environ.get("HOST", "0.0.0.0")
PORT = int(os.environ.get("PORT", "3109"))
# Speaker icon as SVG data URI
SPEAKER_ICON = Icon(
src="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 24 24' fill='none' stroke='currentColor' stroke-width='2'%3E%3Cpolygon points='11 5 6 9 2 9 2 15 6 15 11 19 11 5'/%3E%3Cpath d='M15.54 8.46a5 5 0 0 1 0 7.07'/%3E%3Cpath d='M19.07 4.93a10 10 0 0 1 0 14.14'/%3E%3C/svg%3E",
mimeType="image/svg+xml",
)
mcp = FastMCP("Say Demo", icons=[SPEAKER_ICON])
# Global TTS model (loaded on startup)
tts_model: TTSModel | None = None
# ------------------------------------------------------
# TTS Queue State Management
# ------------------------------------------------------
@dataclass
class AudioChunkData:
"""Audio chunk with timing metadata."""
index: int
audio_base64: str
char_start: int
char_end: int
duration_ms: float
@dataclass
class TTSQueueState:
"""State for a TTS generation queue."""
id: str
voice: str
sample_rate: int
status: Literal["active", "complete", "error"] = "active"
error_message: str | None = None
# Text queue
text_queue: asyncio.Queue = field(default_factory=asyncio.Queue)
end_signaled: bool = False
# Audio output
audio_chunks: list[AudioChunkData] = field(default_factory=list)
chunks_delivered: int = 0
# Tracking
created_at: float = field(default_factory=time.time)
last_activity: float = field(default_factory=time.time) # Last text or end signal
lock: asyncio.Lock = field(default_factory=asyncio.Lock)
task: asyncio.Task | None = None
# Active TTS queues
tts_queues: dict[str, TTSQueueState] = {}
# Queue timeout: if no activity for this long, mark as error
QUEUE_TIMEOUT_SECONDS = 30
# ------------------------------------------------------
# Public Tool: say
# ------------------------------------------------------
DEFAULT_TEXT = """Hello! I'm a text-to-speech demonstration. This speech is being generated in real-time as you watch. The words you see highlighted are synchronized with the audio playback, creating a karaoke-style reading experience. You can click to pause or resume, and use the reset button to restart from the beginning. Pretty neat, right?"""
# Predefined voices from pocket-tts (mapped to HuggingFace files)
# See: https://huggingface.co/kyutai/tts-voices
PREDEFINED_VOICES = {
"alba": "hf://kyutai/tts-voices/alba-mackenna/casual.wav",
"marius": "hf://kyutai/tts-voices/alba-mackenna/merchant.wav",
"javert": "hf://kyutai/tts-voices/alba-mackenna/announcer.wav",
"jean": "hf://kyutai/tts-voices/alba-mackenna/a-moment-by.wav",
"fantine": "hf://kyutai/tts-voices/vctk/p225_023_mic1.wav",
"cosette": "hf://kyutai/tts-voices/vctk/p226_023_mic1.wav",
"eponine": "hf://kyutai/tts-voices/vctk/p227_023_mic1.wav",
"azelma": "hf://kyutai/tts-voices/vctk/p228_023_mic1.wav",
}
DEFAULT_VOICE = "cosette"
@mcp.tool()
def list_voices() -> list[types.TextContent]:
"""List available TTS voices.
Returns the predefined voice names that can be used with the say tool.
You can also use HuggingFace URLs (hf://kyutai/tts-voices/...) or local file paths.
"""
import json
voice_info = {
"predefined_voices": list(PREDEFINED_VOICES.keys()),
"default_voice": DEFAULT_VOICE,
"custom_voice_formats": [
"hf://kyutai/tts-voices/<collection>/<file>.wav",
"/path/to/local/voice.wav",
],
"collections": [
"alba-mackenna (CC-BY 4.0) - voice-acted characters",
"vctk (CC-BY 4.0) - VCTK dataset speakers",
"cml-tts/fr (CC-BY 4.0) - French voices",
"voice-donations (CC0) - public domain community voices",
"expresso (CC-BY-NC 4.0) - expressive (NON-COMMERCIAL ONLY)",
"ears (CC-BY-NC 4.0) - emotional (NON-COMMERCIAL ONLY)",
],
}
return [types.TextContent(type="text", text=json.dumps(voice_info, indent=2))]
@mcp.tool(meta={
"ui":{"resourceUri": WIDGET_URI},
"ui/resourceUri": WIDGET_URI, # legacy support
})
def say(
text: Annotated[str, Field(description="The English text to speak aloud")] = DEFAULT_TEXT,
voice: Annotated[str, Field(
description="Voice to use. Can be a predefined name (alba, marius, cosette, etc.), "
"a HuggingFace URL (hf://kyutai/tts-voices/...), or a local file path."
)] = DEFAULT_VOICE,
autoPlay: Annotated[bool, Field(
description="Whether to start playing automatically. Note: browsers may block autoplay until user interaction."
)] = True,
) -> list[types.TextContent]:
"""Speak English text aloud using text-to-speech.
Use when the user wants text read or spoken aloud:
- "say ...", "speak ...", "read ... out loud"
- "...; say it", "...; read it to me", "...; speak it"
- "narrate ...", "read this aloud"
Audio streams in real-time as text is provided.
Use list_voices() for voice options.
Note: English only. Non-English text may produce poor or garbled results.
"""
# Generate a unique ID for this widget instance (used for speak lock coordination)
widget_uuid = uuid.uuid4().hex[:12]
# This is a no-op - the widget handles everything via ontoolinputpartial
# The tool exists to:
# 1. Trigger the widget to load
# 2. Provide the resourceUri metadata
# 3. Show the final text in the tool result
# 4. Provide widget UUID for multi-player coordination
return [types.TextContent(
type="text",
text=f"Displayed a TTS widget with voice '{voice}'. Click to play/pause, use toolbar to restart or fullscreen.",
_meta={"widgetUUID": widget_uuid},
)]
# ------------------------------------------------------
# Private Tools: TTS Queue Management
# ------------------------------------------------------
@mcp.tool(meta={"ui":{"visibility":["app"]}})
async def create_tts_queue(voice: str = "cosette") -> list[types.TextContent]:
"""Create a TTS generation queue. Returns queue_id and sample_rate.
Args:
voice: Voice to use (cosette, alba, brenda, etc.)
"""
if tts_model is None:
return [types.TextContent(type="text", text='{"error": "TTS model not loaded"}')]
queue_id = uuid.uuid4().hex[:12]
sample_rate = tts_model.config.mimi.sample_rate
state = TTSQueueState(
id=queue_id,
voice=voice,
sample_rate=sample_rate,
)
tts_queues[queue_id] = state
# Start background TTS processing task
state.task = asyncio.create_task(_run_tts_queue(state))
logger.info(f"Created TTS queue {queue_id}")
import json
return [types.TextContent(
type="text",
text=json.dumps({"queue_id": queue_id, "sample_rate": sample_rate})
)]
@mcp.tool(meta={"ui":{"visibility":["app"]}})
def add_tts_text(queue_id: str, text: str) -> list[types.TextContent]:
"""Add text to a TTS queue.
Args:
queue_id: The queue ID from create_tts_queue
text: Text to add (incremental, not cumulative)
"""
state = tts_queues.get(queue_id)
if not state:
return [types.TextContent(type="text", text='{"error": "Queue not found"}')]
if state.end_signaled:
return [types.TextContent(type="text", text='{"error": "Queue already ended"}')]
# Queue the text (non-blocking)
try:
state.text_queue.put_nowait(text)
state.last_activity = time.time() # Update activity timestamp
except asyncio.QueueFull:
return [types.TextContent(type="text", text='{"error": "Queue full"}')]
return [types.TextContent(type="text", text='{"queued": true}')]
@mcp.tool(meta={"ui":{"visibility":["app"]}})
def end_tts_queue(queue_id: str) -> list[types.TextContent]:
"""Signal that no more text will be sent to a queue.
Args:
queue_id: The queue ID from create_tts_queue
"""
state = tts_queues.get(queue_id)
if not state:
logger.warning(f"end_tts_queue called for unknown queue: {queue_id}")
return [types.TextContent(type="text", text='{"error": "Queue not found"}')]
if state.end_signaled:
logger.info(f"end_tts_queue called for already-ended queue: {queue_id}")
return [types.TextContent(type="text", text='{"already_ended": true}')]
state.end_signaled = True
state.last_activity = time.time() # Update activity timestamp
try:
state.text_queue.put_nowait(None) # EOF marker
except asyncio.QueueFull:
pass
logger.info(f"end_tts_queue called for queue: {queue_id}")
return [types.TextContent(type="text", text='{"ended": true}')]
@mcp.tool(meta={"ui":{"visibility":["app"]}})
def cancel_tts_queue(queue_id: str) -> list[types.TextContent]:
"""Cancel and cleanup a TTS queue. Use before creating a new queue to avoid overlapping playback.
Args:
queue_id: The queue ID from create_tts_queue
"""
state = tts_queues.pop(queue_id, None)
if not state:
return [types.TextContent(type="text", text='{"error": "Queue not found"}')]
# Cancel the background task
if state.task and not state.task.done():
state.task.cancel()
logger.info(f"Cancelled TTS queue {queue_id}")
# Signal end to unblock any waiting consumers
state.end_signaled = True
try:
state.text_queue.put_nowait(None)
except asyncio.QueueFull:
pass
state.status = "complete"
return [types.TextContent(type="text", text='{"cancelled": true}')]
@mcp.tool(meta={"ui":{"visibility":["app"]}})
def poll_tts_audio(queue_id: str) -> list[types.TextContent]:
"""Poll for available audio chunks from a TTS queue.
Returns base64-encoded audio chunks with timing metadata.
Call repeatedly until done=true.
Args:
queue_id: The queue ID from create_tts_queue
"""
import json
import time
state = tts_queues.get(queue_id)
if not state:
return [types.TextContent(type="text", text='{"error": "Queue not found"}')]
# Update last activity to prevent timeout during active polling
state.last_activity = time.time()
# Get new chunks (use sync approach since we can't await in tool)
# The lock is async, so we need to be careful here
# For simplicity, just grab what's available without locking
new_chunks = state.audio_chunks[state.chunks_delivered:]
state.chunks_delivered = len(state.audio_chunks)
# Consider queues with errors as "done" so widget stops polling
done = (state.status == "complete" or state.status == "error") and state.chunks_delivered >= len(state.audio_chunks)
response = {
"chunks": [
{
"index": c.index,
"audio_base64": c.audio_base64,
"char_start": c.char_start,
"char_end": c.char_end,
"duration_ms": c.duration_ms,
}
for c in new_chunks
],
"done": done,
"status": state.status,
}
# Include error message if present
if state.error_message:
response["error"] = state.error_message
# Clean up completed or errored queues
if done:
# Schedule cleanup after a delay
async def cleanup():
await asyncio.sleep(60)
tts_queues.pop(queue_id, None)
try:
asyncio.get_event_loop().create_task(cleanup())
except RuntimeError:
pass
return [types.TextContent(type="text", text=json.dumps(response))]
# ------------------------------------------------------
# Background TTS Processing
# ------------------------------------------------------
class StreamingTextChunker:
"""Buffers streaming text and emits chunks when ready for TTS processing.
Chunks are emitted when:
- Token count reaches max_tokens threshold (at a sentence boundary if possible)
- flush() is called (end of stream)
This matches the chunking behavior of split_into_best_sentences() but works
incrementally as text arrives.
"""
def __init__(self, tokenizer, max_tokens: int = 50, min_tokens: int = 15):
"""
Args:
tokenizer: SentencePiece tokenizer from flow_lm.conditioner.tokenizer
max_tokens: Maximum tokens per chunk (default 50, matches existing)
min_tokens: Minimum tokens before considering emission
"""
self.tokenizer = tokenizer
self.max_tokens = max_tokens
self.min_tokens = min_tokens
self.buffer = ""
# Cache end-of-sentence token IDs for boundary detection
_, *eos_tokens = tokenizer(".!...?").tokens[0].tolist()
self.eos_tokens = set(eos_tokens)
def add_text(self, text: str) -> list[str]:
"""Add text to buffer, return any complete chunks ready for processing.
Args:
text: Incremental text to add (e.g., from LLM token)
Returns:
List of text chunks ready for TTS (may be empty if still buffering)
"""
self.buffer += text
return self._extract_ready_chunks()
def flush(self) -> list[str]:
"""Flush remaining buffer as final chunk(s).
Call this when the text stream ends to process any remaining text.
Returns:
List of final text chunks (may be empty if buffer was empty)
"""
if not self.buffer.strip():
return []
# Force emit whatever remains
chunks = self._extract_ready_chunks(force_emit=True)
if self.buffer.strip():
chunks.append(self.buffer.strip())
self.buffer = ""
return chunks
def _extract_ready_chunks(self, force_emit: bool = False) -> list[str]:
"""Extract chunks that are ready for processing."""
chunks = []
while True:
chunk = self._try_extract_chunk(force_emit and not chunks)
if chunk is None:
break
chunks.append(chunk)
return chunks
def _try_extract_chunk(self, force_emit: bool = False) -> str | None:
"""Try to extract one chunk from buffer."""
text = self.buffer.strip()
if not text:
return None
tokens = self.tokenizer(text).tokens[0].tolist()
num_tokens = len(tokens)
# Not enough tokens yet
if num_tokens < self.min_tokens and not force_emit:
return None
# Under max and not forcing - check for complete sentence worth emitting
if num_tokens < self.max_tokens and not force_emit:
# Only emit early if we have a complete sentence at a good length
if num_tokens >= self.min_tokens and self._ends_with_sentence_boundary(tokens):
# Found a complete sentence - emit it
chunk = text
self.buffer = ""
return chunk
return None
# Over max_tokens or force_emit - find best split point
split_idx = self._find_best_split(tokens, force_emit)
if split_idx == 0:
if force_emit:
chunk = text
self.buffer = ""
return chunk
return None
# Decode tokens up to split point
chunk_text = self.tokenizer.sp.decode(tokens[:split_idx])
remaining_text = self.tokenizer.sp.decode(tokens[split_idx:])
self.buffer = remaining_text
return chunk_text.strip()
def _find_best_split(self, tokens: list[int], force_emit: bool = False) -> int:
"""Find the best token index to split at (sentence boundary near max_tokens)."""
# Find all sentence boundaries (position AFTER the punctuation)
boundaries = []
prev_was_eos = False
for i, token in enumerate(tokens):
if token in self.eos_tokens:
prev_was_eos = True
elif prev_was_eos:
boundaries.append(i)
prev_was_eos = False
# Also consider end of tokens if it ends with punctuation
if tokens and tokens[-1] in self.eos_tokens:
boundaries.append(len(tokens))
if not boundaries:
# No sentence boundaries - split at max_tokens if we're over
if len(tokens) >= self.max_tokens:
return self.max_tokens
return len(tokens) if force_emit else 0
# Find boundary closest to max_tokens without going too far over
best_boundary = 0
for boundary in boundaries:
if boundary <= self.max_tokens:
best_boundary = boundary
elif best_boundary == 0:
# First boundary is past max - use it anyway
best_boundary = boundary
break
else:
# We have a good boundary before max, stop
break
return best_boundary
def _ends_with_sentence_boundary(self, tokens: list[int]) -> bool:
"""Check if token sequence ends with sentence-ending punctuation."""
if not tokens:
return False
return tokens[-1] in self.eos_tokens
@property
def buffered_text(self) -> str:
"""Current buffered text (for debugging/monitoring)."""
return self.buffer
@property
def buffered_token_count(self) -> int:
"""Approximate token count in buffer."""
if not self.buffer.strip():
return 0
return len(self.tokenizer(self.buffer).tokens[0].tolist())
async def _run_tts_queue(state: TTSQueueState):
"""Background task: consume text queue, produce audio chunks."""
if tts_model is None:
state.status = "error"
state.error_message = "TTS model not loaded"
return
model_state = tts_model._cached_get_state_for_audio_prompt(state.voice, truncate=True)
chunker = StreamingTextChunker(tts_model.flow_lm.conditioner.tokenizer)
chunk_index = 0
char_offset = 0
try:
while True:
# Wait for text with timeout to detect stale queues
try:
text_item = await asyncio.wait_for(
state.text_queue.get(),
timeout=5.0 # Check every 5 seconds
)
except asyncio.TimeoutError:
# Check if queue is stale (no activity for too long)
if time.time() - state.last_activity > QUEUE_TIMEOUT_SECONDS:
logger.warning(f"TTS queue {state.id} timeout after {QUEUE_TIMEOUT_SECONDS}s of inactivity")
state.status = "error"
state.error_message = f"Queue timeout: no activity for {QUEUE_TIMEOUT_SECONDS}s"
break
# Continue waiting - queue might still be active
continue
if text_item is None:
# EOF - flush remaining text
remaining = chunker.flush()
for chunk_text in remaining:
await _process_tts_chunk(state, chunk_text, chunk_index, char_offset, model_state)
char_offset += len(chunk_text)
chunk_index += 1
state.status = "complete"
logger.info(f"TTS queue {state.id} complete: {chunk_index} chunks")
break
# Feed text to chunker
ready_chunks = chunker.add_text(text_item)
for chunk_text in ready_chunks:
await _process_tts_chunk(state, chunk_text, chunk_index, char_offset, model_state)
char_offset += len(chunk_text)
chunk_index += 1
except Exception as e:
logger.error(f"TTS queue {state.id} error: {e}")
state.status = "error"
state.error_message = str(e)
async def _process_tts_chunk(
state: TTSQueueState,
text: str,
chunk_index: int,
char_offset: int,
model_state: dict,
):
"""Process a text chunk and add audio to state."""
if tts_model is None:
return
loop = asyncio.get_event_loop()
audio_bytes_list: list[bytes] = []
total_samples = 0
def generate_sync():
nonlocal total_samples
_, frames_after_eos = prepare_text_prompt(text)
frames_after_eos += 2
for audio_chunk in tts_model._generate_audio_stream_short_text(
model_state=model_state,
text_to_generate=text,
frames_after_eos=frames_after_eos,
copy_state=True,
):
audio_int16 = (audio_chunk * 32767).to(torch.int16)
audio_bytes_list.append(audio_int16.cpu().numpy().tobytes())
total_samples += len(audio_chunk)
await loop.run_in_executor(None, generate_sync)
combined_audio = b"".join(audio_bytes_list)
duration_ms = (total_samples / state.sample_rate) * 1000
chunk_data = AudioChunkData(
index=chunk_index,
audio_base64=base64.b64encode(combined_audio).decode(),
char_start=char_offset,
char_end=char_offset + len(text),
duration_ms=duration_ms,
)
async with state.lock:
state.audio_chunks.append(chunk_data)
logger.debug(f"TTS queue {state.id}: chunk {chunk_index} ready ({duration_ms:.0f}ms)")
# ------------------------------------------------------
# Widget Resource
# ------------------------------------------------------
# Embedded widget HTML for standalone execution via `uv run <url>`
# Uses Babel standalone for in-browser JSX transpilation
# This is a copy of widget.html - keep them in sync!
EMBEDDED_WIDGET_HTML = """<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Say Widget</title>
<script src="https://unpkg.com/@babel/standalone@7.26.10/babel.min.js"></script>
<script type="importmap">
{
"imports": {
"react": "https://esm.sh/react@19.2.0",
"react-dom/client": "https://esm.sh/react-dom@19.2.0/client",
"@modelcontextprotocol/ext-apps/react": "https://esm.sh/@modelcontextprotocol/ext-apps@0.4.1/react?deps=zod@3.25.1&external=react,react-dom"
}
}
</script>
<style>
:root {
/* Fallback values if host doesn't provide */
--font-sans: system-ui, -apple-system, sans-serif;
--color-text-primary: #333;
--color-text-secondary: #999;
}
* { box-sizing: border-box; margin: 0; padding: 0; }
body { font-family: var(--font-sans); }
.container { padding: 16px; min-height: 100px; position: relative; outline: none; }
.textWrapper { position: relative; }
.textDisplay {
font-size: 16px; line-height: 1.6; padding: 8px; border-radius: 6px;
}
/* Fullscreen mode: enable scrolling */
.container.fullscreen .textDisplay {
max-height: calc(100vh - 100px);
overflow-y: auto;
}
.spoken { color: var(--color-text-primary); }
.pending { color: var(--color-text-secondary); }
/* Toolbar - top right, visible on hover */
.toolbar {
position: absolute;
top: 8px; right: 8px;
display: flex;
gap: 4px;
opacity: 0;
transition: opacity 0.2s;
z-index: 10;
}
.container:hover .toolbar { opacity: 0.8; }
.toolbar:hover { opacity: 1; }
.controlBtn {
width: 32px; height: 32px; border: none; border-radius: 6px;
background: rgba(0, 0, 0, 0.5); color: white; cursor: pointer;
display: flex; align-items: center; justify-content: center;
transition: background 0.2s;
font-size: 14px;
}
.controlBtn:hover { background: rgba(0, 0, 0, 0.8); }
.controlBtn svg { width: 16px; height: 16px; }
.fullscreenBtn { display: none; }
.fullscreenBtn.available { display: flex; }
.fullscreenBtn .collapseIcon { display: none; }
.container.fullscreen .fullscreenBtn .expandIcon { display: none; }
.container.fullscreen .fullscreenBtn .collapseIcon { display: block; }
/* Info button - bottom right */
.infoBtn {
position: absolute;
bottom: 8px; right: 8px;
width: 24px; height: 24px;
border: none; border-radius: 50%;
background: rgba(128, 128, 128, 0.4);
color: white; cursor: pointer;
display: flex; align-items: center; justify-content: center;
font-size: 12px; font-weight: bold; font-style: italic; font-family: serif;
opacity: 0.5;
transition: opacity 0.2s, background 0.2s;
z-index: 10;
}
.container:hover .infoBtn { opacity: 0.8; }
.infoBtn:hover { opacity: 1; background: rgba(128, 128, 128, 0.7); }
/* Info popup */
.infoPopup {
position: absolute;
bottom: 40px; right: 8px;
background: rgba(0, 0, 0, 0.9);
color: white;
padding: 12px 16px;
border-radius: 8px;
font-size: 12px;
line-height: 1.5;
max-width: 280px;
z-index: 20;
box-shadow: 0 4px 12px rgba(0,0,0,0.3);
}
.infoPopup h4 { margin: 0 0 8px 0; font-size: 13px; }
.infoPopup a {
color: #6cb6ff;
text-decoration: none;
cursor: pointer;
}
.infoPopup a:hover { text-decoration: underline; }
.infoPopup ul { margin: 0; padding-left: 16px; }
.infoPopup li { margin: 4px 0; }
@media (prefers-color-scheme: dark) {
:root {
--color-text-primary: #eee;
--color-text-secondary: #666;
}
}
</style>
</head>
<body>
<div id="root"></div>
<script type="text/babel" data-type="module">
import React, { useState, useCallback, useEffect, useRef, StrictMode } from 'react';
import { createRoot } from 'react-dom/client';
import { useApp } from '@modelcontextprotocol/ext-apps/react';
function SayWidget() {
const [hostContext, setHostContext] = useState(undefined);
const [displayText, setDisplayText] = useState("");
const [charPosition, setCharPosition] = useState(0);
const [status, setStatus] = useState("idle"); // idle | playing | paused | finished
const [hasPendingChunks, setHasPendingChunks] = useState(false);
const [displayMode, setDisplayMode] = useState("inline");
const [fullscreenAvailable, setFullscreenAvailable] = useState(false);
const [autoPlay, setAutoPlay] = useState(true); // Default to autoPlay, can be overridden by tool input
const [showInfo, setShowInfo] = useState(false);
const voiceRef = useRef("cosette"); // Current voice, updated from tool input
const widgetUuidRef = useRef(null); // Widget UUID for speak lock coordination
const speakLockIntervalRef = useRef(null); // Polling interval for speak lock
const queueIdRef = useRef(null);
const audioContextRef = useRef(null);
const sampleRateRef = useRef(24000);
const nextPlayTimeRef = useRef(0);
const playbackStartTimeRef = useRef(0);
const chunkTimingsRef = useRef([]);
const pendingChunksRef = useRef([]);
const allAudioReceivedRef = useRef(false);
const isPollingRef = useRef(false);
const lastTextRef = useRef("");
const fullTextRef = useRef("");
const progressIntervalRef = useRef(null);
const appRef = useRef(null);
const lastModelContextUpdateRef = useRef(0);
const audioOperationInProgressRef = useRef(false);
const initQueuePromiseRef = useRef(null);
const pendingModelContextUpdateRef = useRef(null);
// Speak lock: coordinates multiple TTS widgets so only one plays at a time
const SPEAK_LOCK_KEY = "mcp-tts-playing";
const announcePlayback = useCallback(() => {
if (!widgetUuidRef.current) return;
localStorage.setItem(SPEAK_LOCK_KEY, JSON.stringify({
uuid: widgetUuidRef.current,
timestamp: Date.now()
}));
}, []);
const clearSpeakLock = useCallback(() => {
if (!widgetUuidRef.current) return;
try {
const current = JSON.parse(localStorage.getItem(SPEAK_LOCK_KEY) || "null");
if (current?.uuid === widgetUuidRef.current) {
localStorage.removeItem(SPEAK_LOCK_KEY);
}
} catch {}
}, []);
const startSpeakLockPolling = useCallback((onStolenCallback) => {
if (speakLockIntervalRef.current) clearInterval(speakLockIntervalRef.current);
speakLockIntervalRef.current = setInterval(() => {
try {
const current = JSON.parse(localStorage.getItem(SPEAK_LOCK_KEY) || "null");
if (current && current.uuid !== widgetUuidRef.current) {
// Someone else started playing - yield
console.log('[TTS] Another widget started playing, pausing');
onStolenCallback();
}
} catch {}
}, 200);
}, []);
const stopSpeakLockPolling = useCallback(() => {
if (speakLockIntervalRef.current) {
clearInterval(speakLockIntervalRef.current);
speakLockIntervalRef.current = null;
}
}, []);
const roundToWordEnd = useCallback((pos) => {
const text = lastTextRef.current;
if (pos >= text.length) return text.length;
if (pos <= 0) return 0;
if (text[pos] === " " || text[pos] === "\\n") return pos;
let end = pos;
while (end < text.length && text[end] !== " " && text[end] !== "\\n") end++;
return end;
}, []);
const getCharacterPosition = useCallback((currentTime) => {
const timings = chunkTimingsRef.current;
let rawPos = 0;
if (timings.length === 0) {
rawPos = Math.floor((currentTime - playbackStartTimeRef.current) * 12);
} else {
for (const chunk of timings) {
if (currentTime >= chunk.audioStartTime && currentTime < chunk.audioEndTime) {
const duration = chunk.audioEndTime - chunk.audioStartTime;
if (duration <= 0) { rawPos = chunk.charStart; }
else {
const progress = (currentTime - chunk.audioStartTime) / duration;
rawPos = Math.floor(chunk.charStart + progress * (chunk.charEnd - chunk.charStart));
}
break;
}
}
if (rawPos === 0 && timings.length > 0) {
if (currentTime < timings[0].audioStartTime) rawPos = 0;
else {
const last = timings[timings.length - 1];
if (currentTime >= last.audioEndTime) rawPos = last.charEnd;
}
}
}
return roundToWordEnd(rawPos);
}, [roundToWordEnd]);
const finishPlayback = useCallback(() => {
setStatus("finished");
if (progressIntervalRef.current) {
clearInterval(progressIntervalRef.current);
progressIntervalRef.current = null;
}
setCharPosition(lastTextRef.current.length);
}, []);
const startProgressTracking = useCallback(() => {
if (progressIntervalRef.current) return;
progressIntervalRef.current = setInterval(() => {
const ctx = audioContextRef.current;
if (!ctx) return;
setCharPosition(getCharacterPosition(ctx.currentTime));
if (allAudioReceivedRef.current && ctx.currentTime >= nextPlayTimeRef.current - 0.05) {
finishPlayback();
}
}, 50);
}, [getCharacterPosition, finishPlayback]);
const scheduleAudioChunkInternal = useCallback(async (chunk) => {
const ctx = audioContextRef.current;
if (!ctx) return;
const binaryString = atob(chunk.audio_base64);
const bytes = new Uint8Array(binaryString.length);
for (let i = 0; i < binaryString.length; i++) bytes[i] = binaryString.charCodeAt(i);
const int16Array = new Int16Array(bytes.buffer);
const float32Array = new Float32Array(int16Array.length);
for (let i = 0; i < int16Array.length; i++) float32Array[i] = int16Array[i] / 32768;
const audioBuffer = ctx.createBuffer(1, float32Array.length, sampleRateRef.current);
audioBuffer.getChannelData(0).set(float32Array);
const source = ctx.createBufferSource();
source.buffer = audioBuffer;
source.connect(ctx.destination);
const startTime = Math.max(ctx.currentTime, nextPlayTimeRef.current);
const duration = audioBuffer.duration;
if (chunkTimingsRef.current.length === 0) {
playbackStartTimeRef.current = startTime;
setStatus("playing");
startProgressTracking();
}
source.start(startTime);
nextPlayTimeRef.current = startTime + duration;
chunkTimingsRef.current.push({
charStart: chunk.char_start, charEnd: chunk.char_end,
audioStartTime: startTime, audioEndTime: nextPlayTimeRef.current,
});
const thisBufferEndTime = nextPlayTimeRef.current;
source.onended = () => {
if (!audioContextRef.current) return;
const ct = audioContextRef.current.currentTime;
if (allAudioReceivedRef.current && thisBufferEndTime >= nextPlayTimeRef.current - 0.01 && ct >= nextPlayTimeRef.current - 0.05) {
finishPlayback();
}
};
}, [startProgressTracking, finishPlayback]);
const scheduleAudioChunk = useCallback(async (chunk) => {
const ctx = audioContextRef.current;
if (!ctx) return;
// Only defer to pendingChunks if suspended AND playback hasn't started yet
// (i.e., autoplay is blocked). If we're paused by user (chunkTimings exists),
// schedule normally - the audio will queue up and play when resumed.
if (ctx.state === "suspended" && chunkTimingsRef.current.length === 0) {
pendingChunksRef.current.push(chunk);
setHasPendingChunks(true);
return;
}
await scheduleAudioChunkInternal(chunk);
}, [scheduleAudioChunkInternal]);
const startPolling = useCallback(async () => {
const app = appRef.current;
if (isPollingRef.current || !app) return;
isPollingRef.current = true;
let emptyPollCount = 0;
while (queueIdRef.current) {
try {
const result = await app.callServerTool({ name: "poll_tts_audio", arguments: { queue_id: queueIdRef.current } });
const data = JSON.parse(result.content[0].text);
if (data.error) {
console.log('[TTS] Queue error:', data.error);
break;
}
for (const chunk of data.chunks) await scheduleAudioChunk(chunk);
if (data.done) { allAudioReceivedRef.current = true; break; }
// Adaptive backoff: faster when streaming, slower when waiting
if (data.chunks.length > 0) {
emptyPollCount = 0; // Reset - we're getting chunks
await new Promise(r => setTimeout(r, 20)); // Fast poll during streaming
} else {
emptyPollCount++;
// Exponential backoff for empty polls: 50ms, 100ms, 150ms max
const delay = Math.min(50 + (emptyPollCount * 50), 150);
await new Promise(r => setTimeout(r, delay));
}
} catch (err) {
console.log('[TTS] Polling error:', err);
break;
}
}
isPollingRef.current = false;
}, [scheduleAudioChunk]);
const cancelCurrentQueue = useCallback(async () => {
const app = appRef.current;
if (queueIdRef.current && app) {
try { await app.callServerTool({ name: "cancel_tts_queue", arguments: { queue_id: queueIdRef.current } }); }
catch (err) {}
}
}, []);
const initTTSQueue = useCallback(async () => {
console.log('[TTS] initTTSQueue called, queueIdRef:', queueIdRef.current);
// Already initialized
if (queueIdRef.current) { console.log('[TTS] already initialized'); return true; }
// Wait for in-progress initialization
if (initQueuePromiseRef.current) {
await initQueuePromiseRef.current;
return !!queueIdRef.current;
}
const app = appRef.current;
if (!app) return false;
// Start initialization with promise lock
initQueuePromiseRef.current = (async () => {
try {
// Close any existing audio context from previous session
if (audioContextRef.current) {
try { await audioContextRef.current.close(); } catch {}
audioContextRef.current = null;
}
if (progressIntervalRef.current) {
clearInterval(progressIntervalRef.current);
progressIntervalRef.current = null;
}
// Reset state for new session
chunkTimingsRef.current = [];
pendingChunksRef.current = [];
allAudioReceivedRef.current = false;
setCharPosition(0);
setStatus("idle");
// Create new queue
console.log('[TTS] creating new queue');
const result = await app.callServerTool({ name: "create_tts_queue", arguments: { voice: voiceRef.current } });
console.log('[TTS] create_tts_queue result:', result);
const text = result.content?.[0]?.text;
if (!text) { console.error('[TTS] No text in result'); return false; }
let data;
try { data = JSON.parse(text); }
catch (e) { console.error('[TTS] Failed to parse result:', text); return false; }
if (data.error) { console.log('[TTS] queue creation error:', data.error); return false; }
queueIdRef.current = data.queue_id;
sampleRateRef.current = data.sample_rate || 24000;
console.log('[TTS] creating new AudioContext');
audioContextRef.current = new AudioContext({ sampleRate: sampleRateRef.current });
nextPlayTimeRef.current = 0;
startPolling();
return true;
} catch (err) { console.error('[TTS] initTTSQueue error:', err); return false; }
finally { initQueuePromiseRef.current = null; }
})();
return initQueuePromiseRef.current;
}, [startPolling]);
const sendTextToTTS = useCallback(async (text) => {
const app = appRef.current;
if (!queueIdRef.current || !app) return;
if (text.length > lastTextRef.current.length) {
const diff = text.slice(lastTextRef.current.length);
lastTextRef.current = text;
try { await app.callServerTool({ name: "add_tts_text", arguments: { queue_id: queueIdRef.current, text: diff } }); }
catch (err) {}
}
}, []);
const ensureAudioContextResumed = useCallback(async () => {
const ctx = audioContextRef.current;
if (ctx && ctx.state === "suspended") {
await ctx.resume();
if (pendingChunksRef.current.length > 0) {
// This is only reached during initial autoplay unblocking (before any audio played).
// Reset nextPlayTimeRef to current time so chunks start immediately.
nextPlayTimeRef.current = ctx.currentTime;
const chunks = pendingChunksRef.current;
pendingChunksRef.current = [];
setHasPendingChunks(false);
for (const chunk of chunks) await scheduleAudioChunkInternal(chunk);
}
}
}, [scheduleAudioChunkInternal]);
const restartPlayback = useCallback(async () => {
console.log('[TTS] restartPlayback called');
// Prevent concurrent audio operations
if (audioOperationInProgressRef.current) { console.log('[TTS] restartPlayback blocked'); return; }
audioOperationInProgressRef.current = true;
try {
if (progressIntervalRef.current) { clearInterval(progressIntervalRef.current); progressIntervalRef.current = null; }
await cancelCurrentQueue();
if (audioContextRef.current) { await audioContextRef.current.close(); audioContextRef.current = null; }
const textToReplay = fullTextRef.current || lastTextRef.current;
if (!textToReplay) return;
queueIdRef.current = null; lastTextRef.current = ""; isPollingRef.current = false;
nextPlayTimeRef.current = 0; playbackStartTimeRef.current = 0;
setStatus("idle"); chunkTimingsRef.current = []; allAudioReceivedRef.current = false;
setCharPosition(0); pendingChunksRef.current = []; setHasPendingChunks(false);
setDisplayText(textToReplay);
const app = appRef.current;
if (!app) return;
const result = await app.callServerTool({ name: "create_tts_queue", arguments: { voice: voiceRef.current } });
const data = JSON.parse(result.content[0].text);
if (data.error) return;
queueIdRef.current = data.queue_id;
sampleRateRef.current = data.sample_rate || 24000;
audioContextRef.current = new AudioContext({ sampleRate: sampleRateRef.current });
nextPlayTimeRef.current = 0;
await app.callServerTool({ name: "add_tts_text", arguments: { queue_id: queueIdRef.current, text: textToReplay } });
lastTextRef.current = textToReplay;
await app.callServerTool({ name: "end_tts_queue", arguments: { queue_id: queueIdRef.current } });
startPolling();
} catch (err) {
} finally {
audioOperationInProgressRef.current = false;
}
}, [cancelCurrentQueue, startPolling]);
const togglePlayPause = useCallback(async () => {
console.log('[TTS] togglePlayPause called, status:', status, 'ctx:', audioContextRef.current?.state);
// Prevent concurrent audio operations
if (audioOperationInProgressRef.current) { console.log('[TTS] blocked by audioOpInProgress'); return; }
let ctx = audioContextRef.current;
try {
if (status === "finished") { console.log('[TTS] finished, calling restartPlayback'); await restartPlayback(); return; }
// If no context yet, wait for init to complete (up to 3s)
if (!ctx) {
console.log('[TTS] no ctx, waiting for init');
for (let i = 0; i < 30 && !audioContextRef.current; i++) {
await new Promise(r => setTimeout(r, 100));
}
ctx = audioContextRef.current;
if (!ctx) { console.log('[TTS] still no ctx, giving up'); return; }
}
if (ctx.state === "suspended" || pendingChunksRef.current.length > 0) {
console.log('[TTS] resuming via ensureAudioContextResumed');
await ensureAudioContextResumed(); setStatus("playing"); return;
}
if (status === "paused") { console.log('[TTS] resuming paused'); await ctx.resume(); setStatus("playing"); }
else if (status === "playing") { console.log('[TTS] pausing'); await ctx.suspend(); setStatus("paused"); }
} catch (err) { console.error('[TTS] togglePlayPause error:', err); }
}, [status, restartPlayback, ensureAudioContextResumed]);
const toggleFullscreen = useCallback(async () => {
const app = appRef.current;
if (!app) return;
const newMode = displayMode === "fullscreen" ? "inline" : "fullscreen";
try {
const result = await app.requestDisplayMode({ mode: newMode });
setDisplayMode(result.mode);
} catch (err) {}
}, [displayMode]);
const { app, error } = useApp({
appInfo: { name: "Say Widget", version: "1.0.0" },
capabilities: {},
onHostContextChanged: (ctx) => {
if (ctx.availableDisplayModes?.includes("fullscreen")) {
setFullscreenAvailable(true);
}
if (ctx.displayMode) {
setDisplayMode(ctx.displayMode);
}
},
onAppCreated: (app) => {
appRef.current = app;
app.ontoolinputpartial = async (params) => {
console.log('[TTS] ontoolinputpartial called, queueId:', queueIdRef.current);
const newText = params.arguments?.text;
if (!newText) return;
// Detect new session: text doesn't continue from where we left off
const isNewSession = lastTextRef.current.length > 0 && !newText.startsWith(lastTextRef.current);
if (isNewSession) {
console.log('[TTS] new session detected in partial - resetting queue');
// Reset for new session
queueIdRef.current = null;
lastTextRef.current = "";
}
setDisplayText(newText);
if (!queueIdRef.current && !(await initTTSQueue())) {
console.log('[TTS] initTTSQueue failed in partial');
return;
}
await sendTextToTTS(newText);
};
app.ontoolinput = async (params) => {
console.log('[TTS] ontoolinput called, queueId:', queueIdRef.current);
const text = params.arguments?.text;
if (!text) return;
// Read voice setting (defaults to cosette)
const voice = params.arguments?.voice || "cosette";
voiceRef.current = voice;
// Read autoPlay setting (defaults to true, but browser may block autoplay)
const shouldAutoPlay = params.arguments?.autoPlay !== false;
setAutoPlay(shouldAutoPlay);
// Detect new session: text doesn't continue from where we left off
const isNewSession = lastTextRef.current.length > 0 && !text.startsWith(lastTextRef.current);
if (isNewSession) {
console.log('[TTS] new session detected in input - resetting queue');
queueIdRef.current = null;
lastTextRef.current = "";
}
setDisplayText(text);
if (!queueIdRef.current && !(await initTTSQueue())) {
console.log('[TTS] initTTSQueue failed in input');
return;
}
await sendTextToTTS(text);
};
app.ontoolresult = async (params) => {
console.log('[TTS] ontoolresult called, queueId:', queueIdRef.current);
fullTextRef.current = lastTextRef.current;
// Read widget UUID from tool result _meta for speak lock coordination
const resultUuid = params.content?.[0]?._meta?.widgetUUID;
if (resultUuid) {
widgetUuidRef.current = resultUuid;
console.log('[TTS] Widget UUID:', resultUuid);
}
if (queueIdRef.current) {
console.log('[TTS] Calling end_tts_queue for:', queueIdRef.current);
try { await app.callServerTool({ name: "end_tts_queue", arguments: { queue_id: queueIdRef.current } }); }
catch (err) {
console.log('[TTS] end_tts_queue error:', err);
}
} else {
console.log('[TTS] No queueId to end in ontoolresult');
}
// DON'T reset here - let audio continue playing
// New session detection happens in ontoolinputpartial via text comparison
};
app.onteardown = async () => {
if (progressIntervalRef.current) clearInterval(progressIntervalRef.current);
stopSpeakLockPolling();
clearSpeakLock();
await cancelCurrentQueue();
if (audioContextRef.current) { await audioContextRef.current.close(); audioContextRef.current = null; }
return {};
};
app.onhostcontextchanged = (params) => {
setHostContext(prev => ({ ...prev, ...params }));
// Sync displayMode when host changes it (e.g., user exits fullscreen via host UI)
if (params.displayMode) {
setDisplayMode(params.displayMode);
}
};
},
});
useEffect(() => {
if (!app) return;
const ctx = app.getHostContext();
setHostContext(ctx);
if (ctx?.availableDisplayModes?.includes("fullscreen")) {
setFullscreenAvailable(true);
}
if (ctx?.displayMode) {
setDisplayMode(ctx.displayMode);
}
}, [app]);
// Speak lock: coordinate with other TTS widgets
useEffect(() => {
if (status === "playing") {
// Announce that we're playing
announcePlayback();
// Poll to detect if another widget starts
startSpeakLockPolling(() => {
// Another widget took over - pause
const ctx = audioContextRef.current;
if (ctx && ctx.state === "running") {
ctx.suspend();
setStatus("paused");
}
});
} else {
// Not playing - stop polling and clear lock if we own it
stopSpeakLockPolling();
if (status === "paused" || status === "finished") {
clearSpeakLock();
}
}
return () => stopSpeakLockPolling();
}, [status, announcePlayback, startSpeakLockPolling, stopSpeakLockPolling, clearSpeakLock]);
useEffect(() => {
if (!app || !displayText || status === "idle") return;
const caps = app.getHostCapabilities();
if (!caps?.updateModelContext) return;
const now = Date.now();
const timeSince = now - lastModelContextUpdateRef.current;
const DEBOUNCE_MS = 2000;
const doUpdate = () => {
lastModelContextUpdateRef.current = Date.now();
pendingModelContextUpdateRef.current = null;
const snippetStart = Math.max(0, charPosition - 30);
const snippetEnd = Math.min(displayText.length, charPosition + 10);
const snippet = `...` + displayText.slice(snippetStart, charPosition) + `█` + displayText.slice(charPosition, snippetEnd) + `...`;
let statusText;
if (status === "finished") statusText = `Finished playing ` + displayText.length + ` chars.`;
else if (status === "paused") statusText = `PAUSED at "` + snippet + `" (` + charPosition + `/` + displayText.length + `)`;
else statusText = `Playing: "` + snippet + `" (` + charPosition + `/` + displayText.length + `)`;
app.updateModelContext({ content: [{ type: "text", text: statusText }] }).catch(() => {});
};
if (pendingModelContextUpdateRef.current) { clearTimeout(pendingModelContextUpdateRef.current); pendingModelContextUpdateRef.current = null; }
if (timeSince >= DEBOUNCE_MS) doUpdate();
else pendingModelContextUpdateRef.current = setTimeout(doUpdate, DEBOUNCE_MS - timeSince);
return () => { if (pendingModelContextUpdateRef.current) clearTimeout(pendingModelContextUpdateRef.current); };
}, [app, status, charPosition, displayText]);
if (error) return <div><strong>ERROR:</strong> {error.message}</div>;
if (!app) return <div>Connecting...</div>;
const spokenText = displayText.slice(0, charPosition);
const pendingText = displayText.slice(charPosition);
return (
<main
className={`container` + (displayMode === "fullscreen" ? ` fullscreen` : ``)}
style={{
paddingTop: hostContext?.safeAreaInsets?.top,
paddingRight: hostContext?.safeAreaInsets?.right,
paddingBottom: showInfo ? 140 : hostContext?.safeAreaInsets?.bottom,
paddingLeft: hostContext?.safeAreaInsets?.left,
}}
tabIndex={0}
onKeyDown={(e) => {
if (e.key === "Escape" && displayMode === "fullscreen") {
toggleFullscreen();
}
}}
>
<div className="textWrapper">
<div className="textDisplay" onClick={togglePlayPause} style={{cursor: "pointer"}}>
<span className="spoken">{spokenText}</span>
<span className="pending">{pendingText}</span>
</div>
</div>
{/* Toolbar - top right */}
<div className="toolbar">
<button className="controlBtn" onClick={togglePlayPause} title="Play/Pause">
{status === "playing" ? (
<svg viewBox="0 0 24 24" fill="currentColor"><rect x="6" y="4" width="4" height="16"/><rect x="14" y="4" width="4" height="16"/></svg>
) : status === "finished" ? (
<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2"><path d="M3 12a9 9 0 1 0 9-9 9.75 9.75 0 0 0-6.74 2.74L3 8"/><path d="M3 3v5h5"/></svg>
) : (
<svg viewBox="0 0 24 24" fill="currentColor"><polygon points="5,3 19,12 5,21"/></svg>
)}
</button>
<button className="controlBtn" onClick={restartPlayback} title="Restart">
<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<path d="M3 12a9 9 0 1 0 9-9 9.75 9.75 0 0 0-6.74 2.74L3 8"/>
<path d="M3 3v5h5"/>
</svg>
</button>
<button className={`controlBtn fullscreenBtn` + (fullscreenAvailable ? ` available` : ``)} onClick={toggleFullscreen} title="Toggle fullscreen">
<svg className="expandIcon" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<path d="M8 3H5a2 2 0 0 0-2 2v3m18 0V5a2 2 0 0 0-2-2h-3m0 18h3a2 2 0 0 0 2-2v-3M3 16v3a2 2 0 0 0 2 2h3"/>
</svg>
<svg className="collapseIcon" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2">
<path d="M8 3v3a2 2 0 0 1-2 2H3m18 0h-3a2 2 0 0 1-2-2V3m0 18v-3a2 2 0 0 1 2-2h3M3 16h3a2 2 0 0 1 2 2v3"/>
</svg>
</button>
</div>
{/* Info button - bottom right */}
<button className="infoBtn" onClick={() => setShowInfo(!showInfo)} title="Attribution info">i</button>
{showInfo && (
<div className="infoPopup">
<h4>Powered by Pocket TTS</h4>
<ul>
<li><a onClick={() => { app.openLink({ url: "https://github.com/kyutai-labs/pocket-tts" }); }}>Pocket TTS Repository</a> (Apache 2.0)</li>
<li><a onClick={() => { app.openLink({ url: "https://huggingface.co/kyutai/pocket-tts" }); }}>Pocket TTS Model</a> (CC BY 4.0)</li>
<li><a onClick={() => { app.openLink({ url: "https://huggingface.co/kyutai/tts-voices" }); }}>Voice Samples</a> (CC BY 4.0)</li>
</ul>
</div>
)}
</main>
);
}
createRoot(document.getElementById('root')).render(<StrictMode><SayWidget /></StrictMode>);
</script>
</body>
</html>"""
def get_widget_html() -> str:
"""Get the widget HTML, preferring built version from dist/."""
# Prefer built version from dist/ (local development with npm run build)
dist_path = Path(__file__).parent / "dist" / "mcp-app.html"
if dist_path.exists():
return dist_path.read_text()
# Fallback to embedded widget (for `uv run <url>` or unbundled usage)
return EMBEDDED_WIDGET_HTML
# IMPORTANT: all the external domains used by app must be listed
# in the meta.ui.csp.resourceDomains - otherwise they will be blocked by CSP policy
@mcp.resource(
WIDGET_URI,
mime_type="text/html;profile=mcp-app",
meta={"ui": {"csp": {"resourceDomains": ["https://esm.sh", "https://unpkg.com"]}}},
)
def widget() -> str:
"""Widget HTML resource with CSP metadata for external dependencies."""
return get_widget_html()
# ------------------------------------------------------
# Startup
# ------------------------------------------------------
def load_tts_model():
"""Load the TTS model on startup."""
global tts_model
logger.info("Loading TTS model...")
tts_model = TTSModel.load_model()
logger.info("TTS model loaded")
def create_app():
"""Create the ASGI app (for uvicorn reload mode)."""
load_tts_model()
app = mcp.streamable_http_app(stateless_http=True)
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_methods=["*"],
allow_headers=["*"],
)
return app
if __name__ == "__main__":
logging.basicConfig(level=logging.INFO)
if "--stdio" in sys.argv:
# Claude Desktop mode
load_tts_model()
mcp.run(transport="stdio")
elif "--reload" in sys.argv:
# Reload mode - pass app as string so uvicorn can reimport
print(f"Say Server listening on http://{HOST}:{PORT}/mcp (reload mode)")
uvicorn.run("server:create_app", host=HOST, port=PORT, reload=True, factory=True)
else:
# HTTP mode
app = create_app()
print(f"Say Server listening on http://{HOST}:{PORT}/mcp")
uvicorn.run(app, host=HOST, port=PORT)