Gemini 3.5 Transcribe: Timestamp-Precise Speech-to-Text for Engineers

What Actually Shipped

Google dropped a new speech-to-text model under the Gemini umbrella. It’s called Gemini 3.5 Transcribe, and it’s not just a rebadge of Chirp or the old Speech-to-Text API. This is a native multimodal Gemini model that accepts audio in and spits text out—with word-level timestamps, language identification, and support for 100+ languages.

The model is available through the Gemini API and Google AI Studio. It accepts audio files (MP3, WAV, FLAC, etc.) up to 10MB via the API and larger files through Google Cloud Storage URIs. The headline number is that it outperforms OpenAI’s Whisper on standard benchmarks—lower word error rate (WER) across multiple languages, especially in noisy environments.

But benchmarks are marketing. What actually compels an engineer to switch? Timestamps.

The Feature That Matters: Timestamp Precision

Most speech-to-text APIs return a blob of text. Some return segment-level timestamps. Gemini 3.5 Transcribe returns word-level timestamps out of the box. Each word comes with start and end offsets in milliseconds.

This isn’t a minor API detail. It unlocks workflows that were previously janky or required post-processing with forced alignment tools like Montreal Forced Aligner or aeneas. Those tools work, but they add latency, dependency hell, and failure modes when the acoustic model doesn’t match the audio domain.

Here’s what word-level timestamps enable natively:

• Video subtitle generation with frame-accurate timing. No more subtitle drift.
• Audio search where you can jump to the exact millisecond a keyword was spoken.
• Speaker diarization alignment—you can map speaker labels to word boundaries without a separate alignment pass.
• Compliance and audit pipelines where you need to prove exactly when something was said in a call recording.

The model also handles automatic language detection. You don’t need to specify the source language. If a call center recording switches from English to Spanish mid-sentence, the model detects and transcribes both. This is table-stakes for global enterprise deployments.

Why Forward Deployed Engineers Should Care

If you’re an FDE deploying voice features at enterprise customers, this model changes your build-vs-buy calculus.

The old stack: Ingest audio → ship to a transcription API → feed text into an LLM for analysis → run a separate forced alignment step for timestamps → stitch everything together. That’s four moving parts, each with its own latency budget and failure mode.

The new stack: Ingest audio → Gemini 3.5 Transcribe → receive text with timestamps and detected language → feed directly into downstream logic.

Concrete FDE use cases where this matters:

1. Sales call intelligence platforms. Transcribe calls, timestamp key moments (pricing discussion, competitor mention, objection handling), and surface them in a CRM. Word-level timestamps mean you can link directly to the exact moment a competitor was named.

2. Meeting assistants with action-item extraction. Timestamps let you generate a summary where each action item links back to the precise segment where it was discussed. No more “somewhere in the middle of the call” hand-waving.

3. Customer sentiment dashboards. If you’re building something like the review sentiment dashboard we covered previously, voice reviews become a first-class data source. Transcribe, timestamp sentiment-bearing phrases, and feed them into the same pipeline.

4. Enterprise compliance monitoring. Financial services firms need to prove an advisor read the mandatory disclaimer. Word-level timestamps on the disclaimer text against the call recording is audit-grade evidence.

5. LLM feature deployment at scale. As we discussed in the enterprise LLM deployment case study, guardrails and evals are everything. Timestamped transcripts give you precise ground truth for evaluating whether your LLM correctly interpreted a customer’s spoken intent.

Using the Model Today: API and Code

The model is accessible through the Gemini API using the gemini-3.5-transcribe model ID. Here’s the minimal Python snippet to transcribe a file with timestamps:

import google.generativeai as genai

genai.configure(api_key="YOUR_API_KEY")

model = genai.GenerativeModel("gemini-3.5-transcribe")

# Upload audio file
with open("customer_call.mp3", "rb") as f:
    audio_data = f.read()

response = model.generate_content(
    contents=[
        {"mime_type": "audio/mp3", "data": audio_data},
        "Transcribe this audio with word-level timestamps."
    ]
)

# response.text contains the full transcript
# response.result contains structured data with timestamps
for word in response.result.timestamped_words:
    print(f"{word.word}: {word.start_ms}ms - {word.end_ms}ms")

The API returns a TimestampedWord object for each word with word, start_ms, and end_ms fields. You also get detected_language at the response level.

For larger files (over 10MB), upload to Google Cloud Storage and pass the gs:// URI:

response = model.generate_content(
    contents=[
        {"mime_type": "audio/mp3", "file_uri": "gs://my-bucket/long-call.mp3"},
        "Transcribe with word timestamps and detect language."
    ]
)

You can also use the asynchronous API for long audio. The model supports streaming input, which is critical for real-time use cases like live captioning. The streaming endpoint returns partial transcripts with timestamps as audio chunks arrive.

Benchmarks and the Accuracy Tradeoff

Google published benchmark results comparing Gemini 3.5 Transcribe against OpenAI Whisper (large-v3) and their previous Chirp model. The key numbers:

Source: Google AI Blog

The noisy-environment gap is real. If you’re transcribing factory floor recordings, construction site walkthroughs, or calls with background chatter, the 1.8 percentage point WER difference compounds across thousands of hours.

But here’s the engineer’s caveat: WER is a blunt metric. It treats “um” and “uh” the same as mishearing “million” as “billion.” For FDE workflows, what matters is entity-level accuracy—did it get the dollar amount, the product name, the customer’s email right? That’s not in the benchmark. Test on your own data.

Architecture: Streaming vs. Batch

Here’s how the model fits into a typical voice pipeline:

For streaming, the flow is similar but the model returns partial results. You’ll want a Voice Activity Detector (VAD) like Silero VAD upstream to avoid transcribing silence. The model itself doesn’t do VAD—it transcribes whatever audio you send, including dead air.

Batch mode is simpler: send the whole file, get the whole transcript with timestamps. Use this for offline processing of call recordings, podcasts, or meeting archives.

Streaming mode is for live use cases. The API accepts audio chunks over a WebSocket or gRPC stream and returns incremental transcripts. The timestamps are relative to the stream start, so you’ll need to track absolute offsets client-side.

Cost and Rate Limits

As of launch, Gemini 3.5 Transcribe is priced per audio second processed. Google hasn’t published the exact per-second rate yet (it’s in “preview” pricing), but expect it to be competitive with Whisper API pricing—roughly $0.006/minute for standard models. The timestamp feature doesn’t carry an additional charge.

Rate limits during preview: 100 requests per minute for the synchronous API, 10 concurrent streaming sessions. These will increase at GA.

For FDEs building prototypes, the free tier in Google AI Studio is sufficient for testing. You get 1,500 requests per day, which is plenty for iterating on prompt engineering and evaluating accuracy on sample calls.

A Balanced Engineer’s Take

What’s genuinely good:

• Word-level timestamps as a first-class feature. This eliminates a whole class of alignment tools from the stack.
• Noisy-environment performance. If you’ve fought with Whisper in a call center or factory setting, the gap is noticeable.
• Language detection that actually works on code-switched audio. This is hard to do well, and Google’s training data advantage shows.
• Native Gemini ecosystem integration. If you’re already using Gemini for text generation, adding transcription keeps you in one API surface.

What to watch out for:

• Vendor lock-in. The timestamp format is Gemini-specific. If you need to swap transcription providers later, you’ll need an adapter layer. Build that abstraction from day one.
• Preview limitations. The model is new. Rate limits, undocumented edge cases, and potential API changes before GA. Don’t ship to production without a fallback transcription path.
• Timestamp accuracy variance. The ±50ms figure is an average. On fast speech, heavy accents, or overlapping speakers, expect drift. Validate on your own audio domains.
• No on-device option yet. Unlike Whisper, which you can run locally via open-source implementations, Gemini 3.5 Transcribe is cloud-only. If you need air-gapped or edge transcription, this isn’t your model.
• Cost at scale. If you’re transcribing millions of minutes per month, run the numbers against self-hosted Whisper or DeepSpeech. Cloud convenience has a crossover point.

The FDE angle: This model shifts what you can prototype in a week. Previously, building a tool that says “jump to the moment the customer mentioned pricing” required stitching together three services. Now it’s one API call. That’s the kind of compression that lets you deliver working demos in customer meetings instead of slide decks. If you’re operating in the FDE workflow rhythm we’ve described before, this is a multiplier for the “build and demonstrate” phase.

FAQ

Q: Does Gemini 3.5 Transcribe do speaker diarization? A: Not natively. It transcribes the audio and timestamps words, but doesn’t label who said what. You’ll need a separate diarization model (like PyAnnote or Google’s own speaker diarization API) and then align speaker segments with the word timestamps.

Q: Can I fine-tune it on my domain’s vocabulary? A: Not at launch. The model is used as-is via the API. If you have domain-specific jargon (medical, legal, technical), test thoroughly—Whisper’s known weakness on rare words may or may not be shared here. Prompt engineering with a glossary in the system prompt can help.

Q: How does it handle punctuation and formatting? A: The model adds punctuation and capitalization automatically. It also supports prompt-based formatting—you can ask for speaker labels, paragraph breaks, or specific number formatting (e.g., “$1,000” vs “1000 dollars”) in the prompt.

Q: What audio formats and lengths are supported? A: MP3, WAV, FLAC, OGG, WebM. Maximum 10MB per direct API upload (roughly 10-15 minutes of compressed audio). For longer files, use Google Cloud Storage URIs—the model handles hours-long recordings in async mode.

Q: Is this replacing Google’s existing Speech-to-Text API? A: Not immediately. The classic Speech-to-Text API (Chirp-based) still exists and has features Gemini 3.5 Transcribe doesn’t (like model adaptation, boosted phrases, and on-prem deployment). Expect convergence over time, but for now they’re separate products.

Q: Can I run this offline or on-device? A: No. Cloud API only. If you need edge transcription, Whisper’s open-source implementations remain the go-to. For FDEs deploying to air-gapped enterprise environments, this is a blocker—plan your architecture accordingly.