From Transcript JSON to Clean Text: Parsing Tips

Raw YouTube transcript JSON is not clean prose. The segments are time-sliced by the caption renderer, not by grammar, and auto-generated captions skip punctuation almost entirely. Before you pass a transcript to an LLM, index it in a search engine, or run any NLP pipeline, you need to reshape that JSON into something more useful.

This post walks through the exact steps: understanding what the API returns, stripping the common artifacts, merging segments correctly, preserving timestamps when they matter, and getting text ready for downstream use. Code examples are in Python and JavaScript.

What the transcript JSON actually looks like

A call to the YouTube Transcriber transcript endpoint returns a structure like this:

curl "https://getyoutubetranscriber.com/api/v2/transcript?video_url=dQw4w9WgXcQ&include_timestamp=true" \
  -H "Authorization: Bearer ytt_your_key_here"

{
  "video_id": "dQw4w9WgXcQ",
  "language": "en",
  "transcript": [
    { "text": "we're we're talking about", "start": 0.0, "duration": 2.08 },
    { "text": "building scalable systems uh", "start": 2.08, "duration": 2.56 },
    { "text": "and you know the the main thing", "start": 4.64, "duration": 2.72 }
  ]
}

Three things stand out immediately. First, the segment boundaries are determined by the caption timing engine, not by sentence structure. A segment ends when a timing window closes, which means a single sentence routinely spans three or four segments. Second, filler words ("uh", "you know") are transcribed literally. Third, there is no punctuation.

If you join these segments naively with newlines and hand the result to a language model or a text classifier, you get mediocre results. The model sees fragmented pseudo-sentences with noise baked in.

Why raw caption JSON is a problem for NLP

Auto-generated captions are optimized for display, not for machine consumption. A few specific issues come up repeatedly:

Segment fragmentation. Each segment typically covers two to four seconds of audio. "The database connection pool exhausted all available connections" might arrive as five separate text fields. Treating each field as a sentence will confuse any sentence-level model.

Missing punctuation. Auto-captions either omit punctuation entirely or insert it inconsistently. A ten-minute tutorial transcript can arrive as a single run-on block. Sentence boundary detection breaks down without periods or question marks to anchor on.

Filler words and stutters. Speech recognition captures every "um", "uh", "like", and false start. These inflate token counts and introduce noise into embeddings and keyword extractions.

Repeated words. Self-corrections produce patterns like "we're we're going to" or "the the main issue". A simple deduplication step on adjacent identical words handles most of these.

Mangled technical terms. Proper nouns, library names, and acronyms are frequent casualties of speech recognition. "Kubernetes" becomes "cube" or "cube ernettes". You cannot fix this automatically without a domain vocabulary, but you can flag segments with low confidence scores if your use case supports it.

Merging segments and stripping artifacts

The first step is always to join the segment texts into one string, then clean at the document level rather than the segment level. Cleaning segment-by-segment loses context and makes filler-word removal harder.

Python

import re

def clean_transcript(segments: list[dict]) -> str:
    # Join all segment text into one string
    raw = " ".join(seg["text"] for seg in segments)

    # Remove filler words (word-boundary aware)
    fillers = r"\b(um|uh|er|ah|like|you know|I mean|I guess|kind of|sort of)\b"
    text = re.sub(fillers, "", raw, flags=re.IGNORECASE)

    # Remove repeated adjacent words: "the the" -> "the"
    text = re.sub(r"\b(\w+)( \1\b)+", r"\1", text, flags=re.IGNORECASE)

    # Collapse multiple spaces left by removals
    text = re.sub(r" {2,}", " ", text).strip()

    return text


def segments_to_sentences(segments: list[dict]) -> list[str]:
    """Best-effort sentence splitting without a punctuation restoration model."""
    raw_text = clean_transcript(segments)

    # If the transcript already has some punctuation, use it
    sentences = re.split(r"(?<=[.!?])\s+", raw_text)

    # For unpunctuated text, split on long pauses heuristically by word count
    if len(sentences) <= 1:
        words = raw_text.split()
        chunk_size = 30  # rough words-per-sentence
        sentences = [
            " ".join(words[i : i + chunk_size])
            for i in range(0, len(words), chunk_size)
        ]

    return [s.strip() for s in sentences if s.strip()]

For better sentence splitting, run the joined text through a punctuation restoration model like deepmultilingualpunctuation before splitting. The regex approach above is good enough for many use cases but will miss complex sentence boundaries.

JavaScript

function cleanTranscript(segments) {
  // Join all segments
  const raw = segments.map((s) => s.text).join(" ");

  // Remove common filler words
  const fillerPattern =
    /\b(um|uh|er|ah|like|you know|I mean|I guess|kind of|sort of)\b/gi;
  let text = raw.replace(fillerPattern, "");

  // Remove adjacent repeated words
  text = text.replace(/\b(\w+)(\s+\1\b)+/gi, "$1");

  // Collapse extra whitespace
  text = text.replace(/\s{2,}/g, " ").trim();

  return text;
}

function segmentsToChunks(segments, wordsPerChunk = 30) {
  const cleaned = cleanTranscript(segments);
  const words = cleaned.split(/\s+/);
  const chunks = [];

  for (let i = 0; i < words.length; i += wordsPerChunk) {
    chunks.push(words.slice(i, i + wordsPerChunk).join(" "));
  }

  return chunks;
}

Preserving timestamps when you need them

Some use cases require you to keep the start time so you can link back to the exact video moment. A summarizer that surfaces key quotes should tell the user where to find them. A RAG pipeline over video content should return a YouTube timestamp URL, not just a text excerpt.

The pattern here is to enrich segments rather than discard them. Clean the text field in place, but keep the start and duration.

Python with timestamp preservation

def clean_segments(segments: list[dict]) -> list[dict]:
    filler_re = re.compile(
        r"\b(um|uh|er|ah|like|you know|I mean|I guess|kind of|sort of)\b",
        re.IGNORECASE,
    )
    repeat_re = re.compile(r"\b(\w+)( \1\b)+", re.IGNORECASE)

    cleaned = []
    for seg in segments:
        text = filler_re.sub("", seg["text"])
        text = repeat_re.sub(r"\1", text)
        text = re.sub(r" {2,}", " ", text).strip()
        if text:  # drop segments that become empty
            cleaned.append({"text": text, "start": seg["start"], "duration": seg["duration"]})

    return cleaned


def timestamp_url(video_id: str, start_seconds: float) -> str:
    t = int(start_seconds)
    return f"https://www.youtube.com/watch?v={video_id}&t={t}s"

JavaScript with timestamp preservation

function cleanSegments(segments) {
  const fillerRe = /\b(um|uh|er|ah|like|you know|I mean|I guess|kind of|sort of)\b/gi;
  const repeatRe = /\b(\w+)(\s+\1\b)+/gi;

  return segments
    .map((seg) => {
      let text = seg.text.replace(fillerRe, "");
      text = text.replace(repeatRe, "$1");
      text = text.replace(/\s{2,}/g, " ").trim();
      return { ...seg, text };
    })
    .filter((seg) => seg.text.length > 0);
}

function timestampUrl(videoId, startSeconds) {
  return `https://www.youtube.com/watch?v=${videoId}&t=${Math.floor(startSeconds)}s`;
}

When you build a RAG retrieval result, attach the timestamp URL to every chunk so the user can jump straight to the source. This is especially useful for long interviews or conference talks where a single video might yield dozens of relevant passages.

For a deeper look at building document pipelines over transcript text, the post on building a RAG pipeline on YouTube transcripts covers chunking strategies, embedding choices, and retrieval patterns in detail.

When to skip timestamps entirely

If you only need clean prose and have no downstream use for timing data, pass include_timestamp=false in your request. The API still returns the transcript array, but each object contains only the text field. This simplifies your parsing code because you have nothing to preserve.

curl "https://getyoutubetranscriber.com/api/v2/transcript?video_url=VIDEO_ID&include_timestamp=false&format=json" \
  -H "Authorization: Bearer ytt_your_key_here"

Alternatively, set format=text and the API returns a plain string with segments already joined. That removes the parsing step entirely, though you lose any opportunity to inspect or selectively discard segments before joining.

Preparing text for LLM and search use

The cleaning steps above get you most of the way there, but LLM input and search indexing have different requirements. It helps to think about them separately.

For LLM prompts and context windows

Modern large language models are trained on enormous amounts of informal and noisy web text. They handle contractions, missing punctuation, and casual phrasing without losing coherence. Aggressive preprocessing for LLM input often does more harm than good because it strips cues the model uses to understand sentence structure and speaker intent.

A practical checklist for LLM-ready transcript text:

• Remove obvious filler words ("um", "uh") that add zero semantic content
• Collapse repeated adjacent words ("we we" to "we")
• Strip leading and trailing whitespace from the final string
• Keep contractions, informal phrasing, and domain vocabulary intact
• Do not lowercase the text unless you have a specific reason; casing carries information

For token efficiency, break long transcripts into overlapping chunks of roughly 500 to 800 tokens rather than sending a full hour-long transcript as one string. Overlap of 50 to 100 tokens between chunks helps the model maintain context across chunk boundaries.

For search indexing

Search indexes benefit from heavier preprocessing than LLM prompts:

• Lowercase the entire text
• Remove stopwords if your search engine does not handle them natively
• Lemmatize or stem depending on your retrieval method
• Normalize whitespace
• Consider stripping punctuation for keyword indexes, but keep it for semantic search

For BM25-style keyword search, lemmatization matters. "Configuring" and "configured" should both match a query for "configuration". Tools like spaCy's linguistic features pipeline make lemmatization straightforward in Python.

For vector search, skip most preprocessing. The embedding model expects natural language, and cleaning away too much text degrades the semantic quality of the resulting vector. Join segments, remove extreme noise (empty segments, pure punctuation), and embed.

A comparison of what to do for each use case:

| Preprocessing step | LLM context | BM25 search | Vector search | |--|--|--|--| | Remove fillers | Yes | Yes | Yes | | Lowercase | No | Yes | No | | Remove stopwords | No | Yes | No | | Lemmatize | No | Yes | No | | Strip punctuation | No | Depends | No | | Chunk text | Yes | By document | Yes | | Preserve timestamps | Optional | No | Yes (as metadata) |

Handling multilingual transcripts

The API supports transcripts in 125+ languages via the lang parameter. When you clean multilingual text, be careful with regex filler-word removal: the patterns above are English-specific. Build a separate filler list per language, or skip that step entirely and rely on stopword removal instead.

Sentence boundary detection is also language-specific. Many NLP libraries default to English tokenization rules. For non-Latin scripts or right-to-left languages, verify that your sentence splitter handles the script correctly before deploying.

If you are building multilingual content pipelines, check the guide to YouTube transcripts in 125+ languages on the blog for language selection, fallback strategies, and auto-translation patterns.

A note on what you cannot fix in post-processing

Cleaning handles structural noise but cannot correct recognition errors. If the speech recognition misheard "Redis" as "read this" or "Kafka" as "coffee", regex cannot recover the original word without domain knowledge. A few strategies help here:

• Maintain a vocabulary list of known technical terms for your domain and do a fuzzy-match pass
• If you have the video URL, check the metadata.title field (available when send_metadata=true) for hints about domain vocabulary
• Flag segments with suspiciously short duration relative to word count, which often indicates run-on recognition errors
• For production pipelines that need high accuracy, consider Whisper as a complementary transcription source for audio where auto-captions underperform

Getting started

The transcript endpoint is available with 100 free credits and no credit card required. If you are building a pipeline from scratch, start with include_timestamp=true even if you do not need timestamps immediately. It costs nothing extra and preserves optionality if your requirements change.

The full parameter reference, including language codes and response schema details, is in the transcript API documentation at getyoutubetranscriber.com/docs.

Clean text is the foundation that everything else depends on: summaries, embeddings, search indexes, and LLM prompts all perform better when the input is well-formed. Getting the parsing step right once means you do not have to debug it in every downstream system separately.