Semantic Video Search with Transcript Embeddings

Keyword search breaks the moment a user asks a question the speaker never said word for word. Embedding-based semantic search solves this: instead of matching tokens, it matches meaning, so a query like "how do you handle database migrations" finds the segment where the speaker says "when you need to update your schema in production" even though no keyword overlaps.

This guide walks through the full pipeline: fetch YouTube transcripts as structured JSON, chunk the segments, generate embeddings, store them in a vector database, and return timestamped results for any natural language query. The approach works for a single video or a corpus of thousands.

Why Keyword Search Falls Short for Video Content

Full-text search engines like Elasticsearch or Postgres tsvector match tokens. For short, well-edited blog posts that is often enough. Video transcripts are different: speakers ramble, use pronouns instead of nouns, rephrase the same idea three times, and rarely use the exact term a user will type into a search box.

Embedding models convert text into dense vectors in a high-dimensional space where semantically similar passages cluster together. A user query gets embedded into the same space, and the nearest transcript segments come back as results regardless of word overlap. The retrieval quality jump is significant enough that RAG (retrieval-augmented generation) now dominates enterprise AI implementations, jumping from 31% of deployments in 2023 to 51% in 2024, according to Menlo Ventures data cited by MarketsandMarkets in their vector database market report.

For video specifically, the added value is the timestamp. You do not just find the right video; you land the user at the right second.

Step 1: Pull Transcripts as JSON

Before you can embed anything, you need clean, structured text with timing data. The YouTube Transcriber API returns exactly that from a single GET request:

curl "https://getyoutubetranscriber.com/api/v2/transcript?video_url=dQw4w9WgXcQ&include_timestamp=true" \
  -H "Authorization: Bearer YOUR_API_KEY"

The response is a JSON object with a transcript array. Each element is a segment:

{
  "video_id": "dQw4w9WgXcQ",
  "language": "en",
  "transcript": [
    {
      "text": "when you need to update your schema in production",
      "start": 142300,
      "duration": 3800
    },
    {
      "text": "you have a few options depending on your database engine",
      "start": 146100,
      "duration": 3200
    }
  ],
  "metadata": {
    "title": "Database Migrations at Scale",
    "author_name": "Some Channel",
    "author_url": "https://www.youtube.com/channel/...",
    "thumbnail_url": "https://i.ytimg.com/vi/dQw4w9WgXcQ/hqdefault.jpg"
  }
}

Note that start and duration are in milliseconds. Keep them: they are what makes the search results linkable to the right video moment.

If you are building pipelines that transcribe YouTube videos for AI at scale, you should also request the lang parameter when you want a specific language track rather than the default auto-detected one. The API handles proxy rotation, retries, and anti-bot challenges, so you get a clean result without writing retry logic yourself.

Step 2: Chunk by Segment

You have two chunking options: use the raw segments as chunks (each is typically 3 to 8 seconds of speech), or merge adjacent segments into larger windows.

Raw segments are short, often 10 to 30 words. That granularity is precise but too narrow for embedding: a single sentence rarely conveys enough context to be semantically useful on its own.

A practical approach merges segments into fixed-token windows with overlap:

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

def chunk_transcript(segments, max_tokens=300, overlap_tokens=50):
    chunks = []
    current_tokens = []
    current_texts = []
    current_start = None
    token_count = 0

    for seg in segments:
        seg_tokens = enc.encode(seg["text"])
        if current_start is None:
            current_start = seg["start"]

        if token_count + len(seg_tokens) > max_tokens and current_texts:
            chunks.append({
                "text": " ".join(current_texts),
                "start": current_start,
                "token_count": token_count
            })
            # slide the window back by overlap_tokens worth of segments
            while token_count > overlap_tokens and current_texts:
                removed = current_texts.pop(0)
                removed_tokens = enc.encode(removed)
                token_count -= len(removed_tokens)
            if current_texts:
                # update start to the start of the oldest remaining segment
                # (approximate, since we dropped the mapping)
                current_start = seg["start"]
            else:
                current_start = seg["start"]

        current_texts.append(seg["text"])
        token_count += len(seg_tokens)

    if current_texts:
        chunks.append({
            "text": " ".join(current_texts),
            "start": current_start,
            "token_count": token_count
        })

    return chunks

This preserves the start of the earliest segment in each chunk, which you will store as metadata in the vector database.

Step 3: Generate Embeddings and Store in a Vector DB

With chunks in hand, call an embedding model. OpenAI's text-embedding-3-small is a reliable choice for English; for multilingual transcript corpora, it also supports multiple languages and fits within reasonable cost budgets.

import openai
import pinecone

openai_client = openai.OpenAI(api_key="YOUR_OPENAI_KEY")
pc = pinecone.Pinecone(api_key="YOUR_PINECONE_KEY")
index = pc.Index("youtube-transcripts")

def embed_and_upsert(video_id, chunks):
    texts = [c["text"] for c in chunks]
    response = openai_client.embeddings.create(
        model="text-embedding-3-small",
        input=texts
    )
    vectors = []
    for i, (chunk, embedding_obj) in enumerate(zip(chunks, response.data)):
        vectors.append({
            "id": f"{video_id}_{i}",
            "values": embedding_obj.embedding,
            "metadata": {
                "video_id": video_id,
                "text": chunk["text"],
                "start_ms": chunk["start"]
            }
        })
    index.upsert(vectors=vectors)

The start_ms metadata field is the key to making results actionable. When you retrieve a match, you read start_ms, divide by 1000 to get seconds, and append ?t={seconds} to the YouTube URL.

For local development, Chroma works without any cloud account:

import chromadb

client = chromadb.Client()
collection = client.create_collection("transcripts")

collection.add(
    ids=[f"{video_id}_{i}" for i in range(len(chunks))],
    documents=[c["text"] for c in chunks],
    metadatas=[{"video_id": video_id, "start_ms": c["start"]} for c in chunks]
)

Chroma handles embedding internally if you pass a function, or you can pass precomputed vectors from the OpenAI call above.

Step 4: Query with Natural Language and Return Timestamped Hits

A query follows the same embedding step, then a nearest-neighbor lookup:

def search(query, top_k=5):
    response = openai_client.embeddings.create(
        model="text-embedding-3-small",
        input=[query]
    )
    query_vector = response.data[0].embedding

    results = index.query(
        vector=query_vector,
        top_k=top_k,
        include_metadata=True
    )

    hits = []
    for match in results.matches:
        meta = match.metadata
        start_seconds = int(meta["start_ms"]) // 1000
        hits.append({
            "score": match.score,
            "text": meta["text"],
            "video_url": f"https://www.youtube.com/watch?v={meta['video_id']}&t={start_seconds}s",
            "start_seconds": start_seconds
        })
    return hits

A result looks like this:

[
  {
    "score": 0.891,
    "text": "when you need to update your schema in production you have a few options depending on your database engine",
    "video_url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ&t=142s",
    "start_seconds": 142
  }
]

The user clicks the link and lands at second 142, where the speaker addresses exactly the question they typed.

For a complete implementation of a conversational interface on top of this search layer, feeding retrieved chunks into an LLM prompt to generate grounded answers is the natural next step after the search layer is working.

Gotcha: Chunk Size vs Recall Trade-offs

The single most common performance problem in this pipeline is chunk size. Here is the trade-off in plain terms:

Smaller chunks (64 to 128 tokens)

• Higher retrieval precision: the matched text is tightly relevant.
• Lower recall: a question that spans two ideas across a paragraph boundary may not match any single chunk well.
• Works best when queries are narrow and specific.

Larger chunks (512 to 1024 tokens)

• Better recall: more context per chunk means more surface area for a query to match against.
• Lower precision: you retrieve the right chunk, but it contains a lot of off-topic content around the relevant sentence.
• Works best when users ask broad, open-ended questions.

A practical default for YouTube transcripts is 256 to 512 tokens with 10 to 20 percent overlap. Research on metadata-enriched retrieval shows that adding structured metadata alongside embeddings pushes precision to 82.5% compared to 73.3% for content-only approaches, according to an IEEE study cited by Atlan's chunking strategy guide. For transcripts, that metadata is the timestamp, the video title, and the language code.

One transcript-specific gotcha: speakers often finish a sentence in the next caption segment. The auto-generated captions YouTube produces break at fixed time intervals, not at sentence boundaries. If you split strictly on segment borders, you will frequently cut sentences in half. The overlap window mitigates this, but you can also pre-process by concatenating all segment texts and then re-splitting on sentence boundaries using spaCy or NLTK before applying the token window.

A second gotcha: very long videos (conference talks, full courses) generate transcripts with thousands of segments. At 300 tokens per chunk, a 3-hour video can produce 150 to 200 chunks. That is manageable for a single video but adds up fast across a channel's entire back-catalog. If you are indexing at that scale, consider filtering to only the most-viewed videos first, or running a full-text pre-filter before embedding so you only embed videos that pass a relevance threshold.

For teams transcribing YouTube videos for AI fine-tuning datasets rather than search, the chunking rules differ: you typically want longer, self-contained passages rather than overlapping windows, because training examples need consistent context windows without duplication.

Putting It Together

A minimal end-to-end script combining all four steps:

import requests
import openai
import pinecone
import tiktoken

API_KEY = "YOUR_YOUTUBE_TRANSCRIBER_KEY"
OPENAI_KEY = "YOUR_OPENAI_KEY"
PINECONE_KEY = "YOUR_PINECONE_KEY"

enc = tiktoken.get_encoding("cl100k_base")
openai_client = openai.OpenAI(api_key=OPENAI_KEY)
pc = pinecone.Pinecone(api_key=PINECONE_KEY)
index = pc.Index("youtube-transcripts")


def get_transcript(video_id):
    resp = requests.get(
        "https://getyoutubetranscriber.com/api/v2/transcript",
        params={"video_url": video_id, "include_timestamp": "true"},
        headers={"Authorization": f"Bearer {API_KEY}"}
    )
    resp.raise_for_status()
    return resp.json()["transcript"]


def chunk(segments, max_tokens=300, overlap=50):
    chunks, buf_texts, buf_tokens, start = [], [], 0, None
    for seg in segments:
        toks = enc.encode(seg["text"])
        if start is None:
            start = seg["start"]
        if buf_tokens + len(toks) > max_tokens and buf_texts:
            chunks.append({"text": " ".join(buf_texts), "start": start})
            while buf_tokens > overlap and buf_texts:
                buf_tokens -= len(enc.encode(buf_texts.pop(0)))
            start = seg["start"]
        buf_texts.append(seg["text"])
        buf_tokens += len(toks)
    if buf_texts:
        chunks.append({"text": " ".join(buf_texts), "start": start})
    return chunks


def index_video(video_id):
    segments = get_transcript(video_id)
    chunks = chunk(segments)
    texts = [c["text"] for c in chunks]
    embeddings = openai_client.embeddings.create(
        model="text-embedding-3-small", input=texts
    ).data
    vectors = [
        {
            "id": f"{video_id}_{i}",
            "values": e.embedding,
            "metadata": {"video_id": video_id, "text": c["text"], "start_ms": c["start"]}
        }
        for i, (c, e) in enumerate(zip(chunks, embeddings))
    ]
    index.upsert(vectors=vectors)
    print(f"Indexed {len(vectors)} chunks for {video_id}")


def search(query, top_k=5):
    vec = openai_client.embeddings.create(
        model="text-embedding-3-small", input=[query]
    ).data[0].embedding
    results = index.query(vector=vec, top_k=top_k, include_metadata=True)
    return [
        {
            "score": m.score,
            "text": m.metadata["text"],
            "url": f"https://youtube.com/watch?v={m.metadata['video_id']}&t={int(m.metadata['start_ms'])//1000}s"
        }
        for m in results.matches
    ]


# Usage
index_video("dQw4w9WgXcQ")
hits = search("how to handle database schema migrations safely")
for h in hits:
    print(h["score"], h["url"])
    print(h["text"][:120])
    print()

This script uses the YouTube Transcriber API for YouTube transcript for LLM pipelines: the JSON output is clean and ready to embed with no pre-processing step to strip HTML or fix encoding issues, which is a real time cost when working with raw caption files from the YouTube Data API v3.

If you are building this inside a serverless function to keep infrastructure costs down, pay attention to cold-start times: the embedding API call adds latency on first invocation, so warming the connection or batching indexing jobs separately from query handling keeps response times acceptable.

Next Step

The transcript endpoint documentation at getyoutubetranscriber.com/docs shows the full parameter list, error codes, and rate limit headers. New accounts start with 100 free credits and no card required, which is enough to index 100 videos and validate the pipeline before you commit to anything.

If your use case is monitoring a specific channel's new uploads and automatically re-indexing them as they publish, the channel uploads and new video tracking endpoints in the same API cover that without additional tooling.