YouTube Transcript API: Designing a Resilient ETL Pipeline

YouTube uploads over 500 hours of video every minute (Hootsuite, 2026). If your product ingests transcripts, monitors channels, or feeds video text into a downstream model, you are building a pipeline that must handle that scale reliably. And yet most YouTube data pipelines are built as a single script: fetch a list of videos, loop, write to a database, pray nothing breaks.

When something does break, and it will, the result is a mess. You do not know which videos failed, you cannot replay just the failed ones, and you have no idea how much it cost. According to Monte Carlo's 2026 State of Data Quality survey, 68% of data teams take four or more hours to detect a pipeline issue, and the average resolution time is 15 hours. That is almost a full working day lost per incident.

A well-designed ETL pipeline fixes this by separating concerns, tolerating partial failures, and giving you visibility into what is actually happening.

This post walks through building exactly that for YouTube transcript data.

---

The Four Stages: Discover, Fetch, Transform, Load

The core principle is that each stage should do exactly one thing and hand off its output to the next stage through a queue. This decoupling is what makes individual stage failures recoverable without restarting the whole job.

Stage 1: Discover

The discover stage produces a list of video IDs that need processing. Its only job is to write messages onto a queue. It does not fetch transcripts or touch the database.

import httpx
import json

def discover_channel_videos(channel_handle: str, queue):
    url = "https://getyoutubetranscriber.com/api/v2/channel/videos"
    headers = {"Authorization": "Bearer ytt_your_key_here"}
    continuation = None

    while True:
        params = {"channel": channel_handle}
        if continuation:
            params["continuation"] = continuation

        resp = httpx.get(url, params=params, headers=headers, timeout=30)
        resp.raise_for_status()
        data = resp.json()

        for video in data["results"]:
            if video.get("has_captions"):
                queue.enqueue({"video_id": video["video_id"], "channel_id": video["channel_id"]})

        if not data.get("has_more"):
            break
        continuation = data["continuation_token"]

Note the has_captions check. Attempting to fetch a transcript for a video with no captions wastes a credit. Filter at discovery time.

For channel monitoring, swap the channel/videos endpoint for channel-latest, which is backed by YouTube's public RSS feed and is free to call. Poll it every 15 minutes and compare returned video IDs against a set of already-processed IDs in Redis or your database. See Monitor a YouTube Channel for New Videos via API for a deeper walkthrough of that pattern.

Stage 2: Fetch

The fetch worker reads messages from the queue, calls the transcript API, and writes raw JSON to a staging area (a database table, object storage, or even a second queue). It does not transform or normalize anything.

def fetch_transcript(video_id: str, lang: str = "en") -> dict:
    url = "https://getyoutubetranscriber.com/api/v2/transcript"
    headers = {"Authorization": "Bearer ytt_your_key_here"}
    params = {
        "video_url": video_id,
        "lang": lang,
        "send_metadata": "true",
        "include_timestamp": "true",
    }

    resp = httpx.get(url, params=params, headers=headers, timeout=60)
    resp.raise_for_status()
    return resp.json()

A successful response looks like this:

{
  "video_id": "dQw4w9WgXcQ",
  "language": "en",
  "transcript": [
    { "text": "We're no strangers to love", "start": 0.0, "duration": 3.5 },
    { "text": "You know the rules and so do I", "start": 3.5, "duration": 3.2 }
  ],
  "metadata": {
    "title": "Rick Astley - Never Gonna Give You Up",
    "author_name": "Rick Astley",
    "author_url": "https://www.youtube.com/@RickAstleyYT",
    "thumbnail_url": "https://i.ytimg.com/vi/dQw4w9WgXcQ/hqdefault.jpg"
  }
}

Write this entire object to a raw_transcripts staging table before doing anything else. The raw stage is your replay buffer.

Stage 3: Transform

The transform stage reads from the raw staging table and produces a cleaned, normalized record. This is where you join timestamps, concatenate segments into a full-text string, strip formatting artifacts, detect language, and apply any business logic your product needs.

def transform_transcript(raw: dict) -> dict:
    segments = raw.get("transcript", [])
    full_text = " ".join(s["text"] for s in segments)

    return {
        "video_id": raw["video_id"],
        "language": raw["language"],
        "full_text": full_text,
        "segment_count": len(segments),
        "duration_seconds": sum(s.get("duration", 0) for s in segments),
        "title": raw.get("metadata", {}).get("title"),
        "channel": raw.get("metadata", {}).get("author_name"),
        "segments": segments,
    }

Keep transforms pure functions. They should not call external APIs. This makes them trivial to unit test and safe to replay.

Stage 4: Load

The load stage writes the transformed record to your production store. The critical detail here is using an upsert rather than a plain insert so replaying a message is always safe.

INSERT INTO transcripts (
  video_id, language, full_text, segment_count,
  duration_seconds, title, channel, segments, fetched_at
)
VALUES (
  %(video_id)s, %(language)s, %(full_text)s, %(segment_count)s,
  %(duration_seconds)s, %(title)s, %(channel)s, %(segments)s::jsonb, NOW()
)
ON CONFLICT (video_id, language)
DO UPDATE SET
  full_text = EXCLUDED.full_text,
  segments = EXCLUDED.segments,
  fetched_at = EXCLUDED.fetched_at;

---

Decoupling with Queues and Idempotent Workers

The queue between each stage is what turns a fragile script into a resilient system. Workers can crash, restart, and replay messages without corrupting data, because each message is processed idempotently.

The simplest production-ready setup for most teams is PostgreSQL with a job queue library (like pgqueuer in Python or pg-boss in Node.js). You already have Postgres. No extra infrastructure.

For higher throughput, Redis Streams or a managed queue like SQS work well. The pattern is the same regardless of backend.

Key design rules for workers:

• Each worker processes one message at a time and acknowledges it only after a successful write.
• Use video_id plus language as your idempotency key throughout.
• Workers should be stateless. All state lives in the queue and the database.
• Run multiple worker instances for parallelism. Since each processes a different video, there are no locking conflicts.

---

Handling Partial Failures and Retries

Every video fetch can fail for a different reason: the video was deleted, captions are unavailable in the requested language, there was a network timeout, or the API returned a transient 503. Your retry logic needs to handle all of these without treating them the same.

Classify errors before retrying

| Error type | HTTP status | Action | |--|--|--| | Transient network error | 500, 502, 503, 504 | Retry with backoff | | Rate limited | 429 | Retry after the Retry-After header delay | | No captions available | 404 or API error | Mark as no_captions, do not retry | | Invalid video ID | 400 | Mark as invalid, do not retry | | Video deleted/private | 403 | Mark as unavailable, do not retry |

Non-retryable errors should be written to your database with a status column set to the error class, not sent to the dead-letter queue. You still want a record that you attempted this video.

Retry with exponential backoff

import time

def fetch_with_retry(video_id: str, max_attempts: int = 3) -> dict:
    delays = [30, 300, 1800]  # 30s, 5min, 30min

    for attempt, delay in enumerate(delays[:max_attempts], start=1):
        try:
            return fetch_transcript(video_id)
        except httpx.HTTPStatusError as e:
            if e.response.status_code in (400, 403, 404):
                raise  # non-retryable
            if attempt == max_attempts:
                raise
            time.sleep(delay)

After three failed attempts on a retryable error, move the message to a dead-letter queue. A dead-letter queue is just a separate table or queue where you park jobs that need human review. Check it daily, not by waking up at 2 AM to alerts.

For bulk transcript jobs across large channels, also read Download YouTube Transcripts in Bulk Without Getting Blocked for guidance on rate pacing.

---

Schema Design for Transcript Storage

How you store transcripts depends on how you query them.

Option A: JSONB column (recommended for most cases)

CREATE TABLE transcripts (
  id              BIGSERIAL PRIMARY KEY,
  video_id        TEXT NOT NULL,
  language        TEXT NOT NULL DEFAULT 'en',
  title           TEXT,
  channel         TEXT,
  full_text       TEXT,
  segment_count   INTEGER,
  duration_seconds NUMERIC,
  segments        JSONB,
  status          TEXT NOT NULL DEFAULT 'ok',
  fetched_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  UNIQUE (video_id, language)
);

-- Index for fast lookups by channel
CREATE INDEX ON transcripts (channel);

-- Index for full-text search on the concatenated transcript
CREATE INDEX ON transcripts USING GIN (to_tsvector('english', full_text));

The UNIQUE (video_id, language) constraint is what makes upserts safe. The segments JSONB column stores the original segment array verbatim, which gives you timestamp data when you need it without denormalizing into a separate table.

Option B: Normalized segments table (for analytics workloads)

If you are running aggregation queries across segment-level data (average segment duration, word frequency per channel, etc.), a normalized schema performs better.

CREATE TABLE videos (
  video_id     TEXT PRIMARY KEY,
  title        TEXT,
  channel      TEXT,
  fetched_at   TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE TABLE transcript_segments (
  id         BIGSERIAL PRIMARY KEY,
  video_id   TEXT NOT NULL REFERENCES videos(video_id),
  language   TEXT NOT NULL,
  seq        INTEGER NOT NULL,
  text       TEXT NOT NULL,
  start_sec  NUMERIC NOT NULL,
  duration   NUMERIC NOT NULL,
  UNIQUE (video_id, language, seq)
);

This schema works well when you join segments to external tables (ad markers, chapter boundaries, speaker diarization results). The trade-off is that inserting a single transcript requires many row inserts instead of one.

---

Monitoring, Alerting, and Cost Tracking

A pipeline you cannot observe is a pipeline you cannot trust. You need three things: a job status view, per-error alerting, and credit cost tracking.

Job status table

Add a pipeline_runs table to track every batch:

CREATE TABLE pipeline_runs (
  id            BIGSERIAL PRIMARY KEY,
  run_id        UUID NOT NULL DEFAULT gen_random_uuid(),
  channel       TEXT,
  started_at    TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  completed_at  TIMESTAMPTZ,
  total_videos  INTEGER DEFAULT 0,
  fetched_ok    INTEGER DEFAULT 0,
  failed        INTEGER DEFAULT 0,
  credits_used  INTEGER DEFAULT 0,
  status        TEXT NOT NULL DEFAULT 'running'
);

Workers increment fetched_ok, failed, and credits_used as they process messages. A simple dashboard query gives you a live view of any running batch.

Credit cost tracking

The YouTube Transcriber API charges per successful call. You can track this by incrementing a counter when you write a successful transcript row:

UPDATE pipeline_runs
SET fetched_ok    = fetched_ok + 1,
    credits_used  = credits_used + 1
WHERE run_id = %(run_id)s;

Set a pre-flight budget check before starting a large batch: estimate the number of videos, compare against your remaining credit balance, and abort if the batch would exceed your budget. This prevents surprise charges when someone accidentally queues a channel with 10,000 videos.

For a deeper look at controlling API costs, Caching YouTube Transcripts to Cut API Costs covers caching patterns that can reduce credit consumption by 60% or more on repeated fetches.

Alerting thresholds

Keep alerting simple. Three metrics cover most failures:

• Error rate above 5% in a 10-minute window: something systematic is wrong (wrong API key, YouTube blocking your range, misconfigured language parameter).
• Dead-letter queue depth above 20: more jobs are failing than you are resolving manually.
• Pipeline run older than 2x its expected duration with status still running: a worker probably crashed without releasing its job.

Route these alerts to Slack or PagerDuty depending on severity. Error rate spikes wake someone up; DLQ depth sends a Slack message.

---

Putting It Together

A complete pipeline for a single channel run looks like this:

1. Discover worker pages through channel/videos, filters by has_captions, enqueues video IDs.
2. Fetch workers pull from the queue, call /api/v2/transcript, write raw JSON to raw_transcripts.
3. Transform workers read raw_transcripts, produce clean records, write to a second queue or process inline.
4. Load workers upsert into transcripts, increment run counters.
5. After the run completes, a summary job marks the pipeline_runs row as completed and posts a Slack notification with the credit cost.

This design scales horizontally: add more fetch workers to process faster, without changing anything else. It also tolerates worker restarts cleanly, because every stage is idempotent and the queue retains unacknowledged messages.

---

Getting Started

YouTube Transcriber provides a single Bearer token, 100 free credits with no card required, and handles IP blocks, proxy rotation, and anti-bot challenges on your behalf. The four API calls this pipeline uses (/api/v2/transcript, /api/v2/channel/videos, /api/v2/channel-latest, and /api/v2/playlist-videos) cover the full discover-and-fetch cycle.

Start with the free tier, wire up the discover and fetch stages against a small channel, and add the queue and retry logic before you scale. The full documentation is at getyoutubetranscriber.com/docs.