Download YouTube Transcripts in Bulk Without Getting Blocked

Pulling a single transcript is trivial. Pulling ten thousand is a different problem.

At scale, YouTube aggressively rate-limits and IP-bans scrapers. Cloud provider address ranges—AWS, GCP, Azure—are on YouTube's radar by default. The open-source youtube-transcript-api library is a popular starting point, but it breaks under load for well-documented reasons. And YouTube's official Data API caps caption access at roughly 50 transcripts per day on the free quota (each caption download costs ~200 of your 10,000 daily units).

This post covers what actually happens when you go bulk, how to design a pipeline that doesn't blow up, and how to keep costs predictable.

---

Why Bulk Scraping Triggers IP Bans Fast

YouTube does not publish a hard request limit, but the pattern is consistent across engineering teams that have hit it: a few hundred requests in a short window from a single IP triggers a soft block (HTTP 429), and sustained hammering leads to a harder IP ban that lasts 24–72 hours or longer.

Several factors make bulk runs especially fragile:

• Cloud IPs are pre-flagged. AWS, GCP, and Azure address ranges are well-known to YouTube. A fresh EC2 instance has a higher block probability than a residential IP.
• No auth = no quota bucket. Scraping without an API key gives you no rate-limit feedback loop. You find out you're blocked only when requests start failing.
• Uniform timing looks robotic. Sending exactly one request per second, every second, for an hour is a clear signal. Real browsers have variance.
• Retrying too fast compounds the problem. Hammering a 429 response without backing off escalates a soft block into a hard ban.

The practical conclusion: at any scale above a few hundred videos, self-managed scraping is fragile. The engineering cost of rotating residential proxies, handling CAPTCHAs, and managing IP pools often exceeds the cost of a managed API.

---

Concurrency, Rate Limits, and Queue Design

The goal is maximum throughput without triggering blocks. That means bounded concurrency, a durable job queue, and honest handling of rate-limit headers.

Pick a concurrency number and stick to it

Start with 5–10 concurrent workers. More workers mean faster throughput but also a higher burst rate that looks suspicious. Once your pipeline is stable, you can measure actual throughput and tune upward.

workers = 8          # concurrent requests in flight
queue   = job_queue  # persistent list of video IDs to process

Respect rate-limit headers

A well-designed API returns IETF draft-7 rate-limit headers:

RateLimit-Limit: 300
RateLimit-Remaining: 247
RateLimit-Reset: 1719187200
Retry-After: 4

Read Retry-After on every 429. Pause all workers for that duration before resuming. Do not retry immediately.

Use a persistent queue, not an in-memory list

If your process crashes at video #4,000 of 10,000, you want to resume from #4,001—not restart. Use Redis, SQLite, or a simple database table as your job queue. Track three states per job: pending, done, failed.

A minimal schema:

CREATE TABLE jobs (
  video_id   TEXT PRIMARY KEY,
  status     TEXT DEFAULT 'pending',  -- pending | done | failed
  retries    INTEGER DEFAULT 0,
  error      TEXT,
  created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
  updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
);

---

Code Example: Processing Thousands of Videos

The example below uses Python with asyncio and httpx to process a large list of video IDs against the YouTube Transcriber API. It respects rate limits, tracks failures, and never blocks the whole queue on a single bad video.

import asyncio
import httpx
import sqlite3
import random
import time

API_KEY = "YOUR_API_KEY"
BASE_URL = "https://getyoutubetranscriber.com/api/v2/transcript"
CONCURRENCY = 8
MAX_RETRIES = 4

db = sqlite3.connect("jobs.db")
db.execute("""
    CREATE TABLE IF NOT EXISTS jobs (
        video_id TEXT PRIMARY KEY,
        status   TEXT DEFAULT 'pending',
        retries  INTEGER DEFAULT 0,
        error    TEXT
    )
""")
db.commit()

def seed_jobs(video_ids: list[str]):
    db.executemany(
        "INSERT OR IGNORE INTO jobs (video_id) VALUES (?)",
        [(vid,) for vid in video_ids]
    )
    db.commit()

def get_pending():
    rows = db.execute(
        "SELECT video_id FROM jobs WHERE status = 'pending' AND retries < ?",
        (MAX_RETRIES,)
    ).fetchall()
    return [r[0] for r in rows]

def mark_done(video_id: str):
    db.execute(
        "UPDATE jobs SET status='done', updated_at=CURRENT_TIMESTAMP WHERE video_id=?",
        (video_id,)
    )
    db.commit()

def mark_failed(video_id: str, error: str, increment_retries: bool = True):
    db.execute(
        """UPDATE jobs
           SET retries = retries + ?,
               error = ?,
               status = CASE WHEN retries + ? >= ? THEN 'failed' ELSE 'pending' END
           WHERE video_id = ?""",
        (1 if increment_retries else 0, error,
         1 if increment_retries else 0, MAX_RETRIES, video_id)
    )
    db.commit()

async def fetch_transcript(client: httpx.AsyncClient, video_id: str, sem: asyncio.Semaphore):
    async with sem:
        attempt = 0
        delay = 1.0
        while attempt < MAX_RETRIES:
            try:
                resp = await client.get(
                    BASE_URL,
                    params={"videoId": video_id},
                    headers={"Authorization": f"Bearer {API_KEY}"},
                    timeout=30,
                )
                if resp.status_code == 200:
                    # process resp.json() here — save to file, DB, vector store, etc.
                    mark_done(video_id)
                    return
                elif resp.status_code == 429:
                    retry_after = float(resp.headers.get("Retry-After", delay))
                    jitter = retry_after * random.uniform(0.8, 1.2)
                    await asyncio.sleep(jitter)
                    delay = min(delay * 2, 60)
                    attempt += 1
                elif resp.status_code in (404, 400):
                    # permanent failure — no captions or bad video ID
                    mark_failed(video_id, f"HTTP {resp.status_code}", increment_retries=False)
                    return
                else:
                    mark_failed(video_id, f"HTTP {resp.status_code}")
                    delay = min(delay * 2 + random.uniform(0, 1), 60)
                    attempt += 1
            except httpx.RequestError as exc:
                mark_failed(video_id, str(exc))
                await asyncio.sleep(delay + random.uniform(0, 1))
                delay = min(delay * 2, 60)
                attempt += 1

        mark_failed(video_id, "max retries exceeded")

async def run(video_ids: list[str]):
    seed_jobs(video_ids)
    sem = asyncio.Semaphore(CONCURRENCY)
    pending = get_pending()
    print(f"Processing {len(pending)} videos with {CONCURRENCY} workers")
    async with httpx.AsyncClient() as client:
        await asyncio.gather(*[fetch_transcript(client, vid, sem) for vid in pending])

if __name__ == "__main__":
    # Load your video IDs from a file, database, or channel listing
    with open("video_ids.txt") as f:
        ids = [line.strip() for line in f if line.strip()]
    asyncio.run(run(ids))

A few things worth noting:

• The semaphore caps in-flight requests to CONCURRENCY. Every worker waits for a slot before sending a request.
• 404 and 400 are permanent. Don't retry them—they consume retries and credits without hope of success.
• 429 uses the Retry-After header from the API, not a hardcoded value. This is the right way to handle rate limits.
• Jitter (0.8–1.2×) spreads retries across workers so they don't all wake up at the same second.

To get the video IDs for an entire channel, call the channel/videos endpoint first:

curl "https://getyoutubetranscriber.com/api/v2/channel/videos?channelId=UC_x5XG1OV2P6uZZ5FSM9Ttw&maxResults=50" \
  -H "Authorization: Bearer YOUR_API_KEY"

Page through the results using the nextPageToken in the response until you have all the IDs, then feed them into the pipeline above.

---

Handling Failures and Retrying Only Failed Calls

The worst bulk pipeline design is "retry everything from scratch." It wastes credits, takes longer, and risks re-blocking your IP on work that already succeeded.

The SQLite schema above tracks each video independently. After a run completes, query your failed jobs:

SELECT video_id, retries, error
FROM jobs
WHERE status = 'failed'
ORDER BY retries DESC;

Common failure patterns and what they mean:

| Error | Cause | Action | |---|---|---| | HTTP 404 | No captions available | Permanent skip — no credits charged | | HTTP 429 | Rate limited | Already retried with backoff; check concurrency setting | | HTTP 403 | IP blocked or auth error | Verify API key; reduce concurrency | | HTTP 500/503 | Server error | Transient; re-queue for another run | | RequestError | Network timeout | Usually transient; re-queue |

To re-run only failed jobs, reset their status and run again:

UPDATE jobs
SET status = 'pending', retries = 0, error = NULL
WHERE status = 'failed' AND error NOT LIKE 'HTTP 40%';

This skips permanent 400/404 errors and only re-queues transient failures.

For the full story on why open-source libraries struggle here and how a managed API solves the retry problem at the infrastructure layer, see Why youtube-transcript-api Gets Blocked (and What to Do).

---

Paying Only for Successful Calls

At scale, cost control matters. A flat-subscription model charges you whether a video has captions or not. Pay-per-successful-call aligns cost with actual output.

YouTube Transcriber charges credits only on successful transcript responses. A 404 (video has no captions), a 429 you never got to resolve, or a network timeout does not consume a credit. That means you can safely run a pipeline against a channel where 20–30% of videos might lack captions without paying for those misses.

The free tier starts with 100 credits, no credit card required, which is enough to validate your pipeline architecture before committing to production volume.

To estimate costs before running a large job: pull the video list first (channel/videos is a separate, lower-cost call), count total videos, subtract an estimated no-caption rate (~15–20% for most channels), and multiply by your credit rate.

---

Best-Practice Checklist

Before you run a bulk job in production, verify each of these:

• [ ] Persistent queue — job state survives process restarts (SQLite, Redis, or a DB table)
• [ ] Bounded concurrency — start at 5–10 workers; don't send unbounded parallel requests
• [ ] Exponential backoff with jitter — double delay on each retry, add ±20–30% randomness
• [ ] Respect Retry-After — pause all workers, not just the one that got 429'd
• [ ] Separate permanent from transient failures — 404/400 → skip; 5xx/429 → retry
• [ ] Per-video error logging — you need to know which videos failed and why
• [ ] Idempotent writes — writing a transcript twice should not corrupt your output store
• [ ] Resume support — re-running the script processes only pending and retryable failed jobs
• [ ] Dry-run test — validate with 10–20 videos before launching thousands
• [ ] Monitor credit usage — set an alert before you hit your budget ceiling

---

Putting It Together

Bulk transcript downloads are an infrastructure problem as much as an API problem. The code itself is straightforward; the hard part is handling the failure cases that only appear at scale—IP blocks, partial caption availability, transient network errors, and the cost of re-running failed jobs.

The pattern above—persistent queue, bounded concurrency, exponential backoff, permanent-vs-transient failure separation—works whether you're processing 500 videos for a content repurposing tool or 500,000 videos for an LLM training dataset. Once you've validated the pipeline with a small batch, scaling up is mostly a matter of adjusting concurrency and monitoring your rate-limit headroom.

If you're building a pipeline that feeds transcripts into an AI workflow, the YouTube Transcript API: Build a Video Summarizer with GPT post walks through the downstream processing side once you have clean JSON transcripts in hand.

Start with the free 100 credits to validate your pipeline, then check the API docs for the full endpoint reference including bulk, channel, and playlist endpoints.