Skip to main content
Open Source Docs

Scrape the web.
Feed your AI.

Tentacrawl scrapes and crawls modern, JavaScript-rendered websites into clean, structured data — Markdown, JSON, HTML, and screenshots ready for RAG pipelines, automation, and analytics. Site-specific adaptation — proxy rotation, captcha solving, fingerprinting, login — plugs in through the open source Challenger extension framework, without touching core code.

Get StartedView on GitHub

What it's for​

  • Structured data extraction — turn any page into Markdown, JSON, HTML, metadata, or a screenshot.
  • AI and LLM pipelines — Markdown and metadata artefacts are ready for RAG ingestion and knowledge bases without extra cleanup.
  • Monitoring and change detection — recurring crawls with discovered-link tracking across a site.
  • Anti-bot and WAF-protected sites — proxy rotation, captcha solving, fingerprinting, and login all plug in through the Challenger framework, with no changes to core code.

Tentacrawl scrapes and crawls JavaScript-rendered websites with real browser automation (Playwright), not a static HTTP fetch — pages that need a login, infinite scroll, or client-side rendering work the same as pages that don't. It is a pnpm monorepo with three applications (apps/api, apps/worker, apps/web) built around a YAML DSL for scripted browser flows.

The core idea​

Site-specific adaptation — proxy rotation, captcha handling, fingerprinting, login, user-behavior mimicry — cannot live in the core scraping engine, because every site needs something different. Tentacrawl solves this with the Challenger framework: a single, contract-based extension surface that lets a module observe and mutate every stage of a run (navigation, requests and responses, session state, artifacts, DSL steps) without editing core code. packages/proxy is the reference implementation; everything else you might want to build — a captcha solver, a fingerprint rotator, a login flow — follows the same contract.

Where to go next​

  • Getting Started — install the platform and run your first scrape.
  • Guides — task-oriented instructions for operating a deployment.
  • Extending Tentacrawl — write a Challenger extension.
  • Architecture — how the pieces fit together and why.
  • Reference — environment variables, the HTTP API, the Challenger contract, the DSL grammar.

The full setup and command reference also lives in the repository README; these docs go deeper on the parts the README only summarizes.