LLM Scraper

LLM Scraper is an open-source TypeScript library, published under the MIT license by GitHub user mishushakov, that turns webpages into structured data using large language models. It runs on top of Playwright and validates output against schemas you define. The README notes that version 2.0 adds Vercel AI SDK 6 support and updated examples.

What It Is

LLM Scraper is a developer library for web data extraction. You open a page with Playwright, define the shape of the data you want with a Zod or JSON Schema, and pass both to the scraper along with an LLM. The model returns typed data that matches the schema, with full TypeScript type safety.

How the Extraction Workflow Works

Setup involves installing zod, playwright and llm-scraper from npm, then adding a model provider package such as OpenAI, Anthropic, Google, Groq (through the OpenAI-compatible provider) or Ollama. You create a scraper instance with the chosen model and call run on a page with a schema and a format option. The README example extracts the top five Hacker News stories with title, points, author and comments URL.

Six formatting modes control what the model sees:

• html for pre-processed HTML
• raw_html for unprocessed HTML
• markdown
• text, extracted with Readability.js
• image, a screenshot, for multi-modal models only
• custom, using your own function

Streaming and Code Generation

Calling stream instead of run returns a partial object stream, so results can be consumed as the model produces them. The generate function asks the model to write a reusable Playwright script for a given schema. You can then evaluate that script on the page and parse the result with the schema, without calling the LLM on every run.