# LLM Scraper

> TypeScript library that extracts structured data from any webpage using LLMs and Playwright.

LLM Scraper is an open-source TypeScript library, published under the MIT license by GitHub user mishushakov, that turns webpages into structured data using large language models. It runs on top of Playwright and validates output against schemas you define. The README notes that version 2.0 adds Vercel AI SDK 6 support and updated examples.

## What It Is

LLM Scraper is a developer library for web data extraction. You open a page with Playwright, define the shape of the data you want with a Zod or JSON Schema, and pass both to the scraper along with an LLM. The model returns typed data that matches the schema, with full TypeScript type safety.

## How the Extraction Workflow Works

Setup involves installing zod, playwright and llm-scraper from npm, then adding a model provider package such as OpenAI, Anthropic, Google, Groq (through the OpenAI-compatible provider) or Ollama. You create a scraper instance with the chosen model and call `run` on a page with a schema and a format option. The README example extracts the top five Hacker News stories with title, points, author and comments URL.

Six formatting modes control what the model sees:
- `html` for pre-processed HTML
- `raw_html` for unprocessed HTML
- `markdown`
- `text`, extracted with Readability.js
- `image`, a screenshot, for multi-modal models only
- `custom`, using your own function

## Streaming and Code Generation

Calling `stream` instead of `run` returns a partial object stream, so results can be consumed as the model produces them. The `generate` function asks the model to write a reusable Playwright script for a given schema. You can then evaluate that script on the page and parse the result with the schema, without calling the LLM on every run.

## Features
- Structured data extraction from any webpage
- Schemas defined with Zod or JSON Schema
- Full TypeScript type safety
- Built on Playwright
- Streaming partial objects
- Code generation of reusable Playwright scripts
- Six formatting modes: html, raw_html, markdown, text, image, custom
- Supports GPT, Sonnet, Gemini, Llama and Qwen model series

## Integrations
Playwright, Zod, Vercel AI SDK, OpenAI, Anthropic, Google Gemini, Groq, Ollama, Readability.js

## Platforms
API, DEVELOPER_SDK

## Pricing
Open Source

## Version
2.0

## Links
- Website: https://github.com/mishushakov/llm-scraper
- Repository: https://github.com/mishushakov/llm-scraper
- EveryDev.ai: https://www.everydev.ai/tools/llm-scraper