ScrapeGraphAI

ScrapeGraphAI is a Python web scraping library from the ScrapeGraphAI team that uses LLMs and direct graph logic to build scraping pipelines for websites and local documents such as XML, HTML, JSON and Markdown. Users state which information they want, and the library extracts it. The open-source library is MIT licensed, and the team also offers a separate managed cloud API with Python and Node.js SDKs.

What It Is

ScrapeGraphAI is a prompt-driven scraping library. You supply a natural-language prompt and a source (a URL or a local file), configure an LLM, and run a graph pipeline that returns the extracted information, for example as a JSON dictionary. It installs with pip and uses Playwright to fetch website content.

Scraping Pipelines and Models

The README documents several standard pipelines:

• SmartScraperGraph extracts information from a single page given a prompt and a source.
• SearchGraph extracts information from the top search results of a search engine.
• SpeechGraph scrapes a page and generates an audio file.
• ScriptCreatorGraph scrapes a page and generates a Python script.
• SmartScraperMultiGraph and ScriptCreatorMultiGraph handle multiple pages from a single prompt and a list of sources.

Multi versions of each graph make LLM calls in parallel. Models can be accessed through APIs such as OpenAI, Groq, Azure, Gemini and MiniMax, or run locally with Ollama.

Open Source vs Managed API

The README contrasts the self-hosted library with the managed cloud API. With the library, you run it on your own infrastructure, bring your own LLM, and configure browsers, proxies and scaling yourself. The managed API, called through the Python and JS/TS SDKs, is hosted by ScrapeGraphAI and handles JavaScript rendering, proxies and anti-bot measures, and adds Scrape, Extract, Search, Crawl, Monitor and History capabilities.

Integrations

The README lists integrations with LangChain, LlamaIndex, Crew.ai, Agno and CamelAI, low-code tools such as Pipedream, Bubble, Zapier, n8n, Dify and Toolhouse, and an MCP server. The README states the project is intended for data exploration and research purposes.