Trafilatura

Trafilatura is an open-source Python package and command-line tool by Adrien Barbaresi that gathers text from the web and turns raw HTML into structured data. It covers web crawling, downloads, scraping, and extraction of main text, metadata, and comments. The current release is v2.3.1, distributed under the Apache 2.0 license.

What It Is

Trafilatura is a web text discovery and extraction toolkit. It takes live URLs, previously downloaded HTML files, or parsed HTML trees and returns the main content while avoiding recurring elements such as headers, footers, and boilerplate. No database is required, and output can be converted to commonly used formats.

How Extraction Works

The main text extractor is rule-based, with jusText and readability-lxml as fallbacks. It also extracts metadata (title, author, date, site name, categories, tags) and preserves structure such as paragraphs, titles, lists, quotes, code, and in-line formatting. Optional elements include comments, links, images, and tables, plus add-ons such as language detection. Output formats include TXT, Markdown, CSV, JSON, HTML, XML, and XML-TEI.

Discovery and Crawling

The toolkit supports sitemaps (TXT, XML) and feeds (ATOM, JSON, RSS), with URL filtering and deduplication. It processes download queues in parallel and politely. It can be used from Python, the command line, R, and a graphical interface.

Research and Pipeline Use

The project began as a PhD effort in linguistics and NLP, building text corpora at the Berlin-Brandenburg Academy of Sciences. The documentation includes tutorials for building web corpora, text embedding with ChromaDB, retrieval-augmented generation, and Hugging Face Datasets training corpora. The project states that it outperforms other open-source libraries in text extraction benchmarks.