AutoScraper

AutoScraper is an open-source Python library by alirezamika for automatic web scraping. You give it a URL or raw HTML plus sample values you want to extract, and it learns the scraping rules and returns similar elements. The latest listed release is v1.1.14 under the MIT License.

What It Is

AutoScraper is a lightweight scraping module for Python 3. Instead of writing selectors by hand, you supply a page and a wanted list of example data, which can be text, URLs, or any HTML tag value. The library learns rules that match those examples and returns similar elements from the page.

How the workflow works

Calling build with a URL (or HTML content via the html parameter) and a wanted list creates a learned scraper object. That object can then be applied to new URLs in two ways:

• get_result_similar returns content similar to the examples, such as related post titles on other pages.
• get_result_exact returns the exact matching elements in the same order as the wanted list, such as a stock price and market cap.

Custom requests parameters such as proxies or headers can be passed through request_args.

Saving and reusing models

A built scraper can be saved to a file with save and restored with load, so learned rules can be reused later. The README links to a gist with advanced usage and a tutorial on building an API from any website with Flask.

Setup path

Install from PyPI with pip install autoscraper, directly from the Git repository with pip, or from source with python setup.py install.