Colly

Colly is an open-source scraping and crawling framework for the Go language, developed by the Colly Team under the gocolly organization. It gives Go developers an API for making network requests and handling the returned content, such as traversing the HTML DOM. The latest listed release is v2.2.0, published on 2025-03-27.

What It Is

Colly is a library that Go developers import to build scrapers of varying complexity, from simple scripts to asynchronous crawlers that process many pages. The project says it can be used to extract structured data for data mining, data processing, or archiving. It is installed with go get github.com/gocolly/colly/v2.

How a Scraper Is Built

A developer creates a collector, registers callbacks such as OnHTML and OnRequest, and then visits a starting URL. In the documented example, an OnHTML callback selects links and visits each one, while an OnRequest callback logs each URL being visited. Scraped elements are accessed through HTMLElement objects, so attributes and content can be pulled out of matching nodes.

Built-in Crawling Controls

The documented features include request delay and maximum concurrency management per domain, automatic cookie and session handling, and sync, async, and parallel scraping. Colly also supports caching, automatic encoding of non-unicode responses, robots.txt support, distributed scraping, configuration through environment variables, and extensions. The project states that it can handle more than 1,000 requests per second on a single core.

Licensing and Community

The repository is licensed under Apache 2.0, and the project site describes it as free for commercial use and community driven. Documentation covers installation, getting started, and a basic scraper example, with API reference on GoDoc and further examples in the repository.