Katana

Katana is an open-source crawling and spidering framework from the ProjectDiscovery team, written in Go and released under the MIT License. It takes URLs, lists or piped input and crawls them in standard or headless Chrome mode, producing discovered endpoints as plain text, JSON Lines or files. The latest listed release is v1.8.0.

What It Is

Katana is a command-line web crawler built for automation pipelines. Input can come from a single URL, comma-separated URLs, a list file or STDIN, so it can be chained with other tools such as httpx. It can also be embedded as a Go library by populating an options struct and calling the crawler's Crawl method.

Crawling Modes and Coverage

Standard mode uses Go's HTTP library without browser overhead, which is faster but does not render JavaScript or the DOM. Headless mode runs requests inside a Chrome browser context, so the client presents a browser-like TLS and user-agent fingerprint and endpoints rendered by JavaScript are also discovered. Headless options include page load strategies for single-page apps, using a system Chrome, and connecting to an already-logged-in browser session. Katana can also parse JavaScript files for endpoints, crawl robots.txt and sitemap.xml, and detect technologies.

Scope, Filtering and Rate Control

Scope can be set by field (root domain, FQDN or domain keyword) or by in-scope and out-of-scope regexes. Output can be matched or filtered by extension, regex or DSL conditions on response data. Similar-URL filtering and page content similarity (simhash, TF-IDF, BM25) reduce redundant crawling. Concurrency, parallelism, delays and global or per-host rate limits are configurable, and custom regex fields can extract data such as emails from responses.

Authenticated Crawling and Knowledge Base

Authenticated crawling is supported through custom headers and cookies, automatic login in headless mode, and replay of Chrome DevTools Recorder login flows. An optional knowledge base classifies pages by type, such as login, error or captcha, and identifies forms using an automatically downloaded model. It can also surface secrets and classify REST, GraphQL, SOAP or XHR endpoints. Headless crawling can also use an external captcha solver.

Setup Path

Install with go install (the README states Go 1.26+), download a pre-compiled binary, or use the projectdiscovery/katana Docker image. Headless mode needs Chrome, either installed by katana or a local system Chrome.