OpenSpiderOpen instead of licence keys
Open source (Apache 2.0) and free – no URL limit, no sign-up, no subscription. You can read, audit and build the code yourself. No feature can be taken away from you later.
OpenSpider
Features for technical SEO
A tool for investigating your website, from URLs and content to repeatable audits.
OpenSpider is under active development. This overview describes the documented feature scope; the scope of a beta build is set out in its release notes.
Crawling & analysis tabs
Explore websites in spider mode or from a URL list. Tabs and filters organise internal and external URLs, response codes, content, images and links. Technical analysis covers indexability, robots.txt, sitemaps, canonicals and hreflang.
Content & technical signals
Check titles, meta descriptions and headings; investigate duplicates and thin content. Inspect redirect chains and conflicting indexability signals alongside the affected URLs.
Exports & open data
Export selected tabs and filters as CSV or XLSX. Bulk exports and reports support further analysis. Every crawl is a versioned SQLite file with an .ospider extension.
Comparison & Change Detection
Compare two saved crawls: inspect new and missing URLs and changes in filters and issues. Change Detection tracks page element changes such as titles, status codes and canonicals.
CLI & scheduled tasks
The command line supports headless crawls, loading saved crawls and exports. Task scheduling is designed for recurring crawls with a configuration and export selection.
MCP for AI agents
List saved crawls, query overviews and issues, and read URL details. MCP supports STDIO and authenticated local HTTP. Only read tools are enabled by default; crawl and write tools require explicit permission.
What OpenSpider does differently
Built for AI agents
A built-in MCP server, usable without the desktop app. Claude, Codex, Cursor & co. query crawls, compare them and dig deeper with read-only SQL. Read-only by default; content from crawled pages is treated as untrusted. A skill and a Claude Code plugin are included.
Your crawls belong to you
Every crawl is a single SQLite file with documented, stable query views. Open it with Python, DuckDB, Excel Power Query or your BI tool – no export and no OpenSpider required. Internals may change; the views stay the same within a major version.
Nothing leaves your computer without you
No telemetry. The only connection the app makes on its own is the update check – and you can turn it off. Updates are signed and verified before installation. Feedback is only sent when you send it yourself.
Made for automation
Crawl, export, generate reports, schedule runs: everything is also available from the command line. Desktop app for macOS and Windows; on Linux, the command line and MCP server run without a desktop app.
What comes next
Planned after version 1.0:
- AI checks per page: page type, search intent, content quality, hidden error pages – with your own key or locally via Ollama, within a fixed cost budget.
- Suggestions, not just findings: titles, meta descriptions, alt texts and redirect mapping for relaunches.
- Visibility in AI search: AI crawlers in robots.txt, checking and generating llms.txt.
- Crawls meet search data: rankings, backlinks and citations in AI overviews per URL – for example “not indexable, but ranking” or “in the AI overview, but not cited”.
BETA
The open beta is live
Try OpenSpider 1.0 for free on macOS or Windows and tell us what you notice. Installers, test tasks and the feedback form are on the beta page.