Back to the work
Data Engineering · Builder

Website Watcher.

A website archiver and change-detection service: it discovers pages, snapshots them with ArchiveBox, detects content changes by hash, makes every version full-text searchable, and seals versions with Merkle trees and OpenTimestamps anchoring.

Role
Builder
When
2024
Stack
Python, Flask, ArchiveBox, SQLite (FTS5)
Scale
26 modules Python service
View live demo ↗GitHub ↗
Website Watcher previewArchiveBox · Merkle · OpenTimestamps
26 modulesPython service
SHA-256change detection
FTS5full-text search
every 2hscheduled crawl

The problem

Web content changes or disappears, and a plain archive does not prove that a snapshot is genuine or tell you what changed between versions. The goal was a service that discovers and snapshots pages on a schedule, detects content changes by hash, makes every captured version searchable, and can produce a verifiable record that a given version existed at a given time.

What it does

  1. A four-stage core pipeline: discovery (sitemap parsing and internal-link crawling, robots-compliant) into ArchiveBox snapshots, then SHA-256 content-hash change detection, then SQLite FTS5 full-text search across every archived version.
  2. A Flask web UI and API (search, health, Prometheus metrics) plus a CLI (add-site, run, web, search, proof-worker, status), with crawls scheduled through APScheduler.
  3. A verifiable-provenance layer: Merkle trees over page versions, OpenTimestamps anchoring of content hashes, and optional IPFS storage, with a content-hash chain and proof state tracked per version in the database.
  4. Operational from the start: Docker Compose (with a production profile for Prometheus and Grafana) and systemd timer/service units, with health checks and metrics.

Impact

  • Turns ad-hoc archiving into a scheduled, queryable system: every version of a page is captured, hashed for change detection, and searchable by full text, faceted by site and date.
  • Provenance integrations degrade gracefully, OpenTimestamps and IPFS use the external tool when present and fall back to a safe local path otherwise, so the service runs without them and gains verifiability when they are available.
  • 26 Python modules across crawling, archiving, search, Merkle/anchor provenance, and operations, with a test suite covering search, ArchiveBox integration, and Merkle ordering.