I built a platform to track prices across Dutch retailers. The engineering held up. The economics of scraping did not. This is the honest version, with the production numbers.

The bet

Dutch retailers change prices and promotions constantly, and no single place shows those changes side by side. The bet behind SaleSpotter: collect prices from four of them steadily enough, and comparing them becomes a product. The four were Albert Heijn, Jumbo, Etos and Kruidvat. The hard requirement hid in one word, steadily. A price tracker that misses days is not a tracker.

The first architecture

I split the system into three parts that could each run alone: a scraper, a backend API, and the infrastructure underneath.

The scraper is TypeScript and Playwright, organized as a Bronze to Silver to Gold pipeline. Bronze fetched raw HTML and JSON and saved it to disk gzipped, with rate limiting, cookies and retries handled at that layer. Silver parsed those files with Cheerio and validated the extracted prices and titles against Zod schemas. Gold aggregated every scraper's output, checked quality, and loaded the result into the database through a service that watched for new Silver data.

The API is Go. It used Gin for routing and FX for dependency injection. Huma generated the OpenAPI contract from the handlers. GORM mapped the models onto MySQL. Google OAuth handled sign-in, and a direct integration with Albert Heijn's own API covered weekly product updates with no scraping at all.

Production ran on Hetzner Cloud, described end to end as code. Terraform provisioned the load balancer, the database server and the app servers. Ansible configured the rest, from the private-network NAT gateway to Fail2Ban and Auditd. Prometheus collected metrics; Grafana displayed them; Loki stored logs. GitHub Actions built Docker images and deployed them.

What production said

Tuned per retailer, the pipeline held a steady pace. Etos ran at about ten products a minute, and one session extracted more than 2,300 products. Kruidvat surfaced more than 7,300 products across 15 categories, but only at a crawl: about eight seconds per product, paced that slowly on purpose to stay under detection thresholds. Those two numbers frame the whole retrospective. The system could collect a full catalog, or it could collect quickly, and the anti-bot systems decided which.

The anti-bot wall

The scraper's version history is really a history of countermeasures. The first version used custom TLS fingerprinting with mTLS and Crawlee. It was technically satisfying, too slow for the scale I needed, and detectable anyway. The second fetched pages as Markdown through Crawl4AI, which simplified Silver parsing and got me about 3,000 products before serious blocking started. The version that lasted moved parsing to gpt-4o-mini. The model extracted structured attributes from the Markdown, and it kept working on dynamic, inconsistent pages where my CSS selectors kept breaking.

Kruidvat sits behind DataDome. Even with Playwright Stealth, a large run had two options: pay for residential proxies, or pace requests slowly enough to imitate a human. One costs money, the other costs freshness, and every anti-bot upgrade raised both prices. The pages themselves leaned on Shadow DOM and AJAX-hydrated content, so a plain fetch often missed the price without network interception or visual waits. Storing raw HTML for thousands of items pushed bandwidth costs up. And retailer HTML changed often enough that selector maintenance never ended.

What held up

The parts I worried about at design time were never the problem. The pipeline, the API and the infrastructure ran through every scraper rewrite without structural change, and the monitoring outlived every scraping strategy. What failed was the assumption underneath the bet: that steady collection from four retailers was mainly an engineering problem. It is a pricing problem, and DataDome sets the price.

The architecture survived contact with production; the economics did not. Every scraped price cost either proxy fees or human-speed pacing, and each anti-bot upgrade raised that cost. The one source that never fought back was the retailer's own API. A smaller rebuild would start from API-first sources and treat scraping as the exception.

I'm Maxat, a software engineer. Since this project I have been testing architecture research against real repositories in the Measuring software architecture series. Corrections and questions are welcome; email me.

Stack appendix

  • Scraper: TypeScript, Playwright, Cheerio, Zod, Crawl4AI, gpt-4o-mini
  • API: Go, Gin, Uber FX, Huma, GORM, MySQL, Google OAuth
  • Infrastructure: Hetzner Cloud, Terraform, Ansible, Docker, GitHub Actions
  • Observability: Prometheus, Grafana, Loki, Promtail