Web data underpins a surprising amount of commercial decision-making: competitor pricing, marketplace assortment, tender notices, commodity quotes, regulatory filings. The information is public and collecting it reliably at scale is genuinely difficult, which is why most in-house attempts stall at the point where the script works on a laptop and fails in production.
This is the architecture we run at up to twelve million pages a week, and the disciplines that keep it working for years rather than weeks.
The architecture that survives
The frontier is the heart of it: a queue of URLs with priority, next-fetch time and retry state. Priority is driven by business value and volatility — a competitor price on a fast-moving SKU refetched hourly, a company profile page monthly. This is what lets a system scale to millions of pages without crawling everything constantly.
Workers pull from the frontier and fetch. Parsing is deliberately strict: every field has an expected type and validation rule, and a page yielding a null where a value is expected raises a parse failure rather than storing a blank.
Storage keeps the raw HTML alongside the parsed record. When a parser is later found wrong, or a new field becomes interesting, history can be reprocessed rather than recollected — faster, and considerably more polite to the source.
Browser or plain HTTP
Always prefer plain HTTP where it works. A headless browser costs roughly an order of magnitude more per page in CPU and memory, and a great deal of content that appears dynamic is actually server-rendered and available with a single request.
Check before assuming. View source rather than inspecting the DOM — if the data is in the initial HTML, no browser is needed. Check whether the site has an internal JSON API that the page itself calls, which is frequently cleaner to consume than the rendered markup.
Where a browser is genuinely required, Playwright over Selenium for new work: auto-waiting removes the largest source of flakiness, browser contexts make concurrency cheap in memory terms, network interception lets you block images and trackers entirely, and the trace viewer makes a failure diagnosable rather than mysterious.
In practice
Politeness is a design constraint
A collection system that hammers a target will be blocked, and it deserves to be. We configure per-domain concurrency and delay based on the target's size and observed response behaviour, back off automatically when latency rises or error rates increase, honour robots directives and crawl-delay, and prefer sitemap-driven discovery where a site offers one.
This is not only ethics. Polite collection is more reliable collection — a system tuned to the target's tolerance runs for years, while an aggressive one enters a cycle of blocking, evasion and re-blocking that consumes engineering time indefinitely.
Where a site has clearly signalled that it does not want automated access, the correct response is to stop and, where a commercial relationship is plausible, ask about an API or a data licence. Several of our clients now receive data by agreement, which is cheaper and far more stable for everyone.
The monitoring that actually matters
Per-field fill rate against a historical baseline. This single metric catches the failure mode that destroys datasets: the system keeps running, keeps reporting success, and quietly returns nulls for weeks after a layout change.
A price field dropping from 99% populated to 40% should trigger an alert within the hour. Distribution shift matters too — if a numeric field's median moves by an order of magnitude, something has changed in the parsing even if the field is still populated.
Alongside that: parse success rate per source, page size anomalies, response code patterns, and duration trends. A source that has been getting slower for three weeks is heading somewhere, and seeing it early is cheaper than discovering it at capacity.
Every catastrophic scraping failure we have fixed had the same shape
The system reported success while returning empty fields, for weeks, until someone noticed a gap in a report. Fill-rate monitoring against a baseline catches it within the hour. It is the single highest-value thing you can build.
Proxies, and where the line is
Proxies serve two legitimate purposes: distributing load so no single exit point burdens a target, and accessing geographically varied content, since a marketplace legitimately shows different pricing and availability in different regions.
Manage pools with health monitoring and automatic rotation away from degraded exits. Match the pool type to the need — datacentre exits for high volume on tolerant targets, residential where geography genuinely matters.
What proxies should not be is a way to evade a target's explicit refusal. If a site has deployed aggressive anti-bot measures, that is a signal about intent, and escalating an arms race against it is both ethically questionable and a poor use of engineering budget.
The legal questions your counsel will ask
We are not lawyers and this is not legal advice, but we have had this conversation many times and the questions land in consistent places.
Is the data publicly accessible without circumventing an access control? Publicly accessible factual data collected at a polite rate is the least contentious position. Does it include personal data? If so, India's Digital Personal Data Protection Act is in scope regardless of public availability. Is the content factual or copyrightable? Prices and availability are facts; articles and images are not. Did access require accepting terms of use? That may create contractual obligations independent of copyright.
Operationally we document the basis for each source, keep collection within stated limits, avoid personal data unless the client has established a lawful basis, retain raw evidence of what was collected and when, and stop immediately on a request from a target. Where a source looks contentious we say so before building rather than after.
| Factor | Lower risk | Higher risk |
|---|---|---|
| Access | Public, no login required | Behind authentication you are not entitled to |
| Content type | Facts — prices, availability, dates | Articles, images, substantial creative text |
| Personal data | None collected | Names, contacts, identifiers — DPDP applies |
| Rate | Polite, backing off on latency | Aggressive enough to affect the target |
| Terms | No acceptance required for access | Terms accepted to gain access |
Key takeaways
- A prioritised frontier is what allows scale without crawling everything constantly.
- Prefer plain HTTP; a browser costs roughly ten times as much per page and is often unnecessary.
- Politeness is a reliability strategy as much as an ethical one — polite systems run for years.
- Per-field fill-rate monitoring against a baseline is the single highest-value thing to build.
- Retain raw HTML so a fixed parser can be applied retroactively and no permanent gap results.
Frequently asked
There is no blanket answer and we are not lawyers. The practical position: collecting publicly accessible factual information without circumventing access controls, at a polite rate, is the least contentious activity. Risk rises with personal data, with copyrightable content rather than facts, and where terms of use were accepted to gain access. We document the basis for every source and will tell you before building if a target looks contentious.
Fill-rate monitoring detects it, usually within the hour, rather than waiting for a crash. Median repair in working hours is under two. Because raw HTML is retained, the repaired parser can be applied retroactively to the broken window, so in most cases there is no permanent gap in the dataset.
Mostly by not triggering them — polite pacing, realistic browser behaviour, sensible session handling and geographically appropriate exits avoid the great majority. Where a site deploys aggressive measures, we read that as intent and recommend approaching them for an API or data agreement rather than escalating. Several clients now buy data by arrangement, which is cheaper and far more stable.
Yes. We deploy on your Kubernetes cluster or cloud account so collected data never leaves your environment, and provide the code, deployment manifests and runbooks. We can operate it or hand it to your team.