Guide · 9 min read
Why web scrapers fail on complex websites
Most scrapers work on day one. The trouble starts on day thirty: an empty result that nobody notices, a redesign that breaks every selector, a session that expires halfway through a run. Here are the seven ways complex websites defeat scrapers, why teams lose the maintenance fight, and what it takes to keep collection running.
Published · NexqData engineering
| What breaks | What you see | What keeps it working |
|---|---|---|
| 1. The data isn't in the HTML | An empty page, or a loading spinner where the data should be. | Collect from the data layer the application itself calls. |
| 2. The record is four screens deep | Works in testing, fails partway through a real run. | Model each step, and check where the agent landed after every one. |
| 3. Sessions expire mid-run | The first pages work; later ones return errors or the login screen. | Treat the session lifecycle as part of the design, not an afterthought. |
| 4. The source protects its capacity | Rate-limit errors, challenge pages, or a sudden block. | Pace to what the source can sustain, and decline what isn't feasible. |
| 5. Layouts change without notice | Zero records, no error. A quiet failure looks like a quiet day. | Validate output and compare every run with its baseline. |
| 6. Pagination at volume | Counts that are close but never quite right. | Measure completeness against the expected total, not “reached the last page.” |
| 7. The record is spread across pages | Flat rows with missing children, or duplicates. | Follow the relationships and deliver one structured record. |
The scraper that works on day one
A first scraper is quick to write. You open the page, find the elements that hold the data, write a few selectors, and the rows come back. On a simple, static site that may be all it ever needs.
The sources that matter to a business are rarely simple. They’re product catalogues rendered by JavaScript, booking engines behind multi-step searches, government registries built on decade-old frameworks, supplier portals behind a login. On those, the first version is the easy part. What takes the time is keeping it correct through every redesign, outage, and quiet change the source makes over the following months.
Each of the seven problems below is manageable on its own. Most production sources have several at once, and court records have nearly all of them.
Seven ways complex websites break scrapers
1. The data isn’t in the HTML
Modern sites are applications. The HTML your HTTP client downloads is a shell, and the records arrive afterwards, through API calls the page makes once JavaScript runs. A scraper that parses the initial HTML finds nothing.
Running a full headless browser for every page works, but it’s slow, expensive at volume, and still breaks when the page’s structure moves. Usually the more robust approach is to find the data layer the application itself calls and collect from that, because the data behind a page changes less often than its appearance. It still changes, though, so it still needs watching.
Where you meet it · retail and pricing pages, single-page dashboards, travel search results, listings that load as you scroll.
2. The record is four screens deep
The data you need often sits behind a journey: fill in a search form, page through a results list, open a detail view, then follow a link to related records. Every step is a place a run can go wrong, and a scraper that assumes each step succeeded will happily parse an error page as if it were a result.
Reliable agents model the journey explicitly. After each step they check where they actually landed, so a missing results page is reported as a failure instead of becoming an empty dataset.
Where you meet it · government registries, booking and availability searches, licensing lookups, B2B supplier catalogues.
3. Sessions expire mid-run
Many sources tie every request to the one before it: a session cookie, a form token, a server-side view state. Some expire after a few minutes. Others invalidate the session if requests arrive in an order a person wouldn’t produce. The first hundred pages work, then the rest come back as errors or the sign-in screen.
The session lifecycle has to be designed in: when it starts, how it’s kept valid, and how a run recovers without starting over. Where a source needs credentials, they belong in a secure store, never hard-coded in the scraper.
Where you meet it · partner and supplier portals, legacy enterprise web apps, account-based marketplaces, court and permit systems.
4. The source protects its capacity
Websites limit how much traffic they accept. Rate limits, challenge pages, and web application firewalls exist because a source has finite capacity, and a collector that ignores that fails, or gets turned away. Many teams discover this only in production, when a job that worked at ten requests a minute is blocked at a thousand.
The answer isn’t to push harder. It’s to assess a source before committing to it: what it permits, what cadence it can sustain, and whether collecting it is appropriate at all. Then pace collection to match. Sometimes the right answer is that a source shouldn’t be collected, and saying so early is cheaper than finding out after the pipeline depends on it.
Where you meet it · high-traffic marketplaces, airline and hotel sites, ticketing, any source with a published rate limit.
5. Layouts change without notice
This is the most expensive failure, because it’s silent. A site renames a CSS class, moves a field, or restyles its results page. The scraper doesn’t crash. It finds no matches, writes zero rows, and exits successfully. Downstream, an empty day looks exactly like a quiet day, until someone notices the numbers are wrong weeks later.
Catching it takes more than checking that the job ran. You validate what came back: expected fields present, record counts in a plausible range, and runtime compared with the source’s normal behaviour. A run that finishes in forty seconds when it usually takes six minutes is telling you something, even if it reported success.
6. Pagination at volume
At a few pages, pagination is trivial. At thousands it isn’t: results shift while you page through them, cursors expire, the last page is reported wrongly, and one dropped page among ten thousand is invisible in the totals. The collected count ends up close to the real one, but never quite right.
Volume needs concurrency tuned to each source, and completion measured against the number of records the source says it has, not “we reached a page with no next link.”
Where you meet it · job boards, real estate listings, product catalogues, public filings and registries.
7. The record is spread across pages
Real records have structure. A listing has photos and price history, a company has officers and filings, a product has variants and sellers. Each part lives on a different page or comes from a different request. Flatten it carelessly and you get duplicate rows, orphaned children, and joins that only work on the sample you tested.
The collector should follow those relationships and deliver one structured record with its children attached, so whoever uses the data doesn’t have to rebuild the source’s structure afterwards.
Where you meet it · company and financial filings, property records, marketplaces with sellers and variants, healthcare provider directories.
Why teams lose the maintenance fight
None of this is unknown to experienced engineers. Teams still lose, because the cost is in maintenance, and maintenance grows with every source you add.
Each source changes on its own schedule. If a source breaks a few times a year, that’s easy to absorb. Thirty sources at that rate means something is broken most weeks, usually discovered late, and often by the people who use the data rather than the people who collect it. The engineer who wrote the scraper has moved on, the fix competes with roadmap work, and collection quietly becomes the team’s most fragile system.
Three things make the difference between a scraper and a collection system you can depend on:
- Detection. Every run is checked, so a break is caught on the first bad run, not discovered downstream.
- Ownership. Someone is responsible for fixing each source, with the context to do it quickly.
- Versioning. A fix ships as a new version you can trace, not an untracked edit to a script on someone’s machine.
A checklist for reliable collection
Whether you build it yourself or bring someone in, a collection setup that holds up in production does these things:
- 1
Assess each source first
Know what it permits, what cadence it can sustain, and what you'd decline, before anything depends on it.
- 2
Engineer per source
One collector per source, built around how that source actually works. Generic templates break on the first unusual page.
- 3
Validate every run
Check fields, record counts, and completeness. An empty result is a finding, not a success.
- 4
Compare against a baseline
Flag runs that are unusually slow or fast. Runtime drift is often the first sign a source changed.
- 5
Retry what's transient, surface what isn't
Timeouts deserve a retry. A changed page deserves a person, quickly.
- 6
Pace to the source
Tune concurrency per source, and back off when it signals strain.
- 7
Version and maintain
Treat collectors as software: reviewed, versioned, and updated when the source changes.
- 8
Deliver structured data
Records with their relationships intact, in a format your systems can take directly.
How NexqData handles it
We build and run collection for the sources teams find hardest to keep working. Our engineers assess each source, build one agent for it, and maintain that agent as the source changes. When a source changes, we ship a new agent version; you don’t file a ticket to keep it alive.
Every run is monitored. Failures are surfaced for attention, and runs that are unusually slow or fast against an agent’s normal runtime are flagged, which is how silent breaks get caught. You start agents from the dashboard, on a schedule, or from your own code through the API, and you can get a webhook when a job finishes instead of polling.
The details of how we approach each kind of complexity are on complex sources, and the API is documented at docs.nexqdata.io.