Validate pages and follow links within a boundary

JSON syntax is weaker than an application schema

A JSON parser can produce dictionaries, lists, strings, numbers, Booleans and null values. A syntactically valid object such as {"message":"sign in"} is not an asset page. Validate required keys, types and domains before adding anything to the collected population.

Our parser also rejects duplicate JSON properties and non-standard numeric constants. Some parsers accept duplicate properties by taking a later value, and Python's default JSON behavior permits special numeric constants beyond strict JSON. Those convenient behaviors would make a strict total contract ambiguous. The code explicitly chooses rejection.

The Content-Type check prevents an obvious representation mismatch such as a 200 HTML login page. It does not prove that a JSON body is trustworthy or that a provider's header is truthful. Schema and identity checks still apply. A malformed UTF-8 body is rejected instead of decoded with replacement characters that could alter an identifier.

A maximum accepted body size limits what this parsing layer accepts, but the offline transport already holds bytes in memory. A real HTTP adapter must enforce a bounded download before materializing an enormous body. Checking size after an unbounded network read is not the same as controlling memory during collection.

Understand cursor pagination

The provider's next link is opaque collection state. Follow the documented link or token rather than guessing that page2 can be incremented to page3. Offset and cursor APIs have different contracts, and a provider may require preserving particular query parameters across pages.

This fictional API allows only the cursor query parameter with one non-empty value of bounded length. The client accepts a next link only if it uses HTTPS, the exact approved origin and the same resource path. It rejects credential-bearing or foreign origins, unexpected paths, duplicate cursor parameters and fragments. These are deliberate limits of the lab adapter, not a universal rule that every API uses this URL structure.

A real provider may return relative links, separate cursor tokens or links containing additional fixed query parameters. Extend the adapter from its documentation and test the accepted forms. Do not relax origin checks merely because a first attempt fails. Authentication belongs to a defined destination.

Detect cycles by meaningful state

A server bug can return a next link that repeats a page already visited. A loop that keeps requesting that link can run indefinitely or accumulate duplicate records. The client records the accepted cursor state and rejects a repeat before another request.

It compares the parsed cursor state rather than raw URL spelling. Encoding differences can make two URL strings look different while selecting the same cursor. Our contract's single cursor parameter makes that canonical state simple. More complex provider state needs a carefully defined canonical request identity.

The client also has a finite page budget. A non-repeating sequence of bad cursors should not escape the cycle check and continue forever. Under the default budget it accepts at most ten pages, with at most three attempts per page. Hitting a budget causes collection failure, not a successful shortened inventory.

Reconcile records and advertised totals

After reaching a terminal page, the number of distinct accepted items must equal the advertised total. The total must remain unchanged across pages. More collected items than the declared total causes immediate rejection. A duplicate item identity also causes rejection; the code does not silently overwrite or remove it.

These choices make changes during collection visible, but they do not guarantee a transactionally consistent snapshot. A provider can return three different valid records while keeping total 3. A directory could change between pages. A provider's snapshot token, versioning guarantees or collection design would be needed for stronger consistency.

A completed authorized collection is also not automatically an organization-wide inventory. The token may be scoped to one group or business unit. The advertised total is the population visible to this operation. Preserve query and authorization-scope evidence when connecting the collector to an operational report.

Page-fault exercise

Trace four proposed second pages. Page A repeats A-002 instead of supplying A-003. Page B supplies A-003 but changes total from 3 to 4. Page C supplies no items, null next and total 3. Page D supplies A-003 but sends a next URL on another origin.

All four should fail the collection contract. A is duplicate identity, B is inconsistent total, C is terminal count mismatch, and D is an unapproved next link. In D, the client must not request the foreign URL. A test that merely checks the final exception misses that important boundary; inspect transport-call history too.

Evidence laboratory — attempt before reading the solution

A mocked ticket API receives POST create-ticket with client reference R-71. The server creates ticket T-9, but the response is lost. The client sees a timeout. The fictional API supports a documented idempotency key: identical requests using the same key return the original result during a stated retention window. A faulty client generates a new key for every retry. Explain the uncertainty and design a correct retry sequence within the stated window.

Official reference for further study