HTML scraping is sometimes necessary, but it should not be the first choice when the data already exists as JSON. A structured endpoint is easier to validate, less sensitive to layout changes, and usually clearer about pagination and field names.
HTTPX is a modern Python HTTP client with synchronous and asynchronous APIs. For a small script, create a client, set a timeout, request the endpoint, check the status, and decode JSON. For repeated calls, reuse the client so connections can be pooled.
Treat the response as untrusted input. Check the content type, handle malformed JSON, validate required fields, and record the endpoint’s status code. Respect authentication, rate limits, robots policies where relevant, and the service’s terms. Never place an access token directly in a repository.
An API may use cursor pagination rather than page numbers. Read the documented next-cursor field and stop when no cursor remains. Add retry logic only for temporary failures, and make sure a retried request will not create duplicate side effects.
Direct JSON access is not a license to bypass access controls. Use public or authorized APIs, identify your client when required, and cache responses when the data does not need to be live.