Skip to content

feat: rewrite hyndburn as a plain http scraper - #2312

Open
InertiaUK wants to merge 1 commit into
robbrad:masterfrom
InertiaUK:feat/hyndburn-http-scraper
Open

InertiaUK wants to merge 1 commit into
robbrad:masterfrom
InertiaUK:feat/hyndburn-http-scraper

Conversation

@InertiaUK

Copy link
Copy Markdown
Contributor

Summary

The Selenium version keeps timing out. The collection day page is a React front end over iTouchVision's JSON endpoints, the same ones the Newport scraper already talks to (same AES key and IV), so I've dropped the browser altogether.

The scraper reads the client from the uuid in the URL (gdsv5/util/igetclientdetails), resolves the address with kmbd/address when no UPRN is given, then fetches kmbd/collectionDay. UPRN and postcode plus house number both work. I've removed web_driver from the input.json entry and updated the wiki note.

Testing

  • UPRN 100010448773 (input.json): 6 collections
  • BB1 4DJ / 50: 6 collections

@coderabbitai

coderabbitai Bot commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 59 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 2 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: cf30e022-ee85-4393-a631-16475a15edd4
Commits

Reviewing files that changed from the base of the PR and between 0d1b26a and 0dac7d9.

Files selected for processing: 2
  • uk_bin_collection/tests/input.json
  • uk_bin_collection/uk_bin_collection/councils/HyndburnBoroughCouncil.py
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Oct 26, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.57%. Comparing base (0d1b26a) to head (0dac7d9).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #2312   +/-   ##
=======================================
  Coverage   83.57%   83.57%           
=======================================
  Files          43       43           
  Lines        3380     3380           
=======================================
  Hits         2825     2825           
  Misses        555      555           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant