Resolve relative inline links to fully-qualified URLs during scrape - #129
Open
arimu1 wants to merge 2 commits into
Open
Resolve relative inline links to fully-qualified URLs during scrape#129arimu1 wants to merge 2 commits into
arimu1 wants to merge 2 commits into
Conversation
Relative hrefs from source HTML (e.g. /self-hosted/latest/migration/entire-database/) were being carried verbatim into the markdown, producing links that LLMs can't resolve on their own. Rewrite <a href> links against each page's URL before converting to markdown, across the Tiger, PostGIS, and Postgres ingest pipelines. Closes timescale#12
The spider method had its own divergent implementation of link resolution (conditional rewrite + debug counter) alongside the shared ingest.utils.beautiful_soup.resolve_relative_links (unconditional rewrite) used by the other two pipelines. Same name, same job, two bodies that could silently drift. The spider method is now a thin wrapper that delegates to the shared util and keeps its debug-level resolved-link count.
Author
|
Addressed the review feedback:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #12
When scraping documentation pages, relative
<a href>links in the source HTML (e.g./self-hosted/latest/migration/entire-database/) were carried verbatim into the generated markdown. Per the issue, this produces broken-looking inline links when an LLM quotes the returned markdown chunks, since the link target isn't resolvable without the source page's base URL.This resolves relative links against the HTML source (per the issue's suggestion), before the markdown conversion step, across all three ingest pipelines:
ingest/tiger_docs.py— newresolve_relative_linksmethod onSitemapMarkdownSpider, called inparse()with the current page'sresponse.urlas the base, right after the existingstrip_data_imagesstep.ingest/postgis_docs.py— calls the new sharedresolve_relative_linksutil with each page's resolvedfull_urlas the base.ingest/postgres_docs.py— calls the same shared util with a per-page URL built fromPOSTGRES_BASE_URL, the version, and the page'sslug(threadedversionintobuild_markdown, which didn't have it before).ingest/utils/beautiful_soup.py— adds the sharedresolve_relative_links(soup, base_url)helper (urljoinwas already imported here but unused).Already-absolute links,
mailto:/tel:links, etc. are left untouched byurljoin.Note: this also expands
#fragment-only anchors (same-page links) to fully-qualified URLs, sinceurljointreats them like any other relative reference. The targets are correct, and it's arguably desirable in a standalone markdown corpus, but PostgreSQL/PostGIS docs are anchor-heavy, so it will visibly increase diff volume in the regenerated output — flagging so it's not a surprise.Test plan
No Python test harness exists in this repo (
ingest/has no test files and CI (.github/workflows/test.yml) only runsbun test), so I verified behavior manually against the exact example from the issue:urljoinuvx ruff check ./uvx ruff format --check .show no new issues introduced by this change (diffed againstmain, all remaining warnings pre-exist onmain)python -m astparses all four changed files without error🤖 Generated with Claude Code