Add bounded website feed discovery

[cardmagic]

Add SimpleRSS.discover so an application can start with a website URL, choose among advertised RSS/Atom/JSON feeds, and then use the existing parser. Discovery returns candidate metadata and makes only the initial request plus bounded redirects.

Closes #61. Based on latest master at 3c1f114, including JSON Feed support from #66.

Example: discover, choose, and fetch

Add the optional HTML parser to the application:

gem "simple-rss"
gem "nokogiri", ">= 1.16", "< 2"

require "simple-rss"

candidates = SimpleRSS.discover("https://example.com/blog")
# => [{ url: "https://example.com/feed.xml", title: "News",
#       format: :rss, media_type: "application/rss+xml",
#       source: :html_link, verified: false }, ...]

candidate = candidates.first
if candidate
  feed = SimpleRSS.fetch(candidate.fetch(:url), network_policy: :public)
  feed.normalized_entries.each { |entry| puts entry.title || entry.identifier }
else
  puts "No advertised feeds found."
end

The application chooses a candidate. Advertised type hints are unverified; a direct feed response is parsed and returned with source: :document and verified: true, including empty feeds.

Bare domains now default to HTTPS:

SimpleRSS.discover("lucascarlson.net")
# => [{ url: "https://lucascarlson.net/feed.xml", title: "Lucas Carlson",
#       format: :rss, media_type: "application/rss+xml",
#       source: :html_link, verified: false }]

This exact call was verified live. Bare domains with paths and protocol-relative inputs also use HTTPS. Explicit schemes retain their meaning and unsupported schemes are rejected. Include the scheme when specifying a port; discovery does not fall back to HTTP after an HTTPS failure.

Example: configure request limits

candidates = SimpleRSS.discover(
  "https://example.com/blog",
  timeout: 5,
  max_bytes: 1_048_576,
  max_redirects: 3,
  headers: { "User-Agent" => "Example Reader", "Accept-Language" => "en" }
)

Defaults are a 10-second total HTTP/DNS budget, five redirects, and 2 MiB of transferred/decompressed body data across the redirect chain. Candidate URLs, pagination, articles, scripts, and assets are not fetched.

An application can explicitly configure an internal-feed policy:

require "ipaddr"

internal_policy = lambda do |uri, address|
  uri.hostname == "feeds.internal.example" &&
    IPAddr.new("10.20.0.0/24").include?(address)
end
SimpleRSS.discover("https://feeds.internal.example/", network_policy: internal_policy)

Behavior and compatibility

• Nokogiri HTML5 handles head metadata, attribute syntax/case, entities, HTML bases, and script/comment/template isolation. It is optional and requires CRuby; core parsing and fetching retain zero mandatory runtime gem dependencies.
• Return all distinct supported advertised URLs in source order. Preserve titles, format hints, and meaningful queries; remove fragments. No path guessing or crawling.
• Share one HTTP client between fetch and discovery. Ordinary fetch retains its defaults; network_policy: :public opts into the same bounded transport as discovery.
• Check every DNS answer at every hop, reject prohibited addresses, and pin the approved connection address while retaining TLS hostname verification. Policy mode disables environment proxies and automatic retries.
• Strip credentials and conditional validators across origins. Validate URLs and controlled headers; allow an application-owned address policy or explicit :unrestricted policy.
• Stream single gzip/deflate responses with both wire and decompressed byte limits. Reject truncated responses, trailing compressed data, and unsupported partial responses. Preserve conditional 304 handling, including representation-encoding headers on bodyless responses.
• Return [] for successful pages with no supported advertisements; expose separate HTTP, policy, timeout, redirect, size, transport, dependency, and parsing errors.
• Accept self-closing empty RSS channels and Atom feeds in the shared parser.
• Include README contracts, an acceptance corpus, runnable discovery and Feedbag examples, a tested Feedbag integration recipe, and unreleased changelog notes.

Validation

• Reproduced missing discovery API, self-closing empty-feed rejection, truncated Content-Length acceptance, and bodyless 304 decompression errors before fixing them.
• Full suite: 228 tests, 1,035 assertions, zero failures/errors; three existing external-network omissions.
• Local servers exercise discovery ordering, format recognition, redirects, cross-origin headers, byte limits, gzip/deflate, timeouts, and conditional requests.
• Controlled DNS/socket fixtures cover public/private/mixed answers, rebinding across redirects, and connection pinning without contacting real internal services.
• A local TLS server verifies that a pinned connection accepts the original hostname and rejects a different hostname.
• A fresh Ruby process verifies core parsing works without loading Nokogiri and missing discovery dependencies fail before HTTP requests.
• RuboCop, generated RBS validation, and Steep pass. Built gem contains the implementation and examples with no mandatory runtime gem dependencies.

[cardmagic]

@greptileai Please review current head 334fac1fda76b44dfdc28fb5ac54c6d249b0892a for #61, including optional HTML parsing, candidate verification, shared transport compatibility, destination policy and DNS pinning, TLS hostname verification, cross-origin headers, decompressed body limits, and timeout/redirect behavior. Please provide an updated confidence score.

[cardmagic]

Hound is applying defaults that conflict with this repository’s .rubocop.yml: double quotes are required, the line limit is 160, and OrderedGems, FrozenStringLiteralComment, and Documentation are disabled. All 16 comments concern those configuration differences; the project’s RuboCop check passes locally and in CI. Resolving these threads according to the committed configuration.

[greptile-apps]

Greptile Summary

Adds bounded website feed discovery backed by the shared policy-controlled HTTP transport.

• Defaults bare domains and protocol-relative discovery inputs to HTTPS while preserving explicit schemes.
• Discovers ordered RSS, Atom, and JSON Feed advertisements or verifies direct feed documents.
• Enforces destination policy, redirect, timeout, header, and compressed/decompressed body limits.
• Keeps Nokogiri optional so core parsing and ordinary fetching retain no mandatory runtime gem dependency.
• Extends parsing to accept self-closing empty RSS and Atom containers.

Confidence Score: 5/5

The current head appears safe to merge with no outstanding correctness, security, or repository-rule findings.

The added HTTPS normalization matches the documented discovery contract and remains subject to the existing URL validation, destination policy, TLS hostname verification, redirect bounds, and request limits. The previous conditional-structure thread is manually resolved, and no new actionable issue remains.

Important Files Changed

Filename	Overview
lib/simple-rss/discovery.rb	Implements feed discovery and now normalizes bare-domain and protocol-relative inputs to HTTPS.
lib/simple-rss/http_client.rb	Provides the shared bounded HTTP transport with redirect, timeout, byte-limit, compression, and header controls.
lib/simple-rss/request_policy.rb	Validates HTTP destinations, checks all resolved addresses, and supplies a policy-approved pinned address.
lib/simple-rss.rb	Exposes discovery through the public API, shares the HTTP client with fetch, and accepts self-closing empty feed containers.
test/base/discovery_transport_test.rb	Covers HTTPS normalization, TLS hostname verification, destination policy, redirects, headers, limits, and transport failures.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Discovery input] --> B{Explicit scheme?}
  B -- No --> C[Prepend HTTPS]
  B -- Yes --> D[Retain scheme]
  C --> E[Validate URL and destination policy]
  D --> E
  E --> F[Bounded HTTP request]
  F --> G{Redirect?}
  G -- Yes --> H[Validate next hop and enforce redirect limit]
  H --> F
  G -- No --> I{Direct feed?}
  I -- Yes --> J[Return verified document candidate]
  I -- No --> K[Parse HTML head with Nokogiri HTML5]
  K --> L[Return ordered deduplicated advertised candidates]

Loading

_{Reviews (3): Last reviewed commit: "feat: default bare discovery domains to ..." | Re-trigger Greptile}

[cardmagic]

@greptileai Please review current head 2d5b2ae6401ae8336a0e0c47a1b6fc5df0d00aef. The nested redirect-limit branch is now a single guard with the exception class selected once before the loop. Full suite still passes (228 tests, 1,002 assertions), along with RuboCop, RBS validation, and Steep. Please confirm the finding is addressed and provide an updated confidence score for this SHA.

[cardmagic]

The new Hound comments also use defaults instead of .rubocop.yml: the committed limits are 160 for line length, 80 for ABC size, 60 for method length, and 30 for cyclomatic complexity. Each reported value is below its configured limit; the remaining comments prefer single quotes where this project requires double quotes. Resolving these configuration mismatches; CI lint and all tests pass on 2d5b2ae.

[cardmagic]

@greptileai Please review current head 9f0858e4f5102f2f2a0c1d315c95a36c4d0e1351. Discovery now prepends HTTPS to bare domains and protocol-relative inputs, as requested. Explicit schemes retain existing validation; the policy-controlled HTTP client is unchanged. Local TLS regressions cover bare domains, paths and queries, and protocol-relative inputs; prohibited-address and unsupported-scheme rejection remain covered. All 228 tests / 1,035 assertions, RuboCop, RBS validation, and Steep pass. The live SimpleRSS.discover("lucascarlson.net") call returns its RSS feed. Please provide an updated confidence score for this SHA.

[cardmagic]

The latest 16 Hound threads have the same configuration mismatch: .rubocop.yml requires double quotes, disables Documentation, ClassAndModuleChildren, and FrozenStringLiteralComment, and sets metric limits above every reported value (line 160, ABC 80, method 60, cyclomatic 30, perceived 35). Resolving those threads according to the committed configuration. RuboCop and all CI checks pass on 9f0858e.