Add bounded website feed discovery

AGENTS.md

@@ -28,6 +28,13 @@ structure and preserves the original document; `json_entry_normalizer.rb` maps
 JSON fields into the same immutable entry type. Keep format-specific extraction
 separate and use the existing JSON standard library dependency.
 
+`http_client.rb` is shared by ordinary fetching and website discovery. Explicit
+network policies enable bounded requests; preserve legacy fetch defaults.
+`request_policy.rb` checks resolved addresses and pins the connection target.
+`discovery.rb` uses optional Nokogiri HTML5 parsing for head metadata. Keep core
+parsing usable without Nokogiri, and verify new transport behavior with local
+servers and controlled DNS/connection fixtures.
+
 **Tag Syntax** (extend via `SimpleRSS.item_tags <<`):
 - `tag` - simple element extraction
 - `tag#attr` - attribute value (e.g., `media:content#url` → `media_content_url`)

CHANGELOG.md

@@ -7,6 +7,18 @@ repository begins with a 1.1 import, so earlier releases are not reconstructed.
 
 ## Unreleased
 
+- Add `SimpleRSS.discover` for advertised RSS, Atom, and JSON Feed links, using
+  optional Nokogiri HTML5 parsing. Return ordered, deduplicated candidates with
+  titles, type hints, and verification status; recognize direct empty feeds.
+  Default bare domains and protocol-relative discovery inputs to HTTPS.
+  Apply destination checks, address pinning, redirect credential stripping,
+  total HTTP timeouts, and streamed wire/decompressed-body limits. Existing
+  `fetch` callers can opt into the shared bounded transport with `network_policy`;
+  ordinary fetching retains its behavior. Include a tested Feedbag alternative.
+  ([#61](https://github.com/cardmagic/simple-rss/issues/61))
+- Accept self-closing empty RSS channels and Atom feeds in `parse` and `fetch`,
+  allowing direct-feed discovery to recognize them without requiring items.
+
 - Parse JSON Feed 1.0 and 1.1 through `parse` and `fetch`, using the same
   normalized entry interface as RSS/Atom. Support titleless and empty feeds,
   inherited authors, opaque/numeric IDs, separate content and dates, tags, and

Gemfile

@@ -4,6 +4,8 @@ gemspec
 
 group :test do
   gem "simplecov", require: false
+  gem "nokogiri", ">= 1.16", "< 2", require: false
+  gem "feedbag", require: false
 end
 
 group :development do

README.md

@@ -11,9 +11,10 @@ A simple, flexible, extensible, and liberal RSS, Atom, and JSON Feed reader for
 - Parses RSS, Atom, and JSON Feed 1.0/1.1
 - Tolerant of malformed XML (regex-based parsing)
 - Built-in URL fetching with conditional GET support (ETags, Last-Modified)
+- Explicit website feed discovery with request limits and destination policies
 - JSON and XML serialization
 - Extensible tag definitions
-- Zero runtime dependencies
+- No mandatory runtime gem dependencies; website discovery uses optional Nokogiri
 
 ## What's New in 2.x
 
@@ -97,6 +98,189 @@ feed = SimpleRSS.fetch(
 # Returns nil if feed hasn't changed (304 Not Modified)
 ```
 
+### Discovering Feeds from a Website
+
+Add Nokogiri to applications that use discovery. Parsing and ordinary fetching
+work without it:
+
+```ruby
+gem "simple-rss"
+gem "nokogiri", ">= 1.16", "< 2"
+```
+
+`discover` uses Nokogiri's HTML5 parser, which requires CRuby. Missing HTML5
+support raises `SimpleRSS::DiscoveryDependencyError` before making a request.
+
+Bare domains and paths default to HTTPS: `SimpleRSS.discover("example.com/blog")`
+requests `https://example.com/blog`. Protocol-relative inputs such as
+`//example.com/blog` also use HTTPS. Explicit HTTP/HTTPS URLs are preserved;
+other schemes remain unsupported. Include the scheme when specifying a port.
+Discovery does not retry over HTTP if HTTPS fails.
+
+```ruby
+require "simple-rss"
+
+candidates = SimpleRSS.discover("https://example.com/blog")
+# => [{ url: "https://example.com/feed.xml", title: "News",
+#       format: :rss, media_type: "application/rss+xml",
+#       source: :html_link, verified: false }, ...]
+
+candidate = candidates.first
+if candidate
+  feed = SimpleRSS.fetch(candidate.fetch(:url), network_policy: :public)
+  feed.normalized_entries.each { |entry| puts entry.title || entry.identifier }
+else
+  puts "No advertised feeds found."
+end
+```
+
+The application chooses among candidates. HTML results follow document order;
+duplicate normalized URLs keep the first record. Fragments are removed, queries
+are preserved, and relative/protocol-relative references use the final response
+URL plus the first direct `head` base URL, when usable. Malformed, credentialed,
+and non-HTTP link URLs are ignored. An invalid first base falls back to the
+response URL; later base tags do not override it.
+
+Only direct `head` links with an `alternate` relation token and a supported
+media type are considered. Names, relation tokens, and media types are
+case-insensitive; quoted/unquoted attributes and HTML entities follow HTML5
+parsing. Supported types are `application/rss+xml`, `application/rdf+xml`,
+`application/atom+xml`, `application/feed+json`, and `application/json`.
+Scripts, comments, styles, templates, noscript content, and body links are
+excluded. HTML parsing limits tree depth and attributes per element to 128.
+
+| Candidate field | Meaning |
+| --- | --- |
+| `url` | Absolute HTTP/HTTPS URL, without a fragment |
+| `title` | Advertised title or parsed feed title; may be nil |
+| `format` | `:rss`, `:atom`, or `:json_feed` |
+| `media_type` | Advertised supported type, or canonical type for a direct feed |
+| `source` | `:html_link` for an advertisement, `:document` for a direct feed |
+| `verified` | Whether this response was successfully parsed as a recognized feed |
+
+An advertised type is a hint, and advertised destinations are not resolved or
+fetched. They may be unreachable or prohibited by the application's policy.
+Use the same network policy when fetching the chosen URL. A direct RSS, Atom,
+or JSON Feed response returns one verified candidate at its final URL, even
+when empty. Verification means SimpleRSS parsed it, not that it passed a full
+standards validator. RSS/Atom root recognition and JSON structure take
+precedence over server Content-Type. Empty self-closing RSS channels and Atom
+feeds are also parseable.
+
+Discovery makes one request plus permitted redirects. It never guesses paths,
+executes scripts, fetches candidate feeds, follows pagination, or crawls links.
+The [discovery example](examples/discover.rb) prints every candidate:
+
+```bash
+ruby -Ilib examples/discover.rb https://example.com/blog
+```
+
+#### Request limits and destination policy
+
+Discovery defaults to a 10-second total HTTP/DNS budget, at most five redirects,
+and a 2 MiB body budget. The byte budget covers both transferred and decompressed
+body data, accumulated across the redirect chain. Streaming stops when either
+budget is exceeded. Gzip and zlib-wrapped deflate are supported as single streams;
+truncated streams, trailing compressed data, unsupported content encodings, and
+partial responses fail explicitly. The time budget includes connection, TLS,
+response reads, DNS resolution, and redirects; HTML/feed parsing follows the
+bounded download.
+
+```ruby
+candidates = SimpleRSS.discover(
+  "https://example.com/blog",
+  timeout: 5,
+  max_bytes: 1_048_576,
+  max_redirects: 3,
+  headers: { "User-Agent" => "Example Feed Reader", "Accept-Language" => "en" }
+)
+```
+
+The default `network_policy: :public` checks every resolved address at every
+hop and pins an approved address for the actual connection. Mixed public/private
+DNS answers are rejected. TLS still verifies the original hostname; environment
+proxies are disabled for policy-controlled requests. The conservative policy
+excludes IPv4 private, loopback, link-local, shared, documentation, benchmark,
+multicast, and reserved blocks. IPv6 permits global unicast `2000::/3`, excluding
+special-purpose, documentation, and 6to4 ranges. Mapped/translated addresses and
+other IPv6 ranges are excluded. See the
+[IANA IPv4](https://www.iana.org/assignments/iana-ipv4-special-registry/) and
+[IPv6 registries](https://www.iana.org/assignments/iana-ipv6-special-registry/).
+
+For an application-controlled internal feed, provide a policy that returns true
+for each permitted address:
+
+```ruby
+require "ipaddr"
+
+internal_policy = lambda do |uri, address|
+  uri.hostname == "feeds.internal.example" &&
+    IPAddr.new("10.20.0.0/24").include?(address)
+end
+candidates = SimpleRSS.discover("https://feeds.internal.example/",
+                                network_policy: internal_policy)
+```
+
+`network_policy: :unrestricted` deliberately allows any destination address while
+retaining URL checks, pinning, time/byte limits, TLS verification, and redirect
+rules. Keep this choice in application configuration. Policies receive a URI and
+an IPAddr; invalid policy names raise `ArgumentError`.
+
+On cross-origin redirects, custom headers are reduced to `Accept`,
+`Accept-Language`, and `User-Agent`. Authorization, cookies, custom credential
+headers, and conditional validators are removed and are not restored on a later
+redirect back. Same-origin redirects retain them. A changed scheme or port is a
+changed origin. URL credentials and unsupported destination schemes are rejected.
+`Host`, proxy/connection/framing headers, `Range`, and `Accept-Encoding` are
+transport-controlled and cannot be supplied in policy mode. `follow_redirects:
+false` reports the initial redirect as an HTTP error during discovery.
+
+Existing `fetch(url, options)` behavior is retained unless `network_policy` is
+explicitly supplied. Opting in uses the same transport and defaults as discovery,
+without requiring Nokogiri. `max_bytes` and `max_redirects` require a network
+policy. Ordinary `fetch` still expects a feed and never performs discovery.
+Parsing a supplied string or IO remains network-free.
+
+| Outcome | Result |
+| --- | --- |
+| Successful HTML page with no supported advertisements | `[]` |
+| Non-success HTTP status, including an unsolicited 304 | `SimpleRSS::HTTPError`, with `status_code` |
+| Rejected URL or destination | `SimpleRSS::PolicyError` |
+| Redirect loop or limit | `SimpleRSS::RedirectError` |
+| Timeout | `SimpleRSS::RequestTimeout` |
+| Body size limit | `SimpleRSS::ResponseTooLarge` |
+| DNS, connection, TLS, compression, or HTTP transport failure | `SimpleRSS::RequestError` |
+| Unrecognized or unparseable response, or HTML parser limit | `SimpleRSS::DiscoveryError` |
+| Missing optional parser | `SimpleRSS::DiscoveryDependencyError` |
+
+These errors inherit from `SimpleRSSError`. `fetch` retains its existing
+`SimpleRSSError` for non-success HTTP statuses and returns nil on conditional
+304 responses, including with an explicit network policy.
+
+#### Feedbag alternative
+
+Applications already using [Feedbag](https://github.com/damog/feedbag) can keep
+it for discovery and pass its results to SimpleRSS:
+
+```ruby
+require "feedbag"
+require "simple-rss"
+
+Feedbag.find("https://example.com/blog", open_timeout: 10, read_timeout: 10).each do |url|
+  feed = SimpleRSS.fetch(url, network_policy: :public)
+  puts feed.title
+end
+```
+
+Install the separate `feedbag` gem for this recipe; it is not a SimpleRSS runtime
+dependency. The [Feedbag example](examples/feedbag.rb) and integration test cover
+this workflow. Feedbag owns its discovery transport, URL heuristics, and error
+handling; SimpleRSS's network policy applies only to the subsequent `fetch`.
+The built-in API provides candidate metadata and explicit error/limit semantics
+for applications that need them. Its acceptance corpus is in
+[test/data/discovery.html](test/data/discovery.html), with discovery and transport
+cases under `test/base/`.
+
 ### Accessing Feed Data
 
 SimpleRSS provides both RSS and Atom style accessors:

Steepfile

@@ -10,4 +10,9 @@ target :lib do
   library "json"
   library "uri"
   library "net-http"
+  library "ipaddr"
+  library "resolv"
+  library "timeout"
+  library "zlib"
+  library "openssl"
 end

examples/discover.rb

@@ -0,0 +1,14 @@
+require "simple-rss"
+
+abort "Usage: ruby -Ilib examples/discover.rb WEBSITE_URL" unless ARGV.size == 1
+
+candidates = SimpleRSS.discover(ARGV.first)
+if candidates.empty?
+  puts "No advertised feeds found."
+  exit
+end
+
+candidates.each do |candidate|
+  verification = candidate[:verified] ? "parsed feed" : "advertised link"
+  puts [candidate[:title], candidate[:format], candidate[:url], verification].compact.join(" | ")
+end

examples/feedbag.rb

@@ -0,0 +1,10 @@
+require "simple-rss"
+require "feedbag"
+
+abort "Usage: ruby -Ilib examples/feedbag.rb WEBSITE_URL" unless ARGV.size == 1
+
+Feedbag.find(ARGV.first, open_timeout: 10, read_timeout: 10).each do |url|
+  puts url
+  feed = SimpleRSS.fetch(url, network_policy: :public, timeout: 10)
+  feed&.normalized_entries&.each { |entry| puts entry.title || entry.identifier }
+end

lib/simple-rss.rb

@@ -314,8 +314,8 @@ def fetch(url, options = {})
       require "net/http"
       require "uri"
 
-      uri = URI.parse(url)
-      response, final_uri = perform_fetch(uri, options)
+      require_relative "simple-rss/http_client"
+      response, final_uri = HTTPClient.new(options).get(url)
 
       return nil if response.is_a?(Net::HTTPNotModified)
 
@@ -328,61 +328,10 @@ def fetch(url, options = {})
       feed
     end
 
-    private
-
-    # @rbs (untyped, Hash[Symbol, untyped]) -> untyped
-    def perform_fetch(uri, options)
-      http = build_http(uri, options)
-      request = build_request(uri, options)
-
-      response = http.request(request)
-      handle_redirect(response, uri, options) || [response, uri]
-    end
-
-    # @rbs (untyped, Hash[Symbol, untyped]) -> untyped
-    def build_http(uri, options)
-      host = uri.host || raise(SimpleRSSError, "Invalid URL: missing host")
-      http = Net::HTTP.new(host, uri.port)
-      http.use_ssl = uri.scheme == "https"
-
-      timeout = options[:timeout]
-      if timeout
-        http.open_timeout = timeout
-        http.read_timeout = timeout
-      end
-
-      http
-    end
-
-    # @rbs (untyped, Hash[Symbol, untyped]) -> untyped
-    def build_request(uri, options)
-      request = Net::HTTP::Get.new(uri)
-      request["Accept"] = "application/feed+json, application/rss+xml, application/atom+xml, application/json, application/xml, text/xml, */*"
-      request["User-Agent"] = "SimpleRSS/#{VERSION}"
-
-      # Conditional GET headers
-      request["If-None-Match"] = options[:etag] if options[:etag]
-      request["If-Modified-Since"] = options[:last_modified] if options[:last_modified]
-
-      # Custom headers
-      options[:headers]&.each { |key, value| request[key] = value }
-
-      request
-    end
-
-    # @rbs (untyped, untyped, Hash[Symbol, untyped]) -> untyped
-    def handle_redirect(response, uri, options)
-      return nil unless response.is_a?(Net::HTTPRedirection)
-      return nil if options[:follow_redirects] == false
-
-      location = response["Location"]
-      return nil unless location
-
-      redirects = (options[:_redirects] || 0) + 1
-      raise SimpleRSSError, "Too many redirects" if redirects > 5
-
-      new_options = options.merge(_redirects: redirects)
-      perform_fetch(URI.join(uri.to_s, location), new_options)
+    # @rbs (String, ?Hash[Symbol, untyped]) -> Array[Hash[Symbol, untyped]]
+    def discover(url, options = {})
+      require_relative "simple-rss/discovery"
+      Discovery.new(options).discover(url)
     end
   end
 
@@ -426,7 +375,7 @@ def parse
 
   # @rbs () -> void
   def parse_xml
-    raise SimpleRSSError, "Poorly formatted feed" unless @source =~ %r{<(channel|feed).*?>.*?</(channel|feed)>}mi
+    raise SimpleRSSError, "Poorly formatted feed" unless @source =~ %r{<(channel|feed).*?>.*?</(channel|feed)>|<(channel|feed)\b[^>]*?/\s*>}mi
 
     # Feed's title and link
     feed_content = Regexp.last_match(1) if @source =~ %r{(.*?)<(rss:|atom:)?(item|entry).*?>.*?</(rss:|atom:)?(item|entry)>}mi
@@ -962,3 +911,5 @@ def unescape(content)
 
 class SimpleRSSError < StandardError # rubocop:disable Style/OneClassPerFile
 end
+
+require_relative "simple-rss/request_errors"