diff --git a/AGENTS.md b/AGENTS.md index 623aebc..a5cd79e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -28,6 +28,13 @@ structure and preserves the original document; `json_entry_normalizer.rb` maps JSON fields into the same immutable entry type. Keep format-specific extraction separate and use the existing JSON standard library dependency. +`http_client.rb` is shared by ordinary fetching and website discovery. Explicit +network policies enable bounded requests; preserve legacy fetch defaults. +`request_policy.rb` checks resolved addresses and pins the connection target. +`discovery.rb` uses optional Nokogiri HTML5 parsing for head metadata. Keep core +parsing usable without Nokogiri, and verify new transport behavior with local +servers and controlled DNS/connection fixtures. + **Tag Syntax** (extend via `SimpleRSS.item_tags <<`): - `tag` - simple element extraction - `tag#attr` - attribute value (e.g., `media:content#url` → `media_content_url`) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4512fb4..935f219 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,18 @@ repository begins with a 1.1 import, so earlier releases are not reconstructed. ## Unreleased +- Add `SimpleRSS.discover` for advertised RSS, Atom, and JSON Feed links, using + optional Nokogiri HTML5 parsing. Return ordered, deduplicated candidates with + titles, type hints, and verification status; recognize direct empty feeds. + Default bare domains and protocol-relative discovery inputs to HTTPS. + Apply destination checks, address pinning, redirect credential stripping, + total HTTP timeouts, and streamed wire/decompressed-body limits. Existing + `fetch` callers can opt into the shared bounded transport with `network_policy`; + ordinary fetching retains its behavior. Include a tested Feedbag alternative. + ([#61](https://github.com/cardmagic/simple-rss/issues/61)) +- Accept self-closing empty RSS channels and Atom feeds in `parse` and `fetch`, + allowing direct-feed discovery to recognize them without requiring items. + - Parse JSON Feed 1.0 and 1.1 through `parse` and `fetch`, using the same normalized entry interface as RSS/Atom. Support titleless and empty feeds, inherited authors, opaque/numeric IDs, separate content and dates, tags, and diff --git a/Gemfile b/Gemfile index 4acddc1..286db53 100644 --- a/Gemfile +++ b/Gemfile @@ -4,6 +4,8 @@ gemspec group :test do gem "simplecov", require: false + gem "nokogiri", ">= 1.16", "< 2", require: false + gem "feedbag", require: false end group :development do diff --git a/README.md b/README.md index cd658e3..ed6fd6d 100644 --- a/README.md +++ b/README.md @@ -11,9 +11,10 @@ A simple, flexible, extensible, and liberal RSS, Atom, and JSON Feed reader for - Parses RSS, Atom, and JSON Feed 1.0/1.1 - Tolerant of malformed XML (regex-based parsing) - Built-in URL fetching with conditional GET support (ETags, Last-Modified) +- Explicit website feed discovery with request limits and destination policies - JSON and XML serialization - Extensible tag definitions -- Zero runtime dependencies +- No mandatory runtime gem dependencies; website discovery uses optional Nokogiri ## What's New in 2.x @@ -97,6 +98,189 @@ feed = SimpleRSS.fetch( # Returns nil if feed hasn't changed (304 Not Modified) ``` +### Discovering Feeds from a Website + +Add Nokogiri to applications that use discovery. Parsing and ordinary fetching +work without it: + +```ruby +gem "simple-rss" +gem "nokogiri", ">= 1.16", "< 2" +``` + +`discover` uses Nokogiri's HTML5 parser, which requires CRuby. Missing HTML5 +support raises `SimpleRSS::DiscoveryDependencyError` before making a request. + +Bare domains and paths default to HTTPS: `SimpleRSS.discover("example.com/blog")` +requests `https://example.com/blog`. Protocol-relative inputs such as +`//example.com/blog` also use HTTPS. Explicit HTTP/HTTPS URLs are preserved; +other schemes remain unsupported. Include the scheme when specifying a port. +Discovery does not retry over HTTP if HTTPS fails. + +```ruby +require "simple-rss" + +candidates = SimpleRSS.discover("https://example.com/blog") +# => [{ url: "https://example.com/feed.xml", title: "News", +# format: :rss, media_type: "application/rss+xml", +# source: :html_link, verified: false }, ...] + +candidate = candidates.first +if candidate + feed = SimpleRSS.fetch(candidate.fetch(:url), network_policy: :public) + feed.normalized_entries.each { |entry| puts entry.title || entry.identifier } +else + puts "No advertised feeds found." +end +``` + +The application chooses among candidates. HTML results follow document order; +duplicate normalized URLs keep the first record. Fragments are removed, queries +are preserved, and relative/protocol-relative references use the final response +URL plus the first direct `head` base URL, when usable. Malformed, credentialed, +and non-HTTP link URLs are ignored. An invalid first base falls back to the +response URL; later base tags do not override it. + +Only direct `head` links with an `alternate` relation token and a supported +media type are considered. Names, relation tokens, and media types are +case-insensitive; quoted/unquoted attributes and HTML entities follow HTML5 +parsing. Supported types are `application/rss+xml`, `application/rdf+xml`, +`application/atom+xml`, `application/feed+json`, and `application/json`. +Scripts, comments, styles, templates, noscript content, and body links are +excluded. HTML parsing limits tree depth and attributes per element to 128. + +| Candidate field | Meaning | +| --- | --- | +| `url` | Absolute HTTP/HTTPS URL, without a fragment | +| `title` | Advertised title or parsed feed title; may be nil | +| `format` | `:rss`, `:atom`, or `:json_feed` | +| `media_type` | Advertised supported type, or canonical type for a direct feed | +| `source` | `:html_link` for an advertisement, `:document` for a direct feed | +| `verified` | Whether this response was successfully parsed as a recognized feed | + +An advertised type is a hint, and advertised destinations are not resolved or +fetched. They may be unreachable or prohibited by the application's policy. +Use the same network policy when fetching the chosen URL. A direct RSS, Atom, +or JSON Feed response returns one verified candidate at its final URL, even +when empty. Verification means SimpleRSS parsed it, not that it passed a full +standards validator. RSS/Atom root recognition and JSON structure take +precedence over server Content-Type. Empty self-closing RSS channels and Atom +feeds are also parseable. + +Discovery makes one request plus permitted redirects. It never guesses paths, +executes scripts, fetches candidate feeds, follows pagination, or crawls links. +The [discovery example](examples/discover.rb) prints every candidate: + +```bash +ruby -Ilib examples/discover.rb https://example.com/blog +``` + +#### Request limits and destination policy + +Discovery defaults to a 10-second total HTTP/DNS budget, at most five redirects, +and a 2 MiB body budget. The byte budget covers both transferred and decompressed +body data, accumulated across the redirect chain. Streaming stops when either +budget is exceeded. Gzip and zlib-wrapped deflate are supported as single streams; +truncated streams, trailing compressed data, unsupported content encodings, and +partial responses fail explicitly. The time budget includes connection, TLS, +response reads, DNS resolution, and redirects; HTML/feed parsing follows the +bounded download. + +```ruby +candidates = SimpleRSS.discover( + "https://example.com/blog", + timeout: 5, + max_bytes: 1_048_576, + max_redirects: 3, + headers: { "User-Agent" => "Example Feed Reader", "Accept-Language" => "en" } +) +``` + +The default `network_policy: :public` checks every resolved address at every +hop and pins an approved address for the actual connection. Mixed public/private +DNS answers are rejected. TLS still verifies the original hostname; environment +proxies are disabled for policy-controlled requests. The conservative policy +excludes IPv4 private, loopback, link-local, shared, documentation, benchmark, +multicast, and reserved blocks. IPv6 permits global unicast `2000::/3`, excluding +special-purpose, documentation, and 6to4 ranges. Mapped/translated addresses and +other IPv6 ranges are excluded. See the +[IANA IPv4](https://www.iana.org/assignments/iana-ipv4-special-registry/) and +[IPv6 registries](https://www.iana.org/assignments/iana-ipv6-special-registry/). + +For an application-controlled internal feed, provide a policy that returns true +for each permitted address: + +```ruby +require "ipaddr" + +internal_policy = lambda do |uri, address| + uri.hostname == "feeds.internal.example" && + IPAddr.new("10.20.0.0/24").include?(address) +end +candidates = SimpleRSS.discover("https://feeds.internal.example/", + network_policy: internal_policy) +``` + +`network_policy: :unrestricted` deliberately allows any destination address while +retaining URL checks, pinning, time/byte limits, TLS verification, and redirect +rules. Keep this choice in application configuration. Policies receive a URI and +an IPAddr; invalid policy names raise `ArgumentError`. + +On cross-origin redirects, custom headers are reduced to `Accept`, +`Accept-Language`, and `User-Agent`. Authorization, cookies, custom credential +headers, and conditional validators are removed and are not restored on a later +redirect back. Same-origin redirects retain them. A changed scheme or port is a +changed origin. URL credentials and unsupported destination schemes are rejected. +`Host`, proxy/connection/framing headers, `Range`, and `Accept-Encoding` are +transport-controlled and cannot be supplied in policy mode. `follow_redirects: +false` reports the initial redirect as an HTTP error during discovery. + +Existing `fetch(url, options)` behavior is retained unless `network_policy` is +explicitly supplied. Opting in uses the same transport and defaults as discovery, +without requiring Nokogiri. `max_bytes` and `max_redirects` require a network +policy. Ordinary `fetch` still expects a feed and never performs discovery. +Parsing a supplied string or IO remains network-free. + +| Outcome | Result | +| --- | --- | +| Successful HTML page with no supported advertisements | `[]` | +| Non-success HTTP status, including an unsolicited 304 | `SimpleRSS::HTTPError`, with `status_code` | +| Rejected URL or destination | `SimpleRSS::PolicyError` | +| Redirect loop or limit | `SimpleRSS::RedirectError` | +| Timeout | `SimpleRSS::RequestTimeout` | +| Body size limit | `SimpleRSS::ResponseTooLarge` | +| DNS, connection, TLS, compression, or HTTP transport failure | `SimpleRSS::RequestError` | +| Unrecognized or unparseable response, or HTML parser limit | `SimpleRSS::DiscoveryError` | +| Missing optional parser | `SimpleRSS::DiscoveryDependencyError` | + +These errors inherit from `SimpleRSSError`. `fetch` retains its existing +`SimpleRSSError` for non-success HTTP statuses and returns nil on conditional +304 responses, including with an explicit network policy. + +#### Feedbag alternative + +Applications already using [Feedbag](https://github.com/damog/feedbag) can keep +it for discovery and pass its results to SimpleRSS: + +```ruby +require "feedbag" +require "simple-rss" + +Feedbag.find("https://example.com/blog", open_timeout: 10, read_timeout: 10).each do |url| + feed = SimpleRSS.fetch(url, network_policy: :public) + puts feed.title +end +``` + +Install the separate `feedbag` gem for this recipe; it is not a SimpleRSS runtime +dependency. The [Feedbag example](examples/feedbag.rb) and integration test cover +this workflow. Feedbag owns its discovery transport, URL heuristics, and error +handling; SimpleRSS's network policy applies only to the subsequent `fetch`. +The built-in API provides candidate metadata and explicit error/limit semantics +for applications that need them. Its acceptance corpus is in +[test/data/discovery.html](test/data/discovery.html), with discovery and transport +cases under `test/base/`. + ### Accessing Feed Data SimpleRSS provides both RSS and Atom style accessors: diff --git a/Steepfile b/Steepfile index aea3ace..5d628a1 100644 --- a/Steepfile +++ b/Steepfile @@ -10,4 +10,9 @@ target :lib do library "json" library "uri" library "net-http" + library "ipaddr" + library "resolv" + library "timeout" + library "zlib" + library "openssl" end diff --git a/examples/discover.rb b/examples/discover.rb new file mode 100644 index 0000000..59d65e6 --- /dev/null +++ b/examples/discover.rb @@ -0,0 +1,14 @@ +require "simple-rss" + +abort "Usage: ruby -Ilib examples/discover.rb WEBSITE_URL" unless ARGV.size == 1 + +candidates = SimpleRSS.discover(ARGV.first) +if candidates.empty? + puts "No advertised feeds found." + exit +end + +candidates.each do |candidate| + verification = candidate[:verified] ? "parsed feed" : "advertised link" + puts [candidate[:title], candidate[:format], candidate[:url], verification].compact.join(" | ") +end diff --git a/examples/feedbag.rb b/examples/feedbag.rb new file mode 100644 index 0000000..b895b0c --- /dev/null +++ b/examples/feedbag.rb @@ -0,0 +1,10 @@ +require "simple-rss" +require "feedbag" + +abort "Usage: ruby -Ilib examples/feedbag.rb WEBSITE_URL" unless ARGV.size == 1 + +Feedbag.find(ARGV.first, open_timeout: 10, read_timeout: 10).each do |url| + puts url + feed = SimpleRSS.fetch(url, network_policy: :public, timeout: 10) + feed&.normalized_entries&.each { |entry| puts entry.title || entry.identifier } +end diff --git a/lib/simple-rss.rb b/lib/simple-rss.rb index 906a826..92f0d2b 100644 --- a/lib/simple-rss.rb +++ b/lib/simple-rss.rb @@ -314,8 +314,8 @@ def fetch(url, options = {}) require "net/http" require "uri" - uri = URI.parse(url) - response, final_uri = perform_fetch(uri, options) + require_relative "simple-rss/http_client" + response, final_uri = HTTPClient.new(options).get(url) return nil if response.is_a?(Net::HTTPNotModified) @@ -328,61 +328,10 @@ def fetch(url, options = {}) feed end - private - - # @rbs (untyped, Hash[Symbol, untyped]) -> untyped - def perform_fetch(uri, options) - http = build_http(uri, options) - request = build_request(uri, options) - - response = http.request(request) - handle_redirect(response, uri, options) || [response, uri] - end - - # @rbs (untyped, Hash[Symbol, untyped]) -> untyped - def build_http(uri, options) - host = uri.host || raise(SimpleRSSError, "Invalid URL: missing host") - http = Net::HTTP.new(host, uri.port) - http.use_ssl = uri.scheme == "https" - - timeout = options[:timeout] - if timeout - http.open_timeout = timeout - http.read_timeout = timeout - end - - http - end - - # @rbs (untyped, Hash[Symbol, untyped]) -> untyped - def build_request(uri, options) - request = Net::HTTP::Get.new(uri) - request["Accept"] = "application/feed+json, application/rss+xml, application/atom+xml, application/json, application/xml, text/xml, */*" - request["User-Agent"] = "SimpleRSS/#{VERSION}" - - # Conditional GET headers - request["If-None-Match"] = options[:etag] if options[:etag] - request["If-Modified-Since"] = options[:last_modified] if options[:last_modified] - - # Custom headers - options[:headers]&.each { |key, value| request[key] = value } - - request - end - - # @rbs (untyped, untyped, Hash[Symbol, untyped]) -> untyped - def handle_redirect(response, uri, options) - return nil unless response.is_a?(Net::HTTPRedirection) - return nil if options[:follow_redirects] == false - - location = response["Location"] - return nil unless location - - redirects = (options[:_redirects] || 0) + 1 - raise SimpleRSSError, "Too many redirects" if redirects > 5 - - new_options = options.merge(_redirects: redirects) - perform_fetch(URI.join(uri.to_s, location), new_options) + # @rbs (String, ?Hash[Symbol, untyped]) -> Array[Hash[Symbol, untyped]] + def discover(url, options = {}) + require_relative "simple-rss/discovery" + Discovery.new(options).discover(url) end end @@ -426,7 +375,7 @@ def parse # @rbs () -> void def parse_xml - raise SimpleRSSError, "Poorly formatted feed" unless @source =~ %r{<(channel|feed).*?>.*?}mi + raise SimpleRSSError, "Poorly formatted feed" unless @source =~ %r{<(channel|feed).*?>.*?|<(channel|feed)\b[^>]*?/\s*>}mi # Feed's title and link feed_content = Regexp.last_match(1) if @source =~ %r{(.*?)<(rss:|atom:)?(item|entry).*?>.*?}mi @@ -962,3 +911,5 @@ def unescape(content) class SimpleRSSError < StandardError # rubocop:disable Style/OneClassPerFile end + +require_relative "simple-rss/request_errors" diff --git a/lib/simple-rss/discovery.rb b/lib/simple-rss/discovery.rb new file mode 100644 index 0000000..c5bc205 --- /dev/null +++ b/lib/simple-rss/discovery.rb @@ -0,0 +1,105 @@ +# rbs_inline: enabled + +require_relative "http_client" + +class SimpleRSS::Discovery + MEDIA_TYPES = { + "application/rss+xml" => :rss, "application/rdf+xml" => :rss, + "application/atom+xml" => :atom, "application/feed+json" => :json_feed, "application/json" => :json_feed + }.freeze + + # @rbs @options: Hash[Symbol, untyped] + # @rbs @parser: untyped + + # @rbs (Hash[Symbol, untyped]) -> void + def initialize(options) + @options = { network_policy: :public }.merge(options) + require "nokogiri" + @parser = Object.const_get(:Nokogiri) + unless @parser.const_defined?(:HTML5) + raise SimpleRSS::DiscoveryDependencyError, "Discovery requires Nokogiri HTML5 support (available on CRuby)" + end + rescue LoadError + raise SimpleRSS::DiscoveryDependencyError, 'Install the optional "nokogiri" gem (>= 1.16, < 2) to use SimpleRSS.discover' + end + + # @rbs (String) -> Array[Hash[Symbol, untyped]] + def discover(url) + url = "https:#{url}" if url.start_with?("//") + url = "https://#{url}" unless url.match?(/\A[a-z][a-z\d+.-]*:/i) + response, uri = SimpleRSS::HTTPClient.new(@options).get(url) + raise SimpleRSS::HTTPError, response.code.to_i unless response.is_a?(Net::HTTPSuccess) + + candidates(response.body, uri, response.content_type, response.type_params["charset"]) + end + + private + + # @rbs (String, untyped, String?, String?) -> Array[Hash[Symbol, untyped]] + def candidates(body, uri, media_type, encoding) + prefix = body.b.sub(/\A\xEF\xBB\xBF/n, "").lstrip + return [feed_candidate(body, uri, :json_feed)] if prefix.start_with?("{", "[") + + document = @parser::XML.parse(body, uri.to_s, nil, @parser::XML::ParseOptions::NONET | @parser::XML::ParseOptions::RECOVER) + root = document.root + format = root && xml_format(root) + return [feed_candidate(body, uri, format)] if format + + unless root&.name&.casecmp?("html") || %w[text/html application/xhtml+xml].include?(media_type) || prefix.match?(/\A(?: e + raise SimpleRSS::DiscoveryError, "Cannot parse discovery response: #{e.message}" + end + + # @rbs (untyped) -> Symbol? + def xml_format(root) + return :rss if root.name == "rss" && root.namespace.nil? + return :atom if root.name == "feed" && [SimpleRSS::ATOM_NAMESPACE, "http://purl.org/atom/ns#"].include?(root.namespace&.href) + return :rss if root.name == "RDF" && root.namespace&.href == "http://www.w3.org/1999/02/22-rdf-syntax-ns#" + + nil + end + + # @rbs (String, untyped, Symbol) -> Hash[Symbol, untyped] + def feed_candidate(body, uri, format) + feed = SimpleRSS.parse(body, source_url: uri.to_s) + title = feed.instance_variable_get(:@title) + { url: uri.to_s, title: title, format: format, media_type: MEDIA_TYPES.key(format), source: :document, verified: true } + end + + # @rbs (String, untyped, String?) -> Array[Hash[Symbol, untyped]] + def html_candidates(body, uri, encoding) + document = @parser::HTML5.parse(body, uri.to_s, encoding, max_tree_depth: 128, max_attributes: 128) + head = document.at_css("html > head") + return [] unless head + + base = resolve_url(head.at_xpath("./base[@href]")&.[]("href"), uri) || uri + candidates = head.xpath("./link[@rel][@type][@href]").filter_map do |link| + next unless link["rel"].downcase.split.include?("alternate") + + media_type = link["type"].split(";", 2).first.to_s.strip.downcase + format = MEDIA_TYPES[media_type] + next unless format + + target = resolve_url(link["href"], base) + next unless target + + { url: target.to_s, title: link["title"], format: format, media_type: media_type, source: :html_link, verified: false } + end + candidates.uniq { |candidate| candidate[:url] } + end + + # @rbs (String?, untyped) -> untyped + def resolve_url(value, base) + return if value.nil? || value.strip.empty? + + SimpleRSS::RequestPolicy.parse_url(URI.join(base.to_s, value.strip).to_s) + rescue URI::Error, SimpleRSS::PolicyError + nil + end +end diff --git a/lib/simple-rss/http_client.rb b/lib/simple-rss/http_client.rb new file mode 100644 index 0000000..e79e0f5 --- /dev/null +++ b/lib/simple-rss/http_client.rb @@ -0,0 +1,216 @@ +# rbs_inline: enabled + +require "net/http" +require "timeout" +require "openssl" +require_relative "request_errors" +require_relative "request_policy" + +class SimpleRSS::HTTPClient + ACCEPT = "application/feed+json, application/rss+xml, application/atom+xml, application/json, application/xml, text/xml, */*".freeze + REDIRECT_CODES = %w[301 302 303 307 308].freeze + CROSS_ORIGIN_HEADERS = %w[accept accept-language user-agent].freeze + PROTECTED_HEADERS = %w[host connection proxy-authorization proxy-connection accept-encoding range transfer-encoding content-length te trailer upgrade + expect].freeze + + # @rbs @options: Hash[Symbol, untyped] + # @rbs @headers: Hash[untyped, untyped] + # @rbs @policy: SimpleRSS::RequestPolicy? + # @rbs @remaining_bytes: Integer + # @rbs @remaining_wire_bytes: Integer + # @rbs @redirect_limit: Integer + # @rbs @timeout: untyped + + # @rbs (Hash[Symbol, untyped]) -> void + def initialize(options) + @options = options.dup + @headers = (@options[:headers] || {}).dup + @policy = @options.key?(:network_policy) ? SimpleRSS::RequestPolicy.new(@options[:network_policy]) : nil + @remaining_bytes = @options.fetch(:max_bytes, 2 * 1024 * 1024) + @remaining_wire_bytes = @remaining_bytes + @redirect_limit = @options.fetch(:max_redirects, 5) + @timeout = @options.fetch(:timeout, @policy ? 10 : nil) + validate_options + end + + # @rbs (String) -> [untyped, untyped] + def get(url) + return perform(URI.parse(url)) unless @policy + + Timeout.timeout(@timeout, SimpleRSS::RequestTimeout, "HTTP operation exceeded its timeout") do + perform(SimpleRSS::RequestPolicy.parse_url(url)) + end + rescue Timeout::Error + raise unless @policy + + raise SimpleRSS::RequestTimeout, "HTTP operation exceeded its timeout" + rescue IOError, SystemCallError, SocketError, OpenSSL::SSL::SSLError, Net::HTTPBadResponse, Net::HTTPHeaderSyntaxError, Resolv::ResolvError, Zlib::Error => e + raise unless @policy + + raise SimpleRSS::RequestError, "HTTP transport failed: #{e.message}" + end + + private + + # @rbs () -> void + def validate_options + if !@policy && (@options.key?(:max_bytes) || @options.key?(:max_redirects)) + raise ArgumentError, "max_bytes and max_redirects require network_policy" + end + return unless @policy + + unless @timeout.is_a?(Numeric) && @timeout.real? && @timeout.finite? && @timeout.positive? + raise ArgumentError, "timeout must be a positive finite number" + end + unless @remaining_bytes.is_a?(Integer) && @remaining_bytes.positive? + raise ArgumentError, "max_bytes must be a positive integer" + end + unless @redirect_limit.is_a?(Integer) && @redirect_limit >= 0 + raise ArgumentError, "max_redirects must be a nonnegative integer" + end + raise ArgumentError, "headers must be a hash" unless @headers.is_a?(Hash) + + @headers.each do |name, value| + if PROTECTED_HEADERS.include?(name.to_s.downcase) + raise ArgumentError, "The transport controls the #{name} header" + end + unless name.to_s.match?(/\A[!#$%&'*+.^_`|~0-9A-Za-z-]+\z/) && value.is_a?(String) && !value.match?(/[\r\n\x00]/) + raise ArgumentError, "Invalid request header" + end + end + end + + # @rbs (untyped) -> [untyped, untyped] + def perform(uri) + visited = {} #: Hash[String, bool] + redirects = @policy ? 0 : (@options[:_redirects] || 0) + redirect_error = @policy ? SimpleRSS::RedirectError : SimpleRSSError + loop do + raise SimpleRSS::RedirectError, "Redirect loop detected" if @policy && visited[uri.to_s] + + visited[uri.to_s] = true + response = request(uri) + return [response, uri] unless redirect?(response) + + location = response["Location"] + return [response, uri] unless location + + redirects += 1 + raise redirect_error, "Too many redirects" if redirects > @redirect_limit + + next_uri = URI.join(uri.to_s, location) + next_uri = SimpleRSS::RequestPolicy.parse_url(next_uri.to_s) if @policy + strip_credentials if @policy && origin(uri) != origin(next_uri) + uri = next_uri + end + rescue URI::Error + raise unless @policy + + raise SimpleRSS::PolicyError, "Malformed redirect URL" + end + + # @rbs (untyped) -> bool + def redirect?(response) + return false if @options[:follow_redirects] == false + return REDIRECT_CODES.include?(response.code) if @policy + + response.is_a?(Net::HTTPRedirection) + end + + # @rbs (untyped) -> untyped + def request(uri) + http = build_http(uri) + request = Net::HTTP::Get.new(uri) + request["Accept"] = ACCEPT + request["User-Agent"] = "SimpleRSS/#{SimpleRSS::VERSION}" + request["If-None-Match"] = @options[:etag] if @options[:etag] + request["If-Modified-Since"] = @options[:last_modified] if @options[:last_modified] + @headers.each { |name, value| request[name] = value } + return http.request(request) unless @policy + + request["Accept-Encoding"] = "gzip, deflate, identity" + http.request(request) { |response| read_response(response) } + end + + # @rbs (untyped) -> untyped + def build_http(uri) + host = uri.hostname || raise(SimpleRSSError, "Invalid URL: missing host") + policy = @policy + http = policy ? Net::HTTP.new(host, uri.port, nil) : Net::HTTP.new(host, uri.port) + http.use_ssl = uri.scheme == "https" + if @timeout + http.open_timeout = @timeout + http.read_timeout = @timeout + end + return http unless policy + + http.ipaddr = policy.address(uri) + http.write_timeout = @timeout + http.max_retries = 0 + http.verify_mode = OpenSSL::SSL::VERIFY_PEER + http.verify_hostname = true + http + end + + # @rbs (untyped) -> void + def read_response(response) + return unless response.class.body_permitted? + + encoding = response["Content-Encoding"].to_s.downcase + unless ["", "identity", "none", "gzip", "x-gzip", "deflate"].include?(encoding) + raise SimpleRSS::RequestError, "Unsupported Content-Encoding" + end + raise SimpleRSS::RequestError, "Partial HTTP responses are not supported" if response["Content-Range"] || response.code == "206" + + body = +"".b + wire_bytes = 0 + inflater = Zlib::Inflate.new(32 + Zlib::MAX_WBITS) if %w[gzip x-gzip deflate].include?(encoding) + response.read_body do |chunk| + if chunk.bytesize > @remaining_wire_bytes + raise SimpleRSS::ResponseTooLarge, "HTTP response bodies exceeded max_bytes" + end + + @remaining_wire_bytes -= chunk.bytesize + wire_bytes += chunk.bytesize + unless inflater + append_body(body, chunk) + next + end + inflater.inflate(chunk) { |decoded| append_body(body, decoded) } + raise SimpleRSS::RequestError, "Trailing data in compressed HTTP response" if inflater.total_in != wire_bytes + end + raise SimpleRSS::RequestError, "Incomplete compressed HTTP response" if inflater && !inflater.finished? + + declared_length = response.content_length + if declared_length && !response.chunked? && wire_bytes != declared_length + raise SimpleRSS::RequestError, "Incomplete HTTP response body" + end + + response.delete("Content-Encoding") + response["Content-Length"] = body.bytesize.to_s if declared_length + response.body = body + ensure + inflater&.close + end + + # @rbs (String, String) -> nil + def append_body(body, chunk) + raise SimpleRSS::ResponseTooLarge, "HTTP response bodies exceeded max_bytes" if chunk.bytesize > @remaining_bytes + + @remaining_bytes -= chunk.bytesize + body << chunk + nil + end + + # @rbs (untyped) -> Array[untyped] + def origin(uri) + [uri.scheme.downcase, uri.hostname.downcase, uri.port] + end + + # @rbs () -> void + def strip_credentials + @headers = @headers.select { |name, _value| CROSS_ORIGIN_HEADERS.include?(name.to_s.downcase) } + @options.delete(:etag) + @options.delete(:last_modified) + end +end diff --git a/lib/simple-rss/request_errors.rb b/lib/simple-rss/request_errors.rb new file mode 100644 index 0000000..1b902b5 --- /dev/null +++ b/lib/simple-rss/request_errors.rb @@ -0,0 +1,21 @@ +# rbs_inline: enabled + +class SimpleRSS + class RequestError < SimpleRSSError; end + class PolicyError < RequestError; end + class RequestTimeout < RequestError; end + class ResponseTooLarge < RequestError; end + class RedirectError < RequestError; end + class DiscoveryError < SimpleRSSError; end + class DiscoveryDependencyError < DiscoveryError; end + + class HTTPError < RequestError + attr_reader :status_code #: Integer + + # @rbs (Integer) -> void + def initialize(status_code) + @status_code = status_code + super("HTTP #{status_code} during feed discovery") + end + end +end diff --git a/lib/simple-rss/request_policy.rb b/lib/simple-rss/request_policy.rb new file mode 100644 index 0000000..f3fa219 --- /dev/null +++ b/lib/simple-rss/request_policy.rb @@ -0,0 +1,72 @@ +# rbs_inline: enabled + +require "ipaddr" +require "resolv" +require "uri" + +class SimpleRSS::RequestPolicy + BLOCKED_IPV4 = %w[ + 0.0.0.0/8 10.0.0.0/8 100.64.0.0/10 127.0.0.0/8 169.254.0.0/16 172.16.0.0/12 + 192.0.0.0/24 192.0.2.0/24 192.88.99.0/24 192.168.0.0/16 198.18.0.0/15 + 198.51.100.0/24 203.0.113.0/24 224.0.0.0/4 240.0.0.0/4 + ].map { |range| IPAddr.new(range).freeze }.freeze + GLOBAL_IPV6 = IPAddr.new("2000::/3").freeze + BLOCKED_IPV6 = %w[2001::/23 2001:db8::/32 2002::/16 3fff::/20].map { |range| IPAddr.new(range).freeze }.freeze + + # @rbs @policy: untyped + + # @rbs (untyped) -> void + def initialize(policy) + unless %i[public unrestricted].include?(policy) || policy.respond_to?(:call) + raise ArgumentError, "network_policy must be :public, :unrestricted, or a callable" + end + + @policy = policy + end + + # @rbs (String) -> untyped + def self.parse_url(value) + uri = URI.parse(value) + raise SimpleRSS::PolicyError, "Expected an absolute HTTP or HTTPS URL" unless uri.is_a?(URI::HTTP) + + unless uri.hostname && !uri.hostname.to_s.empty? && uri.port&.between?(1, 65_535) + raise SimpleRSS::PolicyError, "Expected a valid host and port" + end + raise SimpleRSS::PolicyError, "Credentials in URLs are not supported" if uri.userinfo + + uri.fragment = nil + uri.path = "/" if uri.path.to_s.empty? + uri.normalize + rescue URI::Error, TypeError + raise SimpleRSS::PolicyError, "Malformed HTTP or HTTPS URL" + end + + # @rbs (untyped) -> String + def address(uri) + addresses = resolve(uri.hostname.to_s) + raise SimpleRSS::RequestError, "No destination addresses were resolved" if addresses.empty? + unless addresses.all? { |address| allowed?(uri, address) } + raise SimpleRSS::PolicyError, "Destination address is prohibited by network_policy" + end + + (addresses.find(&:ipv4?) || addresses.fetch(0)).to_s + end + + private + + # @rbs (String) -> Array[untyped] + def resolve(hostname) + [IPAddr.new(hostname)] + rescue IPAddr::InvalidAddressError + Resolv.getaddresses(hostname).uniq.map { |address| IPAddr.new(address) } + end + + # @rbs (untyped, untyped) -> bool + def allowed?(uri, address) + return true if @policy == :unrestricted + return @policy.call(uri.dup.freeze, address.dup.freeze) == true if @policy.respond_to?(:call) + return BLOCKED_IPV4.none? { |range| range.include?(address) } if address.ipv4? + + GLOBAL_IPV6.include?(address) && BLOCKED_IPV6.none? { |range| range.include?(address) } + end +end diff --git a/test/base/discovery_dependency_test.rb b/test/base/discovery_dependency_test.rb new file mode 100644 index 0000000..2dfb3c5 --- /dev/null +++ b/test/base/discovery_dependency_test.rb @@ -0,0 +1,29 @@ +require "test_helper" +require "open3" + +class DiscoveryDependencyTest < Test::Unit::TestCase + def test_core_parsing_works_without_loading_nokogiri_and_discovery_explains_the_dependency + script = <<~RUBY_SCRIPT + require "simple-rss" + abort "Nokogiri loaded by core" if defined?(Nokogiri) + Kernel.prepend(Module.new do + def require(path) + raise LoadError, "optional dependency is unavailable" if path == "nokogiri" + + super + end + end) + feed = SimpleRSS.parse('Example') + abort "Core parsing failed" unless feed.title == "Example" + begin + SimpleRSS.discover("http://127.0.0.1/", timeout: 0.1) + abort "Expected a dependency error" + rescue SimpleRSS::DiscoveryDependencyError => error + puts error.message + end + RUBY_SCRIPT + output, status = Open3.capture2e(RbConfig.ruby, "-Ilib", "-e", script) + assert status.success?, output + assert_include output, 'Install the optional "nokogiri" gem' + end +end diff --git a/test/base/discovery_test.rb b/test/base/discovery_test.rb new file mode 100644 index 0000000..d631c99 --- /dev/null +++ b/test/base/discovery_test.rb @@ -0,0 +1,151 @@ +require "test_helper" +require_relative "../support/http_server" + +class DiscoveryTest < Test::Unit::TestCase + include HTTPServer + + def test_discovers_advertised_feeds_without_fetching_them + html = '' + with_server([[200, { "Content-Type" => "text/html" }, html]]) do |url, requests| + candidates = SimpleRSS.discover(url, network_policy: :unrestricted, timeout: 1) + assert_equal [{ url: "#{url}/feed.xml", title: "News", format: :rss, media_type: "application/rss+xml", source: :html_link, verified: false }], candidates + assert_equal 1, requests.size + end + end + + def test_acceptance_corpus_preserves_formats_order_titles_and_distinct_queries + html = File.read(File.join(__dir__, "../data/discovery.html")) + with_server([[200, { "Content-Type" => "text/html; charset=utf-8" }, html]]) do |url, requests| + candidates = SimpleRSS.discover("#{url}/blog/index.html", network_policy: :unrestricted, timeout: 1) + assert_equal(["news.xml?edition=1&lang=en", "atom.xml", "feed.json", "legacy.json", "feed.rdf", "news.xml?edition=2&lang=en"], candidates.map { |candidate| candidate[:url].delete_prefix("#{url}/syndication/") }) + assert_equal(%i[rss atom json_feed json_feed rss rss], candidates.map { |candidate| candidate[:format] }) + assert_equal "News & updates", candidates.first[:title] + assert_equal "application/rss+xml", candidates.first[:media_type] + assert_equal [false], candidates.map { |candidate| candidate[:verified] }.uniq + assert_equal [:html_link], candidates.map { |candidate| candidate[:source] }.uniq + assert_equal 1, requests.size + end + end + + def test_direct_empty_feeds_are_verified_without_instance_valid + sources = { + rss: 'RSS', + atom: 'Atom', + json_feed: '{"version":"https://jsonfeed.org/version/1.1","title":"JSON Feed","items":[]}' + } + sources.each do |format, body| + with_server([[200, { "Content-Type" => "text/plain" }, body]]) do |url, requests| + candidate = SimpleRSS.discover(url, network_policy: :unrestricted, timeout: 1).first + assert_equal "#{url}/", candidate[:url] + assert_equal format, candidate[:format] + assert_equal :document, candidate[:source] + assert_equal true, candidate[:verified] + assert_equal 1, requests.size + end + end + end + + def test_empty_self_closing_feed_containers_are_recognized + ['', ''].each do |body| + with_server([[200, {}, body]]) do |url, _requests| + candidates = SimpleRSS.discover(url, network_policy: :unrestricted, timeout: 1) + assert_equal true, candidates.first[:verified] + assert_empty SimpleRSS.parse(body).normalized_entries + end + end + end + + def test_rdf_feed_and_xhtml_metadata_are_supported + rdf = 'RDF' + with_server([[200, {}, rdf]]) do |url, _requests| + candidate = SimpleRSS.discover(url, network_policy: :unrestricted, timeout: 1).first + assert_equal :rss, candidate[:format] + assert_equal "RDF", candidate[:title] + end + xhtml = '' + with_server([[200, { "Content-Type" => "application/xhtml+xml" }, xhtml]]) do |url, _requests| + assert_equal(["#{url}/feed.xml"], SimpleRSS.discover(url, network_policy: :unrestricted, timeout: 1).map { |candidate| candidate[:url] }) + end + end + + def test_redirects_base_urls_and_protocol_relative_links_preserve_queries + html = '' + responses = [[302, { "Location" => "../pages/index?view=1#head" }, ""], [200, { "Content-Type" => "text/html" }, html]] + with_server(responses) do |url, requests| + candidates = SimpleRSS.discover("#{url}/start/page", network_policy: :unrestricted, timeout: 1) + assert_equal(["#{url}/feeds/news.xml?edition=1", "http://example.com/atom?q=1"], candidates.map { |candidate| candidate[:url] }) + assert_equal ["GET /start/page HTTP/1.1", "GET /pages/index?view=1 HTTP/1.1"], requests.map(&:first) + end + end + + def test_invalid_links_are_ignored_and_invalid_first_base_uses_document_url + html = '' \ + '' \ + '' \ + '' \ + '' \ + '' \ + '' + with_server([[200, { "Content-Type" => "text/html" }, html]]) do |url, requests| + assert_equal(["#{url}/valid.xml"], SimpleRSS.discover(url, network_policy: :unrestricted, timeout: 1).map { |candidate| candidate[:url] }) + assert_equal 1, requests.size + end + end + + def test_omitted_head_tags_and_html_entities_are_parsed_as_html + html = 'Example

Post

' + with_server([[200, { "Content-Type" => "text/html; charset=utf-8" }, html]]) do |url, _requests| + candidate = SimpleRSS.discover(url, network_policy: :unrestricted, timeout: 1).first + assert_equal "#{url}/feed?q=1&b=2", candidate[:url] + assert_equal "Café & ☀", candidate[:title] + end + end + + def test_no_feeds_is_distinct_from_http_and_parse_failures + with_server([[200, { "Content-Type" => "text/html" }, "

No feeds here

"]]) do |url, _requests| + assert_empty SimpleRSS.discover(url, network_policy: :unrestricted, timeout: 1) + end + with_server([[404, { "Content-Type" => "text/html" }, "

Not found

"]]) do |url, _requests| + error = assert_raise(SimpleRSS::HTTPError) { SimpleRSS.discover(url, network_policy: :unrestricted, timeout: 1) } + assert_equal 404, error.status_code + end + ['{"version":', '{"hello":"world"}', 'broken', "not a feed", ""].each do |body| + with_server([[200, { "Content-Type" => "application/octet-stream" }, body]]) do |url, _requests| + assert_raise(SimpleRSS::DiscoveryError, body) { SimpleRSS.discover(url, network_policy: :unrestricted, timeout: 1) } + end + end + end + + def test_feed_markup_in_html_scripts_does_not_become_a_direct_feed + html = '' + with_server([[200, { "Content-Type" => "application/rss+xml" }, html]]) do |url, _requests| + assert_empty SimpleRSS.discover(url, network_policy: :unrestricted, timeout: 1) + end + end + + def test_foreign_namespaces_do_not_fabricate_verified_feeds + ['', ''].each do |body| + with_server([[200, { "Content-Type" => "text/html" }, body]]) do |url, _requests| + assert_empty SimpleRSS.discover(url, network_policy: :unrestricted, timeout: 1) + end + end + end + + def test_parser_limits_fail_clearly + html = "" + ("
" * 140) + with_server([[200, { "Content-Type" => "text/html" }, html]]) do |url, _requests| + assert_raise(SimpleRSS::DiscoveryError) { SimpleRSS.discover(url, network_policy: :unrestricted, timeout: 1) } + end + end + + def test_advertised_private_urls_are_only_unverified_metadata + html = '' + with_server([[200, { "Content-Type" => "text/html" }, html]]) do |url, requests| + candidate = SimpleRSS.discover(url, network_policy: :unrestricted, timeout: 1).first + assert_equal false, candidate[:verified] + assert_equal "http://127.0.0.1:9/feed", candidate[:url] + assert_raise(SimpleRSS::PolicyError) { SimpleRSS.fetch(candidate[:url], network_policy: :public) } + assert_equal 1, requests.size + end + end +end diff --git a/test/base/discovery_transport_test.rb b/test/base/discovery_transport_test.rb new file mode 100644 index 0000000..fc226cb --- /dev/null +++ b/test/base/discovery_transport_test.rb @@ -0,0 +1,400 @@ +require "test_helper" +require "simple-rss/http_client" +require "zlib" +require "stringio" +require_relative "../support/http_server" +require_relative "../support/replace_method" + +class DiscoveryTransportTest < Test::Unit::TestCase + include HTTPServer + include ReplaceMethod + + PAGE = [200, { "Content-Type" => "text/html" }, "Example"].freeze + FEED = 'Example'.freeze + + def test_public_policy_rejects_prohibited_ipv4_and_ipv6_before_connecting + prohibited = %w[ + 0.0.0.0 10.0.0.1 100.64.0.1 127.0.0.1 169.254.169.254 172.16.0.1 192.0.0.1 192.0.2.1 + 192.88.99.1 192.168.1.1 198.18.0.1 198.51.100.1 203.0.113.1 224.0.0.1 255.255.255.255 + [::] [::1] [::ffff:127.0.0.1] [64:ff9b::7f00:1] [100::1] [2001::1] [2001:db8::1] + [2002:7f00:1::] [3fff::1] [fc00::1] [fe80::1] [ff02::1] + ] + with_replaced_method(TCPSocket, :open, ->(*) { flunk "Prohibited address reached the socket" }) do + prohibited.each do |host| + assert_raise(SimpleRSS::PolicyError, host) { SimpleRSS.discover("http://#{host}/", timeout: 1) } + assert_raise(SimpleRSS::PolicyError, host) { SimpleRSS.discover(host, timeout: 1) } + end + end + end + + def test_malformed_urls_and_schemes_fail_before_connecting + with_replaced_method(TCPSocket, :open, ->(*) { flunk "Invalid URL reached the socket" }) do + ["", "/relative", "ftp://example.com/feed", "file:///tmp/feed", "mailto:reader@example.com", "javascript:alert(1)", "http://", "http://[broken", "http://example.com:0", "http://example.com:65536", "https://user:password@example.com/", "user:password@example.com"].each do |url| + assert_raise(SimpleRSS::PolicyError, url) { SimpleRSS.discover(url) } + end + end + end + + def test_mixed_public_and_private_dns_answers_are_rejected + with_replaced_method(Resolv, :getaddresses, ->(_host) { ["8.8.8.8", "127.0.0.1"] }) do + with_replaced_method(TCPSocket, :open, ->(*) { flunk "Mixed DNS answers reached the socket" }) do + assert_raise(SimpleRSS::PolicyError) { SimpleRSS.discover("http://site.example/") } + end + end + end + + def test_checked_dns_address_is_pinned_and_host_header_is_preserved + resolutions = 0 + resolver = lambda do |host| + assert_equal "site.example", host + resolutions += 1 + resolutions == 1 ? ["8.8.8.8"] : ["127.0.0.1"] + end + with_server([PAGE]) do |local_url, requests| + with_pinned_connection(local_url, resolver) do |connections| + assert_empty SimpleRSS.discover("http://site.example/", timeout: 1) + assert_equal [["8.8.8.8", 80]], connections + assert_equal 1, resolutions + assert_include requests.first, "Host: site.example" + end + end + end + + def test_dns_policy_is_rechecked_on_each_redirect + resolutions = 0 + resolver = lambda do |_host| + resolutions += 1 + resolutions == 1 ? ["8.8.8.8"] : ["127.0.0.1"] + end + with_server([[302, { "Location" => "/next" }, ""]]) do |local_url, requests| + with_pinned_connection(local_url, resolver) do |connections| + assert_raise(SimpleRSS::PolicyError) { SimpleRSS.discover("http://site.example/", timeout: 1) } + assert_equal 2, resolutions + assert_equal 1, connections.size + assert_equal 1, requests.size + end + end + end + + def test_redirect_to_a_private_literal_is_blocked + with_server([[302, { "Location" => "http://127.0.0.1:9/feed" }, ""]]) do |local_url, requests| + with_pinned_connection(local_url, ->(_host) { ["8.8.8.8"] }) do |connections| + assert_raise(SimpleRSS::PolicyError) { SimpleRSS.discover("http://site.example/", timeout: 1) } + assert_equal 1, connections.size + assert_equal 1, requests.size + end + end + end + + def test_public_ipv6_addresses_are_checked_and_pinned + with_server([PAGE]) do |local_url, _requests| + with_pinned_connection(local_url, ->(_host) { ["2606:4700:4700::1111"] }) do |connections| + assert_empty SimpleRSS.discover("http://site.example/", timeout: 1) + assert_equal [["2606:4700:4700::1111", 80]], connections + end + end + end + + def test_an_application_policy_can_allow_one_internal_destination + with_server([PAGE]) do |url, requests| + policy = ->(uri, address) { uri.hostname == "127.0.0.1" && IPAddr.new("127.0.0.1/32").include?(address) } + assert_empty SimpleRSS.discover(url, network_policy: policy, timeout: 1) + assert_equal 1, requests.size + end + end + + def test_cross_origin_redirects_strip_credentials_and_keep_safe_headers + headers = { "Authorization" => "Bearer fixture", "Cookie" => "session=fixture", "X-Api-Key" => "fixture", "User-Agent" => "Feed tests", "Accept" => "text/html", "Accept-Language" => "en" } + options = { network_policy: :unrestricted, timeout: 1, headers: headers, etag: '"fixture"', last_modified: "Sat, 12 Sep 2026 10:00:00 GMT" } + with_server([PAGE]) do |destination, destination_requests| + with_server([[302, { "Location" => destination }, ""]]) do |source, source_requests| + assert_empty SimpleRSS.discover(source, options) + assert_include source_requests.first, "Authorization: Bearer fixture" + assert_include source_requests.first, "Cookie: session=fixture" + assert_include source_requests.first, 'If-None-Match: "fixture"' + assert_empty destination_requests.first.grep(/Authorization:|Cookie:|Api-Key:|If-None-Match:|If-Modified-Since:/i) + assert_include destination_requests.first, "User-Agent: Feed tests" + assert_include destination_requests.first, "Accept: text/html" + assert_include destination_requests.first, "Accept-Language: en" + assert_equal "Bearer fixture", options[:headers]["Authorization"] + assert_equal '"fixture"', options[:etag] + end + end + end + + def test_same_origin_redirects_keep_authorization + with_server([[302, { "Location" => "/next" }, ""], PAGE]) do |url, requests| + SimpleRSS.discover(url, network_policy: :unrestricted, timeout: 1, headers: { "Authorization" => "Bearer fixture" }) + requests.each { |request| assert_include request, "Authorization: Bearer fixture" } + end + end + + def test_redirect_loops_and_limits_are_bounded + with_server([[302, { "Location" => "/" }, ""]]) do |url, requests| + assert_raise(SimpleRSS::RedirectError) { SimpleRSS.discover(url, network_policy: :unrestricted, timeout: 1) } + assert_equal 1, requests.size + end + with_server([[302, { "Location" => "/next" }, ""]]) do |url, requests| + assert_raise(SimpleRSS::RedirectError) { SimpleRSS.discover(url, network_policy: :unrestricted, timeout: 1, max_redirects: 0) } + assert_equal 1, requests.size + end + ["file:///tmp/feed", "ftp://example.com/feed", "http://[invalid", "https://user:password@example.com/"].each do |location| + with_server([[302, { "Location" => location }, ""]]) do |url, requests| + assert_raise(SimpleRSS::PolicyError) { SimpleRSS.discover(url, network_policy: :unrestricted, timeout: 1) } + assert_equal 1, requests.size + end + end + end + + def test_bodyless_conditional_responses_ignore_representation_encoding + response = [304, { "Content-Encoding" => "gzip", "ETag" => '"fixture"' }, ""] + with_server([response, response]) do |url, _requests| + assert_nil SimpleRSS.fetch(url, network_policy: :unrestricted, timeout: 1, max_bytes: 1, etag: '"fixture"') + error = assert_raise(SimpleRSS::HTTPError) { SimpleRSS.discover(url, network_policy: :unrestricted, timeout: 1) } + assert_equal 304, error.status_code + end + end + + def test_truncated_content_length_is_a_transport_failure + response = lambda do |client, _request| + client.write("HTTP/1.1 200 OK\r\nContent-Type: text/html\r\nContent-Length: #{PAGE.last.bytesize + 20}\r\nConnection: close\r\n\r\n#{PAGE.last}") + end + with_server([response]) do |url, _requests| + assert_raise(SimpleRSS::RequestError) { SimpleRSS.discover(url, network_policy: :unrestricted, timeout: 1) } + end + end + + def test_no_follow_option_reports_the_redirect_status + with_server([[302, { "Location" => "/next" }, ""]]) do |url, requests| + error = assert_raise(SimpleRSS::HTTPError) { SimpleRSS.discover(url, network_policy: :unrestricted, follow_redirects: false, timeout: 1) } + assert_equal 302, error.status_code + assert_equal 1, requests.size + end + end + + def test_response_byte_limit_accepts_the_boundary_and_rejects_overflow + with_server([PAGE, PAGE]) do |url, requests| + assert_empty SimpleRSS.discover(url, network_policy: :unrestricted, timeout: 1, max_bytes: PAGE.last.bytesize) + assert_raise(SimpleRSS::ResponseTooLarge) { SimpleRSS.discover(url, network_policy: :unrestricted, timeout: 1, max_bytes: PAGE.last.bytesize - 1) } + assert_equal 2, requests.size + end + end + + def test_chunked_body_is_stopped_before_its_end + response = lambda do |client, _request| + client.write("HTTP/1.1 200 OK\r\nContent-Type: text/html\r\nTransfer-Encoding: chunked\r\n\r\n100\r\n" + ("x" * 256) + "\r\n") + wait_for_disconnect(client) + end + with_server([response]) do |url, requests| + assert_raise(SimpleRSS::ResponseTooLarge) { SimpleRSS.discover(url, network_policy: :unrestricted, timeout: 1, max_bytes: 128) } + assert_equal 1, requests.size + end + end + + def test_gzip_and_deflate_are_decoded_with_a_decompressed_size_limit + { "gzip" => gzip(PAGE.last), "deflate" => Zlib::Deflate.deflate(PAGE.last) }.each do |encoding, compressed| + with_server([[200, PAGE[1].merge("Content-Encoding" => encoding), compressed]]) do |url, _requests| + assert_empty SimpleRSS.discover(url, network_policy: :unrestricted, timeout: 1) + end + end + compressed = gzip("" + ("x" * 200_000) + "") + assert_operator compressed.bytesize, :<, 1024 + with_server([[200, PAGE[1].merge("Content-Encoding" => "gzip"), compressed]]) do |url, _requests| + assert_raise(SimpleRSS::ResponseTooLarge) { SimpleRSS.discover(url, network_policy: :unrestricted, timeout: 1, max_bytes: 1024) } + end + end + + def test_body_budget_is_shared_across_redirects + responses = [[302, { "Location" => "/next" }, "x" * 100], PAGE] + with_server(responses) do |url, requests| + assert_raise(SimpleRSS::ResponseTooLarge) { SimpleRSS.discover(url, network_policy: :unrestricted, timeout: 1, max_bytes: 100 + PAGE.last.bytesize - 1) } + assert_equal 2, requests.size + end + end + + def test_invalid_compression_and_partial_responses_fail_clearly + responses = [ + [200, PAGE[1].merge("Content-Encoding" => "br"), "wrong"], + [200, PAGE[1].merge("Content-Encoding" => "gzip"), "wrong"], + [200, PAGE[1].merge("Content-Encoding" => "gzip"), gzip(PAGE.last).byteslice(0, 15)], + [200, PAGE[1].merge("Content-Encoding" => "gzip"), gzip(PAGE.last) + gzip("ignored")], + [206, PAGE[1].merge("Content-Range" => "bytes 0-9/100"), "0123456789"] + ] + responses.each do |response| + with_server([response]) do |url, _requests| + assert_raise(SimpleRSS::RequestError) { SimpleRSS.discover(url, network_policy: :unrestricted, timeout: 1) } + end + end + end + + def test_read_timeout_is_visible_and_does_not_retry + with_server([->(client, _request) { wait_for_disconnect(client) }]) do |url, requests| + assert_raise(SimpleRSS::RequestTimeout) { SimpleRSS.discover(url, network_policy: :unrestricted, timeout: 0.1) } + assert_equal 1, requests.size + end + end + + def test_dns_resolution_is_inside_the_total_timeout + with_replaced_method(Resolv, :getaddresses, ->(_host) { Queue.new.pop }) do + assert_raise(SimpleRSS::RequestTimeout) { SimpleRSS.discover("http://site.example/", timeout: 0.1) } + end + end + + def test_missing_dns_results_and_connection_failures_are_not_empty_candidates + with_replaced_method(Resolv, :getaddresses, ->(_host) { [] }) do + assert_raise(SimpleRSS::RequestError) { SimpleRSS.discover("http://site.example/") } + end + with_replaced_method(Resolv, :getaddresses, ->(_host) { ["8.8.8.8"] }) do + with_replaced_method(TCPSocket, :open, ->(*) { raise Errno::ECONNREFUSED }) do + assert_raise(SimpleRSS::RequestError) { SimpleRSS.discover("http://site.example/") } + end + end + end + + def test_policy_disables_environment_proxies_and_keeps_tls_verification + constructor = Net::HTTP.method(:new) + sessions = [] + replacement = lambda do |*arguments| + assert_equal ["site.example", 443, nil], arguments + session = constructor.call(*arguments) + sessions << session + session + end + with_replaced_method(Resolv, :getaddresses, ->(_host) { ["8.8.8.8"] }) do + with_replaced_method(Net::HTTP, :new, replacement) do + with_replaced_method(TCPSocket, :open, ->(*) { raise Errno::ECONNREFUSED }) do + assert_raise(SimpleRSS::RequestError) { SimpleRSS.discover("https://site.example/", timeout: 1) } + end + end + end + assert_equal false, sessions.first.proxy? + assert_equal "site.example", sessions.first.address + assert_equal "8.8.8.8", sessions.first.ipaddr + assert_equal OpenSSL::SSL::VERIFY_PEER, sessions.first.verify_mode + assert_equal true, sessions.first.verify_hostname + end + + def test_pinned_https_preserves_certificate_hostname_verification + certificate, private_key = test_certificate + store = OpenSSL::X509::Store.new + store.add_cert(certificate) + context = OpenSSL::SSL::SSLContext.new + context.cert = certificate + context.key = private_key + server = TCPServer.new("127.0.0.1", 0) + ssl_server = OpenSSL::SSL::SSLServer.new(server, context) + requests = [] + worker = Thread.new do + 5.times do + client = nil + begin + client = ssl_server.accept + request = [] + while (line = client.gets) + break if line == "\r\n" + + request << line.strip + end + requests << request + client.write("HTTP/1.1 200 OK\r\nContent-Type: text/html\r\nContent-Length: #{PAGE.last.bytesize}\r\nConnection: close\r\n\r\n#{PAGE.last}") + rescue OpenSSL::SSL::SSLError + next + ensure + client&.close + end + end + end + constructor = Net::HTTP.method(:new) + replacement = lambda do |*arguments| + session = constructor.call(*arguments) + session.cert_store = store + session + end + with_replaced_method(Net::HTTP, :new, replacement) do + with_pinned_connection("http://127.0.0.1:#{server.addr[1]}", ->(_host) { ["8.8.8.8"] }) do |connections| + assert_empty SimpleRSS.discover("https://site.example/", timeout: 1) + assert_empty SimpleRSS.discover("site.example", timeout: 1) + assert_empty SimpleRSS.discover("site.example/blog?edition=1#entries", timeout: 1) + assert_empty SimpleRSS.discover("//site.example/blog", timeout: 1) + error = assert_raise(SimpleRSS::RequestError) { SimpleRSS.discover("https://wrong.example/", timeout: 1) } + assert_kind_of OpenSSL::SSL::SSLError, error.cause + assert_equal Array.new(5) { ["8.8.8.8", 443] }, connections + end + end + worker.value + assert_equal ["GET / HTTP/1.1", "GET / HTTP/1.1", "GET /blog?edition=1 HTTP/1.1", "GET /blog HTTP/1.1"], requests.map(&:first) + assert_include requests.first, "Host: site.example" + ensure + worker&.kill + worker&.join + server&.close + end + + def test_bounded_fetch_reuses_the_policy_and_conditional_get + headers = { "ETag" => '"fixture"', "Last-Modified" => "Sat, 12 Sep 2026 10:00:00 GMT" } + with_server([[200, headers, FEED], [304, headers, ""]]) do |url, requests| + feed = SimpleRSS.fetch(url, network_policy: :unrestricted, timeout: 1) + assert_equal "Example", feed.title + assert_equal "#{url}/", feed.source_url + assert_nil SimpleRSS.fetch(url, network_policy: :unrestricted, timeout: 1, etag: feed.etag, last_modified: feed.last_modified) + assert_include requests.last, 'If-None-Match: "fixture"' + assert_include requests.last, "If-Modified-Since: #{headers["Last-Modified"]}" + end + end + + def test_invalid_options_and_controlled_headers_fail_before_connecting + options = [ + { network_policy: nil }, { network_policy: :unknown }, { timeout: 0 }, { timeout: nil }, { timeout: Float::INFINITY }, + { max_bytes: 0 }, { max_bytes: 1.5 }, { max_redirects: -1 }, { max_redirects: 1.5 }, { headers: [] }, + { headers: { "X-Value" => "bad\r\nInjected: header" } }, { headers: { "Bad Header" => "value" } } + ] + options += %w[Host Proxy-Authorization Accept-Encoding Range Connection Transfer-Encoding].map { |name| { headers: { name => "value" } } } + with_replaced_method(TCPSocket, :open, ->(*) { flunk "Invalid options reached the socket" }) do + options.each do |override| + assert_raise(ArgumentError, override.inspect) { SimpleRSS.discover("https://site.example/", override) } + end + end + end + + private + + def test_certificate + private_key = OpenSSL::PKey::RSA.new(2048) + certificate = OpenSSL::X509::Certificate.new + certificate.version = 2 + certificate.serial = 1 + certificate.subject = OpenSSL::X509::Name.parse("/CN=site.example") + certificate.issuer = certificate.subject + certificate.public_key = private_key.public_key + certificate.not_before = Time.now - 60 + certificate.not_after = Time.now + 3600 + factory = OpenSSL::X509::ExtensionFactory.new + factory.subject_certificate = certificate + factory.issuer_certificate = certificate + certificate.add_extension(factory.create_extension("basicConstraints", "CA:TRUE", true)) + certificate.add_extension(factory.create_extension("subjectAltName", "DNS:site.example")) + certificate.sign(private_key, OpenSSL::Digest.new("SHA256")) + [certificate, private_key] + end + + def gzip(body) + output = StringIO.new + writer = Zlib::GzipWriter.new(output) + writer.write(body) + writer.close + output.string + end + + def with_pinned_connection(local_url, resolver) + socket_open = TCPSocket.method(:open) + connections = [] + local_port = URI.parse(local_url).port + with_replaced_method(Resolv, :getaddresses, resolver) do + replacement = lambda do |address, port, *_arguments| + connections << [address, port] + socket_open.call("127.0.0.1", local_port) + end + with_replaced_method(TCPSocket, :open, replacement) { yield connections } + end + end +end diff --git a/test/base/feedbag_integration_test.rb b/test/base/feedbag_integration_test.rb new file mode 100644 index 0000000..f9795ce --- /dev/null +++ b/test/base/feedbag_integration_test.rb @@ -0,0 +1,21 @@ +require "test_helper" +require "feedbag" +require_relative "../support/http_server" + +class FeedbagIntegrationTest < Test::Unit::TestCase + include HTTPServer + + def test_feedbag_candidates_can_be_consumed_by_simple_rss + html = '' + xml = 'ExamplePost' + responses = [[200, { "Content-Type" => "text/html" }, html], [200, { "Content-Type" => "application/rss+xml" }, xml]] + with_server(responses) do |url, requests| + candidates = Feedbag.find(url, open_timeout: 1, read_timeout: 1) + assert_equal ["#{url}/feed.xml"], candidates + assert_equal 1, requests.size + feed = SimpleRSS.fetch(candidates.first, timeout: 1) + assert_equal "Post", feed.normalized_entries.first.title + assert_equal ["GET / HTTP/1.1", "GET /feed.xml HTTP/1.1"], requests.map(&:first) + end + end +end diff --git a/test/data/discovery.html b/test/data/discovery.html new file mode 100644 index 0000000..b7af5a6 --- /dev/null +++ b/test/data/discovery.html @@ -0,0 +1,28 @@ + + + + + + + + + + + + + + + + + + + + + + + +
+

<link rel=alternate type=application/rss+xml href=text.xml>

+ Feed + + diff --git a/test/support/http_server.rb b/test/support/http_server.rb new file mode 100644 index 0000000..657db7e --- /dev/null +++ b/test/support/http_server.rb @@ -0,0 +1,44 @@ +require "socket" + +module HTTPServer + def wait_for_disconnect(client) + client.read + rescue Errno::ECONNRESET + nil + end + + def with_server(responses) + server = TCPServer.new("127.0.0.1", 0) + base_url = "http://127.0.0.1:#{server.addr[1]}" + requests = [] + worker = Thread.new do + responses.each do |response| + client = server.accept + begin + request = [] + while (line = client.gets) + break if line == "\r\n" + + request << line.strip + end + requests << request + if response.respond_to?(:call) + response.call(client, request) + next + end + status, headers, body = response + response_headers = headers.merge("Content-Length" => body.bytesize.to_s, "Connection" => "close") + client.write("HTTP/1.1 #{status} Test\r\n" + response_headers.map { |name, value| "#{name}: #{value}\r\n" }.join + "\r\n" + body) + ensure + client.close + end + end + end + yield base_url, requests + worker.value + ensure + worker&.kill + worker&.join + server&.close + end +end diff --git a/test/support/replace_method.rb b/test/support/replace_method.rb new file mode 100644 index 0000000..f7f60fa --- /dev/null +++ b/test/support/replace_method.rb @@ -0,0 +1,14 @@ +module ReplaceMethod + def with_replaced_method(object, name, replacement) + own_method = object.singleton_methods(false).include?(name) + original = object.method(name) + object.define_singleton_method(name) { |*arguments, **keywords, &block| replacement.call(*arguments, **keywords, &block) } + yield + ensure + if own_method + object.define_singleton_method(name, original) + else + object.singleton_class.remove_method(name) + end + end +end