Respect robots.txt by default

[jstmrby]

These changes aim to address #614.

Ignoring the robots.txt file by default raises ethical concerns.

Additionally, the documentation regarding the default configuration is arguably incomplete. Considering this, the existence of IgnoreRobotsTxt() function implies that the default behavior is to respect it in the first place.

Important

This alters default behavior and so presumably warrants a note in the release notes whenever it is released. A collector created through NewCollector() now consults robots.txt and will decline URLs its target disallows.

Two further consequences seem worth stating plainly, as neither is obvious. A 5xx response to /robots.txt renders a host entirely uncrawlable, since it is read as a full disallow, and a failed connection causes the request to error out. A 404 maps to allow-all and changes nothing. None of this is new logic, it simply was not reachable by default until now.

The previous behavior remains available per collector

c := colly.NewCollector(colly.IgnoreRobotsTxt())

or globally through the environment

export COLLY_IGNORE_ROBOTSTXT=y

The change itself is confined to Collector.Init(), which now leaves IgnoreRobotsTxt at false. The field's documentation has been amended to state that default explicitly. Tests covering the new default have been added, and the one existing test that relied on the old one now opts out explicitly.

The changes can be exercised with the focused tests

go test -race -run 'Robots|TestNewCollector' -v .
go test -race -run TestQueue -v ./queue

and the full suite, together with the remaining checks CI performs, with

gofmt -l -d ./
golint -set_exit_status
go vet ./...
go test -race -coverprofile=coverage.txt -covermode=atomic ./...

[StantonMatt]

I checked this against #614 locally. The direction looks consistent with the issue, but the full suite needs one follow-up before this can merge cleanly.

What I ran:

GOCACHE=.codex-tmp/go-cache GOTMPDIR=.codex-tmp/go-tmp GOMODCACHE=.codex-tmp/go-mod go test . -run 'TestRobots|Robots|IgnoreRobots|NewCollector' -count=1 -v
GOCACHE=.codex-tmp/go-cache GOTMPDIR=.codex-tmp/go-tmp GOMODCACHE=.codex-tmp/go-mod go test ./... -count=1
git diff --check upstream/master...HEAD

The focused robots/default tests pass, and git diff --check is clean. GitHub reports no checks for this PR at the moment.

go test ./... fails in queue/TestQueue:

queue_test.go:72: wrong Queue implementation: items = 3000, requests = 0, success = 0, failure = 0

That test creates an httptest server that only handles /delay. With robots respected by default, the collector probes /robots.txt, the test server closes that unsupported route, and the queued URLs never reach OnRequest. I temporarily added colly.IgnoreRobotsTxt() to that test collector and confirmed go test ./queue -run TestQueue -count=1 -v passes, so this looks like a test-suite alignment issue rather than a blocker to the behavior change itself.

I’d suggest updating tests that intentionally expect the old ignore-robots behavior to opt out explicitly, and adding a small assertion around the new NewCollector() default. Since this changes default crawler behavior, a docs/changelog note would also help downstream users notice the migration path.

[jstmrby]

Confirmed. The queue test server only answers /delay, and everything else hits shutdown, which hijacks the connection and closes it. So the robots probe errors out, checkRobots returns that error, and requestCheck bails before OnRequest ever fires — hence 3000 items and 0 requests.

I went with colly.IgnoreRobotsTxt() on that collector rather than giving the server a /robots.txt route, since the test is about queue mechanics and shouldn't really care either way.

For the default, I added two tests: TestNewCollector/Defaults checks the field, and TestRobotsDisallowedByDefault goes through Visit. The second one matters because all the existing robots tests set IgnoreRobotsTxt themselves, so none of them would catch the default flipping back. Both fail if I revert it.

On docs: the default is now stated on the IgnoreRobotsTxt field. I've called the breaking change out in the description instead, so it's there to pull into the release notes.

Worth flagging while you're looking at it: respecting robots.txt also means a 5xx on /robots.txt makes a host completely uncrawlable, since robotstxt reads that as a full disallow, and a connection failure errors the request out, which TestConnectionErrorOnRobotsTxtResultsInError already covers. A 404 is fine, it maps to allow-all. None of that is new, it just wasn't reachable by default before. It's in the description too.

No checks showing up on the PR, so I ran CI's sequence locally: gofmt, golint and go vet are all clean, and go test -race ./... passes both packages.