Queue Depth functionality + Panic with Async

[KristinnVikar]

Description

This pull request adds support for Depth in Queue and adds a panic when attempting to use Async with Queue, as they are incompatible. The changes ensure that MaxDepth works as expected when using Queues.

I had to figure out by trial and error that Async caused my Queue based crawler to instantly finish since it thought the returned promises were 'completed' requests.

Changes Made

1. Added a default value for Depth for Requests made in AddURL.
2. Implemented a check in the AddRequest method to increase the depth for each nested request.
3. Added a panic in the Queue.Run method to prevent users from using Async with the Queue, as it's not supported.

Checklist

• Added unit tests for change

Please let me know if you require any changes to this Pull Request!
And if there's any interest I did make a UniqueInMemoryQueueStorage struct in my project that only stores non duplicate entries within the Queue.

Thank you!

[WGH-]

I don't understand this change. Could you please clarify? The test you added doesn't catch it if I revert this change.

[KristinnVikar]

Odd, the TestCollectorDepth test should catch that, it's so that the depth increases on each request, so that you can use MaxDepth in Queue-based crawlers. I'll take a look at it.

[StantonMatt]

I found one depth edge case that looks worth fixing before this can be merged.

With this branch, Queue.AddURL() serializes the root request with Depth: 1, and Request.Do() then calls scrape(..., r.Depth+1, ...). That means the first queued URL is checked as depth 2. With colly.MaxDepth(1), the root request is rejected by requestCheck before OnRequest runs, even though direct c.Visit(...) still treats the first request as depth 1.

Local-only repro:

c := colly.NewCollector(colly.MaxDepth(1), colly.IgnoreRobotsTxt())
c.OnRequest(func(req *colly.Request) {
    if req.Depth != 1 {
        t.Fatalf("queued root request depth = %d, want 1", req.Depth)
    }
})

q, _ := queue.New(1, &queue.InMemoryQueueStorage{MaxSize: 10})
_ = q.AddURL(server.URL)
_ = q.Run(c)

On this PR head, that fails because the queued root request never reaches OnRequest:

queued root requests = 0, want 1

The direct c.Visit(server.URL) comparison with MaxDepth(1) still reaches OnRequest at depth 1, so this looks like a queue-specific off-by-one. If Request.Do() is going to increment queued requests, I think AddURL() should probably keep serializing root requests at depth 0, with a regression test for MaxDepth(1).

The branch's existing focused tests and full suite still pass:

go test ./queue -run 'TestQueue|TestCollectorDepth|TestAsyncPanic' -count=1 -v
go test ./... -count=1
git diff --check origin/master...HEAD

GitHub reports no hosted checks for this PR branch.