Skip to content

update examples with v2 #22

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mathieu-aubin wants to merge 4 commits into
base: master
Choose a base branch
from
Open

update examples with v2 #22

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
48e68ee
update examples with v2
mathieu-aubin Jul 22, 2021
941c5f5
adds missing v2
mathieu-aubin Jul 22, 2021
71c4906
more v2 fixes
mathieu-aubin Jul 22, 2021
8190fdc
more v2 changes found
mathieu-aubin Jul 22, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion content/docs/best_practices/crawling.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ Colly's default configuration is optimized for scraping smaller number of sites

## Use persistent storage backend

By default Colly stores cookies and visited URLs in memory. You can replace the built-in in-memory storage backend with any custom backend. See more details [here](https://godoc.org/github.com/gocolly/colly/storage).
By default Colly stores cookies and visited URLs in memory. You can replace the built-in in-memory storage backend with any custom backend. See more details [here](https://godoc.org/github.com/gocolly/colly/v2/storage).


## Use async for long running jobs with recursive calls
Expand Down
8 changes: 4 additions & 4 deletions content/docs/best_practices/debugging.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,13 @@ Sometimes it's enough to place some `log.Println()` function calls to your callb

## Attach debugger to a collector

Attaching a basic logging debugger requires the `debug` (`github.com/gocolly/colly/debug`) package from Colly's repo.
Attaching a basic logging debugger requires the `debug` (`github.com/gocolly/colly/v2/debug`) package from Colly's repo.

```go

import (
"github.com/gocolly/colly"
"github.com/gocolly/colly/debug"
"github.com/gocolly/colly/v2"
"github.com/gocolly/colly/v2/debug"
)

func main() {
Expand All @@ -28,4 +28,4 @@ func main() {

## Implement a custom debugger

You can create any kind of custom debugger by implementing the [debug.Debugger](https://godoc.org/github.com/gocolly/colly/debug#Debugger) interface. A good example is [LogDebugger](https://github.com/gocolly/colly/blob/master/debug/logdebugger.go).
You can create any kind of custom debugger by implementing the [debug.Debugger](https://godoc.org/github.com/gocolly/colly/v2/debug#Debugger) interface. A good example is [LogDebugger](https://github.com/gocolly/colly/blob/master/debug/logdebugger.go).
6 changes: 3 additions & 3 deletions content/docs/best_practices/distributed.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,8 @@ Colly has a built-in proxy switcher which rotates a list of proxies on every req
package main

import (
"github.com/gocolly/colly"
"github.com/gocolly/colly/proxy"
"github.com/gocolly/colly/v2"
"github.com/gocolly/colly/v2/proxy"
)

func main() {
Expand Down Expand Up @@ -70,4 +70,4 @@ An example implementation can be found [here](/docs/examples/scraper_server).

Visited URL and cookie data are stored in-memory by default. This is handy for short living scraper jobs, but it can be a serious limitation when dealing with large scale or long running crawling jobs.

Colly has the ability to replace the default in-memory storage with any storage backend which implements [colly/storage.Storage](https://godoc.org/github.com/gocolly/colly/storage#Storage) interface. Check out [existing storages](/docs/best_practices/storage).
Colly has the ability to replace the default in-memory storage with any storage backend which implements [colly/storage.Storage](https://godoc.org/github.com/gocolly/colly/v2/storage#Storage) interface. Check out [existing storages](/docs/best_practices/storage).
6 changes: 3 additions & 3 deletions content/docs/best_practices/extensions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ menu:
weight: 50
---

Extensions are small helper utilities shipped with Colly. List of plugins is available [here](https://godoc.org/github.com/gocolly/colly/extensions).
Extensions are small helper utilities shipped with Colly. List of plugins is available [here](https://godoc.org/github.com/gocolly/colly/v2/extensions).


## Usage
Expand All @@ -18,8 +18,8 @@ The following example enables the random User-Agent switcher and the Referrer se
import (
"log"

"github.com/gocolly/colly"
"github.com/gocolly/colly/extensions"
"github.com/gocolly/colly/v2"
"github.com/gocolly/colly/v2/extensions"
)

func main() {
Expand Down
6 changes: 3 additions & 3 deletions content/docs/best_practices/storage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,13 @@ menu:
weight: 25
---

Colly has an in-memory storage backend to store cookies and visited URLs, but it can be overwritten by any custom storage backend which implements [colly/storage.Storage](https://godoc.org/github.com/gocolly/colly/storage#Storage).
Colly has an in-memory storage backend to store cookies and visited URLs, but it can be overwritten by any custom storage backend which implements [colly/v2/storage.Storage](https://godoc.org/github.com/gocolly/colly/v2/storage#Storage).

## Existing Storage Backends

### [In-Memory Backend](https://godoc.org/github.com/gocolly/colly/storage#InMemoryStorage)
### [In-Memory Backend](https://godoc.org/github.com/gocolly/colly/v2/storage#InMemoryStorage)

The default backend of Colly. Use [collector.SetStorage()](https://godoc.org/github.com/gocolly/colly#Collector.SetStorage) to override.
The default backend of Colly. Use [collector.SetStorage()](https://godoc.org/github.com/gocolly/colly/v2#Collector.SetStorage) to override.

### [Redis backend](https://github.com/gocolly/redisstorage)

Expand Down
2 changes: 1 addition & 1 deletion content/docs/examples/basic.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ package main
import (
"fmt"

"github.com/gocolly/colly"
"github.com/gocolly/colly/v2"
)

func main() {
Expand Down
2 changes: 1 addition & 1 deletion content/docs/examples/coursera_courses.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ import (
"os"
"strings"

"github.com/gocolly/colly"
"github.com/gocolly/colly/v2"
)

// Course stores information about a coursera course
Expand Down
2 changes: 1 addition & 1 deletion content/docs/examples/cryptocoinmarketcap.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ import (
"log"
"os"

"github.com/gocolly/colly"
"github.com/gocolly/colly/v2"
)

func main() {
Expand Down
2 changes: 1 addition & 1 deletion content/docs/examples/error_handling.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ package main
import (
"fmt"

"github.com/gocolly/colly"
"github.com/gocolly/colly/v2"
)

func main() {
Expand Down
2 changes: 1 addition & 1 deletion content/docs/examples/factbase.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ import (
"io/ioutil"
"strconv"

"github.com/gocolly/colly"
"github.com/gocolly/colly/v2"
)

var baseSearchURL = "https://factba.se/json/json-transcript.php?q=&f=&dt=&p="
Expand Down
2 changes: 1 addition & 1 deletion content/docs/examples/google_groups.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ import (
"os"
"strings"

"github.com/gocolly/colly"
"github.com/gocolly/colly/v2"
)

// Mail is the container of a single e-mail
Expand Down
2 changes: 1 addition & 1 deletion content/docs/examples/hackernews_comments.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ import (
"strconv"
"strings"

"github.com/gocolly/colly"
"github.com/gocolly/colly/v2"
)

type comment struct {
Expand Down
2 changes: 1 addition & 1 deletion content/docs/examples/instagram.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ import (
"os"
"strings"

"github.com/gocolly/colly"
"github.com/gocolly/colly/v2"
)

// found in https://www.instagram.com/static/bundles/en_US_Commons.js/68e7390c5938.js
Expand Down
2 changes: 1 addition & 1 deletion content/docs/examples/login.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ package main
import (
"log"

"github.com/gocolly/colly"
"github.com/gocolly/colly/v2"
)

func main() {
Expand Down
2 changes: 1 addition & 1 deletion content/docs/examples/max_depth.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ package main
import (
"fmt"

"github.com/gocolly/colly"
"github.com/gocolly/colly/v2"
)

func main() {
Expand Down
2 changes: 1 addition & 1 deletion content/docs/examples/multipart.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ import (
"os"
"time"

"github.com/gocolly/colly"
"github.com/gocolly/colly/v2"
)

func generateFormData() map[string][]byte {
Expand Down
2 changes: 1 addition & 1 deletion content/docs/examples/openedx_courses.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ import (
"strings"
"time"

"github.com/gocolly/colly"
"github.com/gocolly/colly/v2"
)

// DATE_FORMAT default format date used in openedx
Expand Down
2 changes: 1 addition & 1 deletion content/docs/examples/parallel.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ package main
import (
"fmt"

"github.com/gocolly/colly"
"github.com/gocolly/colly/v2"
)

func main() {
Expand Down
4 changes: 2 additions & 2 deletions content/docs/examples/proxy_switcher.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ import (
"bytes"
"log"

"github.com/gocolly/colly"
"github.com/gocolly/colly/proxy"
"github.com/gocolly/colly/v2"
"github.com/gocolly/colly/v2/proxy"
)

func main() {
Expand Down
4 changes: 2 additions & 2 deletions content/docs/examples/queue.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@ package main
import (
"fmt"

"github.com/gocolly/colly"
"github.com/gocolly/colly/queue"
"github.com/gocolly/colly/v2"
"github.com/gocolly/colly/v2/queue"
)

func main() {
Expand Down
4 changes: 2 additions & 2 deletions content/docs/examples/random_delay.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ import (
"fmt"
"time"

"github.com/gocolly/colly"
"github.com/gocolly/colly/debug"
"github.com/gocolly/colly/v2"
"github.com/gocolly/colly/v2/debug"
)

func main() {
Expand Down
4 changes: 2 additions & 2 deletions content/docs/examples/rate_limit.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@ package main
import (
"fmt"

"github.com/gocolly/colly"
"github.com/gocolly/colly/debug"
"github.com/gocolly/colly/v2"
"github.com/gocolly/colly/v2/debug"
)

func main() {
Expand Down
2 changes: 1 addition & 1 deletion content/docs/examples/reddit.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ import (
"os"
"time"

"github.com/gocolly/colly"
"github.com/gocolly/colly/v2"
)

type item struct {
Expand Down
4 changes: 2 additions & 2 deletions content/docs/examples/redis_backend.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@ package main
import (
"log"

"github.com/gocolly/colly"
"github.com/gocolly/colly/queue"
"github.com/gocolly/colly/v2"
"github.com/gocolly/colly/v2/queue"
"github.com/gocolly/redisstorage"
)

Expand Down
2 changes: 1 addition & 1 deletion content/docs/examples/request_context.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ package main
import (
"fmt"

"github.com/gocolly/colly"
"github.com/gocolly/colly/v2"
)

func main() {
Expand Down
2 changes: 1 addition & 1 deletion content/docs/examples/scraper_server.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ import (
"log"
"net/http"

"github.com/gocolly/colly"
"github.com/gocolly/colly/v2"
)

type pageInfo struct {
Expand Down
2 changes: 1 addition & 1 deletion content/docs/examples/shopify_sitemap.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ package main
import (
"fmt"

"github.com/gocolly/colly"
"github.com/gocolly/colly/v2"
)

func main() {
Expand Down
2 changes: 1 addition & 1 deletion content/docs/examples/url_filter.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ import (
"fmt"
"regexp"

"github.com/gocolly/colly"
"github.com/gocolly/colly/v2"
)

func main() {
Expand Down
2 changes: 1 addition & 1 deletion content/docs/examples/xkcd_store.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ import (
"log"
"os"

"github.com/gocolly/colly"
"github.com/gocolly/colly/v2"
)

func main() {
Expand Down
2 changes: 1 addition & 1 deletion content/docs/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,4 +12,4 @@ Below you will find some of the most common and helpful pages from our documenta
* [Getting started](/docs/introduction/start/)
* [Basic scraper example](/docs/examples/basic/)

Also, check out [GoDoc](https://godoc.org/github.com/gocolly/colly) for API reference.
Also, check out [GoDoc](https://godoc.org/github.com/gocolly/colly/v2) for API reference.
2 changes: 1 addition & 1 deletion content/docs/introduction/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ Colly is a highly customizable scraping framework. It has sane defaults and prov

## Collector configuration

Full list of collector attributes can be found [here](https://godoc.org/github.com/gocolly/colly#Collector).
Full list of collector attributes can be found [here](https://godoc.org/github.com/gocolly/colly/v2#Collector).
The recommended way to initialize a collector is using `colly.NewCollector(options...)`.

Create a collector with default settings:
Expand Down
4 changes: 2 additions & 2 deletions content/docs/introduction/start.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Let's get started with some simple examples.
First, you need to import Colly to your codebase:

```go
import "github.com/gocolly/colly"
import "github.com/gocolly/colly/v2"
```


Expand All @@ -32,7 +32,7 @@ c := colly.NewCollector()
## Callbacks


You can attach different type of callback functions to a `Collector` to control a collecting job or retrieve information. Check out the [related section](https://godoc.org/github.com/gocolly/colly#Collector.OnError) in the package documentation.
You can attach different type of callback functions to a `Collector` to control a collecting job or retrieve information. Check out the [related section](https://godoc.org/github.com/gocolly/colly/v2#Collector.OnError) in the package documentation.


### Add callbacks to a `Collector`
Expand Down
4 changes: 2 additions & 2 deletions themes/colly/layouts/_default/baseof.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@
{{ range .Site.Menus.main.ByWeight }}
<a class="item" href="{{ .URL }}">{{ .Name }}</a>
{{ end }}
<a class="right item" href="https://godoc.org/github.com/gocolly/colly" target="_blank">GoDoc</a>
<a class="right item" href="https://godoc.org/github.com/gocolly/colly/v2" target="_blank">GoDoc</a>
<a class="item" href="https://github.com/gocolly/colly" target="_blank"><i class="github big icon"></i></a>
</div>
<div class="mobile only row">
Expand All @@ -76,7 +76,7 @@
<a class="item" href="{{ .URL }}">{{ .Name }}</a>
{{ end }}
<div class="ui divider"></div>
<a class="item" href="https://godoc.org/github.com/gocolly/colly" target="_blank">GoDoc</a>
<a class="item" href="https://godoc.org/github.com/gocolly/colly/v2" target="_blank">GoDoc</a>
<a class="item" href="https://github.com/gocolly/colly" target="_blank">GitHub</a>
</div>
</div>
Expand Down