'Scraping a simple website with colly in golang does not return any data

I'm trying to scrape a simple website that looks like this:

<html>

<head>
</head>

<body>
  <pre>
    "Name Surname 1
    Name Surname 2
    Name Surname 3
    Name Surname 4"
  </pre>
</body>

</html>

Wrote a simple go code:

package main

import (
    "fmt"

    "github.com/gocolly/colly"
)

func main() {
    c := colly.NewCollector(
        colly.AllowedDomains("thewebsite.com"),
    )

    c.OnHTML("body", func(e *colly.HTMLElement) {
        fmt.Println(e.Text)
    })

    c.OnResponse(func(r *colly.Response) {
        fmt.Println(r.StatusCode)
    })

    c.OnRequest(func(r *colly.Request) {
        fmt.Println("Visiting", r.URL)
    })

    c.Visit("http://thewebsite.com")
}

When I run this code, I get the below output:

Visiting http://thewebsite.com
200

So everything is OK. The website is being opened successfully, but I do not get any data from it.

I've tried to change the c.OnHTML to pre, body.pre - but none of them worked as I expected to.

What am I missing here?

go web-scraping go-colly

Solution 1:^[1]

I had a similar problem and I had to remove the domain restriction, despite it appearing to be correct. In other words, try commenting out the AllowedDomains() bit, like this:

c := colly.NewCollector(
      //colly.AllowedDomains("thewebsite.com"),
    )

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source
Solution 1	Tom Chantler

'Scraping a simple website with colly in golang does not return any data

Solution 1:[1]

Sources

Related Questions

Solution 1:^[1]