Category "scrapy"

Auth failing - 999- HTTP status code is not handled or not allowed

I using scrapy, and I would like to get Ignoring response URL.I just see in the output console this: DEBUG: Ignoring response <999 https://www.mywebsite.com

scrapy follow external link with one depth only

Imagine I am crawling foo.com. foo.com has several internal links to itself, and it has some external links like: foo.com/hello foo.com/contact bar.com holla.c

Scrapy: How to output items in a specific json format

I output the scraped data in json format. Default scrapy exporter outputs list of dict in json format. Item type looks like: [{"Product Name":"Product1", "Cate

Get all link text and href in a page using scrapy

class LinkSpider(scrapy.Spider): name = "link" def start_requests(self): urlBasang = "https://bloomberg.com" yield scrapy.Request(url =

Which parse method scrapy used to parse start_urls

I want scrapy to scrape some start urls and then follow the links in those pages according to rules. My spider is inherited from CrawlSpider and has start_urls