how to use same cookies over multiple requests when using python requests

Question

I am new to python requests and am using it to scrape a website and get to a certain webpage, first I login and then I do a few requests for other webpages:

import requests

url1 = 'https://ringzer0team.com/login'
s = requests.Session()

request = s.get(url1)
print s.cookies
print request.cookies
request = s.post(url1, auth=('username', 'password'))
print request.cookies

url2 = 'https://ringzer0team.com/home'
request = s.get(url2, cookies =   s.cookies)
print request.cookies

url3 = 'https://ringzer0team.com/challenges'
request = s.get(url3, cookies = s.cookies)
print request.cookies

url4 = 'https://ringzer0team.com/challenges/13'
request = s.get(url3, cookies = s.cookies)
print request.cookies

I believe it is because cookies get lost during the session, here is the output I get:

my output

As you can see, the cookies are lost after I post my credentials to url1 and I can get the contents of url2 but not those of url3 and url4. s.cookies remains the same throughout so I've been trying to use it in all my get requests without any success. Any help would be appreciated.

Answer 1

The Requests docs say, under Advanced Usage:

Note, however, that method-level parameters will not be persisted across requests, even if using a session. [...]

If you want to manually add cookies to your session, use the Cookie utility functions to manipulate Session.cookies.

The referenced resource basically shows what to do to persist cookies throughout the session.

Answer 2

I was working on this same problem. Combining this post with some other posts I was able to find the correct syntax to accomplish this.

s = requests.Session()
response1 = s.get(url1)

#this will work
s.cookies.update(response1.cookies.get_dict())

#this will not work
#response2 = s.get(url2,cookies=response1.cookies)


response2 = s.get(url2)

It's due to the way that cookie objects are "not really dictionaries" or something. IDK, I'm not a dev.