Python selenium cookies to requests

Невозможно передать куки между селеном и запросами, чтобы выполнить очистку, используя последний

Я написал скрипт на python в сочетании с селеном для входа на сайт и передачи файлов cookie из driver в requests , чтобы я мог продолжать использовать requests для выполнения дальнейших действий.

Я использовал item = soup.select_one(«div[class^=’gravatar-wrapper-‘]»).get(«title») эту строку, чтобы проверить, может ли скрипт получить мое имя пользователя, когда все будет сделано.

Это моя попытка до сих пор:

import requests from bs4 import BeautifulSoup from selenium import webdriver from selenium.webdriver.common.keys import Keys url = "https://stackoverflow.com/users/login" driver = webdriver.Chrome() driver.get(url) driver.find_element_by_css_selector("#email").send_keys("your_username") driver.find_element_by_css_selector("#password").send_keys("your_password") driver.find_element_by_css_selector("#submit-button").click() driver_cookies = driver.get_cookies() c = res = requests.get(driver.current_url,cookies=c) soup = BeautifulSoup(res.text,"lxml") item = soup.select_one("div[class^='gravatar-wrapper-']").get("title") print(item) driver.quit() 

Когда я запускаю свой скрипт, он не находит имя пользователя и выдает None в качестве вывода.

Как передать файлы cookie между selenium и requests , чтобы выполнить очистку с помощью requests после того, как я войду в систему с использованием селена?

3 ответа

Вы уже на правильном пути. Все, что вам нужно сделать сейчас, это заставить скрипт немного подождать, пока загрузятся куки. Вот как вы можете получить ответ:

import time import requests from bs4 import BeautifulSoup from selenium import webdriver from selenium.webdriver.common.keys import Keys url = "https://stackoverflow.com/users/login" with webdriver.Chrome() as driver: driver.get(url) driver.find_element_by_css_selector("#email").send_keys("your_username") driver.find_element_by_css_selector("#password").send_keys("your_password") driver.find_element_by_css_selector("#submit-button").click() time.sleep(5) #This is the fix driver_cookies = driver.get_cookies() c = res = requests.get(driver.current_url,cookies=c) soup = BeautifulSoup(res.text,"lxml") item = soup.select_one("div[class^='gravatar-wrapper-']").get("title") print(item) 

Расширяет классы Selenium WebDriver для включения функции запроса из библиотеки запросов, выполняя все необходимые cookie и Обработка заголовков запроса .

В моем случае это помогло мне, дайте нам знать, если это работает в вашем случае ..

 import requests from selenium import webdriver driver = webdriver.Firefox() url = "some_url" #a redirect to a login page occurs driver.get(url) #storing the cookies generated by the browser request_cookies_browser = driver.get_cookies() #making a persistent connection using the requests library params = s = requests.Session() #passing the cookies generated from the browser to the session c = [s.cookies.set(c['name'], c['value']) for c in request_cookies_browser] resp = s.post(url, params) #I get a 200 status_code #passing the cookie of the response to the browser dict_resp_cookies = resp.cookies.get_dict() response_cookies_browser = [ for name, value in dict_resp_cookies.items()] c = [driver.add_cookie(c) for c in response_cookies_browser] #the browser now contains the cookies generated from the authentication driver.get(url) 

Источник

Как преобразовать куки из selenium так чтобы они работали с requests?

Я хочу авторизоваться на сайте через куки.
Использую при этом requests. Куки беру из selenium.
Как мне преобразовать куки которые я взял из selenium к такому формату который понимает requests.
Или как достать куки без selenium?
Изначально авторизоваться на сайте через requests не получается.

import os, json, requests from selenium import webdriver from selenium.common import exceptions from selenium.webdriver.chrome.service import Service from selenium.webdriver.common.keys import Keys from selenium.webdriver.common.by import By from selenium.webdriver.common.desired_capabilities import DesiredCapabilities from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as ec from webdriver_manager.chrome import ChromeDriverManager if __name__ == '__main__': options = webdriver.ChromeOptions() options.add_argument('user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36') options.add_argument('--headless') options.add_argument('--user-data-dir=/app/google-chrome') # В моем случае, это Selenium Grid 4 driver = webdriver.Remote(command_executor=grid_url, options=options, desired_capabilities=<>) driver.get("https://domain.com/some-page-with-cookies") # Делаем авторизацию, если нужна # . # Забираем куки и удаляем сессию driver_cookies = driver.get_cookies() driver.close() driver.quit() cookies = <> for cookie in driver_cookies: cookies[ cookie['name'] ] = cookie['value'] r = requests.get('https://domain.com/target-url', cookies=cookies)

Источник

Passing cookies from Selenium to Python Requests

Reference: https://medium.com/geekculture/how-to-share-cookies-between-selenium-and-requests-in-python-d36c3c8768b Question: i have this code to save cookies from requests i wanna to use it in selenium browser In other words, how can we capture cookies with Selenium and use those cookies in Requests? I tried the suggested answer in this question, but it is not correct and the question is over a year old without any other answers.

Passing cookies from Selenium to Python Requests

I am making a Python 3-script with Selenium 4 and Gecko web driver. I am using cookies = driver.get_cookies() to capture cookies after logging in to a site.

The question is how I can cookies from Selenium in a GET request using the Requests module. In other words, how can we capture cookies with Selenium and use those cookies in Requests?

I tried the suggested answer in this question, but it is not correct and the question is over a year old without any other answers.

cookies = driver.get_cookies() requests_cookies = <> for c in cookies: requests_cookies[c['name']] = c['value'] response = requests.get('http://some-host. ', cookies=requests_cookies) 

Python — Requests, Selenium — passing cookies while logging in, I finally found out what the problem was. Before making the post request with the requests library, I should have passed the cookies of the

Reuse sessions using cookies in Python Selenium

In this quick video I am showing how to reuse cookies in selenium.Present your Github Duration: 10:34

How do I load session and cookies from requests library to Selenium browser in Python?

i have this code to save cookies from requests

pickle.dump(session.cookies.get_dict(), open("cookies.pkl", "wb")) 

i wanna to use it in selenium browser so i used this code but it doesn’t work

cookies = pickle.load(open("cookies.pkl", "rb")) for cookie in cookies: driver.add_cookie(cookie) 

driver.add_cookie(cookie) File "C:\Users\Allah\AppData\Local\Programs\Python\Python38-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 894, in add_cookie self.execute(Command.ADD_COOKIE, ) File "C:\Users\Allah\AppData\Local\Programs\Python\Python38-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute self.error_handler.check_response(response) File "C:\Users\Allah\AppData\Local\Programs\Python\Python38-32\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response raise exception_class(message, screen, stacktrace) selenium.common.exceptions.InvalidArgumentException: Message: invalid type: string "wa-xd-sessionid", expected struct AddCookieParameters at line 1 column 28 

ses = requests.session() ses.get('https://www.google.com/') driver = webdriver.Chrome() driver.get('https://www.google.com/') for item in ses.cookies: driver.add_cookie( ) 

I’m not able to use python requests session cookies in selenium, By clicking “Accept all cookies”, you agree Stack Exchange can store cookies on your device and disclose information in accordance with our

Exporting Requests cookies into webdriver

Essentially I have a program that makes use of the requests library and its post request . I want to take the cookies of the session after the post request has been called and load them into a webdriver. I was thinking of making use of selenium and a chrome binary but I am confused on how to go about it.

Basically what I have this far.

import requests url=www.storeUrl.com session=requests.Session() data= session.cookies.clear() response=session.post(url,data=data) storeResponse=session.request('get','http://www.storeUrl.com') print storeResponse.cookies 

class ‘requests.cookies.RequestsCookieJar’>Cookie _store_session=BAh7CUkiD3Nlc3Npb25faWQGOgZFRkkiJTBiYmY4MmEzNmRmMjZkMjNhZDdiODg4NWVmYWQ5Y2IzBjsAVEkiB3RqBjsARnsLSSIHcDAGOwBGSXU6CVRpbWUNte4cgFPoSgUKOgtAX3pvbmVJIghFU1QGOwBUOg1uYW5vX251bWkCGgE6DW5hbm9fZGVuaQY6DXN1Ym1pY3JvIgcoIDoLb2Zmc2V0af6wuUkiB3AxBjsARjBJIgdwMgY7AEYwSSIHY3MGOwBGMEkiB2NjBjsARjBJIghpcHMGOwBGWwYiETI0LjkxLjIyNi4zNkkiCWNhcnQGOwBGewdpAph7aQY6C2Nvb2tpZUkiHTEgaXRlbS0tJDM2LS0zMTY0MCwxMjY0MwY7AFRJIhBfY3NyZl90b2tlbgY7AEZJIjFxNHI4QWFUQWNWaXZmY2xIVlNPcHRQeUk2ODF2NTVhbm9pREE1YWFSOHpNPQY7AEY%3D—eea073c1f0a4fd19163e39536e75eed04ab788f9 for www.storeUrl.com/>]>

How would I go about loading this cookie into selenium? Any help would be greatly appreciated.

Selenium has built in add_cookie method for adding cookies to current session:

In [4]: browser.add_cookie?? Type: instancemethod String form: > File: /usr/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py Definition: browser.add_cookie(self, cookie_dict) Source: def add_cookie(self, cookie_dict): """ Adds a cookie to your current session. :Args: - cookie_dict: A dictionary object, with required keys - "name" and "value"; optional keys - "path", "domain", "secure", "expiry" Usage: driver.add_cookie() driver.add_cookie() driver.add_cookie() """ self.execute(Command.ADD_COOKIE, ) 

Basically, you need to pass a dict with cookies to add_cookie method and session.cookies.get_dict() returns dict with cookies:

• adding cookies using Selenium : https://stackoverflow.com/a/15058521/2517622
• getting cookies using requests : https://stackoverflow.com/a/25092059/2517622

from selenium import webdriver driver = webdriver.Firefox() cookeis = storeResponse.cookies # cookiejar like in question driver.get('') # catch InvalidCookieDomainException without in for key in cookies.keys(): # print() # just print driver.add_cookie() 

Selenium-requests, Extends Selenium WebDriver classes to include the request function from the Requests library, while doing all the needed cookie and request headers

Источник

Python – Requests, Selenium – passing cookies while logging in

I would like to integrate python Selenium and Requests modules to authenticate on a website.

I am using the following code:

import requests from selenium import webdriver driver = webdriver.Firefox() url = "some_url" #a redirect to a login page occurs driver.get(url) #the login page is displayed #making a persistent connection to authenticate params = 'os_username':'username', 'os_password':'password'> s = requests.Session() resp = s.post(url, params) #I get a 200 status_code #passing the cookies to the driver driver.add_cookie(s.cookies.get_dict()) 

The problem is that when I enter the browser the login authentication is still there when I try to access the url even though I passed the cookies generated from the requests session.

How can I modify the code above to get through the authentication web-page?

Can anyone help me on this issue?
Your help is much appreciated.
Best Regards.

I finally found out what the problem was.
Before making the post request with the requests library, I should have passed the cookies of the browser first.
The code is as follows:

import requests from selenium import webdriver driver = webdriver.Firefox() url = "some_url" #a redirect to a login page occurs driver.get(url) #storing the cookies generated by the browser request_cookies_browser = driver.get_cookies() #making a persistent connection using the requests library params = 'os_username':'username', 'os_password':'password'> s = requests.Session() #passing the cookies generated from the browser to the session c = [s.cookies.set(c['name'], c['value']) for c in request_cookies_browser] resp = s.post(url, params) #I get a 200 status_code #passing the cookie of the response to the browser dict_resp_cookies = resp.cookies.get_dict() response_cookies_browser = ['name':name, 'value':value> for name, value in dict_resp_cookies.items()] c = [driver.add_cookie(c) for c in response_cookies_browser] #the browser now contains the cookies generated from the authentication driver.get(url) 

I had some issues with this code because its set double cookies to the original browser cookie (before login) then I solve this with cleaning the cookies before set the login cookie to original. I used this command:

Источник