Python async requests post

Asynchronous HTTP Requests in Python with aiohttp and asyncio

Asynchronous code has increasingly become a mainstay of Python development. With asyncio becoming part of the standard library and many third party packages providing features compatible with it, this paradigm is not going away anytime soon.

Let’s walk through how to use the aiohttp library to take advantage of this for making asynchronous HTTP requests, which is one of the most common use cases for non-blocking code.

What is non-blocking code?

You may hear terms like «asynchronous», «non-blocking» or «concurrent» and be a little confused as to what they all mean. According to this much more detailed tutorial, two of the primary properties are:

• Asynchronous routines are able to “pause” while waiting on their ultimate result to let other routines run in the meantime.
• Asynchronous code, through the mechanism above, facilitates concurrent execution. To put it differently, asynchronous code gives the look and feel of concurrency.

So asynchronous code is code that can hang while waiting for a result, in order to let other code run in the meantime. It doesn’t «block» other code from running so we can call it «non-blocking» code.

The asyncio library provides a variety of tools for Python developers to do this, and aiohttp provides an even more specific functionality for HTTP requests. HTTP requests are a classic example of something that is well-suited to asynchronicity because they involve waiting for a response from a server, during which time it would be convenient and efficient to have other code running.

Setting up

Make sure to have your Python environment setup before we get started. Follow this guide up through the virtualenv section if you need some help. Getting everything working correctly, especially with respect to virtual environments is important for isolating your dependencies if you have multiple projects running on the same machine. You will need at least Python 3.7 or higher in order to run the code in this post.

Now that your environment is set up, you’re going to need to install some third party libraries. We’re going to use aiohttp for making asynchronous requests, and the requests library for making regular synchronous HTTP requests in order to compare the two later on. Install both of these with the following command after activating your virtual environment:

pip install aiohttp-3.7.4.post0 requests==2.25.1 

With this you should be ready to move on and write some code.

Making an HTTP Request with aiohttp

Let’s start off by making a single GET request using aiohttp, to demonstrate how the keywords async and await work. We’re going to use the Pokemon API as an example, so let’s start by trying to get the data associated with the legendary 151st Pokemon, Mew.

Run the following Python code, and you should see the name «mew» printed to the terminal:

import aiohttp import asyncio async def main(): async with aiohttp.ClientSession() as session: pokemon_url = 'https://pokeapi.co/api/v2/pokemon/151' async with session.get(pokemon_url) as resp: pokemon = await resp.json() print(pokemon['name']) asyncio.run(main()) 

In this code, we’re creating a coroutine called main , which we are running with the asyncio event loop. In here we are opening an aiohttp client session, a single object that can be used for quite a number of individual requests and by default can make connections with up to 100 different servers at a time. With this session, we are making a request to the Pokemon API and then awaiting a response.

This async keyword basically tells the Python interpreter that the coroutine we’re defining should be run asynchronously with an event loop. The await keyword passes control back to the event loop, suspending the execution of the surrounding coroutine and letting the event loop run other things until the result that is being «awaited» is returned.

Making a large number of requests

Making a single asynchronous HTTP request is great because we can let the event loop work on other tasks instead of blocking the entire thread while waiting for a response. But this functionality truly shines when trying to make a larger number of requests. Let’s demonstrate this by performing the same request as before, but for all 150 of the original Pokemon.

Let’s take the previous request code and put it in a loop, updating which Pokemon’s data is being requested and using await for each request:

import aiohttp import asyncio import time start_time = time.time() async def main(): async with aiohttp.ClientSession() as session: for number in range(1, 151): pokemon_url = f'https://pokeapi.co/api/v2/pokemon/' async with session.get(pokemon_url) as resp: pokemon = await resp.json() print(pokemon['name']) asyncio.run(main()) print("--- %s seconds ---" % (time.time() - start_time)) 

This time, we’re also measuring how much time the whole process takes. If you run this code in your Python shell, you should see something like the following printed to your terminal:

8 seconds seems pretty good for 150 requests, but we don’t really have anything to compare it to. Let’s try accomplishing the same thing synchronously using the requests library.

Comparing speed with synchronous requests

Requests was designed to be an HTTP library «for humans» so it has a very beautiful and simplistic API. I highly recommend it for any projects in which speed might not be of primary importance compared to developer-friendliness and easy to follow code.

To print the first 150 Pokemon as before, but using the requests library, run the following code:

import requests import time start_time = time.time() for number in range(1, 151): url = f'https://pokeapi.co/api/v2/pokemon/' resp = requests.get(url) pokemon = resp.json() print(pokemon['name']) print("--- %s seconds ---" % (time.time() - start_time)) 

You should see the same output with a different runtime:

At nearly 29 seconds, this is significantly slower than the previous code. For each consecutive request, we have to wait for the previous step to finish before even beginning the process. It takes much longer because this code is waiting for 150 requests to finish sequentially

Utilizing asyncio for improved performance

So 8 seconds compared to 29 seconds is a huge jump in performance, but we can do even better using the tools that asyncio provides. In the original example, we are using await after each individual HTTP request, which isn’t quite ideal. It’s still faster than the requests example because we are running everything in coroutines, but we can instead run all of these requests «concurrently» as asyncio tasks and then check the results at the end, using asyncio.ensure_future and asyncio.gather .

If the code that actually makes the request is broken out into its own coroutine function, we can create a list of tasks, consisting of futures for each request. We can then unpack this list to a gather call, which runs them all together. When we await this call to asyncio.gather , we will get back an iterable for all of the futures that were passed in, maintaining their order in the list. This way we’re only awaiting one time.

To see what happens when we implement this, run the following code:

import aiohttp import asyncio import time start_time = time.time() async def get_pokemon(session, url): async with session.get(url) as resp: pokemon = await resp.json() return pokemon['name'] async def main(): async with aiohttp.ClientSession() as session: tasks = [] for number in range(1, 151): url = f'https://pokeapi.co/api/v2/pokemon/' tasks.append(asyncio.ensure_future(get_pokemon(session, url))) original_pokemon = await asyncio.gather(*tasks) for pokemon in original_pokemon: print(pokemon) asyncio.run(main()) print("--- %s seconds ---" % (time.time() - start_time)) 

This brings our time down to a mere 1.53 seconds for 150 HTTP requests! That is a vast improvement over even our initial async/await example. This example is completely non-blocking, so the total time to run all 150 requests is going to be roughly equal to the amount of time that the longest request took to run. The exact numbers will vary depending on your internet connection.

Concluding Thoughts

As you can see, using libraries like aiohttp to rethink the way you make HTTP requests can add a huge performance boost to your code and save a lot of time when making a large number of requests. By default, it is a bit more verbose than synchronous libraries like requests, but that is by design as the developers wanted to make performance a priority.

In this tutorial, we have only scratched the surface of what you can do with aiohttp and asyncio, but I hope that this has made starting your journey into the world of asynchronous Python a little easier.

I’m looking forward to seeing what you build. Feel free to reach out and share your experiences or ask any questions.

Источник

Python aiohttp: How to Send POST Requests

To send POST requests with Python aiohttp first create a session object using the aiohttp.ClientSession() method and then use the post() method on that session object. Next, add the POST body and Content-Type using the body and headers parameters.

  import aiohttp import asyncio  async def post_request(): async with aiohttp.ClientSession() as session:  response = await session.post(url="https://httpbin.org/post",  data="key": "value">,  headers="Content-Type": "application/json">) print(await response.json()) asyncio.run(post_request()) 

In this guide for The Python Web Scraping Playbook, we will look at how to make POST requests with the Python aiohttp library.

In this guide we will walk you through the most common ways of sending POST requests with Python aiohttp:

POST JSON Data Using Python aiohttp​

A common scenario for using POST requests is to send JSON data to an API endpoint, etc. Doing this with Python aioHTTP is very simple.

Here we will use Python aiohttp’s Session functionality to send POST requests.

We need to use aiohttp.ClientSession() to create a new instance of the ClientSession class. The async with statement is used to create a context manager for the aiohttp.ClientSession() to manage the life cycle of the session object. Next, we make a POST request using the session.post() method. Then, We simply need to add the data to the request using the json parameter of the POST request:

  import aiohttp import asyncio  async def post_request():  url = "https://httpbin.org/post"  data = "key": "value">  async with aiohttp.ClientSession() as session:  response = await session.post(url, json=data) print(await response.json()) asyncio.run(post_request()) 

The aiohttp library will automatically encode the data as JSON and set the Content-Type header to application/json .

This approach can be simpler and more concise than manually encoding the data and setting the headers. Additionally, it may offer some performance benefits, as the Python aiohttp library can use a more efficient encoding method for JSON data.

POST Form Data Using Python aioHTTP​

Another common use case for using POST requests is to send form data to an endpoint.

We simply just need to add the data to the request using the data parameter of the POST request:

  import aiohttp import asyncio  async def post_request():  url = "https://httpbin.org/post"  data = "key": "value">  async with aiohttp.ClientSession() as session:  response = await session.post(url, data=data) print(await response.text()) asyncio.run(post_request()) 

The aiohttp library will automatically encode the data as JSON and set the Content-Type header to application/x-www-form-urlencoded so you don’t have to set any headers.

Configuring Data Types​

As we’ve seen above when you use the data or json parameter to send data with the POST request is defaults the Content-Type header to either application/json or application/x-www-form-urlencoded .

However, if you would like to override this or send data with another Content-Type then you can do so by just adding the Content-Type header to the POST request.

In the following example, we will send JSON data using the data parameter instead of the json parameter as we did previously.

  import aiohttp import asyncio import json  async def post_request():  url = "https://httpbin.org/post"  data = "key": "value">  async with aiohttp.ClientSession() as session:  headers = "Content-Type": "application/custom-type">  json_data = json.dumps(data)  response = await session.post(url, data=json_data, headers=headers) print(await response.json()) asyncio.run(post_request())  ---  ## More Web Scraping Tutorials So that's how you can send POST requests using **Python aiohttp**. If you would like to learn more about Web Scraping, then be sure to check out [The Web Scraping Playbook](/web-scraping-playbook). Or check out one of our more in-depth guides:  - [How to Scrape The Web Without Getting Blocked Guide](/web-scraping-playbook/web-scraping-without-getting-blocked) - [The State of Web Scraping 2020](https://scrapeops.io/blog/the-state-of-web-scraping-2022) - [The Ethics of Web Scraping](/web-scraping-playbook/ethics-of-web-scraping) 

Источник