Example Domain

Downloading files with the Requests library

The Requests package isn’t part of Python’s standard library. But the way that it wraps up Python’s standard HTTP functionality into a simple, elegant interface makes it one of the most widely used external libraries. This guide contains various examples and use cases.

Quick cheat sheet

A quick guide to common downloading tasks.

Downloading a file

import requests resp = requests.get("http://www.example.com") ## Reading as text resp.text ## Read as bytes resp.content 

Downloading a URL with parameters

To fetch a URL contains a query string, e.g.:

http://www.example.com?name=Daniel&id=123456 

The query string is: ?name=Daniel&id=123456

We can pass a dict into the params argument of the get() method. It will serialize the dict as the query string:

import requests resp = requests.get("http://www.example.com", params = "name":"Daniel", "id": 123456>) print(resp.url) # http://www.example.com/?id=123456&name=Daniel 

About the Requests library

Our primary library for downloading data and files from the Web will be Requests, dubbed «HTTP for Humans».

To bring in the Requests library into your current Python script, use the import statement:

You have to do this at the beginning of every script for which you want to use the Requests library.

Note: If you get an error, i.e. ImportError , it means you don’t have the requests library installed. Email me if you’re having that issue, because it likely means you probably don’t have Anaconda installed properly.

The get method

The get method of the requests module is the one we will use most frequently – which corresponds to how the majority of the HTTP requests your browser makes involve the GET method. Even without knowing much about HTTP, the concept of GET is about as simple as its name: it will get a resource from a web server.

The get() method requires one argument: a web URL, e.g. http://www.example.com . The URL’s scheme – i.e. «http://» – is required, even though you probably never type it out in your browser.

Run this from the interactive prompt:

>>> requests.get("http://www.example.com") Response [200]> 

You might have expected the command to just dump the text contents of http://www.example.com to the screen. But it turns out there’s a lot more to getting a webpage than just getting what you see rendered in your browser.

You can see this for yourself by popping open the Developer Tools (in Chrome, for OSX, the shortcut is: Command-Alt-J), clicking the Network panel, then visiting a page:

What each of those various attributes mean isn’t important to figure out now, it’s just enough to know that they exist as part of every request for a web resource, whether it’s a webpage, image file, data file, etc.

Returning to our previous code snippet, let’s assign the result of the requests.get() command to a variable, then inspect that variable. I like using resp for the variable name – short for «response»

>>> resp = requests.get("http://www.example.com") 

Use the type() function to see what that resp object actually is:

>>> type(resp) requests.models.Response 

If you want to get the text of a successful requests.get() response, use its text attribute:

>>> resp = requests.get("http://www.example.com") >>> print(resp.text) 

The output will look like this:

       # . and so on 

Computational Methods in the Civic Sphere is produced by Dan Nguyen as an online reference for the Stanford Computational Journalism Lab and Masters in Journalism curriculum.

Links

Источник

Python How to Download a File from a URL

To download a file from a URL using Python, use the requests.get() method. For example, let’s download Instagram’s icon:

import requests URL = "https://instagram.com/favicon.ico" response = requests.get(URL) open("instagram.ico", "wb").write(response.content)

This is an example for someone who is looking for a quick answer. But if the above code lines don’t work or make sense, please keep on reading.

More Detailed Steps

To download a file from a URL using Python follow these three steps:

1. Install requests module and import it to your project.
2. Use requests.get() to download the data behind that URL.
3. Write the file to a file in your system by calling open().

Let’s download Instagram’s icon using Python. The icon can be found behind this URL https://instagram.com/favicon.ico.

First, install the requests module by opening up a command line window and running:

Then, you can use it to download the icon behind the URL:

# 1. Import the requests library import requests URL = "https://instagram.com/favicon.ico" # 2. download the data behind the URL response = requests.get(URL) # 3. Open the response into a new file called instagram.ico open("instagram.ico", "wb").write(response.content)

As a result of running this piece of code, you see the Instagram icon appear in the same folder where your program file is.

Other Ways to Download a File in Python

There are other modules that make downloading files possible in Python.

In addition to the requests library, the two commonly used ones are:

How to Download a File Using wget Module

Before you can download files using wget , you need to install the wget module.

Open up a command line window and run:

Then follow these two steps to download a file:

1. Import the wget module into your project.
2. Use wget.download() to download a file from a specific URL and save it on your machine.

As an example, let’s get the Instagram icon using wget :

import wget URL = "https://instagram.com/favicon.ico" response = wget.download(URL, "instagram.ico")

As a result of running the code, you can see an Instagram icon appear in the folder of your program.

How to Download a File Using urllib Module in Python

Before you can download files using urllib , you need to install the module. Open up a command line window and run:

Then follow these two steps to download a file:

1. Import the urllib module into your project.
2. Use urllib‘s request.urlretrieve() method to download a file from a specific URL and save it on your machine.

As an example, let’s get the Instagram icon using urllib :

from urllib import request URL = "https://instagram.com/favicon.ico" response = request.urlretrieve("https://instagram.com/favicon.ico", "instagram.ico")

As a result of running the code, you can see an Instagram icon appear in the folder of your program.

Источник

How to Download Files in Python

Esther Vaati Last updated Dec 29, 2022

Python provides several ways to download files from the internet. This can be done over HTTP using the urllib package or the requests library. This tutorial will discuss how to use these libraries to download files from URLs using Python.

Requests Library

The requests library is one of the most popular libraries in Python. Requests allow you to send HTTP/1.1 requests without the need to manually add query strings to your URLs, or form-encode your POST data.

With the requests library, you can perform a lot of functions including:

• adding form data
• adding multipart files
• accessing the response data of Python

Making Requests

The first you need to do is to install the library, and it’s as simple as:

To test if the installation has been successful, you can do a very easy test in your Python interpreter by simply typing:

If the installation has been successful, there will be no errors.

Making a GET request

Making requests is very easy, as illustrated below.

req = requests.get(“https://www.google.com”) 

The above command will get the google web page and store the information in the req variable. We can then go on to get other attributes as well.

For instance, to know if fetching the Google web page was successful, we will query the status_code .

req = requests.get(“https://www.google.com") 

Источник