Python http error 404

How to catch HTTP 404 error in Python

Hi guys! In this tutorial, we will learn how to catch an HTTP 404 error in Python. There are many instances when we encounter an HTTP error with error code 404 on the internet. This error indicates that the requested page was not found. This tutorial will teach you how you can use Python to spot pages with such an error. Let’s see more on this with example programs.

Catch HTTP 404 error in Python

There are many methods for catching a 404 error. We will import the urllib or urllib3 library in this post for catching a 404 error. These library has required methods and attributes for our purpose. Let’s see the code and understand how it’s done.

Method 1

See the below example to find out a page with an HTTP 404 error.

import urllib3 http = urllib3.PoolManager() error = http.request("GET", "https://www.google.com/404") if (error.status == 404): print("HTTP 404 ERROR")

The output for the above code is:

In the above example, we have used the PoolManager of urllib3 to create an http object. A PoolManager instance is needed to make requests. Then we have used the request method to get an HTTPResponse object for the web page ‘https://www.google.com/404’. This HTTPResponse object contains status, data, and header. We check whether the status for this page is 404 if so we print that this is an HTTP error.

Method 2

In this method, we will import the Python urllib library. Have a look at the given code and try to understand.

import urllib.request, urllib.error try: con = urllib.request.urlopen('http://www.google.com/404') except urllib.error.HTTPError as err: print('HTTP', err.code, 'ERROR')

The output of the program:

As you can see, the urlopen() method opens the given URL for us. We have used try and except blocks to catch the HTTP error using urllib.error.HTTPError. If any error is raised while opening the URL, the control is given to except block and there we print the error code for HTTP error.

Источник

Getting HTTP 404 Error when Web Scraping in Python 3.7

I’m receiving a 404 error code (shown below) when trying to web scrape a certain website. I’ve tried looking for answers on different forums but couldn’t find a solution Anyone have a solution to fix this 404 error? The Website URL is https://www.transfermarkt.com/stoke-city/startseite/verein/512/saison_id/2018.

>>> from urllib.request import urlopen as uReq >>> from urllib.request import Request >>> from bs4 import BeautifulSoup as soup >>> my_url = "https://www.transfermarkt.com/stoke-city/startseite/verein/512/saison_id/2018" >>> uClient = uReq(my_url) Traceback (most recent call last): File "", line 1, in File "C:\Users\King Carmo\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 222, in urlopen return opener.open(url, data, timeout) File "C:\Users\King Carmo\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 531, in open response = meth(req, response) File "C:\Users\King Carmo\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 641, in http_response 'http', request, response, code, msg, hdrs) File "C:\Users\King Carmo\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 569, in error return self._call_chain(*args) File "C:\Users\King Carmo\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 503, in _call_chain result = func(*args) File "C:\Users\King Carmo\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 649, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 404: Not Found 

Источник

HTTP Error 404: Not Found python urllib

Traceback (most recent call last): File «C:/Users/alext/AppData/Local/Programs/Python/Python36/Weather forecast.py», line 9, in data = urllib.request.urlopen(url).read() File «C:\Users\alext\AppData\Local\Programs\Python\Python36\lib\urllib\request.py», line 223, in urlopen return opener.open(url, data, timeout) File «C:\Users\alext\AppData\Local\Programs\Python\Python36\lib\urllib\request.py», line 532, in open response = meth(req, response) File «C:\Users\alext\AppData\Local\Programs\Python\Python36\lib\urllib\request.py», line 642, in http_response ‘http’, request, response, code, msg, hdrs) File «C:\Users\alext\AppData\Local\Programs\Python\Python36\lib\urllib\request.py», line 564, in error result = self._call_chain(*args) File «C:\Users\alext\AppData\Local\Programs\Python\Python36\lib\urllib\request.py», line 504, in _call_chain result = func(*args) File «C:\Users\alext\AppData\Local\Programs\Python\Python36\lib\urllib\request.py», line 756, in http_error_302 return self.parent.open(new, timeout=req.timeout) File «C:\Users\alext\AppData\Local\Programs\Python\Python36\lib\urllib\request.py», line 532, in open response = meth(req, response) File «C:\Users\alext\AppData\Local\Programs\Python\Python36\lib\urllib\request.py», line 642, in http_response ‘http’, request, response, code, msg, hdrs) File «C:\Users\alext\AppData\Local\Programs\Python\Python36\lib\urllib\request.py», line 570, in error return self._call_chain(*args) File «C:\Users\alext\AppData\Local\Programs\Python\Python36\lib\urllib\request.py», line 504, in _call_chain result = func(*args) File «C:\Users\alext\AppData\Local\Programs\Python\Python36\lib\urllib \request.py», line 650, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 404: Not Found

I have checked the url and it is definitely correct. I have seen others with problems like this but am still unsure as to the solution.

Источник

How to Fix HTTPError in Python

The urllib.error.HTTPError is a class in the Python urllib library that represents an HTTP error. An HTTPError is raised when an HTTP request returns a status code that represents an error, such as 4xx (client error) or 5xx (server error).

HTTPError Attributes

The urllib.error.HTTPError class has the following attributes:

• code : The HTTP status code of the error.
• reason : The human-readable reason phrase associated with the status code.
• headers : The HTTP response headers for the request that caused the HTTPError .

What Causes HTTPError

Here are some common reasons why an HTTPError might be raised:

• Invalid or malformed request URL.
• Invalid or malformed request parameters or body.
• Invalid or missing authentication credentials.
• Server internal error or malfunction.
• Server temporarily unavailable due to maintenance or overload.

Python HTTPError Examples

Here are a few examples of HTTP errors in Python:

404 Not Found

 import urllib.request import urllib.error try: response = urllib.request.urlopen('http://httpbin.org/status/404') except urllib.error.HTTPError as err: print(f'A HTTPError was thrown: ') 

In the above example, an invalid URL is attempted to be opened using the urllib.request.urlopen() function. Running the above code raises an HTTPError with code 404:

 A HTTPError was thrown: 404 NOT FOUND 

400 Bad Request

 import urllib.request try: response = urllib.request.urlopen('http://httpbin.org/status/400') except urllib.error.HTTPError as err: if err.code == 400: print('Bad request!') else: print(f'An HTTP error occurred: ') 

In the above example, a bad request is sent to the server. Running the above code raises a HTTPError with code 400:

401 Unauthorized

 import urllib.request import urllib.error try: response = urllib.request.urlopen('http://httpbin.org/status/401') except urllib.error.HTTPError as err: if err.code == 401: print('Unauthorized!') else: print(f'An HTTP error occurred: ') 

In the above example, a request is sent to the server with missing credentials. Running the above code raises a HTTPError with code 401:

500 Internal Server Error

 import urllib.request import urllib.error try: response = urllib.request.urlopen('http://httpbin.org/status/500') except urllib.error.HTTPError as err: if err.code == 500: print('Internal server error!') else: print(f'An HTTP error occurred: ') 

In the above example, the server experiences an error internally. Running the above code raises a HTTPError with code 500:

How to Fix HTTPError in Python

To fix HTTP errors in Python, the following steps can be taken:

1. Check the network connection and ensure it is stable and working.
2. Check the URL being accessed and make sure it is correct and properly formatted.
3. Check the request parameters and body to ensure they are valid and correct.
4. Check whether the request requires authentication credentials and make sure they are included in the request and are correct.
5. If the request and URL are correct, check the HTTP status code and reason returned in the error message. This can give more information about the error.
6. Try adding error handling code for the specific error. For example, the request can be attempted again or missing parameters can be added to the request.

Track, Analyze and Manage Errors With Rollbar

Managing errors and exceptions in your code is challenging. It can make deploying production code an unnerving experience. Being able to track, analyze, and manage errors in real-time can help you to proceed with more confidence. Rollbar automates error monitoring and triaging, making fixing Python errors easier than ever. Try it today!

Источник