What does Python’s socket.recv() return for non-blocking sockets if no data is received until a timeout occurs?
Basically, I’ve read in several places that socket.recv() will return whatever it can read, or an empty string signalling that the other side has shut down (the official docs don’t even mention what it returns when the connection is shut down. great!). This is all fine and dandy for blocking sockets, since we know that recv() only returns when there actually is something to receive, so when it returns an empty string, it MUST mean the other side has closed the connection, right? Okay, fine, but what happens when my socket is non-blocking?? I have searched a bit (maybe not enough, who knows?) and can’t figure out how to tell when the other side has closed the connection using a non-blocking socket. There seems to be no method or attribute that tells us this, and comparing the return value of recv() to the empty string seems absolutely useless. is it just me having this problem? As a simple example, let’s say my socket’s timeout is set to 1.2342342 (whatever non-negative number you like here) seconds and I call socket.recv(1024) , but the other side doesn’t send anything during that 1.2342342 second period. The recv() call will return an empty string and I have no clue as to whether the connection is still standing or not.
«(the official docs don’t even mention what it returns when the connection is shut down. great!)» . Seven years later and still not fixed. This is so frustrating, spent almost an hour trying to figure out why my socket didn’t raise an Error when the connection is closed. Then I was like ok let’s set a timeout then. This also didn’t raise an Exception and i was really confused. Thanks for your post, otherwise i would still be trying to figure this out.
Is this problem still present in Python 3? I assume the OP is likely to have encountered this issue in Python 2, since the question was asked in 2013.
Ed of 2021, the python 3 documentation still does not really provide a clear description of return values for corrupted or dropped connections. Although the example code does look for b» as a logical comparison. Thanks for the question! (and answers) 🙂 docs.python.org/3/library/socket.html
4 Answers 4
In the case of a non blocking socket that has no data available, recv will throw the socket.error exception and the value of the exception will have the errno of either EAGAIN or EWOULDBLOCK. Example:
import sys import socket import fcntl, os import errno from time import sleep s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.connect(('127.0.0.1',9999)) fcntl.fcntl(s, fcntl.F_SETFL, os.O_NONBLOCK) while True: try: msg = s.recv(4096) except socket.error, e: err = e.args[0] if err == errno.EAGAIN or err == errno.EWOULDBLOCK: sleep(1) print 'No data available' continue else: # a "real" error occurred print e sys.exit(1) else: # got a message, do something :) The situation is a little different in the case where you’ve enabled non-blocking behavior via a time out with socket.settimeout(n) or socket.setblocking(False) . In this case a socket.error is stil raised, but in the case of a time out, the accompanying value of the exception is always a string set to ‘timed out’. So, to handle this case you can do:
import sys import socket from time import sleep s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.connect(('127.0.0.1',9999)) s.settimeout(2) while True: try: msg = s.recv(4096) except socket.timeout, e: err = e.args[0] # this next if/else is a bit redundant, but illustrates how the # timeout exception is setup if err == 'timed out': sleep(1) print 'recv timed out, retry later' continue else: print e sys.exit(1) except socket.error, e: # Something else happened, handle error, exit, etc. print e sys.exit(1) else: if len(msg) == 0: print 'orderly shutdown on server end' sys.exit(0) else: # got a message do something :) As indicated in the comments, this is also a more portable solution since it doesn’t depend on OS specific functionality to put the socket into non-blockng mode.
See recv(2) and python socket for more details.
Python – How to Receive Full Data with the recv() Socket function
If you are writing a socket program in python that communicates with some remote machine or server to receive data, then you shall be using the recv() function to receive data on a socket.
The recv() can receive only a specified number of bytes into a string buffer. If the incoming data is larger than the buffer size then only the part of the data is received, and rest gets lost.
This is a problem with the recv function in socket.
Lets take a quick example to understand this:
Code
#Socket client example in python import socket #for sockets import sys #for exit #create an INET, STREAMing socket try: s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) except socket.error: print 'Failed to create socket' sys.exit() print 'Socket Created' host = 'www.google.com'; port = 80; try: remote_ip = socket.gethostbyname( host ) except socket.gaierror: #could not resolve print 'Hostname could not be resolved. Exiting' sys.exit() #Connect to remote server s.connect((remote_ip , port)) print 'Socket Connected to ' + host + ' on ip ' + remote_ip #Send some data to remote server message = "GET / HTTP/1.1\r\n\r\n" try : #Set the whole string s.sendall(message) except socket.error: #Send failed print 'Send failed' sys.exit() print 'Message send successfully' #Now receive data reply = s.recv(4096) print reply
The output of the above code might be something like this :
$ python simple_client.py Socket Created Socket Connected to www.google.com on ip 209.85.175.99 Message send successfully HTTP/1.1 302 Found Location: http://www.google.co.in/ Cache-Control: private Content-Type: text/html; charset=UTF-8 Set-Cookie: expires=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=www.google.com Set-Cookie: path=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=www.google.com Set-Cookie: domain=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=www.google.com Set-Cookie: expires=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=www.google.com Set-Cookie: path=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=www.google.com Set-Cookie: domain=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=www.google.com Set-Cookie: expires=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.www.google.com Set-Cookie: path=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.www.google.com Set-Cookie: domain=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.www.google.com Set-Cookie: expires=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.www.google.com Set-Cookie: path=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.www.google.com Set-Cookie: domain=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.www.google.com Set-Cookie: expires=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=google.com Set-Cookie: path=; expires=Mon, 01-Jan-1990 00:00:00 GMT; $
The problem ? The output is not complete. Some data has been left out. Communication like the above takes places through the TCP/IP protocol.
In this protocol the data transfer takes place in chunks. Lets say a webpage is 500KB in size, but the maximum packet size is only 64KB.
Hence the transfer of the web page will take place in parts or chunks and not the whole thing at once.
Now this is where the problem comes in. The recv function can be made to wait till it receives full data, but for this it must know beforehand the total size of the full data. s.recv(4096 , socket.MSG_WAITALL) will wait till it gets full 4096 bytes.
Now if the actual response size is less than that size, the function will block for a long time before it returns. This is definitely not the desired behaviour we are looking for.
recv() in chunks — The Solution
The solution is to keep looking for data till a decent timeout occurs. And in the next code example we shall precisely do the same.
#Socket client example in python import socket #for sockets import sys #for exit import struct import time #create an INET, STREAMing socket try: s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) except socket.error: print 'Failed to create socket' sys.exit() print 'Socket Created' host = 'www.google.com'; port = 80; try: remote_ip = socket.gethostbyname( host ) except socket.gaierror: #could not resolve print 'Hostname could not be resolved. Exiting' sys.exit() #Connect to remote server s.connect((remote_ip , port)) print 'Socket Connected to ' + host + ' on ip ' + remote_ip #Send some data to remote server message = "GET / HTTP/1.1\r\n\r\n" try : #Set the whole string s.sendall(message) except socket.error: #Send failed print 'Send failed' sys.exit() print 'Message send successfully' def recv_timeout(the_socket,timeout=2): #make socket non blocking the_socket.setblocking(0) #total data partwise in an array total_data=[]; data=''; #beginning time begin=time.time() while 1: #if you got some data, then break after timeout if total_data and time.time()-begin > timeout: break #if you got no data at all, wait a little longer, twice the timeout elif time.time()-begin > timeout*2: break #recv something try: data = the_socket.recv(8192) if data: total_data.append(data) #change the beginning time for measurement begin=time.time() else: #sleep for sometime to indicate a gap time.sleep(0.1) except: pass #join all parts to make final string return ''.join(total_data) #get reply and print print recv_timeout(s) #Close the socket s.close()
The above code will have an output similar to this
Saw the closing html tag at the end ? Now thats complete data and infact that is what the browser displays when google.com is opened. The whole magic takes place inside the function recv_timeout. So lets have a look at how does it work.
def recv_timeout(the_socket,timeout=2): #make socket non blocking the_socket.setblocking(0) #total data partwise in an array total_data=[]; data=''; #beginning time begin=time.time() while 1: #if you got some data, then break after timeout if total_data and time.time()-begin > timeout: break #if you got no data at all, wait a little longer, twice the timeout elif time.time()-begin > timeout*2: break #recv something try: data = the_socket.recv(8192) if data: total_data.append(data) #change the beginning time for measurement begin = time.time() else: #sleep for sometime to indicate a gap time.sleep(0.1) except: pass #join all parts to make final string return ''.join(total_data)
1. Make the socket non-blocking. By doing this, the socket wont wait if there is no data in recv calls. It will continue if there is no data available.
2. Do in a loop the following : keep calling recv, till a timeout occurs or recv finishes up on its own.
Now this is a very simple approach to demonstrate how the recv function ought to work in real applications. The same function can be developed further and made more complex according to the protocol it is working with, for example HTTP.
If you have any feedback or questions let us know in the comments below.
A Tech Enthusiast, Blogger, Linux Fan and a Software Developer. Writes about Computer hardware, Linux and Open Source software and coding in Python, Php and Javascript. He can be reached at [email protected] .
19 Comments
- Anibal May 23, 2022 at 9:09 pm In Python 3.8 this line doesn’t work: return ”.join(total_data) It throws: return ”.join(total_data)
TypeError: sequence item 0: expected str instance, bytes found My data is a bunch of characters (not binary data), but I have control characters in the stream. And if I replace it with: return ”.join(str(total_data)) I’m basically changing the control character \x0b to a literal string with a backslash, followed by x, followed by 0 and then b. That’s not what I want. Not sure how to address this.
- Silver Moon Post author September 28, 2020 at 10:10 pm yes you are correct.
the code is not fully compatible with python 3.
it needs to be edited.