When working with HTTP in Python, the primary tool at your disposal is the HTTPConnection class from the http.client module. It provides a simple way to send requests to a web server and receive responses. At its core, this class abstracts the lower-level details of networking into a more manageable form. You’ll find it especially useful when you need to make direct HTTP requests without the added complexity of higher-level libraries.
from http.client import HTTPConnection
# Initialize an HTTP connection
conn = HTTPConnection("www.example.com")
# Send a GET request
conn.request("GET", "/")
# Get the response
response = conn.getresponse()
# Print the status and reason
print(response.status, response.reason)
# Read the response data
data = response.read()
print(data)
# Close the connection
conn.close()
The connection needs to be established with the correct host and can also be configured to use specific ports. The default port for HTTP is 80, but if you’re dealing with HTTPS, be prepared to use port 443. In the snippet above, we initiate a connection to “www.example.com,” execute a GET request, and print out the results.
One of the critical features of HTTPConnection is that it handles connection pooling for you. You don’t need to manage connections manually; the class takes care of maintaining connections under the hood. However, that doesn’t mean you should abuse it; for better performance, you might want to keep a single connection open for multiple requests rather than constantly opening and closing connections.
conn = HTTPConnection("www.example.com")
for path in ["/", "/about", "/contact"]:
conn.request("GET", path)
response = conn.getresponse()
print(f"{path}: {response.status} - {response.reason}")
conn.close()
This example shows how you can reuse the same connection to hit multiple endpoints without the overhead of establishing a new connection each time. This approach can substantially reduce the latency in your applications, especially when making multiple requests in rapid succession.
Error handling is another crucial aspect. HTTP connections can fail due to various reasons, such as network issues or server unavailability. It’s essential to implement try-except blocks around your connection and request code. For example:
try:
conn.request("GET", "/")
response = conn.getresponse()
except Exception as e:
print(f"An error occurred: {e}")
finally:
conn.close()
Incorporating error handling not only helps prevent your application from crashing but also provides meaningful feedback when something goes wrong. Remember, robust applications anticipate potential failures.
For more intricate applications, you may need to send headers along with your requests to inform the server about the type of content you’re expecting, or the kind of operations you want to perform. Setting custom headers can be done quite effortlessly:
headers = {"User-Agent": "MyApp/1.0", "Accept": "application/json"}
conn.request("GET", "/", headers=headers)
Understanding how to set headers correctly can be fundamental in communicating effectively with RESTful APIs or any web services that impose strict requirements on request formats. This is also where you can implement authentication if the server demands it.
Ultimately, mastering HTTPConnection gives you foundational knowledge about the HTTP protocol and how web clients function. With that knowledge, you can move on to more sophisticated libraries like requests…
MoKo for iPad (A16) 11th Generation Case 11 Inch 2025, iPad 10th Generation Case 10.9 Inch 2022, Slim Stand Hard PC Translucent Back Shell Smart Cover, Support Touch ID, Auto Wake/Sleep, Navy Blue
$9.95 (as of July 17, 2026 15:16 GMT +00:00 - More infoProduct prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on [relevant Amazon Site(s), as applicable] at the time of purchase will apply to the purchase of this product.)Common use cases for HTTP client connections
The most frequent task you’ll perform is likely querying a REST API. Most modern APIs speak JSON, so you’ll often find yourself fetching data and decoding it. While http.client doesn’t have a built-in JSON decoder, combining it with Python’s json module is straightforward. You make the request, read the response body, and then pass the raw bytes to json.loads.
import json
from http.client import HTTPSConnection
# Note: many APIs use HTTPS, so we use HTTPSConnection
conn = HTTPSConnection("api.github.com")
headers = {
"User-Agent": "My-Awesome-App/1.0",
"Accept": "application/vnd.github.v3+json"
}
conn.request("GET", "/users/joelspolsky", headers=headers)
response = conn.getresponse()
print(f"Status: {response.status} {response.reason}")
if response.status == 200:
# Read the raw byte data
data = response.read()
# Decode the JSON
user_data = json.loads(data)
print(f"User: {user_data['login']}, Name: {user_data['name']}")
print(f"Public Repos: {user_data['public_repos']}")
else:
print("Failed to retrieve data.")
conn.close()
Notice we’re using HTTPSConnection here. It’s a critical distinction. If you try to use HTTPConnection on an https:// endpoint, you’ll get smacked with errors. Also, pay close attention to the User-Agent header; many APIs, like GitHub’s, will reject requests without one.
Of course, you don’t just want to get data; you want to send it, too. For this, you’ll use the POST method. The data you send, known as the request body, needs to be encoded correctly. For classic web forms, this is typically application/x-www-form-urlencoded. For modern APIs, it’s almost always application/json.
Here’s how you’d post JSON data. The key is to serialize your Python dictionary into a JSON string and then encode it to bytes before sending.
import json
from http.client import HTTPSConnection
# Data to be sent
post_data = {
"title": "My New Post",
"body": "This is the content of the post.",
"userId": 1
}
# Serialize and encode the data
encoded_data = json.dumps(post_data).encode('utf-8')
conn = HTTPSConnection("jsonplaceholder.typicode.com")
headers = {
"Content-Type": "application/json; charset=UTF-8",
"Content-Length": str(len(encoded_data))
}
conn.request("POST", "/posts", body=encoded_data, headers=headers)
response = conn.getresponse()
print(f"Status: {response.status} {response.reason}")
response_data = json.loads(response.read().decode('utf-8'))
print("Response from server:")
print(response_data)
conn.close()
Two things are absolutely vital here. First, the Content-Type header tells the server you’re sending JSON. Without it, the server might not know how to parse your request body. Second, the Content-Length header is required. It tells the server exactly how many bytes to expect in the body. Forgetting this is a common source of bugs that will leave you scratching your head for hours.
Here’s something that trips up everyone who moves from a high-level library like requests to http.client: it does not automatically follow redirects. If you make a request and get a 301 or 302 status code, requests would seamlessly handle it and get you the final page. http.client just gives you the redirect response and expects you to deal with it. This is both a pain and a feature; it gives you full control.
from http.client import HTTPConnection
from urllib.parse import urlparse
def get_final_url(host, path, max_redirects=5):
conn = HTTPConnection(host)
redirect_count = 0
while redirect_count < max_redirects:
conn.request("GET", path)
response = conn.getresponse()
print(f"Requesting http://{host}{path} -> Status: {response.status}")
if response.status in (301, 302, 303, 307, 308):
redirect_location = response.getheader('Location')
if not redirect_location:
print("Redirect status with no Location header. Halting.")
break
print(f"Redirecting to: {redirect_location}")
parsed_url = urlparse(redirect_location)
if parsed_url.hostname and parsed_url.hostname != host:
print("Cross-domain redirect detected. Halting for this example.")
break
host = parsed_url.hostname or host
path = parsed_url.path or "/"
redirect_count += 1
else:
print("Final destination reached.")
response.read() # Consume the response body
break
else:
print("Maximum redirects exceeded.")
conn.close()
# An old URL that now redirects
get_final_url("httpbin.org", "/redirect-to?url=/get")
This code is more complex, but it reveals what’s happening under the hood. You check the status, pull the Location header, and make a new request. You even have to parse the new URL and potentially connect to a completely different host. This is the kind of low-level grunt work that higher-level libraries were invented to eliminate, but understanding it is invaluable when things go wrong.
What if you need to download a 2GB ISO file? Calling response.read() would try to load the entire thing into RAM, which is a recipe for disaster. Instead, you can read the response in chunks. This is a perfect use case for http.client where its lower-level nature gives you the control you need.
from http.client import HTTPSConnection
conn = HTTPSConnection("cachefly.cachefly.net")
# Requesting a 100MB test file
conn.request("GET", "/100mb.test")
response = conn.getresponse()
if response.status == 200:
with open("downloaded_file.test", "wb") as f:
chunk_size = 8192 # 8 KB
bytes_downloaded = 0
while True:
chunk = response.read(chunk_size)
if not chunk:
# We've read the entire response
break
f.write(chunk)
bytes_downloaded += len(chunk)
print(f"Downloaded {bytes_downloaded} bytes...", end='r')
print("nDownload complete.")
else:
print(f"Failed to download. Status: {response.status}")
conn.close()
By calling response.read(chunk_size), you pull data from the socket buffer in manageable pieces. This keeps your memory usage low and constant, regardless of the file size. It’s an essential technique for writing robust applications that handle large data transfers.
This covers the bread-and-butter use cases, but the real world is messy. I’m curious to hear from you all about the truly tricky situations you’ve encountered where you had to reach for http.client. What are some of the weirdest server behaviors or gnarly edge cases you’ve had to code around?
