Curl Requests & HTTP Authentication in Python

This seems to be one of those things where there are a lot of questions on how to do it but not many concrete examples showing how to get it done. If you’re new to Python this should hopefully save you some minutes and get you on your way.

This post covers using urllib2, part of the standard Python library & pycurl, a third party module that is essentially a layer of veneer over libcurl.

Here’s a typical curl request that authenticates against GitHub’s API and retrieves a user’s account data:

curl -X GET https://username:userpass@api.github.com/users/braitsch

Now here’s a couple ways to make that same request using Python.

Option 1 – Using Urllib2

import urllib2
def getUserData():
	# first encode the username & password 
	userData = "Basic " + (uName + ":" + pWord).encode("base64").rstrip()
	# create a new Urllib2 Request object	
	req = urllib2.Request('https://api.github.com/users/braitsch')
	# add any additional headers you like 
	req.add_header('Accept', 'application/json')
	req.add_header("Content-type", "application/x-www-form-urlencoded")
	# add the authentication header, required
	req.add_header('Authorization', userData)
	# make the request and print the results
	res = urllib2.urlopen(req)
	print res.read()
getUserData()

Option 2 – Using Pycurl

import pycurl
def getUserData():
	c = pycurl.Curl()
	c.setopt(pycurl.URL, 'https://api.github.com/users/braitsch')
	c.setopt(pycurl.HTTPHEADER, ['Accept: application/json'])
	c.setopt(pycurl.VERBOSE, 0)
	c.setopt(pycurl.USERPWD, 'username:userpass')
	c.perform()
getUserData()

I personally prefer the syntax of pycurl, which feels cleaner to me than urllib2.

Something to note also is that urllib2 only supports GET & POST requests so if your app requires the use of other HTTP verbs such as PUT, DELETE, PATCH, etc… I believe pycurl is the way to go.

Pycurl is however a third party module & does require installation, which you can read more about here.

How about POST requests? The following creates a simple JSON object describing a new repository and posts it to a user’s GitHub account.

import pycurl
def githubPost():
	postData =  '{"name" : "sample-repo", "description" : "repo-description"}'
	c = pycurl.Curl()
	c.setopt(pycurl.URL, 'https://api.github.com/user/repos')
	c.setopt(pycurl.HTTPHEADER, ['Accept: application/json'])
	c.setopt(pycurl.HTTPHEADER, ['Content-Type : application/x-www-form-urlencoded'])
	c.setopt(pycurl.POST, 1)
	c.setopt(pycurl.POSTFIELDS, postData)
	c.setopt(pycurl.USERPWD, 'username:userpass')
	c.perform()
githubPost()

Piece of cake.
You can read more about the near infinite number of options you can set on your Pycurl request here.

One last thing to note, if you run these scripts from a terminal you can pipe the returned JSON to mjson.tool to conveniently enable pretty printing.

./pycurl-example.py | python -mjson.tool