Python Requests Library

Learn how to use the Python requests library to make HTTP requests and interact with web APIs. This guide covers GET, POST, PUT, DELETE requests, handling responses, and working with JSON data.

What is the Requests Library?

Requests is a popular Python library for making HTTP requests. It provides a simple, elegant API for sending HTTP requests and handling responses, making it much easier to work with web services than Python’s built-in urllib.

Installation

Install requests using pip:

pip install requests

Basic Usage

Making a GET Request

The simplest way to make a request:

import requests

# Make a GET request
response = requests.get('https://api.example.com/data')

# Check if request was successful
if response.status_code == 200:
    print('Request successful!')
    print(response.text)
else:
    print(f'Error: {response.status_code}')

Response Object

The response object contains all the information about the HTTP response:

import requests

response = requests.get('https://jsonplaceholder.typicode.com/posts/1')

print(f"Status Code: {response.status_code}")
print(f"Headers: {response.headers}")
print(f"Content Type: {response.headers['Content-Type']}")
print(f"Response Text: {response.text}")
print(f"JSON Data: {response.json()}")

Working with JSON Data

Most modern APIs return JSON data. Requests makes it easy to work with JSON:

import requests

# Get JSON data
response = requests.get('https://jsonplaceholder.typicode.com/posts/1')

if response.status_code == 200:
    # Parse JSON response
    data = response.json()
    
    print(f"Title: {data['title']}")
    print(f"Body: {data['body']}")
    print(f"User ID: {data['userId']}")

Handling Complex JSON 

import requests

response = requests.get('https://jsonplaceholder.typicode.com/posts')

if response.status_code == 200:
    posts = response.json()
    
    # Process multiple posts
    for post in posts[:5]:  # First 5 posts
        print(f"Post {post['id']}: {post['title']}")
        print(f"By User {post['userId']}")
        print("-" * 50)

Query Parameters

Add query parameters to your requests:

import requests

# Method 1: Using params parameter
params = {
    'userId': 1,
    'completed': True
}

response = requests.get('https://jsonplaceholder.typicode.com/todos', params=params)

# Method 2: Directly in URL (less recommended)
response = requests.get('https://jsonplaceholder.typicode.com/todos?userId=1&completed=true')

print(response.url)  # Shows the final URL with parameters

Headers and Authentication

Custom Headers 

import requests

headers = {
    'User-Agent': 'MyApp/1.0',
    'Accept': 'application/json',
    'Authorization': 'Bearer your-token-here'
}

response = requests.get('https://api.example.com/data', headers=headers)

Basic Authentication 

import requests

from requests.auth import HTTPBasicAuth

response = requests.get(
    'https://api.example.com/protected',
    auth=HTTPBasicAuth('username', 'password')
)

Bearer Token Authentication 

import requests

token = 'your-bearer-token-here'
headers = {
    'Authorization': f'Bearer {token}',
    'Content-Type': 'application/json'
}

response = requests.get('https://api.example.com/data', headers=headers)

POST Requests

Send data to APIs using POST requests:

import requests
import json

# Data to send
data = {
    'title': 'My New Post',
    'body': 'This is the content of my post',
    'userId': 1
}

headers = {'Content-Type': 'application/json'}

# Make POST request
response = requests.post(
    'https://jsonplaceholder.typicode.com/posts',
    data=json.dumps(data),
    headers=headers
)

if response.status_code == 201:  # 201 Created
    created_post = response.json()
    print(f"Post created with ID: {created_post['id']}")
    print(f"Title: {created_post['title']}")

Form Data 

import requests

# Send form data
form_data = {
    'username': 'john_doe',
    'password': 'secret123',
    'remember_me': 'true'
}

response = requests.post(
    'https://example.com/login',
    data=form_data
)

PUT and PATCH Requests

Update existing data:

import requests
import json

# Update existing post
updated_data = {
    'id': 1,
    'title': 'Updated Title',
    'body': 'Updated content',
    'userId': 1
}

response = requests.put(
    'https://jsonplaceholder.typicode.com/posts/1',
    data=json.dumps(updated_data),
    headers={'Content-Type': 'application/json'}
)

if response.status_code == 200:
    print("Post updated successfully!")

DELETE Requests

Remove data from the server:

import requests

response = requests.delete('https://jsonplaceholder.typicode.com/posts/1')

if response.status_code == 200:
    print("Post deleted successfully!")
elif response.status_code == 404:
    print("Post not found!")

Error Handling

Status Code Checking 

import requests

try:
    response = requests.get('https://api.example.com/data')
    response.raise_for_status()  # Raises exception for 4XX/5XX status codes
    
    data = response.json()
    print("Request successful!")
    
except requests.exceptions.HTTPError as err:
    print(f"HTTP Error: {err}")
except requests.exceptions.ConnectionError as err:
    print(f"Connection Error: {err}")
except requests.exceptions.Timeout as err:
    print(f"Timeout Error: {err}")
except requests.exceptions.RequestException as err:
    print(f"Request Error: {err}")

Timeout Handling 

import requests

try:
    # Set timeout (in seconds)
    response = requests.get('https://api.example.com/data', timeout=5)
    print(response.text)
    
except requests.exceptions.Timeout:
    print("Request timed out!")

Advanced Features

Session Objects

Use sessions for persistent connections and cookies:

import requests

# Create a session
session = requests.Session()

# Set headers for all requests
session.headers.update({
    'User-Agent': 'MyApp/1.0',
    'Authorization': 'Bearer token'
})

# Multiple requests with same session
response1 = session.get('https://api.example.com/user')
response2 = session.get('https://api.example.com/posts')

# Session automatically handles cookies
print(f"Session cookies: {session.cookies}")

File Uploads

Upload files to servers:

import requests

# Upload single file
files = {'file': open('document.pdf', 'rb')}

response = requests.post(
    'https://api.example.com/upload',
    files=files
)

# Upload multiple files
files = [
    ('files', open('doc1.pdf', 'rb')),
    ('files', open('doc2.pdf', 'rb'))
]

response = requests.post(
    'https://api.example.com/upload-multiple',
    files=files
)

Downloading Files

Download and save files:

import requests

# Download small files
response = requests.get('https://example.com/image.jpg')
with open('image.jpg', 'wb') as f:
    f.write(response.content)

# Download large files (streaming)
response = requests.get('https://example.com/large-file.zip', stream=True)

with open('large-file.zip', 'wb') as f:
    for chunk in response.iter_content(chunk_size=8192):
        if chunk:  # filter out keep-alive chunks
            f.write(chunk)

Practical Examples

Weather API Client 

import requests
import os

def get_weather(city, api_key):
    """Get weather information for a city"""
    base_url = "https://api.openweathermap.org/data/2.5/weather"
    
    params = {
        'q': city,
        'appid': api_key,
        'units': 'metric'  # Celsius
    }
    
    try:
        response = requests.get(base_url, params=params)
        response.raise_for_status()
        
        data = response.json()
        
        weather = {
            'city': data['name'],
            'temperature': data['main']['temp'],
            'description': data['weather'][0]['description'],
            'humidity': data['main']['humidity'],
            'wind_speed': data['wind']['speed']
        }
        
        return weather
        
    except requests.exceptions.RequestException as e:
        return {'error': str(e)}

# Usage
api_key = os.getenv('WEATHER_API_KEY')
weather = get_weather('London', api_key)

if 'error' not in weather:
    print(f"Weather in {weather['city']}:")
    print(f"Temperature: {weather['temperature']}°C")
    print(f"Description: {weather['description']}")
    print(f"Humidity: {weather['humidity']}%")
else:
    print(f"Error: {weather['error']}")

GitHub API Client 

import requests

def get_user_repos(username):
    """Get all repositories for a GitHub user"""
    url = f"https://api.github.com/users/{username}/repos"
    
    try:
        response = requests.get(url)
        response.raise_for_status()
        
        repos = response.json()
        
        return [
            {
                'name': repo['name'],
                'description': repo['description'],
                'stars': repo['stargazers_count'],
                'language': repo['language'],
                'url': repo['html_url']
            }
            for repo in repos
        ]
        
    except requests.exceptions.RequestException as e:
        return {'error': str(e)}

# Usage
repos = get_user_repos('octocat')

if 'error' not in repos:
    print(f"Repositories for octocat:")
    for repo in repos[:5]:  # First 5 repos
        print(f"- {repo['name']} ({repo['language']})")
        print(f"  Stars: {repo['stars']}")
        print(f"  {repo['description']}")
        print()
else:
    print(f"Error: {repos['error']}")

Best Practices

  1. Always check status codes - Use response.raise_for_status() or manual checks
  2. Handle exceptions - Wrap requests in try-catch blocks
  3. Use timeouts - Prevent hanging requests
  4. Set User-Agent - Some APIs require a User-Agent header
  5. Use sessions - For multiple requests to the same domain
  6. Stream large files - Use stream=True for downloads
  7. Validate input - Sanitize user input before sending to APIs

Next Steps

Learn about Python file handling to work with local files and data.

External Resources