Introduction

Requests is a popular HTTP library in Python that makes it easy to send HTTP requests and handle responses. It provides a simplified interface for interacting with web services and APIs. However, it is sometimes necessary to extend or enhance the functionality of the Requests library to meet specific requirements. In this blog post, we will introduce some useful Requests extensions that can be used to enhance your web scraping, API testing, and web development workflows.

Requests-HTML

Requests-HTML is a powerful library built on top of Requests, which allows you to parse HTML and extract data from web pages. It provides an easy-to-use API to fetch web pages and access their contents using selectors. Here’s an example of how you can use Requests-HTML to scrape titles of news articles from a web page:

from requests_html import HTMLSession

# Create an HTML session
session = HTMLSession()

# Send a GET request to the URL
response = session.get('https://example.com/news')

# Use CSS selectors to extract data
titles = response.html.find('.news-title')
for title in titles:
    print(title.text)

Requests-HTML provides several features like automatic JavaScript execution, form submission, and session management. It is a great choice for web scraping projects where you need to extract data from HTML pages.

Requests-Mock

Requests-Mock is a library that allows you to mock HTTP requests made by a Python script using Requests. This can be useful when writing unit tests for code that relies on external APIs or web services. With Requests-Mock, you can define custom responses for specific URLs and test different scenarios without actually making real network requests. Here’s an example of how to use Requests-Mock to mock API responses:

import requests_mock

# Create a requests_mock instance
with requests_mock.Mocker() as m:
    # Define a mock response for a specific URL
    m.get('https://api.example.com/data', json={'key': 'value'})

    # Make a request to the mock URL
    response = requests.get('https://api.example.com/data')

    # Assert the response data
    assert response.status_code == 200
    assert response.json()['key'] == 'value'

Using Requests-Mock, you can easily simulate different scenarios and test your code’s behavior without relying on external services.

Requests-Cache

Requests-Cache is a caching library that can cache HTTP responses from Requests to reduce latency and improve performance. It provides a simple interface to enable caching for requests made using Requests library. With Requests-Cache, you can cache responses locally and avoid making the same request multiple times. Here’s an example of how to use Requests-Cache to enable caching:

import requests_cache

# Enable caching with Requests-Cache
requests_cache.install_cache('example_cache')

# Make a request to a URL
response = requests.get('https://example.com/api/data')

# Cache the response for future requests
assert response.from_cache is False

# Make the same request again
response = requests.get('https://example.com/api/data')

# Use the cached response
assert response.from_cache is True

By using Requests-Cache, you can significantly reduce the number of HTTP requests made by your application and improve its overall performance.

Conclusion

In this blog post, we introduced three useful extensions for the Requests library in Python. Requests-HTML provides an easy way to parse HTML and extract data from web pages. Requests-Mock allows you to mock HTTP requests made by your code for testing purposes. Finally, Requests-Cache enables caching of HTTP responses to improve your application’s performance. Next time you work with the Requests library, consider using these extensions to enhance your workflow.