One of the most common tasks in web scraping is submitting forms on web pages. The requests-html library in Python provides a convenient way to submit forms using its built-in HTMLSession class.

In this blog post, we will explore how to submit a form using requests-html and extract the response from the submitted form.

Installation

To get started, make sure you have requests-html installed in your Python environment. You can install it using pip:

pip install requests-html

Importing the necessary modules

First, let’s import the necessary modules from the requests_html library:

from requests_html import HTMLSession

Initializing an HTMLSession

To start making requests and submitting forms, we need to create an instance of the HTMLSession class:

session = HTMLSession()

Submitting a form

To submit a form, we will use the post() method of the HTMLSession class. This method takes the URL of the form submission endpoint and a dictionary containing the form data as parameters.

Here’s an example of submitting a form using requests-html:

# Define the form data as a dictionary
form_data = {
    'username': 'john@example.com',
    'password': 'p@ssw0rd'
}

# Submit the form
response = session.post('https://example.com/login', data=form_data)

Handling the form submission response

After submitting the form, we can access the response using the response object returned by the post() method. We can then extract information from the response, such as status code, content, or specific elements from the submitted form:

# Get the status code of the response
status_code = response.status_code

# Get the content of the response
content = response.text

# Extract specific elements from the response
welcome_message = response.html.find('#welcome-message', first=True).text

Conclusion

In this blog post, we learned how to submit a form using requests-html in Python. We explored how to initialize an HTMLSession, submit a form using the post() method, and extract information from the form submission response.

requests-html provides a powerful and user-friendly way to interact with web forms and automate form submissions. This library is a valuable tool for web scraping and data extraction tasks.

Remember to always respect the terms of service and privacy policies of the websites you are scraping. Happy scraping!