Scrap COVID-19 data using BeautifulSoup in Python

In this tutorial, we will learn how to web scrap COVID-19 data using BeautifulSoup and requests library in Python. We will extract the total number of cases, deaths and recoveries.

BeautifulSoup is a package in the bs4 library of Python. It is an HTML parser that we use to read through HTML and XML files. BeautifulSoup is one of the most commonly used web scraping libraries. It has various inbuilt functions to extract data from websites.

Requests is a Python library that makes HTTP requests in Python. It hides the complexities of making HTTP requests seamlessly. It does the same through its beautiful and simple API. Therefore, it helps the programmer to focus on interacting with services and consuming data.

Parse HTML in Python is a step by step guide to help you get started with web scrapping using BeautifulSoup.

Web Scrapping COVID-19 Statistics

We will scrape the www.worldometers.info to get the statistics of COVID-19 in India. We will extract the following statistics:

  1. Total number of cases
  2. Number of deaths
  3. Number of recoveries

The steps to this are as follows:

  • First, assign or URL to a variable ‘url’ and generate an HTML request for the same using the get function of the requests library.
  • Extract the text of the page and create an HTML parser for it using BeautifulSoup.
  • Go to the website and use the inspect to get the class variables of the desired data.
  • Extract text from the desired varials of the HTML page and print the same.

The following code in Python shows how to scrap COVID-19 data using BeautifulSoup.

import requests
from bs4 import BeautifulSoup

url = 'https://www.worldometers.info/coronavirus/country/india/'

page = requests.get(url)
page = page.text
soup = BeautifulSoup(page, 'html.parser')
        
x1 = soup.findAll('h1')
x2 = soup.findAll("div", {"class": "maincounter-number"})
        
print('so, the total number of cases are')
for i,j in zip(x1[1:], x2):
            print(i.text, j.text)

Similarly, you can try the same for various other countries or for the world data as well. In addition to this, you can also try to scrape many other websites to practice and gain amazing insights.

You may also like the following:
Track Google trends using Pytrends
Perform Google search using Python

Leave a Reply

Your email address will not be published. Required fields are marked *