Parse JSON from URL in Python

By Aditi Deo

It is not easily possible for humans to read the various kinds of JSON data that are available on the internet. Therefore, Python has provided several libraries and methods through which we can open any URL link from the web and read the JSON response available on it. Thus, in this tutorial, we will learn two different methods on how to Parse JSON from URL in Python.

Method 1: Using urlopen() and json.loads() methods

We can parse JSON from the URL using the urllib library available in Python.

The working of the below-mentioned code is as follows:-

  1. Initially, we import the urlopen module from urllib.request library of Python. The urllib library is responsible for manipulating and working with URLs. For opening and reading URLs, we use the urllib.request package.
  2. Next, we use the urlopen() method which simply opens the URLs mentioned in the parameters and returns the raw data from the HTTP protocol, in this case.
  3.  The JSON data consists of key-value pairs enclosed between curly brackets {}.It is platform independent style of representing and storing the data. The json.loads() method is used which parses a JSON string to a dictionary format.
  4. Thus, we finally print the JSON response as an output.
from urllib.request import urlopen

import json

test_url = 'https://api.github.com/users?since=100'

response = urlopen(test_url)

output_json = json.loads(response.read())

print(output_json)

Output:

[{'login': 'jvantuyl', 'id': 101, 'node_id': 'MDQ6VXNlcjEwMQ==', 
'avatar_url': 'https://avatars.githubusercontent.com/u/101?v=4', 
'gravatar_id': '', 'url': 'https://api.github.com/users/jvantuyl', 
'html_url': 'https://github.com/jvantuyl', 
'followers_url': 'https://api.github.com/users/jvantuyl/followers', 
'following_url': 'https://api.github.com/users/jvantuyl/following{/other_user}', 
'gists_url': 'https://api.github.com/users/jvantuyl/gists{/gist_id}', 
'starred_url': 'https://api.github.com/users/jvantuyl/starred{/owner}{/repo}', 
'subscriptions_url': 'https://api.github.com/users/jvantuyl/subscriptions', 
'organizations_url': 'https://api.github.com/users/jvantuyl/orgs', 
'repos_url': 'https://api.github.com/users/jvantuyl/repos', 
'events_url': 'https://api.github.com/users/jvantuyl/events{/privacy}', 
'received_events_url': 'https://api.github.com/users/jvantuyl/received_events', 
'type': 'User', 'site_admin': False}, 
{'login': 'BrianTheCoder', 'id': 102, 'node_id': 'MDQ6VXNlcjEwMg==', 'avatar_url': 'https://avatars.githubusercontent.com/u/102?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/BrianTheCoder', 'html_url': 'https://github.com/BrianTheCoder', 'followers_url': 'https://api.github.com/users/BrianTheCoder/followers', 'following_url': 'https://api.github.com/users/BrianTheCoder/following{/other_user}', 'gists_url': 'https://api.github.com/users/BrianTheCoder/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/BrianTheCoder/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/BrianTheCoder/subscriptions', 'organizations_url': 'https://api.github.com/users/BrianTheCoder/orgs', 'repos_url': 'https://api.github.com/users/BrianTheCoder/repos', 'events_url': 'https://api.github.com/users/BrianTheCoder/events{/privacy}', 'received_events_url': 'https://api.github.com/users/BrianTheCoder/received_events', 'type': 'User', 'site_admin': False}, {'login': 'freeformz', 'id': 103, 'node_id': 'MDQ6VXNlcjEwMw==', 'avatar_url': 'https://avatars.githubusercontent.com/u/103?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/freeformz', 'html_url': 'https://github.com/freeformz', 'followers_url': 'https://api.github.com/users/freeformz/followers', 'following_url': 'https://api.github.com/users/freeformz/following{/other_user}', 'gists_url': 'https://api.github.com/users/freeformz/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/freeformz/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/freeformz/subscriptions', 'organizations_url': 'https://api.github.com/users/freeformz/orgs', 'repos_url': 'https://api.github.com/users/freeformz/repos', 'events_url': 'https://api.github.com/users/freeformz/events{/privacy}', 'received_events_url': 'https://api.github.com/users/freeformz/received_events', 'type': 'User', 'site_admin': False}]

Method 2: Using request.get() and response.json() methods

We can also parse JSON from the URL using the request library in Python. The request library is used to handle HTTP requests in Python. The request.get() method is used to send a GET request to the URL mentioned in the parameters. The output will be an HTTP response. Next, we apply the json() function to the response generated from the request.get() method to convert the response to a JSON object. Finally, we print the JSON object output.

import requests

response = requests.get('https://api.github.com/users?since=100')

print(response.json())

Output:

[{'login': 'jvantuyl', 'id': 101, 'node_id': 'MDQ6VXNlcjEwMQ==', 
'avatar_url': 'https://avatars.githubusercontent.com/u/101?v=4', 
'gravatar_id': '', 'url': 'https://api.github.com/users/jvantuyl', 
'html_url': 'https://github.com/jvantuyl', 
'followers_url': 'https://api.github.com/users/jvantuyl/followers', 
'following_url': 'https://api.github.com/users/jvantuyl/following{/other_user}', 
'gists_url': 'https://api.github.com/users/jvantuyl/gists{/gist_id}', 
'starred_url': 'https://api.github.com/users/jvantuyl/starred{/owner}{/repo}', 
'subscriptions_url': 'https://api.github.com/users/jvantuyl/subscriptions', 
'organizations_url': 'https://api.github.com/users/jvantuyl/orgs', 
'repos_url': 'https://api.github.com/users/jvantuyl/repos', 
'events_url': 'https://api.github.com/users/jvantuyl/events{/privacy}', 
'received_events_url': 'https://api.github.com/users/jvantuyl/received_events', 
'type': 'User', 'site_admin': False}, 
{'login': 'BrianTheCoder', 'id': 102, 'node_id': 'MDQ6VXNlcjEwMg==', 'avatar_url': 'https://avatars.githubusercontent.com/u/102?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/BrianTheCoder', 'html_url': 'https://github.com/BrianTheCoder', 'followers_url': 'https://api.github.com/users/BrianTheCoder/followers', 'following_url': 'https://api.github.com/users/BrianTheCoder/following{/other_user}', 'gists_url': 'https://api.github.com/users/BrianTheCoder/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/BrianTheCoder/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/BrianTheCoder/subscriptions', 'organizations_url': 'https://api.github.com/users/BrianTheCoder/orgs', 'repos_url': 'https://api.github.com/users/BrianTheCoder/repos', 'events_url': 'https://api.github.com/users/BrianTheCoder/events{/privacy}', 'received_events_url': 'https://api.github.com/users/BrianTheCoder/received_events', 'type': 'User', 'site_admin': False}, {'login': 'freeformz', 'id': 103, 'node_id': 'MDQ6VXNlcjEwMw==', 'avatar_url': 'https://avatars.githubusercontent.com/u/103?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/freeformz', 'html_url': 'https://github.com/freeformz', 'followers_url': 'https://api.github.com/users/freeformz/followers', 'following_url': 'https://api.github.com/users/freeformz/following{/other_user}', 'gists_url': 'https://api.github.com/users/freeformz/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/freeformz/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/freeformz/subscriptions', 'organizations_url': 'https://api.github.com/users/freeformz/orgs', 'repos_url': 'https://api.github.com/users/freeformz/repos', 'events_url': 'https://api.github.com/users/freeformz/events{/privacy}', 'received_events_url': 'https://api.github.com/users/freeformz/received_events', 'type': 'User', 'site_admin': False}]

Thus we have reached the end of this tutorial on how to parse JSON from URL in Python. To learn more about JSON in Python click on the following link: Loop through a JSON array in Python

Leave a Reply

Your email address will not be published. Required fields are marked *