File Handling Using Pickle Module in Python with examples

This post primarily focusses on the usage of the pickle module for file handling in python, along with its implementation. Refer here for text file handling using built-in methods.

Prerequisites: Basic idea of classes and objects in Python, how to import a module (refer here) and Error Handling (refer here).

Why File Handling Using Pickle Module?

Python has built-in methods like open, close, read and write functions to store and extract data from files. However, these methods are usually used for textual data and for managing .txt files. Whereas, the pickle module is useful when dealing with objects of classes, both built-in and user-defined.

It helps in storing objects into “.dat” files very easily without delving into its details. The programmer doesn’t have to know the number of bytes to be allocated for each object and search for an object byte-wise and so on.




It will be helpful to understand that “pickling” is defined as the process of converting an object hierarchy in Python into a byte-sequence. “unpickling” is just the reverse of that i.e converting a byte-sequence into an object hierarchy.

The pickle module helps in writing and object to the file and retrieving it back using the functions dump() and load().

Basic Functions in Pickle Module

The two most basic functions in the pickle module include:

  1. dump(obj,file,protocol=None) -> which is used to write an object ‘obj’ into a file ‘file’
  2. load(file,protocol=None) -> which is used to read an object from the file ‘file’

In this post, we deal only with these two functions to read and write objects to a file

Using the Functions

Let us consider a very basic example of using the functions:

import pickle

list1 = list()
for k in range(10):
    list1.append(k+1)
print "Created:",list1

fout = open("Sample.dat",'w')
pickle.dump(list1,fout)
fout.close()

fin = open("Sample.dat",'r')
obj = pickle.load(fin)
fin.close()

print "From File:",obj

In this code, note that list() is an in-built class in python. hence any list we create in a program is simply an object of that class. We can, therefore, use pickle to store this object in a file.

First, create a file named “Sample.dat” and open it in the write(w) mode. We pickle(dump) the list object into the file and close it. Then we open the same file in the read(r) mode, read the first object in the file and store it in obj and close the file. We display the object (obj).

This is the output

File Handling Using Pickle Module in Python

Hence we have managed to write an object into a file and later extract it back and display it.

Some Common Operations of Pickle Module in Python

Some of the most common operations using objects and file include: Adding objects in order, searching for objects, updating details for an object, deleting objects, displaying objects based on specific criteria, etc.

Consider the following class definition. We will perform all the basic operations for objects of this class

class Products: 
    def __init__(self): 
        self.PRODUCT_no = 0 
        self.PRODUCT_name = '' 
        self.PRODUCT_price = 0 
    def Accept_data(self): 
        self.PRODUCT_no = input("Enter Product No.: ") 
        self.PRODUCT_name = raw_input("Enter Product Name: ") 
        self.PRODUCT_price = input("Enter Product Price: ") 
    def Display_data(self): 
        print "Product No.:",self.PRODUCT_no 
        print "Product Name:",self.PRODUCT_name 
        print "Product Price:",self.PRODUCT_price print
    def Change_data(self):
        self.PRODUCT_name = raw_input("Enter Product Name: ") 
        self.PRODUCT_price = input("Enter Product Price: ")

We have first defined a class Products with an Accept_data() function to accept product details and a Display_data() function to display its details. The Change_data() is used to accept only changeable/updatable data from the user during suitable operations.

Searching in pickle module

Consider the following function definition,

def search():
    filename = 'PRODUCT.dat'
    obj = Products()
    in_pnum = input("Enter Product Number to Search: ")
    try:
        fr = open(filename,'rb')
        while True:
            try:
                obj = pickle.load(fr)
                print
                if obj.PRODUCT_no == in_pnum:
                    obj.Display_data()
                    fr.close()
                    break
                else:
                    continue
            except EOFError:
                print
                print "Not Found"
                break
    except IOError:
        print
        print "ERROR in finding file"

In the function definition for search(), which has been defined in the main part of the program, we first accept the product number as input, since we are going to search for a product by product number here. Then we open a file “PRODUCT.dat” in the read mode. For the purpose of this implementation, we assume that such a file already exists and contains a large number of objects of the Product class.

Note that if a file does not exist, an IOError (Input-Output Error) is raised. Hence we try to manage that error using the try, except blocks.

We then start an infinite loop and keep reading objects from the file. If there are no more objects to read, an EOFError (End Of File Error) is raised. When we encounter that, we terminate the loop saying that the product was not found. This happens if we have read and compared all objects in the python with the accepted product number but haven’t found a match.

Inside the while loop, we keep checking whether the read object’s p_num and the accepted value match. If they match, we simply display the details of the product using its member function and terminate the loop.

NOTE: If there can be multiple objects with the same p_num and it is expected to find all of them, we simply don’t terminate the loop after displaying

Inserting in Sorted Manner

It is always more efficient (at least for small amounts of data) to add new elements in a sorted manner so that we can directly implement search methods like binary search directly.

Consider the following function definition

def create_file():
    """Function to write to a file"""
    filename = 'PRODUCT.dat'
    try:
        print
        noc = input("Enter number of products to enter:")
        obj = PhoneBill()
        for k in range(noc):
            fw = open(filename,'ab+')
            tfile = open("temp.dat",'ab+')
            print
            print "Enter Data "
            obj.Accept_data()

            flag = False
            while True:
                try:
                    temp = pickle.load(fw)
                    if not flag:
                        if temp.PRODUCT_no >= obj.PRODUCT_no:
                            pickle.dump(obj,tfile)
                            flag = True
                        pickle.dump(temp,tfile)
                    else:
                        pickle.dump(temp,tfile)
                except EOFError:
                    break
            if not flag:
                pickle.dump(obj,tfile)   #Object added as last entry

            tfile.close()
            fw.close()
            os.remove(filename)
            os.rename('temp.dat',filename)

 

In this function, we open the file always in ab+ mode so that it can create the file on first use, add data to an existing file as well as read from the file. Note that we are maintaining the file in the ascending order in this function.

We then accept the number of entries to be made, run a for loop for that many times. In each iteration, we accept data into the same temporary object – obj and store it in the file. To store the object in the file, we keep comparing each existing object with the new one until we find the right place. The flag is used to indicate whether we have entered the new product or not. If the flag remains False till the end, it means this object must go at the last and we do the same.

Note that we have created a temporary file “temp.dat”. We add all data from the original file into the temp.dat file but including the new one at the right position. Finally, we delete the “PRODUCTS.dat” and rename “temp.dat” as “PRODUCTS.dat” using the s module functions. This method is in fact used in deletion and modification as well.

Update Existing Data: Pickle Module

Consider the following function definition

def update():
    filename = 'PRODUCT.dat'
    in_pnum = input("Enter Product Number: ")
    try:
        fr = open(filename,'rb')
        fw = open('TEMP.dat','wb')
        while True:
            try:
                obj = pickle.load(fr)
                if in_pnum != obj.PRODUCT_no:
                    pickle.dump(obj,fw)
                else:
                    obj = PhoneBill()
                    print
                    obj.Change_data()
                    pickle.dump(obj,fw)
            except EOFError:
                break
        fr.close()
        fw.close()
    except IOError:
        print
        print "ERROR in finding file"
    else:
        pass
    finally:
        os.remove("PRODUCT.dat")
        os.rename("Temp.dat","PRODUCT.dat")

This function is simply a combination of searching and writing a file. We find the file based on the product number entered. We call the Change_data() function to modify changeable data and write it into the temp file. This file is finally renamed to the original file.

Deletion: Pickle Module

Consider the following function definition

def delete():
    filename = 'PRODUCT.dat'
    in_pnum = input("Enter Product Number: ")
    try:
        fr = open(filename,'rb')
        fw = open('TEMP.dat','wb')
        while True:
            try:
                obj = pickle.load(fr)
                if in_pnum != obj.PRODUCT_no:
                    pickle.dump(obj,fw)
                else:
                    continue
            except EOFError:
                break
        fr.close()
        fw.close()
    except IOError:
        print
        print "ERROR in finding file"
    else:
        pass
    finally:
        os.remove("PRODUCT.dat")
        os.rename("Temp.dat","PRODUCT.dat")

This function is exactly the same as updating changeable data, except the ‘continue’ statement. When a suitable product ID is found, the object is simply not written into the text file. Hence we leave out the data that has to be deleted and write everything else into the “Temp.dat” file. The rest is the same.

Displaying: Pickle Module

This function is as simple as reading all data from the file, checking for a condition (if the filter is necessary) and displaying them using a suitable member function. In the following funtion., we are not filtering the contents. All data from the file is displayed. Consider the following function.

def display():
    fr = open("PRODUCT.dat",'rb')
    while True:
        try:
            obj = pickle.load(fr)
            obj.Display_data()
        except EOFError:
            break

NOTE: To filter, simply add an ‘if’ condition inside the while loop and call the displaying function only if the condition is satisfied

 

Hence, in this post, we have introduced the file handling using the pickle module and discussed some common operations using the module.

Feel free to leave behind any sort of feedback, suggestions or doubts below.


Leave a Reply

Your email address will not be published. Required fields are marked *