File Handling Using Pickle Module in Python with examples
This post primarily focusses on the usage of the pickle module for file handling in python, along with its implementation. Refer here for text file handling using built-in methods.
Why File Handling Using Pickle Module?
Python has built-in methods like open, close, read and write functions to store and extract data from files. However, these methods are usually used for textual data and for managing .txt files. Whereas, the pickle module is useful when dealing with objects of classes, both built-in and user-defined.
It helps in storing objects into “.dat” files very easily without delving into its details. The programmer doesn’t have to know the number of bytes to be allocated for each object and search for an object byte-wise and so on.
It will be helpful to understand that “pickling” is defined as the process of converting an object hierarchy in Python into a byte-sequence. “unpickling” is just the reverse of that i.e converting a byte-sequence into an object hierarchy.
The pickle module helps in writing and object to the file and retrieving it back using the functions dump() and load().
Basic Functions in Pickle Module
The two most basic functions in the pickle module include:
- dump(obj,file,protocol=None) -> which is used to write an object ‘obj’ into a file ‘file’
- load(file,protocol=None) -> which is used to read an object from the file ‘file’
In this post, we deal only with these two functions to read and write objects to a file
Using the Functions
Let us consider a very basic example of using the functions:
import pickle list1 = list() for k in range(10): list1.append(k+1) print "Created:",list1 fout = open("Sample.dat",'w') pickle.dump(list1,fout) fout.close() fin = open("Sample.dat",'r') obj = pickle.load(fin) fin.close() print "From File:",obj
In this code, note that list() is an in-built class in python. hence any list we create in a program is simply an object of that class. We can, therefore, use pickle to store this object in a file.
First, create a file named “Sample.dat” and open it in the write(w) mode. We pickle(dump) the list object into the file and close it. Then we open the same file in the read(r) mode, read the first object in the file and store it in obj and close the file. We display the object (obj).
This is the output
Hence we have managed to write an object into a file and later extract it back and display it.
Some Common Operations of Pickle Module in Python
Some of the most common operations using objects and file include: Adding objects in order, searching for objects, updating details for an object, deleting objects, displaying objects based on specific criteria, etc.
Consider the following class definition. We will perform all the basic operations for objects of this class
class Products: def __init__(self): self.PRODUCT_no = 0 self.PRODUCT_name = '' self.PRODUCT_price = 0 def Accept_data(self): self.PRODUCT_no = input("Enter Product No.: ") self.PRODUCT_name = raw_input("Enter Product Name: ") self.PRODUCT_price = input("Enter Product Price: ") def Display_data(self): print "Product No.:",self.PRODUCT_no print "Product Name:",self.PRODUCT_name print "Product Price:",self.PRODUCT_price print def Change_data(self): self.PRODUCT_name = raw_input("Enter Product Name: ") self.PRODUCT_price = input("Enter Product Price: ")
We have first defined a class Products with an Accept_data() function to accept product details and a Display_data() function to display its details. The Change_data() is used to accept only changeable/updatable data from the user during suitable operations.
Searching in pickle module
Consider the following function definition,
def search(): filename = 'PRODUCT.dat' obj = Products() in_pnum = input("Enter Product Number to Search: ") try: fr = open(filename,'rb') while True: try: obj = pickle.load(fr) print if obj.PRODUCT_no == in_pnum: obj.Display_data() fr.close() break else: continue except EOFError: print print "Not Found" break except IOError: print print "ERROR in finding file"
In the function definition for search(), which has been defined in the main part of the program, we first accept the product number as input, since we are going to search for a product by product number here. Then we open a file “PRODUCT.dat” in the read mode. For the purpose of this implementation, we assume that such a file already exists and contains a large number of objects of the Product class.
Note that if a file does not exist, an IOError (Input-Output Error) is raised. Hence we try to manage that error using the try, except blocks.
We then start an infinite loop and keep reading objects from the file. If there are no more objects to read, an EOFError (End Of File Error) is raised. When we encounter that, we terminate the loop saying that the product was not found. This happens if we have read and compared all objects in the python with the accepted product number but haven’t found a match.
Inside the while loop, we keep checking whether the read object’s p_num and the accepted value match. If they match, we simply display the details of the product using its member function and terminate the loop.
NOTE: If there can be multiple objects with the same p_num and it is expected to find all of them, we simply don’t terminate the loop after displaying
Inserting in Sorted Manner
It is always more efficient (at least for small amounts of data) to add new elements in a sorted manner so that we can directly implement search methods like binary search directly.
Consider the following function definition
def create_file(): """Function to write to a file""" filename = 'PRODUCT.dat' try: print noc = input("Enter number of products to enter:") obj = PhoneBill() for k in range(noc): fw = open(filename,'ab+') tfile = open("temp.dat",'ab+') print print "Enter Data " obj.Accept_data() flag = False while True: try: temp = pickle.load(fw) if not flag: if temp.PRODUCT_no >= obj.PRODUCT_no: pickle.dump(obj,tfile) flag = True pickle.dump(temp,tfile) else: pickle.dump(temp,tfile) except EOFError: break if not flag: pickle.dump(obj,tfile) #Object added as last entry tfile.close() fw.close() os.remove(filename) os.rename('temp.dat',filename)
In this function, we open the file always in ab+ mode so that it can create the file on first use, add data to an existing file as well as read from the file. Note that we are maintaining the file in the ascending order in this function.
We then accept the number of entries to be made, run a for loop for that many times. In each iteration, we accept data into the same temporary object – obj and store it in the file. To store the object in the file, we keep comparing each existing object with the new one until we find the right place. The flag is used to indicate whether we have entered the new product or not. If the flag remains False till the end, it means this object must go at the last and we do the same.
Note that we have created a temporary file “temp.dat”. We add all data from the original file into the temp.dat file but including the new one at the right position. Finally, we delete the “PRODUCTS.dat” and rename “temp.dat” as “PRODUCTS.dat” using the s module functions. This method is in fact used in deletion and modification as well.
Update Existing Data: Pickle Module
Consider the following function definition
def update(): filename = 'PRODUCT.dat' in_pnum = input("Enter Product Number: ") try: fr = open(filename,'rb') fw = open('TEMP.dat','wb') while True: try: obj = pickle.load(fr) if in_pnum != obj.PRODUCT_no: pickle.dump(obj,fw) else: obj = PhoneBill() print obj.Change_data() pickle.dump(obj,fw) except EOFError: break fr.close() fw.close() except IOError: print print "ERROR in finding file" else: pass finally: os.remove("PRODUCT.dat") os.rename("Temp.dat","PRODUCT.dat")
This function is simply a combination of searching and writing a file. We find the file based on the product number entered. We call the Change_data() function to modify changeable data and write it into the temp file. This file is finally renamed to the original file.
Deletion: Pickle Module
Consider the following function definition
def delete(): filename = 'PRODUCT.dat' in_pnum = input("Enter Product Number: ") try: fr = open(filename,'rb') fw = open('TEMP.dat','wb') while True: try: obj = pickle.load(fr) if in_pnum != obj.PRODUCT_no: pickle.dump(obj,fw) else: continue except EOFError: break fr.close() fw.close() except IOError: print print "ERROR in finding file" else: pass finally: os.remove("PRODUCT.dat") os.rename("Temp.dat","PRODUCT.dat")
This function is exactly the same as updating changeable data, except the ‘continue’ statement. When a suitable product ID is found, the object is simply not written into the text file. Hence we leave out the data that has to be deleted and write everything else into the “Temp.dat” file. The rest is the same.
Displaying: Pickle Module
This function is as simple as reading all data from the file, checking for a condition (if the filter is necessary) and displaying them using a suitable member function. In the following funtion., we are not filtering the contents. All data from the file is displayed. Consider the following function.
def display(): fr = open("PRODUCT.dat",'rb') while True: try: obj = pickle.load(fr) obj.Display_data() except EOFError: break
NOTE: To filter, simply add an ‘if’ condition inside the while loop and call the displaying function only if the condition is satisfied
Hence, in this post, we have introduced the file handling using the pickle module and discussed some common operations using the module.
Feel free to leave behind any sort of feedback, suggestions or doubts below.