Writing a directory of images into CSV file in Python
In this tutorial, we will learn and implement the simplest way to read a directory full of images without using any frameworks like Pytorch or TensorFlow. In many Kaggle competitions, they provide us with the set of images in the form of CSV files where each row of the file corresponds single image and each cell is filled with pixel value( RBG or grayscale). This is because it is the simplest way to transfer/share huge data without much loss.
Ever wondered how to do it? I did. But couldn’t find a single source which could explain this from scratch, that’s what a beginner always want Right?. So here is the compilation of various small techniques that can be put together to complete our task.
Creating a list Pointing towards each image
Once we know image name its a one-line code to read it into a numpy array. Image.open(image_name).getdata() as simple as that. But while training a deeplearning model we have hundreds of images( at least) and are usually from the different sources hence won’t have any pattern in the names so that a simple loop changing name variable following that pattern wouldn’t help. That’s why we need to point towards each image using its address(root/directory/filename). os is the module which helps us performing many tasks related to our system. Like moving to the specified directory and in a specified order, joining a path with the file name to point at the file in the absolute sense. These are the few of the innumerable task this module can do. you can read about os module here.
import os def ListofFiles(Dir): Files =  for root, dir_name, file_name in os.walk(Dir): #root store address till directory, dir_name stores directory name # file_name stores file name for name in file_name: fullName = os.path.join(root, name) Files.append(fullName) return Files
We will be learning the code in bits and will import modules that will be using for that particular bit only. Here we imported os module of which we used os.walk iterate over different paths and os.path to manipulate the address(pointed by the os.walk).
os.walk gives three outputs root, dir_name and file_name. “root” is the address till current directory(contacting the images). “dir_name” is the name of the current directory and “file_name” is the name of the image. os.path.join combines the two strings “root” and “name” thus creating a complete address of a file in the directory. Now we append each address in a list named “Files”.
Reading each image
From the previous section, we have a list “Files” containing absolute addresses of each image. Now we just have to use these addresses to read data by using PIL.Image module.
To access that “Files” let’s call the function. Our parameter would me address of the directory. Here I am passing on the address of MNIST dataset located in my machine.
FileList = ListofFiles('C:/Users/prem/Downloads/CodeSpeedy/Dataset/\MNIST_Dataset/testSample')
from PIL import Image pixels= for file in FileList: Im = Image.open(file) pixels.append(list(Im.getdata()))
Image.open() does not have any output, it open the image pointed out by the address “file” in the python background. “Im.getdata()” store the pixels values of the image in list, i.e it flattens the 3D or 2D images that’s why it is being appended to the list “pixels”. “pixels” is a list containing list of pixel values of the images.
Writing the pixels to CSV
From [previous section we have list of list containing pixel values. Now we are going to write it to CSV using numpy.savetxt module. Before That we need to convert the list of list to numpy array.
import numpy as np from numpy import savetxt pixels_arr=np.asarray(pixels) print(pixels_arr.shape) savetxt('numbers.csv', pixels_arr, delimiter=',')
pixels is now converted to pixels_arr and is feed to savetxt which is storing the array pixels_arr in the form of CSV. Thus we have succeeded in storing the image in the form of CSV.
But how do we know we succeeded. Lets call the CSV file and convert back it into an image to see what we get.
from numpy import loadtxt Image = loadtxt('numbers.csv', delimiter=',') X=Image[6,:].reshape(28,28) print(Image.shape) plt.imshow(X)
“loadtxt” is used to load the data from CSV file into NumPy array. The loaded data has dimensions which is the same as the saved data. and the image printed can be identified as from MNIST dataset. Thus verifying that we could write the image into CSV successfully.
Thanks for reading. Will be coming up with more exciting and less talked about codes soon….