Wednesday, 05-Feb-20 20:37:25 CET

Archiving program in Python

Can you blog about anything? Like for example, how to create a Python program copying images from the SD card of a camera to OneDrive? I don't know. I really don't know if that is something you can do. So I just have a go and try doing it.

First of all, this is the second time I am making this program. The first time I made it in Perl. I decided to do it another time, this time in Python. The reason for this is that I got another laptop, and on this laptop, I will not install Perl. I will try to keep it free from Perl and use Python instead.

The program is very simple. It is is a function that reads the content of a directory. That directory is on the SD card. The function will get the files in the directory and copy them to the location of OneDrive. The little issue here is that I want to create folders for the days that the picture was taken or filmed for that matter. I will need to look into when the picture was made and make a directory for that if needed. I also need the program to remove the files it has copied so that I can start with a clean SD card.

It would be very nice if the card was removed safely as well. So I would like to mark for safe removal when done so that I am not corrupting the SD card when removing it. My previous laptop had it's own SD card slot. This laptop does not have that. Also, my previous laptop never corrupted the SD card, but this happened already twice with the new laptop, so I am starting to be a little cautious.

from pathlib import Path
import DateTime
import shutil

def _move(ip_self_Path, ip_target_Path):
    """Move a file to a new location"""
    assert ip_self_Path.is_file()
    shutil.move(str(ip_self_Path), str(ip_target_Path))
 
Path.move = _move

def processdir(source_Path):
    """Function moving the files"""
    print("Open dir " + str(source_Path))
    
    for entry in source_Path.iterdir():
        stat = entry.stat()
        if (entry.is_dir()):
            print("Dir:" + str(entry))
            processdir(entry)
        else:
            mtime = stat.st_mtime
            timestamp_str = datetime.datetime.fromtimestamp(mtime).strftime('%Y-%m-%d %H:%M')
            print("File:" + str(entry) + " " + timestamp_str)
            movefile(entry)
            
def movefile(source_Path):
    """Move the file to the target directory, if the target is missing then we create it."""
    global target_dir
    global target_Path
    strTargetYear = datetime.datetime.fromtimestamp(source_Path.stat().st_mtime).strftime('%Y')
    targetYear_Path = Path(str(target_Path) + "\" + strTargetYear)
    if (not(targetYear_Path.exists())):
        print("Creating target year directory: " + str(targetYear_Path))
        targetYear_Path.mkdir()
    strTargetDay = datetime.datetime.fromtimestamp(source_Path.stat().st_mtime).strftime('%Y-%m-%d')
    targetDay_Path = Path(str(target_Path) + "\" + strTargetYear + "\" + strTargetDay)
    if (not(targetDay_Path.exists())):
        print("Creating target day directory: " + str(targetDay_Path))
        targetDay_Path.mkdir()
    Path.move(source_Path, targetDay_Path)

target_dir = "C:\Users\jens\OneDrive\Malmgren\Bilder"
target_Path = Path(target_dir)
processdir(Path("D:\DCIM\"))

The first thing I researched was how to read information about files and directories in Python. I learned that there are several kinds of ways to do this, but I liked pathlib the most. For this, I imported pathlib on line 1. Directories are treelike structures where you are doing similar things at different levels of the structure. For this, I decided on making a recursive function that calls itself when it sees a subdirectory.

To define the recursive function, I wrote def processdir with as argument a path object, line 12. I indicate the type by adding an underscore and "Path," but that is just for me to remind me that it is an object and not a string. The path object has several functions, for example, iterdir to be used in a for-loop, as on line 16, in the body of the loop entry is set to the entries of the directory one by one.

I can check each entry for various things. For example, if the entry is a directory, line 18, then we can call the own function with the subdirectory as an argument, line 20. This calling the own function from within itself is called recursive. This line makes this procedure recursive.

If the entry is not a directory, line 20, then we assume it is a file. When it is a file, we can move the file, but I will come to that later.

Before I move on, I would like to print the path so that I can see that the program is walking the directory correctly. That is done with the str function (I suppose it is a function), we get the object converted back to a string that I can print. On line 24, for example, I print that it is a file and then str(entry) and so on.

For this program, I need to get to different parts of the date. To begin with, I am ok with year month day and the time, and I print that just as a status message. To get the modified time of the file, we call the stat method of the Path object on line 17. In the stat object, we can read the st_mtime member, line 22. It is the last modified time as the number of seconds since the epoch in a local time zone, and the epoch is defined as 1 January 1970.

This timestamp number will work fine until 2038. Hmm… I suppose I will need to reprogram this in 18 years from now. I will be 71 years old. Haha, I put it on the agenda. Good to know we got a new year two-thousand at 2038.

I created a function called movefile at line 27. It is called from line 25 with an argument given the current entry.

I have my images organized by year in the target directory. In that folder, I have directories for the day that the photos where taken. The day folder has the date written as iso date. By doing so, the directory name starts with year then month then day separated by dashes. This way, the folders can be sorted alphabetically. At line 43, I have initiated a variable with the place on disk where I store the year-folders.

I can be pretty sure that the "Bilder" directory exists, so there is no need to test that. That name is Swedish, meaning images. The year directory is another thing. I can not be sure that it exists, and if it is missing, then I will need to create that directory. So how do we do check if a directory exists? With pathlib, we need to instantiate the object with the full path to the file or directory that we wish to check. That is done at line 32.

At first, I forgot to call mkdir with any arguments. That did not work at all. It looks like it is possible to give mkdir several arguments to do different clever things. In my case, I decided to figure out if the year directory is needed, and I am sure the parents are available, so I don't need to send any arguments to the function.

Found the documentation about pathlib mkdir here: https://docs.python.org/3/library/pathlib.html

When the year directory is created, if needed, then we can go on and test the day directory the same way, at line 40.

And then, we can finally move the source to target. How do we do that? I found a really cool way of working with copying the file. We can add the copy function of pathlib. That is nice. Of course, I need to add a move function. Here is the page talking about the copy:

https://stackoverflow.com/questions/33625931/copy-file-with-pathlib-in-python

I changed it to move for the rest; it is the same. The move function is defined on line 5. It is called at line 41.

The program is started at line 45.

Nice, now I can easily archive the pictures from my camera.