Find and Remove Duplicate Files in Dropbox on linux (Ubuntu 14.04)

In this post, I show you how to find a remove duplicate files in Dropbox. I decided to install Ubuntu 14.04 on my notebook, alongside Windows 7 – I am still undecided as to which will be my OS of choice eventually, but since I had decided to do lots and lots of software development this year, I found windows limiting for the task

OK, the deal breaker was getting psychopg2 to install on a virtualenv.

I use GenieTimeline for my backups on Windows and “I have found peace with it.” I’m excited to know they now have cloud backup and I can’t wait to try it out.

The side effect of recovering my Dropbox folder after a system format, is that dropbox recreates every file, appending the file name -excluding the extension, with a digit i.e file-name.1.jpeg, file-name.1.docx, file-name.2.docx (if there was an existing .1. file).

As this would be the third/fourth time I would be undertaking this exercise – my Windows OS usually hits standstill after 365 days, requiring reformatting from my Dell Recovery partition (praises to Dell for this), I was already having ‘.4.xxxx’ files.

This meant I was using up between 2 to 4 times what I should actually do in storage, and I really needed to find and remove those duplicates that dropbox was making. I should report this as a bug to dropbox.

To fix this, I booted up my windows PC and tried to search for applications to help with this task via download.com and sourceforge. I ended up with two apps, but in the end I was disappointed as although both were smart enough to detect duplicates (even had options to use the hash analysis), I still manually had to delete the duplicates one by one. N.B There was over 1GB in duplicate files.

Also, some files especially libraries that get saved when you save a webpage turned up in multiple locations, and they were also listed as duplicates.

I knew I could reliably identify duplicates with the “.1” pattern, so I decided to deal with this problem using linux, yes command line.

Find below the command to identify duplicates in your dropbox folder. cd to your Dropbox folder and run

[code]
find -regextype posix-extended -regex ‘.+(\.[1-9]\.)(\w+)’
[/code]

I understand I could achieved the same using bash, but after tinkering with the regex, I found that the above worked just fine.

You can send the generated file list to a text file for examination before you begin deletion. This helped as I detected some last minute bug right before posting

[code]
touch examine.txt

find -regextype posix-extended -regex ‘.+(\.[1-9]\.)(\w+)’ > ~/examine.txt
[/code]

Please note that I primarily store documents in my dropbox. So, most files end up in the form file-name.1.docx etc. If however you store files with version numbers e.g file-name-0.3.8.tar.gz, this will also be deleted by this code. You are free to drop your particular problem in the comments below and I can send you a command suited to your situation – in my spare time. This command works for me.

[code]
find -type f -regextype posix-extended -regex ‘.+(\.[1-9]\.)(\w+)’ -delete
[/code]

So, there you have it. Go find and remove duplicate files in that Dropbox folder!