Find and remove duplicate files from the terminal
If you have a lot of media files (photo, music etc), then most likely you also have a lot of duplicate files. In this article I’ll show you how to find and remove duplicate files, from the terminal, by using
fdupes — is a programme written by Adrian Lopez to scan directories for duplicate files, with the option to display a list and automatic removal of duplicates. It first compares file sizes, partial MD5 signatures, full MD5 signatures, and then performs a byte-by-byte comparison for verification.
fdupes you can remove duplicate files very easily.
But first you need to install
sudo apt-get install fdupes
To search a directory for duplicate files, simply run the following command:
Progress [5539/39061] 14%
The command will search the directory for any duplicate files and show the progress-bar, and then show the results.
Also, you can redirect the results of serching to a text file:
fdupes -r ./ > duplicates.txt
The fdupes can do more with options like:
- r - search recursively (in subdirectories).
- d - delete duplicates.
- N - used together with
d, preserve the first file in each set of duplicates and delete the others without prompting the user.
To recursively search the directory and delete duplicates:
fdupes -r -d catalog/
You will need to manually select a file for deleting in each set of duplicates.
Or you can run following command for automatically delete the duplicate files:
fdupes -r -d -N catalog/
The results of running the utility:
[+] ./catalog/file.jpg [-] ./catalog/file-1.jpg [-] ./catalog/file-2.jpg
Marked as plus is the kepted file and marked as minus is the deleted files.
Also note that the two following commands do the same things:
fdupes -r -d -N ./
fdupes -rdN ./
If this article has helped you then please leave a comment
Thanks for reading!
Arthur is a designer and full stack software engineer. He is the founder of Space X-Chimp and the blog My Cyber Universe. His personal website can be found at arthurgareginyan.com.