Page 1 of 1
How to avoid redudant, duplicate hardlinks?
Posted: 04 Sep 2020, 04:15
by AbedlaPaille
rednoah wrote: ↑14 Sep 2016, 05:33
Description:
Finding duplicate movie or episode files is easy once all your content has been renamed and xattr tagged with FileBot. This script allows you to view or delete duplicates.
List Logical Duplicates:
Code: Select all
filebot -script fn:duplicates /path/to/files
![Idea :idea:](./images/smilies/icon_idea.gif)
There are
logical duplicates when multiple files refer to the same movie or episode
(e.g. 1080p and 4K versions of the same movie or episode).
List Binary Duplicates:
Code: Select all
filebot -script fn:duplicates --mode binary /input /output /backup
![Idea :idea:](./images/smilies/icon_idea.gif)
There are
binary duplicates when multiple files are identical byte for byte
(i.e. copies of the same file).
NOTE: FileBot 4.9.2 (r7667) or higher is required to efficiently deduplicate links.
Options:
--action delete delete duplicate files
-mediainfo <options> perform
-mediainfo command on duplicate files
-rename <options> perform
-rename command on duplicate files
Advanced Options:
--order input sort duplicates by input argument order
(default in --mode binary)
--order quality sort duplicates by highest quality first
(default in --mode logical)
--order date sort duplicates by least recently created
(according to Media Encoding Date or File Creation Date)
--order time sort duplicates by least recently modified
(according to File Last-Modified Date)
Notes:
- In --mode logical, as per --order quality, the --action delete option will keep only the one file with the highest video quality.
- In --mode logical, this script will not parse or guess any information from the filename. Files that do not contain xattr metadata will be ignored.
- In --mode binary, as per --order input, the --action delete option will keep only the first occurrence of a file according to the input argument order.
- In --mode binary, this script may read files partially or entirely to compute checksums, which are then stored as xattr to speed up subsequent runs.
Say i have a hardlink library to browse by movie rating, those ratings can change. When filebot rescans my plex master structure to create hardlinks, it'll format the hardlinks with the new rating - which i want - but won't delete the older one - which i'd want.
Can
--order time be altered to keep the most recently modified file instead of the least recently modified one?
Always want the new location/name to be kept while the other one gets deleted.
I realize after hardlinking i should run a second process with
fn:duplicates at that hardlink location, but i'm not sure what should be the arguments. I want to keep best quality always, and in case those are binary copies i want to keep the new file/location format.
Re: How to avoid redudant, duplicate hardlinks?
Posted: 04 Sep 2020, 07:36
by rednoah
Why not delete the entire secondary structure at the root folder, and the re-generate it with updated movie ratings?
Re: How to avoid redudant, duplicate hardlinks?
Posted: 04 Sep 2020, 21:34
by AbedlaPaille
Because my library is big (800) and i'm managing a lot of hardlinks (10), most based on complex formats. Deleting and generating i've been doing, but manually every month or so -> it takes a while.
Here i'm trying to automatize my hardlinks creation/management as a follow up to my qBittorent amc post process -> disk usage and processing power grow in importance. Some of those hardlinks are using --filter for that same precious time/energy combo and i'm trying to go further.
Re: How to avoid redudant, duplicate hardlinks?
Posted: 05 Sep 2020, 02:55
by rednoah
So you're saying you have a complex slow format for 1st time processing, and then you have another simple fast format which primarily copies over the current file path, and specifically updates the movie rating?
![Idea :idea:](./images/smilies/icon_idea.gif)
Updating the movie rating does require a network request, so it will always be relatively slow. That's not something we can get around either way.
![Idea :idea:](./images/smilies/icon_idea.gif)
Processing files that have already been processed and xattr-tagged can be re-processed relatively quickly via
Local Xattr Mode. The updated movie rating will still require a slow online request though.
Well, if you already have a folder full of hardlinks, then you can just use move those hardlinks in your update pass. You will end up with empty folders as your files / hardlinks are moved into a new structure
(assuming that folder names change as movie ratings change) but the
cleaner script can take care of those easily.
(NOTE: The GUI will also auto-delete left-behind empty folders, but it's very conservative about it, to make sure you're not accidentally deleting important hidden files and such)
Re: How to avoid redudant, duplicate hardlinks?
Posted: 08 Sep 2020, 14:17
by AbedlaPaille
Interesting idea. Great for a manual pass every once in a while.
The problem is MOVE isn't compatible with new additions from the master structure.
So i'm stuck with hardlinks/duplicates for post processing new additions. Can't i make fn:duplicates pick/keep the most recently created file/folder in case of binary duplicates?
Re: How to avoid redudant, duplicate hardlinks?
Posted: 08 Sep 2020, 14:53
by rednoah
Have you tried the recently added
--order time option yet?
viewtopic.php?p=23171#p23171
e.g. delete least-recently modified binary duplicate
Code: Select all
filebot -script fn:duplicates --mode binary --order time --action delete /a /b /c
![Exclamation :!:](./images/smilies/icon_exclaim.gif)
Run
without --action delete first to make sure it works the way you want before deleting files.
Re: How to avoid redudant, duplicate hardlinks?
Posted: 08 Sep 2020, 21:22
by AbedlaPaille
Amazing, i thought it did the opposite!
Looks like i'm all setup. Thanks for your help.