AMC error: <filename> is an exact copy and already exists

All your suggestions, requests and ideas for future development
Post Reply
howdymilsap
Posts: 35
Joined: 14 Jul 2018, 17:22

AMC error: <filename> is an exact copy and already exists

Post by howdymilsap »

Sometimes I run into this issue when processing a large folder of files using the AMC script (I use the node.js frontend, so some workarounds are limited).

Wondering if a switch could be added to filebot that would enable moving files that are detected as exact copies to the Unsorted location with the original filename. And then continue processing the remaining files in the folder.

Thanks as always for the excellent software and support.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: AMC error: <filename> is an exact copy and already exists

Post by rednoah »

Please read this:
viewtopic.php?f=4&t=6158#p40366

It's currently not possible to disable the anti-abuse measures, and not planned. There may be legitimate use cases where this is an issue, but I have yet to see a convincing one.

e.g. the amc script generally takes care of this issue via the --def excludeList feature, which prevents files from being processed more than once.


:?: What exactly are you doing?

:?: Why are you processing the exact same file into the exact same path more than once?

:?: Why not only process new files?
:idea: Please read the FAQ and How to Request Help.
howdymilsap
Posts: 35
Joined: 14 Jul 2018, 17:22

Re: AMC error: <filename> is an exact copy and already exists

Post by howdymilsap »

rednoah wrote: 12 Dec 2018, 05:58
It's currently not possible to disable the anti-abuse measures, and not planned. There may be legitimate use cases where this is an issue, but I have yet to see a convincing one.
I am not asking to fully process the file, just move it out from the folder it is in and put it into the Unsorted location with the original filename. Then continue to the next file. I am not sure what kind of abuse this would cause. No lookup is needed for metadata, just move it aside and keep going.

I am working to consolidate two drives of data and am running across duplicates identified by filebot. Instead of processing all the files in one shot, I have to go and fix each file that is identified as a duplicate (fix = delete) and then re-run. I was hoping for a simpler solution.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: AMC error: <filename> is an exact copy and already exists

Post by rednoah »

Are you using Windows or Linux / Mac?


What command are you using? If you PM me some command-line logs then I could have a look and think about a better solution.


If you're using the amc script, then there'll be lots and lots of lookups just to make sense of the file, and if all that fails, it'll go into the left over folder. So depending on the situation, a lot of processing may have to happen just to figure out how to deal with this particular file. Only Linux, you can probably script your way around the problem by doing find -exec for each file or folder (depending on your exact requirements) and act accordingly depending on whether or not filebot fails.


If you're searching for duplicate files between drives, other tools might be better for the job though. It would be very easy to write a tool that finds files with the same file size / file hash, so there should be lots of open-source solutions available for this particular task, so you can weed out the physical duplicates with some fully automated tool first, and then process with filebot as usual.
:idea: Please read the FAQ and How to Request Help.
howdymilsap
Posts: 35
Joined: 14 Jul 2018, 17:22

Re: AMC error: <filename> is an exact copy and already exists

Post by howdymilsap »

I am using Ubuntu 18.04 and am running a filebot command using AMC formatting. I interrogate the files and move 4K to a separate folder, HD to another, etc. There are many examples of this kind of script in the forum here.

I am trying to run this script against a large directory of content and it fails periodically with the error I listed in the original post. My reason for using filebot is to simplify the organization of media content, and in this case, it is not simplifying this effort.

You are right, I could script around this and eliminate dupes by comparing hashes and then call filebot on what is left. But that is going to take a bunch of effort on my part - I am not an expert at coding something like that.

I appreciate that you have these anti-abuse rules, but there is a flaw in your logic. You state that there is a bunch of processing in filebot to even determine if something is a duplicate, and it seems like that additional processing is what you want to avoid doing. But I can repeatedly run filebot against the same files and your code will still perform this processing. I can run filebot in a loop if I want and continually hit this error. How does your anti-abuse code actually prevent abuse in this situation?

My request is to add an additional step to that anti-abuse code. Move the file to unsorted, or even delete it. That will actually prevent future abuse and simplify the task that I (and many other people) are trying to perform.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: AMC error: <filename> is an exact copy and already exists

Post by rednoah »

1.
What command are you calling? If you PM me some command-line logs then I can have a look at it and see what we can improve there.


2.
A quick Google search yields duplicate finder tools with nice GUI such as this one:
https://snapcraft.io/speedy-duplicate-finder

Free alternatives:
https://www.howtogeek.com/201140/how-to ... -on-linux/


3.
The current filter does indeed not prevent abuse, but it forces you to fix your setup if certain sanity checks fail. I just makes sure you can't just run it on all your files every 5 minutes, and have that working, which is the most typical abuse scenario. It needs to fail, repeatedly, so users check the logs, and then fix it.

Maybe it can be improved, but touching / moving / deleting files from the source directory is a big nono as far as filebot is concerned, so the destination file really is the only kinda state that can be used to make sure you're not doing the same thing over and over, which understandably backfires in your particular use case.

:idea: Note that the GUI doesn't have the same sanity checks, so if your "merge drives operation" is a one time thing, then using the GUI for that would be an easy workaround.


EDIT:

In r5977 the sanity check has been updated to include File Last-Modified Date as well, so FileBot will only error out if the file is an exact copy AND if the last-modified time stamp is the same as well.
:idea: Please read the FAQ and How to Request Help.
Post Reply