Feature request: AMC Adaptation for remote input locations

[magnet0]

I'd like to suggest a modification to amc.groovy that would ensure that the excludelist is only appended with files that were successfully processed. This would make it more useful for use cases where file copying/moving might not be successful, such as when the input folder is a remote location. With this change, Filebot would become more useful for getting files form a remote location and assembling them locally in a personalised format; for example, if multiple people were sharing a remote source of media.

The motivation for this change is my own use case, where I mount a remote filesystem from another server, then use Filebot AMC to copy files to my local disk. I use CentOS 7 and Filebot 4.7.7 with a bash script which runs periodically with an excludelist, and while it normally works fine, problems occur when the copy doesn't finish successfully. This happens when the internet connection fails, or the server is restarted, etc.. I find that either no file is copied, or an empty file appears locally; either way, the source file appears in the excludelist and hence the copy is never attempted again. To fix this problem, I have to delete the empty file and remove the line from excludelist.

I think the reason is that in amc.groovy, all the input files are appended to excludelist at once at the beginning (lines 301-304), but files are actually processed later (lines 522-527). What if each file were added individually after each process? I don't see how it would jeopardise any other part of the script.

[rednoah]

1.
So if a file can't be processed, it would just be processed again, and again, and again, until it maybe works? This could result in files being processed over and over indefinitely. Infinite loops must not be possible.


2.
What does the log say? Does FileBot know that the copy operation failed, or does FileBot think everything went fine? Depends on how the underlying filesystem driver reports errors to the layers above.

There might be better solutions altogether. Such as processing files locally to some "upload buffer" folder, and then using rsync to mirror files to a remote server. File transfer tools can deal with retries and interrupted connections a lot better than low-level filesystem drivers.


EDIT:

I see you're downloading files from a remote filesystem. In that case I'd definitely use rsync to sync the files to the local machine, and then run filebot to hardlink everything into a new nicely organized structure.