Automation Issue on Synology

[hshah]

So I am probably trying to be too clever for my own good, but I am curious as to what I can do.

I have a Synology NAS where everything is stored (in /Episodes) and have created /EpisodesTEMP for Filebot to read from. Now I tend to work away from home a lot so I download the episodes onto my laptop and then upload them to the /EpisodesTEMP folder via WebDAV (using BitKinex).

I have Filebot running on the NAS every 10 minutes, but what happens is that when a file is still being copied it sometimes ignores it and adds it to the .excludes list and sometimes it just copies the incomplete file which results in a partial file moved to /Episodes. BitKinex realises that the file is gone and therefore starts the transfer again. Obviously by this time the file is in .excludes so Filebot doesn't copy the full file.

I have conflict set to auto so I would have hoped that Filebot would realise that the file in /Episodes is incomplete and that the file in /EpisodesTEMP is full, but since its defined in .excludes it doesn't do anything.

Is there any way of preventing Filebot not attempting to move a file that is incomplete and not add it to .excludes?

[rednoah]

1.
Running heavy programs like FileBot every 10 minutes is a very bad idea. A single run will probably take more than 10 minutes already. I recommend once per day at 4am.


2.
Where are the logs that show filebot failing to copy files?
viewtopic.php?f=6&t=1868


3.
FileBot cannot tell what other processes are doing, but as long as the files don't have a video extension they won't be processed. Most tools use a temporary name (e.g. xyz.upload) when uploading files and only move/rename to the final destination when upload is complete. In this case files will be ignore by merit of file extension, but you could also use a --def ignore pattern.

[hshah]

1.
Running heavy programs like FileBot every 10 minutes is a very bad idea. A single run will probably take more than 10 minutes already. I recommend once per day at 4am.


2.
Where are the logs that show filebot failing to copy files?
viewtopic.php?f=6&t=1868


3.
FileBot cannot tell what other processes are doing, but as long as the files don't have a video extension they won't be processed. Most tools use a temporary name (e.g. xyz.upload) when uploading files and only move/rename to the final destination when upload is complete. In this case files will be ignore by merit of file extension, but you could also use a --def ignore pattern.

1. I have already processed all my files in one run which took hours. Now it is just an incremental whenever I add new files... maybe 3-4 a day. Runs take about 30 seconds

2. I had wiped them so unfortunately don't have anything to upload at the moment. I was playing around with the Copy option instead and then running something else to clean up whatever is stated as skipped in the log file.

3. Unfortunately over WebDAV the files that are being copied do not have a temporary extension which is where my issue lies. I am currently looking at alternative ways to copy files from my laptop to my NAS which will result in a temporary extension.,

[rednoah]

Best to do a find | xargs | filebot amc and have find only pass along files that haven't been modified for x hours.

[hshah]

Best to do a find | xargs | filebot amc and have find only pass along files that haven't been modified for x hours.

Ok, I think I can fairly say I have no idea what you are talking about. Am quite embarrassed to admit it too since I did Computer Science at university and IT/Programming is my job

[rednoah]

Haha, don't they teach shell fu anymore?


List files last-modified more than one hour ago, and less than 7 days ago:

Code: Select all

find /path/to/files -type f -mmin +60 -mtime -7

Take all these files and pass them on to another command-line tool:

Code: Select all

find /path/to/files -type f -mmin +60 -mtime -7 -print0 | xargs -0 filebot -script fn:sysenv

@see http://stackoverflow.com/questions/5439 ... our/543951

[hshah]

It has just been a long time... you tend to forget or make a point of not trying to remember things that you do not use/need

Anyways, how do I hook in what you have said above to this scheduled script on my NAS?

Code: Select all

filebot -script 'fn:amc' /volume1/EpisodesTEMP --output /volume1/Episodes --action copy -non-strict --conflict auto --lang en --def 'ut_label=tv' 'music=y' 'unsorted=y' 'clean=y' 'deleteAfterExtract=y' 'minFileSize=100000000' 'seriesFormat={n}/Season {s.pad(2)}/{n} - {s.pad(2)+'\''x'\''}{episodes*.episode*.pad(2).join("-")} - {t}' --log info --log-file '/volume1/@appstore/filebot-node/filebot.log' >> '/volume1/@appstore/filebot-node/log/1456364210329.log' 2>&1

Obviously that is set to the copy one at the moment, but I could easily change it to the move. As mentioned earlier, I put it to the copy one because the files were being moved mid transfer

[rednoah]

Didn't I already give you a complete solution?

You'll just need to slightly modify it for your input folder and your amc command:

Code: Select all

find /volume1/EpisodesTEMP -type f -mmin +60 -mtime -7 -print0 | xargs -0 filebot -script 'fn:amc' --output /volume1/Episodes --action copy -non-strict --conflict auto --lang en --def 'ut_label=tv' 'music=y' 'unsorted=y' 'clean=y' 'deleteAfterExtract=y' 'minFileSize=100000000' 'seriesFormat={n}/Season {s.pad(2)}/{n} - {s.pad(2)+'\''x'\''}{episodes*.episode*.pad(2).join("-")} - {t}' --log info --log-file '/volume1/@appstore/filebot-node/filebot.log' >> '/volume1/@appstore/filebot-node/log/1456364210329.log' 2>&1

(copy & paste, untested)

[hshah]

Sorry I think the fn:sysenv just confused me.

So tried what you said, with 1 file in the EpisodesTEMP folder that was modified about 20 minutes ago. Not sure why it is still trying to process The Originals - 03x14 - A Streetcar Named Desire.mp4.

Code: Select all

Run script [fn:amc] at [Sat Feb 27 13:04:50 GMT 2016]
Parameter: ut_label = tv
Parameter: music = y
Parameter: unsorted = y
Parameter: clean = y
Parameter: deleteAfterExtract = y
Parameter: minFileSize = 100000000
Parameter: seriesFormat = {n}/Season {s.pad(2)}/{n} - {s.pad(2)+'x'}{episodes*.episode*.pad(2).join("-")} - {t}
Argument: /volume1/EpisodesTEMP
Input: /volume1/EpisodesTEMP/The Originals - 03x14 - A Streetcar Named Desire.mp4
Group: [tvs:the originals] => [The Originals - 03x14 - A Streetcar Named Desire.mp4]
Rename episodes using [TheTVDB]
Auto-detected query: [The Originals]
Fetching episode data for [The Originals]
Fetching episode data for [SAS: The Originals]
Fetching episode data for [The Originals With Emeril]
[MOVE] Rename [/volume1/EpisodesTEMP/The Originals - 03x14 - A Streetcar Named Desire.mp4] to [/volume1/Episodes/The Originals/Season 03/The Originals - 03x14 - A Streetcar Named Desire.mp4]
Processed 1 files
Clean clutter files and empty folders
Done ヾ(＠⌒ー⌒＠)ノ

[rednoah]

This line tells me with absolute certainty that that you fail at copy & paste:

Code: Select all

Argument: /volume1/EpisodesTEMP

If xargs was used, then you'd get lots of arguments for each input file:

Code: Select all

find /volume1/EpisodesTEMP -type f -mmin +60 -mtime -7 -print0 | xargs -0 filebot -script fn:amc

The output above tells me that you still use your old command that looks like this:

Code: Select all

filebot -script fn:amc /volume1/EpisodesTEMP

[hshah]

Ahh crap, you are right. Let me go over to the naughty corner and hang my head in shame!

[rednoah]

Alternatively, you can click the Donate button and donate your way out of purgatory.

[hshah]

Will definitely send you money for a pint if I get this working.

Right, so following your example properly this time and I got an error stating that I should pass a file list or definition but not both, so I removed the --def part of the script.

I then re-ran it and got:

Code: Select all

Run script [fn:amc] at [Sat Feb 27 14:40:09 GMT 2016]
Parameter: ut_label = tv
Parameter: music = y
Parameter: unsorted = y
Parameter: clean = y
Parameter: deleteAfterExtract = y
Parameter: minFileSize = 100000000
Parameter: seriesFormat = {n}/Season {s.pad(2)}/{n} - {s.pad(2)+'x'}{episodes*.episode*.pad(2).join("-")} - {t}
Invalid arguments: pass in either file arguments or ut_dir/ut_file parameters but not both
Failure (°_°)

Do I need to remove the ut_label and music options? They were inserted by the Filebot Node client I have on my NAS.

[rednoah]

This command can lead 0 files:

Code: Select all

find /volume1/EpisodesTEMP -type f -mmin +60 -mtime -7

So the amc script will complain about not getting any arguments. The amc script doesn't know that 0 files is exactly what you want in this case, so it'll give you a noob warning anyway. You may ignore that warning. It'll work if there are files to be processed.

You can probably fine-tune your command-line to only call filebot if there are in fact any files to be processed. That will be your homework:

Code: Select all

BusyBox v1.16.1 (2015-10-28 13:24:11 CST) multi-call binary.

Usage: xargs [OPTIONS] [PROG ARGS]

Run PROG on every item given by stdin

Options:
	-p	Ask user whether to run each command
	-r	Don't run command if input is empty
	-0	Input is separated by NUL characters
	-t	Print the command on stderr before execution
	-e[STR]	STR stops input processing
	-n N	Pass no more than N args to PROG
	-s N	Pass command line of no more than N bytes
	-x	Exit if size is exceeded

[hshah]

This command can lead 0 files:

Code: Select all

find /volume1/EpisodesTEMP -type f -mmin +60 -mtime -7

So the amc script will complain about not getting any arguments. The amc script doesn't know that 0 files is exactly what you want in this case, so it'll give you a noob warning anyway. You may ignore that warning. It'll work if there are files to be processed.

You can probably fine-tune your command-line to only call filebot if there are in fact any files to be processed. That will be your homework:

Code: Select all

BusyBox v1.16.1 (2015-10-28 13:24:11 CST) multi-call binary.

Usage: xargs [OPTIONS] [PROG ARGS]

Run PROG on every item given by stdin

Options:
	-p	Ask user whether to run each command
	-r	Don't run command if input is empty
	-0	Input is separated by NUL characters
	-t	Print the command on stderr before execution
	-e[STR]	STR stops input processing
	-n N	Pass no more than N args to PROG
	-s N	Pass command line of no more than N bytes
	-x	Exit if size is exceeded

Or I could just leave it as it is as it seems to be working now

Code: Select all

Run script [fn:amc] at [Sat Feb 27 15:49:22 GMT 2016]
Parameter: ut_label = tv
Parameter: music = y
Parameter: unsorted = y
Parameter: clean = y
Parameter: deleteAfterExtract = y
Parameter: minFileSize = 100000000
Parameter: seriesFormat = {n}/Season {s.pad(2)}/{n} - {s.pad(2)+'x'}{episodes*.episode*.pad(2).join("-")} - {t}
Argument: /volume1/EpisodesTEMP/The Originals - 03x14 - A Streetcar Named De.mp4
Input: /volume1/EpisodesTEMP/The Originals - 03x14 - A Streetcar Named De.mp4
Group: [tvs:the originals] => [The Originals - 03x14 - A Streetcar Named De.mp4]
Rename episodes using [TheTVDB]
Auto-detected query: [The Originals]
Fetching episode data for [The Originals]
Fetching episode data for [SAS: The Originals]
Fetching episode data for [The Originals With Emeril]
[MOVE] Rename [/volume1/EpisodesTEMP/The Originals - 03x14 - A Streetcar Named De.mp4] to [/volume1/Episodes/The Originals/Season 03/The Originals - 03x14 - A Streetcar Named Desire.mp4]
Processed 1 files
Done ヾ(＠⌒ー⌒＠)ノ

A small donation is on its way to you my friend

[hshah]

And done:

Confirmation number: 73Y41915AA992192W.

[rednoah]

Well, by doing xargs -0 -r filebot ... you could save your NAS a lot of time and energy booting up java/filebot/groovy/amc for just doing nothing.


EDIT:


filebot script doing nothing:

Code: Select all

time find /nothing -print0 | xargs -0 filebot -script "g:println now"
find: /nothing: No such file or directory
Sun Feb 28 00:08:29 GMT+08:00 2016
Done ヾ(＠⌒ー⌒＠)ノ


real	0m13.250s
user	0m11.170s
sys	0m1.820s

sh doing nothing:

Code: Select all

time find /nothing -print0 | xargs -0 -r filebot -script "g:println now"
find: /nothing: No such file or directory

real	0m0.009s
user	0m0.000s
sys	0m0.000s

Since this way of doing nothing is 1472x faster I think it's worth spending 5 seconds on adding the -r option.

[hshah]

Haha, that made me laugh... so much so that I have gone and added it.

The NAS isn't under strain or anything so it wouldn't have made much difference to me or the system, but optimising operations is always good. Part of me liked Filebot doing the check because it wrote out to the logs and I knew it was doing something... now I know you are going to say that I could make the directory check do that, but that is more of a nice to have rather than a need

[hshah]

Made some slight tweaks to it, including adding the Plex update. Posting it here in case anyone else finds it useful

Code: Select all

find /volume1/EpisodesTEMP -type f ! -path "/volume1/EpisodesTEMP/@eaDir*" -mmin +60 -mtime -7 -print0 | xargs -0 -r filebot -script 'fn:amc' --output /volume1/Episodes --action move -non-strict --conflict auto --lang en --def 'ut_label=tv' 'music=y' 'unsorted=y' 'clean=y' 'deleteAfterExtract=y' 'minFileSize=100000000' 'plex=192.168.1.3:XXXYYYZZZ' 'seriesFormat={n}/{episode.special ? "Special" : "Season "+s.pad(2)}/{n} - {episode.special ? "00x"+special.pad(2) : s.pad(2)+"x"+episodes*.episode*.pad(2).join("-")} - {t}' --log all --log-file '/volume1/@appstore/filebot-node/filebot.log' >> '/volume1/@appstore/filebot-node/log/1456576972343.log' 2>&1

One key thing is the exclusion of @eaDir which the Synology Diskstation adds everywhere when you have Cloud Station / Photo Station installed and running. This was causing there to be metadata files within @eaDir and whilst the original episode may have been moved out long ago, these files kept getting passed through to Filebot. Obviously Filebot just put them in its ignore list (probably down to the file size limit I have set) but it will still invoking Filebot every time.

[hshah]

Bah, for some reason when a file already exists it moves it the Unsorted folder. I'm assuming if I use the overwrite option it will just replace it, but what if I wanted to leave it in the EpisodesTEMP folder?

[rednoah]

1.

what if I wanted to leave it in the EpisodesTEMP folder?

Why would you?


2.

Note that if you call this script in intervals every hour or every day you must set --def excludeList or you will get banned.

If you would, then this becomes a much more pressing concern. Although, you should set excludeList either way just in case you didn't think of everything

[hshah]

If it is already there for whatever reason then I would like to be able to take any action myself.

Banned from where?

[rednoah]

1.
That's what the unsorted folder is for. Whatever is in the unsorted folder is for you to handle yourself.


2.
Banned from whatever data source is hit repeatedly with the same queries over and over. FileBot API, TheTVDB API, TheMovieDB API, etc.


Just add it and don't worry about it:

Code: Select all

--def excludeList=amc.txt

[hshah]

Ok so I added that and just now a PROPER release of Better Call Saul came out, it just skipped it

/Edit: Don't worry, set conflict to override

[hshah]

As per the other thread, mine is kind of working on the new Synology beta.

However, whilst manually clicking on Run on the task works, the automatic execution gives:

Code: Select all

find: cannot get current directory: Permission denied

/Edit: Instead of running it as admin I tried as my own user and now I get:

Code: Select all

sh: /vcannot get current directorye/log/1456576972343.log: Permission denied
: Permission denied