Automation Issue on Synology
Automation Issue on Synology
So I am probably trying to be too clever for my own good, but I am curious as to what I can do.
I have a Synology NAS where everything is stored (in /Episodes) and have created /EpisodesTEMP for Filebot to read from. Now I tend to work away from home a lot so I download the episodes onto my laptop and then upload them to the /EpisodesTEMP folder via WebDAV (using BitKinex).
I have Filebot running on the NAS every 10 minutes, but what happens is that when a file is still being copied it sometimes ignores it and adds it to the .excludes list and sometimes it just copies the incomplete file which results in a partial file moved to /Episodes. BitKinex realises that the file is gone and therefore starts the transfer again. Obviously by this time the file is in .excludes so Filebot doesn't copy the full file.
I have conflict set to auto so I would have hoped that Filebot would realise that the file in /Episodes is incomplete and that the file in /EpisodesTEMP is full, but since its defined in .excludes it doesn't do anything.
Is there any way of preventing Filebot not attempting to move a file that is incomplete and not add it to .excludes?
I have a Synology NAS where everything is stored (in /Episodes) and have created /EpisodesTEMP for Filebot to read from. Now I tend to work away from home a lot so I download the episodes onto my laptop and then upload them to the /EpisodesTEMP folder via WebDAV (using BitKinex).
I have Filebot running on the NAS every 10 minutes, but what happens is that when a file is still being copied it sometimes ignores it and adds it to the .excludes list and sometimes it just copies the incomplete file which results in a partial file moved to /Episodes. BitKinex realises that the file is gone and therefore starts the transfer again. Obviously by this time the file is in .excludes so Filebot doesn't copy the full file.
I have conflict set to auto so I would have hoped that Filebot would realise that the file in /Episodes is incomplete and that the file in /EpisodesTEMP is full, but since its defined in .excludes it doesn't do anything.
Is there any way of preventing Filebot not attempting to move a file that is incomplete and not add it to .excludes?
Re: Automation Issue on Synology
1.
Running heavy programs like FileBot every 10 minutes is a very bad idea. A single run will probably take more than 10 minutes already. I recommend once per day at 4am.
2.
Where are the logs that show filebot failing to copy files?
viewtopic.php?f=6&t=1868
3.
FileBot cannot tell what other processes are doing, but as long as the files don't have a video extension they won't be processed. Most tools use a temporary name (e.g. xyz.upload) when uploading files and only move/rename to the final destination when upload is complete. In this case files will be ignore by merit of file extension, but you could also use a --def ignore pattern.
Running heavy programs like FileBot every 10 minutes is a very bad idea. A single run will probably take more than 10 minutes already. I recommend once per day at 4am.
2.
Where are the logs that show filebot failing to copy files?
viewtopic.php?f=6&t=1868
3.
FileBot cannot tell what other processes are doing, but as long as the files don't have a video extension they won't be processed. Most tools use a temporary name (e.g. xyz.upload) when uploading files and only move/rename to the final destination when upload is complete. In this case files will be ignore by merit of file extension, but you could also use a --def ignore pattern.
Please read the FAQ and How to Request Help.
Re: Automation Issue on Synology
1. I have already processed all my files in one run which took hours. Now it is just an incremental whenever I add new files... maybe 3-4 a day. Runs take about 30 secondsrednoah wrote:1.
Running heavy programs like FileBot every 10 minutes is a very bad idea. A single run will probably take more than 10 minutes already. I recommend once per day at 4am.
2.
Where are the logs that show filebot failing to copy files?
viewtopic.php?f=6&t=1868
3.
FileBot cannot tell what other processes are doing, but as long as the files don't have a video extension they won't be processed. Most tools use a temporary name (e.g. xyz.upload) when uploading files and only move/rename to the final destination when upload is complete. In this case files will be ignore by merit of file extension, but you could also use a --def ignore pattern.
2. I had wiped them so unfortunately don't have anything to upload at the moment. I was playing around with the Copy option instead and then running something else to clean up whatever is stated as skipped in the log file.
3. Unfortunately over WebDAV the files that are being copied do not have a temporary extension which is where my issue lies. I am currently looking at alternative ways to copy files from my laptop to my NAS which will result in a temporary extension.,
Re: Automation Issue on Synology
Best to do a find | xargs | filebot amc and have find only pass along files that haven't been modified for x hours.
Please read the FAQ and How to Request Help.
Re: Automation Issue on Synology
Ok, I think I can fairly say I have no idea what you are talking about. Am quite embarrassed to admit it too since I did Computer Science at university and IT/Programming is my jobrednoah wrote:Best to do a find | xargs | filebot amc and have find only pass along files that haven't been modified for x hours.
Re: Automation Issue on Synology
Haha, don't they teach shell fu anymore?
List files last-modified more than one hour ago, and less than 7 days ago:
Take all these files and pass them on to another command-line tool:
@see http://stackoverflow.com/questions/5439 ... our/543951
List files last-modified more than one hour ago, and less than 7 days ago:
Code: Select all
find /path/to/files -type f -mmin +60 -mtime -7
Code: Select all
find /path/to/files -type f -mmin +60 -mtime -7 -print0 | xargs -0 filebot -script fn:sysenv
@see http://stackoverflow.com/questions/5439 ... our/543951
Please read the FAQ and How to Request Help.
Re: Automation Issue on Synology
It has just been a long time... you tend to forget or make a point of not trying to remember things that you do not use/need
Anyways, how do I hook in what you have said above to this scheduled script on my NAS?
Obviously that is set to the copy one at the moment, but I could easily change it to the move. As mentioned earlier, I put it to the copy one because the files were being moved mid transfer
Anyways, how do I hook in what you have said above to this scheduled script on my NAS?
Code: Select all
filebot -script 'fn:amc' /volume1/EpisodesTEMP --output /volume1/Episodes --action copy -non-strict --conflict auto --lang en --def 'ut_label=tv' 'music=y' 'unsorted=y' 'clean=y' 'deleteAfterExtract=y' 'minFileSize=100000000' 'seriesFormat={n}/Season {s.pad(2)}/{n} - {s.pad(2)+'\''x'\''}{episodes*.episode*.pad(2).join("-")} - {t}' --log info --log-file '/volume1/@appstore/filebot-node/filebot.log' >> '/volume1/@appstore/filebot-node/log/1456364210329.log' 2>&1
Re: Automation Issue on Synology
Didn't I already give you a complete solution?
You'll just need to slightly modify it for your input folder and your amc command:
(copy & paste, untested)
You'll just need to slightly modify it for your input folder and your amc command:
Code: Select all
find /volume1/EpisodesTEMP -type f -mmin +60 -mtime -7 -print0 | xargs -0 filebot -script 'fn:amc' --output /volume1/Episodes --action copy -non-strict --conflict auto --lang en --def 'ut_label=tv' 'music=y' 'unsorted=y' 'clean=y' 'deleteAfterExtract=y' 'minFileSize=100000000' 'seriesFormat={n}/Season {s.pad(2)}/{n} - {s.pad(2)+'\''x'\''}{episodes*.episode*.pad(2).join("-")} - {t}' --log info --log-file '/volume1/@appstore/filebot-node/filebot.log' >> '/volume1/@appstore/filebot-node/log/1456364210329.log' 2>&1
Please read the FAQ and How to Request Help.
Re: Automation Issue on Synology
Sorry I think the fn:sysenv just confused me.
So tried what you said, with 1 file in the EpisodesTEMP folder that was modified about 20 minutes ago. Not sure why it is still trying to process The Originals - 03x14 - A Streetcar Named Desire.mp4.
So tried what you said, with 1 file in the EpisodesTEMP folder that was modified about 20 minutes ago. Not sure why it is still trying to process The Originals - 03x14 - A Streetcar Named Desire.mp4.
Code: Select all
Run script [fn:amc] at [Sat Feb 27 13:04:50 GMT 2016]
Parameter: ut_label = tv
Parameter: music = y
Parameter: unsorted = y
Parameter: clean = y
Parameter: deleteAfterExtract = y
Parameter: minFileSize = 100000000
Parameter: seriesFormat = {n}/Season {s.pad(2)}/{n} - {s.pad(2)+'x'}{episodes*.episode*.pad(2).join("-")} - {t}
Argument: /volume1/EpisodesTEMP
Input: /volume1/EpisodesTEMP/The Originals - 03x14 - A Streetcar Named Desire.mp4
Group: [tvs:the originals] => [The Originals - 03x14 - A Streetcar Named Desire.mp4]
Rename episodes using [TheTVDB]
Auto-detected query: [The Originals]
Fetching episode data for [The Originals]
Fetching episode data for [SAS: The Originals]
Fetching episode data for [The Originals With Emeril]
[MOVE] Rename [/volume1/EpisodesTEMP/The Originals - 03x14 - A Streetcar Named Desire.mp4] to [/volume1/Episodes/The Originals/Season 03/The Originals - 03x14 - A Streetcar Named Desire.mp4]
Processed 1 files
Clean clutter files and empty folders
Done ヾ(@⌒ー⌒@)ノ
Re: Automation Issue on Synology
This line tells me with absolute certainty that that you fail at copy & paste:
If xargs was used, then you'd get lots of arguments for each input file:
The output above tells me that you still use your old command that looks like this:
Code: Select all
Argument: /volume1/EpisodesTEMP
Code: Select all
find /volume1/EpisodesTEMP -type f -mmin +60 -mtime -7 -print0 | xargs -0 filebot -script fn:amc
Code: Select all
filebot -script fn:amc /volume1/EpisodesTEMP
Please read the FAQ and How to Request Help.
Re: Automation Issue on Synology
Ahh crap, you are right. Let me go over to the naughty corner and hang my head in shame!
Re: Automation Issue on Synology
Alternatively, you can click the Donate button and donate your way out of purgatory.
Please read the FAQ and How to Request Help.
Re: Automation Issue on Synology
Will definitely send you money for a pint if I get this working.
Right, so following your example properly this time and I got an error stating that I should pass a file list or definition but not both, so I removed the --def part of the script.
I then re-ran it and got:
Do I need to remove the ut_label and music options? They were inserted by the Filebot Node client I have on my NAS.
Right, so following your example properly this time and I got an error stating that I should pass a file list or definition but not both, so I removed the --def part of the script.
I then re-ran it and got:
Code: Select all
Run script [fn:amc] at [Sat Feb 27 14:40:09 GMT 2016]
Parameter: ut_label = tv
Parameter: music = y
Parameter: unsorted = y
Parameter: clean = y
Parameter: deleteAfterExtract = y
Parameter: minFileSize = 100000000
Parameter: seriesFormat = {n}/Season {s.pad(2)}/{n} - {s.pad(2)+'x'}{episodes*.episode*.pad(2).join("-")} - {t}
Invalid arguments: pass in either file arguments or ut_dir/ut_file parameters but not both
Failure (°_°)
Re: Automation Issue on Synology
This command can lead 0 files:
So the amc script will complain about not getting any arguments. The amc script doesn't know that 0 files is exactly what you want in this case, so it'll give you a noob warning anyway. You may ignore that warning. It'll work if there are files to be processed.
You can probably fine-tune your command-line to only call filebot if there are in fact any files to be processed. That will be your homework:
Code: Select all
find /volume1/EpisodesTEMP -type f -mmin +60 -mtime -7
You can probably fine-tune your command-line to only call filebot if there are in fact any files to be processed. That will be your homework:
Code: Select all
BusyBox v1.16.1 (2015-10-28 13:24:11 CST) multi-call binary.
Usage: xargs [OPTIONS] [PROG ARGS]
Run PROG on every item given by stdin
Options:
-p Ask user whether to run each command
-r Don't run command if input is empty
-0 Input is separated by NUL characters
-t Print the command on stderr before execution
-e[STR] STR stops input processing
-n N Pass no more than N args to PROG
-s N Pass command line of no more than N bytes
-x Exit if size is exceeded
Please read the FAQ and How to Request Help.
Re: Automation Issue on Synology
Or I could just leave it as it is as it seems to be working nowrednoah wrote:This command can lead 0 files:So the amc script will complain about not getting any arguments. The amc script doesn't know that 0 files is exactly what you want in this case, so it'll give you a noob warning anyway. You may ignore that warning. It'll work if there are files to be processed.Code: Select all
find /volume1/EpisodesTEMP -type f -mmin +60 -mtime -7
You can probably fine-tune your command-line to only call filebot if there are in fact any files to be processed. That will be your homework:Code: Select all
BusyBox v1.16.1 (2015-10-28 13:24:11 CST) multi-call binary. Usage: xargs [OPTIONS] [PROG ARGS] Run PROG on every item given by stdin Options: -p Ask user whether to run each command -r Don't run command if input is empty -0 Input is separated by NUL characters -t Print the command on stderr before execution -e[STR] STR stops input processing -n N Pass no more than N args to PROG -s N Pass command line of no more than N bytes -x Exit if size is exceeded
Code: Select all
Run script [fn:amc] at [Sat Feb 27 15:49:22 GMT 2016]
Parameter: ut_label = tv
Parameter: music = y
Parameter: unsorted = y
Parameter: clean = y
Parameter: deleteAfterExtract = y
Parameter: minFileSize = 100000000
Parameter: seriesFormat = {n}/Season {s.pad(2)}/{n} - {s.pad(2)+'x'}{episodes*.episode*.pad(2).join("-")} - {t}
Argument: /volume1/EpisodesTEMP/The Originals - 03x14 - A Streetcar Named De.mp4
Input: /volume1/EpisodesTEMP/The Originals - 03x14 - A Streetcar Named De.mp4
Group: [tvs:the originals] => [The Originals - 03x14 - A Streetcar Named De.mp4]
Rename episodes using [TheTVDB]
Auto-detected query: [The Originals]
Fetching episode data for [The Originals]
Fetching episode data for [SAS: The Originals]
Fetching episode data for [The Originals With Emeril]
[MOVE] Rename [/volume1/EpisodesTEMP/The Originals - 03x14 - A Streetcar Named De.mp4] to [/volume1/Episodes/The Originals/Season 03/The Originals - 03x14 - A Streetcar Named Desire.mp4]
Processed 1 files
Done ヾ(@⌒ー⌒@)ノ
Re: Automation Issue on Synology
And done:
Confirmation number: 73Y41915AA992192W.
Re: Automation Issue on Synology
Well, by doing xargs -0 -r filebot ... you could save your NAS a lot of time and energy booting up java/filebot/groovy/amc for just doing nothing.
EDIT:
filebot script doing nothing:
sh doing nothing:
Since this way of doing nothing is 1472x faster I think it's worth spending 5 seconds on adding the -r option.
EDIT:
filebot script doing nothing:
Code: Select all
time find /nothing -print0 | xargs -0 filebot -script "g:println now"
find: /nothing: No such file or directory
Sun Feb 28 00:08:29 GMT+08:00 2016
Done ヾ(@⌒ー⌒@)ノ
real 0m13.250s
user 0m11.170s
sys 0m1.820s
sh doing nothing:
Code: Select all
time find /nothing -print0 | xargs -0 -r filebot -script "g:println now"
find: /nothing: No such file or directory
real 0m0.009s
user 0m0.000s
sys 0m0.000s
Please read the FAQ and How to Request Help.
Re: Automation Issue on Synology
Haha, that made me laugh... so much so that I have gone and added it.
The NAS isn't under strain or anything so it wouldn't have made much difference to me or the system, but optimising operations is always good. Part of me liked Filebot doing the check because it wrote out to the logs and I knew it was doing something... now I know you are going to say that I could make the directory check do that, but that is more of a nice to have rather than a need
The NAS isn't under strain or anything so it wouldn't have made much difference to me or the system, but optimising operations is always good. Part of me liked Filebot doing the check because it wrote out to the logs and I knew it was doing something... now I know you are going to say that I could make the directory check do that, but that is more of a nice to have rather than a need
Re: Automation Issue on Synology
Made some slight tweaks to it, including adding the Plex update. Posting it here in case anyone else finds it useful
One key thing is the exclusion of @eaDir which the Synology Diskstation adds everywhere when you have Cloud Station / Photo Station installed and running. This was causing there to be metadata files within @eaDir and whilst the original episode may have been moved out long ago, these files kept getting passed through to Filebot. Obviously Filebot just put them in its ignore list (probably down to the file size limit I have set) but it will still invoking Filebot every time.
Code: Select all
find /volume1/EpisodesTEMP -type f ! -path "/volume1/EpisodesTEMP/@eaDir*" -mmin +60 -mtime -7 -print0 | xargs -0 -r filebot -script 'fn:amc' --output /volume1/Episodes --action move -non-strict --conflict auto --lang en --def 'ut_label=tv' 'music=y' 'unsorted=y' 'clean=y' 'deleteAfterExtract=y' 'minFileSize=100000000' 'plex=192.168.1.3:XXXYYYZZZ' 'seriesFormat={n}/{episode.special ? "Special" : "Season "+s.pad(2)}/{n} - {episode.special ? "00x"+special.pad(2) : s.pad(2)+"x"+episodes*.episode*.pad(2).join("-")} - {t}' --log all --log-file '/volume1/@appstore/filebot-node/filebot.log' >> '/volume1/@appstore/filebot-node/log/1456576972343.log' 2>&1
Re: Automation Issue on Synology
Bah, for some reason when a file already exists it moves it the Unsorted folder. I'm assuming if I use the overwrite option it will just replace it, but what if I wanted to leave it in the EpisodesTEMP folder?
Re: Automation Issue on Synology
1.
2.
Why would you?what if I wanted to leave it in the EpisodesTEMP folder?
2.
If you would, then this becomes a much more pressing concern. Although, you should set excludeList either way just in case you didn't think of everythingNote that if you call this script in intervals every hour or every day you must set --def excludeList or you will get banned.
Please read the FAQ and How to Request Help.
Re: Automation Issue on Synology
If it is already there for whatever reason then I would like to be able to take any action myself.
Banned from where?
Banned from where?
Re: Automation Issue on Synology
1.
That's what the unsorted folder is for. Whatever is in the unsorted folder is for you to handle yourself.
2.
Banned from whatever data source is hit repeatedly with the same queries over and over. FileBot API, TheTVDB API, TheMovieDB API, etc.
Just add it and don't worry about it:
That's what the unsorted folder is for. Whatever is in the unsorted folder is for you to handle yourself.
2.
Banned from whatever data source is hit repeatedly with the same queries over and over. FileBot API, TheTVDB API, TheMovieDB API, etc.
Just add it and don't worry about it:
Code: Select all
--def excludeList=amc.txt
Please read the FAQ and How to Request Help.
Re: Automation Issue on Synology
Ok so I added that and just now a PROPER release of Better Call Saul came out, it just skipped it
/Edit: Don't worry, set conflict to override
/Edit: Don't worry, set conflict to override
Re: Automation Issue on Synology
As per the other thread, mine is kind of working on the new Synology beta.
However, whilst manually clicking on Run on the task works, the automatic execution gives:
/Edit: Instead of running it as admin I tried as my own user and now I get:
However, whilst manually clicking on Run on the task works, the automatic execution gives:
Code: Select all
find: cannot get current directory: Permission denied
/Edit: Instead of running it as admin I tried as my own user and now I get:
Code: Select all
sh: /vcannot get current directorye/log/1456576972343.log: Permission denied
: Permission denied