need help 'find'ing files less than 24 hours old to process

Any questions? Need some help?
Post Reply
elgallo
Posts: 39
Joined: 29 Apr 2014, 10:33

need help 'find'ing files less than 24 hours old to process

Post by elgallo » 02 Nov 2019, 23:06

specs:
  • Ubuntu 18.04.3 headless.
  • This box is a local seedbox and media server.
  • Files are hardlinked.
  • excludes has been in use for several years.
here is a the script

Code: Select all

#!/bin/bash
   
NOW=$(date +"%c")
LOGDIR="/home/gallo/.log/filebot"
LOGFILE="filebot_$NOW.log"
DST="/zpool/view"
SRC="/zpool/done/ser"
TV_FORMAT="{plex}"

if find $SRC -type f -mtime -1 | read
        then echo "Running Filebot in $SRC"
                filebot -script fn:amc --output $DST \
                        --log-file $LOGDIR/filebot-amc.log \
                        --action hardlink -non-strict \
                        --db TheTVDB "$SRC" --def \
                        excludeList=$LOGDIR/filebot-history.log \
                        minFileSize=1 subtitles=en,es artwork=y \
                        extras=y seriesFormat=$TV_FORMAT \
                fi
                done
                echo "Finished processing for files at $NOW." | tee -a $LOGDIR/$LOGFILE
exit

Purpose
To catch files that deluge-postprocess.sh missed



Problem
The current iteration of the script reads all the files then reads excludes, then processes files. Something broke, in that script either failed to "FIND" <24hr old files. or FIND works, and filebot scans all files anway.
Consequences A. takes 5-8 mins before script BEGINS to process skipped files.
B. The script is locked and cannot process any other files that are finished downloading, causing some files to be missed/skipped inavertdently.

------

Idea 1
Download to new/different folder
Consequences
A. excludes list incrementally grows and restarts the original issue.

Idea 2
cleanup/remove excludes
Consequences
A. list rebuilds fast due to permaseeding all previous downloads.
B. still does not force filebot to only process new files.

Idea 3
alias/abbr to call filebot with flags
Consequences
A. still have to type/tab out file pathname
B. excludes will still be read THEN skipped, THEN files are processed

Suggestions
requesting help to make filebot only process files in the last X (in this case 24 hours)

Feature Request to add

Code: Select all

maxFileAge=
to

Code: Select all

--def

User avatar
rednoah
The Source
Posts: 16782
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: need help 'find'ing files less than 24 hours old to process

Post by rednoah » 03 Nov 2019, 04:54

1.
Adding a --file-filter should do the trick:

Code: Select all

--file-filter "age < 1"
:idea: If you're using the amc script, you may also need to specify -r so that FileBot will interpret the input folder and pass it on to the script as list of files, instead of letting the amc script itself do it (and thus ignore the --file-filter option).


2.
filebot can be used as a more powerful find-equivalent:

Code: Select all

filebot -mediainfo -r /files --file-filter "age < 1" --format "{d} {f}"

3.
Alternatively, using --log FINE instead of --log ALL will give you less logging (e.g. no repeat of arguments, no excludes, etc).


4.
elgallo wrote:
02 Nov 2019, 23:06
To catch files that deluge-postprocess.sh missed
This warrants further investigation. Why are files missed? Files not added via deluge? Files that failed processing? Deluge not working correctly? The amc script not working correctly?
:idea: Please read the FAQ and How to Request Help.

elgallo
Posts: 39
Joined: 29 Apr 2014, 10:33

Re: need help 'find'ing files less than 24 hours old to process

Post by elgallo » 03 Nov 2019, 08:11

1. I will add //--file-filter ...// to script and try/test. Also -r I shall report.

2.Hopefully this works and excludelist will not be necessary.

3. I'll give this a try as well. should I pipe this line with read

Code: Select all

IF filebot -mediainfo -r /files --file-filter "age < 1" --format "{d} {f}" | read
        then...
exit
4. I am simply guessing as to why. Files are added via watch folder (Medusa --only for snatching). Sometimes I manually snatch (magnet link) 4-6 torrents within a few minutes or each other and the torrent files finish downloading in close succession. I imagine that the script would run hundreds of times as a new script call regardless of the progress of another script. However, I have witnessed deluge (GUI) not move a completed file for several seconds and in some cases not at all. In which case i have to manually 'Move Storage' and run a different script.

User avatar
rednoah
The Source
Posts: 16782
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: need help 'find'ing files less than 24 hours old to process

Post by rednoah » 03 Nov 2019, 08:21

3.
I merely gave -mediainfo as an example for testing --file-filter options to see if that does indeed select only the files you want.

If you were to do some kind of find -exec, then there is an -exec option for that:

Code: Select all

filebot -mediainfo -r /files --file-filter "age < 1" --format "{d} {f}" -exec filebot -script fn:sysenv {f} +
:idea: viewtopic.php?f=4&t=11078
:idea: Please read the FAQ and How to Request Help.

elgallo
Posts: 39
Joined: 29 Apr 2014, 10:33

Re: need help 'find'ing files less than 24 hours old to process

Post by elgallo » 03 Nov 2019, 22:49

I ran this script

Code: Select all

#!/bin/bash

NOW=$(date +"%c")
LOGDIR="/home/gallo/.log/filebot"
LOG="filebot_$NOW.log"
DST="/zata/view"
TV_FORMAT="{plex}"
FILEBOT="$(which filebot)"

$FILEBOT -mediainfo -r "/zata/done/ser/" --file-filter "age < 1" --format "{d} {f}" -exec $FILEBOT -script fn:amc {f} \
        --output $DST --log fine --log-file=$LOGDIR/$LOG.log --action hardlink -non-strict \
        --db TheTVDB --conflict skip --def subtitles=en,es artwork=y extras=y seriesFormat=$TV_FORMAT
and this is the error, though --file-filter "age <1" seems to have worked, only attempting to process one file

Code: Select all

Locking /home/gallo/.log/filebot/Sun
Illegal Argument: java.nio.file.NoSuchFileException: /home/gallo/10:41:11 (10:41:11)
Illegal Argument: java.nio.file.NoSuchFileException: /home/gallo/UTC.log.log (UTC.log.log)
Run script [fn:amc] at [Sun Nov 03 22:41:24 UTC 2019]
Argument[4]: File does not exist: 10:41:11
Argument[6]: File does not exist: UTC.log.log
Input: /zata/done/ser/Saturday.Night.Live.S45E05.Kristen.Stewart.1080p.HDTV.x264-CRiMSON.mkv
Looking up subtitles by hash via OpenSubtitles
No matching subtitles found: /zata/done/ser/Saturday.Night.Live.S45E05.Kristen.Stewart.1080p.HDTV.x264-CRiMSON.mkv
Looking up subtitles by hash via OpenSubtitles
No matching subtitles found: /zata/done/ser/Saturday.Night.Live.S45E05.Kristen.Stewart.1080p.HDTV.x264-CRiMSON.mkv
Rename episodes using [TheTVDB]
Auto-detected query: [Saturday Night Live]
Fetching episode data for [Saturday Night Live]
Fetching episode data for [SNL Poland]
Fetching episode data for [Saturday Night Live (JP)]
Fetching episode data for [Saturday Night Live Korea]
Fetching episode data for [Saturday Night Live Arabia]
Processed 0 files
CmdlineException: Failed to process [/zata/done/ser/Saturday.Night.Live.S45E05.Kristen.Stewart.1080p.HDTV.x264-CRiMSON.mkv] because [/zata/view/TV Shows/Saturday Night Live/Season 45/Saturday Night Live - S45E05 - Kristen Stewart Coldplay.mkv] is an exact copy and already exists [Last-Modified: Sun Nov 03 21:12:24 UTC 2019]
Finished without processing any files
[/usr/bin/filebot, -script, fn:amc, /zata/done/ser/Saturday.Night.Live.S45E05.Kristen.Stewart.1080p.HDTV.x264-CRiMSON.mkv, --output, /zata/view, --log, fine, --log-file=/home/gallo/.log/filebot/Sun, 03, Nov, 2019, 10:41:11, PM, UTC.log.log, --action, hardlink, -non-strict, --db, TheTVDB, --def, subtitles=en,es, artwork=y, extras=y, seriesFormat=TV Shows/Saturday Night Live/Season 45/Saturday Night Live - S45E05 - Kristen Stewart Coldplay] failed (4)
net.filebot.ExecuteException: [/usr/bin/filebot, -script, fn:amc, /zata/done/ser/Saturday.Night.Live.S45E05.Kristen.Stewart.1080p.HDTV.x264-CRiMSON.mkv, --output, /zata/view, --log, fine, --log-file=/home/gallo/.log/filebot/Sun, 03, Nov, 2019, 10:41:11, PM, UTC.log.log, --action, hardlink, -non-strict, --db, TheTVDB, --def, subtitles=en,es, artwork=y, extras=y, seriesFormat=TV Shows/Saturday Night Live/Season 45/Saturday Night Live - S45E05 - Kristen Stewart Coldplay] failed (4)
	at net.filebot.Execute.system(Execute.java:82)
	at net.filebot.cli.ExecCommand.executeSequence(ExecCommand.java:48)
	at net.filebot.cli.ExecCommand.execute(ExecCommand.java:36)
	at net.filebot.cli.CmdlineOperations.execute(CmdlineOperations.java:1120)
	at net.filebot.cli.ArgumentProcessor.runCommand(ArgumentProcessor.java:100)
	at net.filebot.cli.ArgumentProcessor.run(ArgumentProcessor.java:33)
	at net.filebot.Main.main(Main.java:132)

Error (o_O)
So it works, though this error made me think there was something wrong with the script. When the error/issue is that the file already exists. Though I am certain in my script without the line mediainfo/find <24h would end with something along the lines of


...
file not processed because it exists in /path/to/processed/file
Done
Last edited by elgallo on 04 Nov 2019, 19:49, edited 2 times in total.

User avatar
rednoah
The Source
Posts: 16782
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: need help 'find'ing files less than 24 hours old to process

Post by rednoah » 03 Nov 2019, 23:31

1.
You're not escaping your arguments correctly:

Code: Select all

--log-file=$LOGDIR/$LOG.log
So you get this gibberish:

Code: Select all

Locking /home/gallo/.log/filebot/Sun
Illegal Argument: java.nio.file.NoSuchFileException: /home/gallo/10:41:11 (10:41:11)
Illegal Argument: java.nio.file.NoSuchFileException: /home/gallo/UTC.log.log (UTC.log.log)
:idea: Please read Cmdline and Argument Passing.


2.
As I said, I gave the -mediainfo command primarily for testing, not for production in combination with the amc script.

I recommend added the --file-filter option to the amc script call:

Code: Select all

filebot -script fn:amc --file-filter "age < 1" -r /path/to/input ...

3.
Exit Code 4 refers to the amc script running and then not having anything to do and then aborting:

Code: Select all

Finished without processing any files
The -mediainfo -exec call then takes this error code as a failure response, and bubbles up the error.


4.
The root cause for (3.) is that you have already processed the files before:

Code: Select all

CmdlineException: Failed to process [...] because [...] is an exact copy and already exists [...]
:!: This sanity check strongly indicates that your process is broken (i.e. processing the same files over and over). If you don't process files that have already been processed, then you will never run into this error.
:idea: Please read the FAQ and How to Request Help.

elgallo
Posts: 39
Joined: 29 Apr 2014, 10:33

Re: need help 'find'ing files less than 24 hours old to process

Post by elgallo » 04 Nov 2019, 20:04

1. the nonsense is caused by $NOW inside of $LOGDIR (improperly escaped) now fixed in updated script.

2. deleting mediainfo from script Filebot will still try to process all files in dir.
running

Code: Select all

$ filebot -script fn:amc --file-filter "age < 1" --format "{d} {f}" -r /zata/done/ser/ --output /zata/view --log-file $HOME/.log/filebot/filebot-amc.log -non-strict --conflict skip --action hardlink --db TheTVDB --def subtitles=en,es artwork=y extras=y format={plex}
also tries to process all files instead of less than 24h

3/4. yes my tunnelvision for processing < 24h files made me miss this until I posted here.


A. I could live with the mediainfo workaround but --conflict skip I imagine only works with filebot options not mediainfo.

User avatar
rednoah
The Source
Posts: 16782
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: need help 'find'ing files less than 24 hours old to process

Post by rednoah » 05 Nov 2019, 06:33

2.
I'll look into it. Maybe that combination just doesn't work.


EDIT: Fixed with FileBot r6859.


5.
-mediainfo calls don't use the --conflict option, so it has no effect. Since there's no rename operation, there's never a file path conflict.
:idea: Please read the FAQ and How to Request Help.

elgallo
Posts: 39
Joined: 29 Apr 2014, 10:33

Re: need help 'find'ing files less than 24 hours old to process

Post by elgallo » 06 Nov 2019, 01:45

where can I get FileBot r6859? on the current Download Page only FileBot 4.8.5 (r6224) is available

User avatar
rednoah
The Source
Posts: 16782
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: need help 'find'ing files less than 24 hours old to process

Post by rednoah » 06 Nov 2019, 05:59

elgallo wrote:
06 Nov 2019, 01:45
where can I get FileBot r6859? on the current Download Page only FileBot 4.8.5 (r6224) is available
:arrow: viewtopic.php?f=7&t=1609
:idea: Please read the FAQ and How to Request Help.

Post Reply