Machine friendly log format

[haarts]

Hi,

This is a friendly feature request nothing more.
I'm using the log file of Filebot to send metrics to an InfluxDB instance via Telegraf (see https://www.influxdata.com/). I want to track how many renames have been successful. However the log contains this:

Code: Select all

[SYMLINK] Rename [some file] to [some normalized file]
[SYMLINK] Failed to rename [some file]

The second line does not show when the renaming is successful. So one log line is insufficient to determine if a rename was successful or not. It would help if the two lines would be collapsed. So a successful rename could be:

Code: Select all

[SYMLINK] Renamed [some file] to [some normalized file]

And an unsuccessful:

Code: Select all

[SYMLINK] Failed to rename [some file]

Bonus points for adding a timestamp!

Best of luck!

[rednoah]

1.
You could use --def exec to add all the extra output your heart desires, for all successfully renamed files anyway:

Code: Select all

--def exec="echo [$(date)] Successfully processed file: {quote f}"

2.
You can also use standard Unix tools to prepend timestamps to each line:
http://stackoverflow.com/a/21620/1514467

[haarts]

Ah! Look at that. I'm certainly going to enjoy myself with "--def". And with that timestamps shouldn't be a problem. This tool continues to amaze.

[haarts]

It's rather tricky finding documentation for '--def'. On the CLI page (http://www.filebot.net/cli.html) nothing is mentioned. And the source is a little too much. And the forum search won't allow searching on '--def'.

From the filebot help output I read that it is used to 'define script variables'. In your example the name is 'exec' and the value seems an interpolated Groovy snippet?

[rednoah]

Sry, I was assuming that you're using the amc script:
viewtopic.php?t=215

--def exec is an amc script option. It will not work for simple -rename calls or other script calls. --def exec is the same as --format except that you're generating shell commands to be executed instead of destination file paths used for moving files.

[haarts]

Ah. Shame, I'm not sure I want to add all the features currently in the amc script. I'll consider this.

[rednoah]

You can make your own script that just lightly wraps around the rename function.

e.g.

Code: Select all

filebot-rename-and-print $FILEBOT_ARGS

filebot-rename-and-print

Code: Select all

#!/usr/bin/env filebot -script

def files = rename(file: args)

def t = new Date().format('yyyy-MM-dd HH-mm-ss')

if (files)
	files.each{ println "[$t] [NEW FILE] $it" }
else
	die "[$t] [FAILURE] Failed to rename any files"

[haarts]

Dude that is amazing. I've donated 5EUR.

[rednoah]

Depending on how your files are organized, it can make sense to call rename(...) per folder. But that really depends on how you're using and if you're running it on big directory trees.

@see https://github.com/filebot/scripts/blob ... all.groovy

[haarts]

My Transmission dumps everything (series only) in one directory. Then I run:

Code: Select all

filebot --log-file /home/john/logs/filebot.log --db TheTVDB -rename /home/john/Downloads/torrents/completed -r --output /home/john/Public/ --action symlink --format '{n}/{s}/{n}-{s00e00}-{t}{"."+lang}' -non-strict

Why would it make sense to run per directory? Is that a speed consideration?

Regarding the script: the CLI args ($FILEBOT_ARGS) are passed as a whole to the `rename` function?

[rednoah]

1.
How does your script ensure that you're not processing every file over and over again indefinitely?


2.
If you're using transmission I'd definitely look into the amc script:
viewtopic.php?f=4&t=215#p3380

If you know that you're only downloading TV shows then you can use --def ut_label=TV. You should also use --def excludeList=amc.excludes to keep track of what files have been processed already.

[haarts]

The script will, over and over again, find out the the destination already exists. I see a lot of 'Skipped [some file] because [destination] already exists', this seems fine to me.

But I'm definitely looking into the amc script!

[rednoah]

The script will, over and over again, find out the the destination already exists. I see a lot of 'Skipped [some file] because [destination] already exists', this seems fine to me.

It's not fine. It's very bad actually. FileBot can't know the destination path unless it's identified each file. In order to identify each file it'll need to send lots and lots of search requests to all the various databases. You're effectively abusing TheTVDB and TheMovieDB web services by spamming them with unnecessary requests for all your files when you really only want to process the 1 or 2 files that are new.

Please use the amc script and make sure to specify --def excludeList=amc.excludes.

[haarts]

Ah, I did not know that. I'll refrain from doing that again.

I've just updated to version 4.7.9 and am wondering what the `ut_*` parameters do.
`ut_dir` = input dir
`ut_label` = whether the input is TV/Movie/etc
`ut_kind` = ?
`ut_title` = ?, something to do with notifications?

Also during a test run it seems to try to unpack mp4 files:

Code: Select all

Extracting files [/home/john/Downloads/torrents/completed/Colony.S01E02.HDTV.x264-FLEET/colony.s01e02.hdtv.x264-fleet/Colony.S01E02.HDTV.x264-FLEET/Colony.S01E02.HDTV.x264-FLEET.mp4]

It seems it is concatenating the source and destination?

I'm running this:

Code: Select all

filebot -script fn:amc --output /home/john/Public/ --action symlink --conflict auto -non-strict --log-file /home/harm/logs/amc.log --def unsorted=y ut_label=TV excludeList=/home/john/logs/amc.excludes ut_dir=/home/john/Downloads/torrents/completed/ ut_kind=multi

[rednoah]

1.
The ut_* parameters are used for passing as much information as possible on to FileBot so it can be used in various ways. The ones you need to know about are documented in the amc manual.

--def ut_label is the only one that matters really, because it allows you to force TV or Movie mode which is a very good thing to do if you already know in advance what you're gonna process.


2.
If you have archives with media files in them, then by default, filebot will extract them. Extracted files will be stored and organized in a way that makes them easy to identify.


3.
If you're using transmission, then I recommend sticking to the transmission instructions:
viewtopic.php?f=4&t=215#p3380

The script you're running should look like this:

Code: Select all

#!/bin/sh -xu

# Input Parameters
ARG_PATH="$TR_TORRENT_DIR/$TR_TORRENT_NAME"
ARG_NAME="$TR_TORRENT_NAME"
ARG_LABEL="N/A"

# Configuration
CONFIG_OUTPUT="$HOME/Media"

/usr/local/bin/filebot -script fn:amc --output "$CONFIG_OUTPUT" --action duplicate --conflict skip -non-strict --log-file amc.log --def unsorted=y music=y artwork=y excludeList=".excludes" ut_dir="$ARG_PATH" ut_kind="multi" ut_title="$ARG_NAME" ut_label="$ARG_LABEL"

4.
Is there any particular reason you're using symlinks instead of hardlinks?

[haarts]

2. Filebot seems to try (and fail) to extract a regular video file:

Code: Select all

Extracting files [/home/harm/Downloads/torrents/completed/Mars.2016.S01E02.HDTV.x264-CROOKS/mars.2016.s01e02.hdtv.x264-crooks/Mars.2016.S01E02.HDTV.x264-CROOKS/mars.2016.s01e02.hdtv.x264-crooks.mkv]
SevenZipException: HRESULT: 0x1 (FALSE). Error extracting all items

Something is amiss.
3. I'm currently testing the amc script but I will use the Transmission integration!
4. No particular reason to use symlinks really. Why would I want to use hardlinks?

[rednoah]

2.
Not all archives are made equal. This is a bad archive. Too bad. You could try a different $EXTRACTOR and see if it works. All of these have pros and cons though.

Code: Select all

# choose extractor
# EXTRACTOR="ApacheVFS"					# use Apache Commons VFS2 with junrar plugin
# EXTRACTOR="SevenZipExecutable"		# use the 7z executable
EXTRACTOR="SevenZipNativeBindings" # use the lib7-Zip-JBinding.so native library

4.
If you have no reason to use symlinks, then you want to use hardlinks. If you want the same file twice, then you create hardlinks. If you want to create shortcuts, then you use symlinks.

[haarts]

I've made progress! The script is now called after a torrent is finished and seems to work as intended.
With one small hiccup. The code below:

Code: Select all

--def exec="echo [$(date)] Successfully processed file: {quote f}"

Doesn't write to the filebot log but writes to standard out. Which I understand, 'echo' kinda does that. Is there a way to write to the filebot log instead?

[rednoah]

Nope, that's not possible because FileBot does not know the console output of processes spawned via --def exec.

You can do your own IO redirection though when calling your top level script.