Including stopped working in 5.0.3

[shadowsoul]

Hope this is the correct forum for this.

Due to my fairly complex naming rules I've been using separate files for my formats to AMC and the UI, but with 5.0.3 it seems something has broken in terms of includes.

If I have the format file (my actual files are faaaar larger):
movieFormat.groovy:

Groovy: Select all

def normalize = include "utility/normalize.groovy"
def transliterate = include "utility/transliterate.groovy"

def isLatin = { java.text.Normalizer.normalize(it, java.text.Normalizer.Form.NFD).replaceAll(/\p{InCombiningDiacriticalMarks}+/, "") ==~ /^\p{InBasicLatin}+$/ }
def useOriginalName = any {audio[0]?.language == "zh"}{false}
def strTitle = normalize(useOriginalName ? primaryTitle : n)
def strOriginalTitle = (useOriginalName || normalize(n) == normalize(primaryTitle)) ? null : normalize(isLatin(primaryTitle) ? primaryTitle : transliterate(primaryTitle))

    allOf 
        {strTitle}
        {"($y)"}
        {strOriginalTitle}
    .join(".")
    .space(".")

This worked fine in 5.0.2 by passing in movieFormat=e:/scripts/filebot/movieFormat.groovy to AMC, and setting it up as {evaluate("e:/scripts/filebot/movieFormat.groovy" as File)} in the GUI.

However, after upgrading to 5.0.3 it no longer works, if I run AMC I just get "Ignore invalid output file name: d:\sorted\Movies\.mkv", and in the UI I now get "Expression yields empty value: java.lang.NullPointerException: Cannot invoke "java.net.URL.getPath()" because the return value of "java.security.CodeSource.getLocation()" is null".

If I change the UI format to be {include 'e:/scripts/filebot/seriesFormat.groovy'} I instead see:
"Expression yields empty value: net.filebot.InvalidInputException: File not found: D:\Utilities\FileBot_5.0.3-portable\data\.\utility\normalize.groovy"


Is there a new correct way to split the format files into multiple ones?

[rednoah]

1. Compile-time Includes in the Format context:

You can share an external format file and use it in both GUI and amc script like so:

e.g. Format Editor usage:

Format: Select all

@/path/to/MyFormat.groovy

e.g.--def name=@file script parameter for amc script usage:

Shell: Select all

--def movieFormat=@/path/to/MyFormat.groovy

See How can I use the same format in both GUI and CLI? for details.





2. Runtime-time Includes in the Groovy context:

You can use evaluate(File) (Groovy-specific) and include(String) (FileBot/Groovy-specific) but you must take care to always use absolute paths here:

Dynamically evaluate external Groovy scripts at runtime:

Format: Select all

{ include '/path/to/TargetFolder.groovy' }
{ include '/path/to/MovieNaming.groovy' }
{ include '/path/to/MediaInfoTags.groovy' }

Console Output: Select all

$ cat /path/to/MovieNaming.groovy
plex.id

See Split code into external *.groovy script files for details.

[shadowsoul]

Hi,

I've read those links and tried to use the @/path syntax, but if I do that the includes don't work from that file (i.e. if I use {include /formats/myMovieFormat.groovy}, the includes within that file fails with a "binding myVariableName no such property".

When I use the runtime includes, it used to work fine in 5.0.2, but it seems that something in the execution context has changed, so that the relative paths I import within my format.groovy file gets imported as though the were in the filebot/data folder, not from the folder where they actually are which must be the behaviour in 5.0.2...

[rednoah]

1.

I've read those links and tried to use the @/path syntax, but if I do that the includes don't work from that file (i.e. if I use {include /formats/myMovieFormat.groovy}, the includes within that file fails with a "binding myVariableName no such property".

Your include() files are groovy code and not format code. Make sure not to confuse format code (top-level code that consists of literal text and one or more {sections} of groovy code) and groovy code.

e.g. format @file:

Format: Select all

@/path/to/format.groovy

Format: Select all

{ plex.id }

** format.groovy contains format code; FileBot merely concatenates text; so you can concatenate bits and pieces of text from different files;


e.g. format that consists of one Groovy code snippet which happens to evaluate some other Groovy code from an external file:

Format: Select all

{ include "/path/to/code.groovy" }

Groovy: Select all

plex.id

** code.groovy contains groovy code; the Groovy engine will read / compile / interpret the source file and return a value; so the code has to be complete and valid;





2.

When I use the runtime includes, it uses to work fine in 5.0.2, but it seems that something in the execution context has changed.

Yes, if you're using relative paths then behaviour may have changed. include() text() csv() etc now universally interpret relative paths as "relative to $HOME" (which is the %EXEDIR%/data folder if you are using the portable ZIP package) for consistency across GUI and CLI usage.


The easiest solution is to just use absolute paths. That's the recommended and documented way of doing Groovy runtime includes in the FileBot context where the code source file is generally undefined.

Groovy: Select all

def normalize = include "E:/scripts/filebot/utility/normalize.groovy"

If you must have the "relative to current script file" behaviour, then you can do that with standard Groovy evaluate(File) calls:

Format: Select all

{
	// this is inline groovy code without a source file
	evaluate("/path/to/main.groovy" as File)
}

Groovy: Select all

// File: main.groovy
// this code only works when evaluated via evaluate(File)
def __file__ = this.class.protectionDomain.codeSource.location.path as File

def result = evaluate("${__file__.parentFile}/fun.groovy" as File)

println result

** This is not documented because it's just plain Groovy code not specific to FileBot. I just made it up on the spot right now, and it works. I would not do it this way though.

[shadowsoul]

Starting to think that me setup has worked by sheer dumb luck for the last few years...

I'm really confused by the difference, from what I understand:
The format.groovy file should not contain groovy code, only a string formatted to include the different actual groovy files.
The utiliities groovy files should not contain any formatting code, but be pure groovy code (I am unsure how they get access to any, allOf and such in that case)...

Previously I had one of my utilities (strVideoCodec.groovy which I now interpret as a groovy file, not a format file) as

Groovy: Select all

allOf
    {hdr}
    { vc.replace('Microsoft', 'VC-1') }
.join(".")

And I used it like this in my movieFormat.groovy (which I assume is then a formatFile):

Groovy: Select all

def formattedCodec = include "strVideoCodec.groovy"

allOf {
  {title}
  {formattedCodec}
}
.join('.)

Which no longer works, even when changing to full path for the include.

I've tried to use --format @/[snipped]/movieFormat.groovy, which results in my files being renamed to "def formattedCode[..snip rest of raw text content]"
If I change my movieFormat.groovy to be:
{include 'fullpath/strVideoCodec.groovy'}
It seems to evaluate the value correctly, _but_ I cannot then use allOf {...}.join('.) to concatenate the different parts...

I did find one way that seems to work, which is:

Edit movieFormat.groovy file and change all "def X = include path/X.groovy" to instead be { include path }.
Create a new movieFormat2.groovy that contains only "{ include movieFormat.groovy }".

Update my format expression to use be @/path/movieFormat2.groovy.

Is that the expected behaviour? I.e. the formatFile only contains a single { }-expression (i tried to have allOf {include 1} {include 2}.join('.') in it, but it resulted in my filenames becoming "allOf Title Metadata.join('.')"....

For reference my updated files are now looking more like:
strMetadata.groovy:

Groovy: Select all

allOf
    {include "D:/FileBot/scripts/sharedExpressions/strEditions.groovy"}
    {vf}
    {include "D:/FileBot/scripts/sharedExpressions/strSource.groovy"}
    {include "D:/FileBot/scripts/sharedExpressions/strVideo.groovy"}
    {include "D:/FileBot/scripts/sharedExpressions/strAudio.groovy"}
.join('.')

movieFormat.groovy (I'm guessing the def stuff here works as they are functions, not expressions):

Groovy: Select all

def normalize = include "D:/FileBot/scripts/utility/normalize.groovy"
def transliterate = include "D:/FileBot/scripts/utility/transliterate.groovy"
def isLatin = { java.text.Normalizer.normalize(it, java.text.Normalizer.Form.NFD).replaceAll(/\p{InCombiningDiacriticalMarks}+/, "") ==~ /^\p{InBasicLatin}+$/ }

def useOriginalName = any {audio[0]?.language == "da"}{false}
def strTitle = normalize(useOriginalName ? primaryTitle : n)
def strOriginalTitle = (useOriginalName || normalize(n) == normalize(primaryTitle)) ? null : normalize(isLatin(primaryTitle) ? primaryTitle : transliterate(primaryTitle))

allOf 
        {strTitle}
        {"($y)"}
        {strOriginalTitle}
        {include "D:/FileBot/scripts/sharedExpressions/strMetadata.groovy"}
.join(".")
.space(".")

movieFormat2.groovy

Groovy: Select all

{include D:/FileBot/scripts/movieFormat.groovy}

And then finally I use --movieFormat=@D:/FileBot/scripts/movieFormat2.groovy.

Is that the expected setup, or am I just lucking into a combination that works again?

Really grateful for the help btw, the concept of a formatfile and real groovy files that are both using groovy is really confusing me.

[rednoah]

This is format code:

Format: Select all

{ n } - { s00e00 } - { t }

1. 1st groovy script:

   Groovy: Select all

   n

2. Literal text:

   Code: Select all

    - 

3. 2nd groovy script:

   Groovy: Select all

   s00e00

4. Literal text:

   Code: Select all

    - 

5. 3rd groovy script:

   Groovy: Select all

   t

FileBot generates a file path by evaluating each script one after another and concatenating the result.

A groovy script can be as simple as one character (i.e. just return the value of a variable) or thousands of lines.

A groovy script will return a value. If the format runs a groovy script then the result will be added to the file path as text. If you run a groovy script from your groovy script then you can do whatever you want with that return value in your groovy code.




I see some glaring syntax errors that could never have worked, e.g. missing ' and missing " here, so I'm not really sure if that's just forum post typos or the actual source of the problem:

Groovy: Select all

.join('.)

Format: Select all

{include D:/FileBot/scripts/movieFormat.groovy}

Here's how it works if you chose the maximum level of indirection:

1. amc script command-line code:

   Shell: Select all

   --def movieFormat=@/path/to/format.groovy

2. FileBot format code:

   Format: Select all

   { include "/path/to/function.groovy" }

3. Groovy function code:

   Groovy: Select all

   return plex.id

Personally, I'd stick to @files because that's just automated copy & paste. Anything you can copy & paste together in the Format Editor, you can @file cobble together just as well. However, since you've already build a complex system based Groovy evaluate() and include() (those two are the same thing really) you'll probably want to avoid rewriting things.

[shadowsoul]

Oops, yeah, that was just some manual editing that I missed for the syntax errors (my actual files are waaaaay bigger), the scripts I have worked (and have worked fine for many years), it was mostly the change to the $HOME dir that surprised me.

But seems I did get the gist of the format code vs the actual groovy code, so it works with the new format file that imports all the rest, I just need to go through and update the include parts.


Thanks a ton for the feedback, have a better idea of how it executes things behind the scenes now which helps a lot

[rednoah]

So I've spent some time debugging things in detail, and the gist is that source file information is generally lost due to bytecode caching, which was introduced in FileBot 5.* for all Groovy code (not an issue since almost all Groovy code in FileBot is inline and not loaded from external source files anyway) except we missed the include() code which erroneously bypassed bytecode caching, and just used evaluate(File) internally, that was fixed in the latest update which then broke your use case.


FileBot r9937 reintroduces the previous behaviour, so that script files can include other script files with relative paths, relative to the current script file. It should behave almost exactly the same as before, except the internal implementation details are completely different. The difference is that there is now a FileBot-specific magic __file__ script context binding that is implicitly used in include() calls. Since this is a FileBot thing, standard Groovy evaluate() calls are not affected, so only use include() calls for consistency. csv() lines() xml() etc calls now also resolve against the current script file. Groovy code without script file source resolve against $HOME as before, and absolute paths are not affected anyway.


Please try the latest beta and confirm that if it is now working as expected again:
viewtopic.php?t=1609

[shadowsoul]

Hi,

Sorry for the late reply, I'd updated all includes to be absolute paths so stopped checking this

I've tested the BETA version now, works as expected when I change back to relative filepaths, only the formatFiles I created seems to need the absolute path now, but thay might be expected?

[rednoah]

Format @files with recursively nested relative @files should work. That is a separate feature implemented some time ago. The top-level format @file must be absolute, and then @file lines in that file can be relative.


What exactly are you doing? Can you provide a test case?


Note that you while you can mix @files that are concatenated at compile-time, and Groovy includes that are resolved at runtime, each method is entirely unaware of the other. Groovy code is not aware of format @files because FileBot will resolve all the @lines and build one large Groovy code file which is then compiled as a single unit.

[shadowsoul]

I was just trying out simplifying the imports.

But the scenario I talked about where I have to provide absolute path in the format file looks something like:

c:/filebot/scripts/normalize.groovy

Groovy: Select all

def normalize = { 
  if (!it) return null
  
  return it.replaceTrailingBrackets()
    // .upperInitial().lowerTrail()
    .replaceAll(/[`´‘’ʻ""“”]/, "'")
    .replaceAll(",", "\u201A") // lower SINGLE LOW-9 QUOTATION MARK
    .colon("\uA789 ") // MODIFIER LETTER COLON. space needed because for some reason we otherwise get "test\uA7892" from "test: 2"
    .replaceAll(/[|]/, "\u2223") // replace with DIVIDES
    .replaceAll(/[?]/, "\uFE56") // small question mark
    .replaceAll("/", "\u2215") // DIVISION SLASH
    .replaceAll(/[*\s]+/, " ")
    .replaceAll(/\b[IiVvXx]+\b/, { it.upper() })
    .replaceAll(/\b[0-9](?i:th|nd|rd)\b/, { it.lower() }) 
}

c:/filebot/scripts/strMovieName.groovy:

Groovy: Select all

def normalize = include 'normalize.groovy'
return normalize(n).upperInitial().lowerTrail()

c:/filebot/movieFormat.groovy:

Groovy: Select all

{ include 'c:/filebot/scripts/strMovieName.groovy' }

And then for AMC/the movie format in GUI I'd do @c:/filebot/movieFormat.groovy.

This works perfect with the BETA version (i.e. strMovieName.groovt does not need full path to normalize.groovy.
But if I change the movieFormat.groovy to instead be just {include 'scripts/strMovieName.groovy'} it doesn't pick up the relative path from movieFormat.groovy.

*edit* Just wanted to say I appreciate the help and feedback on this, but don't feel the need to spend time on it if I'm the only one having issues, it's working fine with the absolute path, I just wasn't aware of that demand for it to work before

[rednoah]

Using this format in the Format Editor:

Format: Select all

@c:/filebot/movieFormat.groovy

is exactly the same as using this format in the Format Editor:

Groovy: Select all

{ include 'c:/filebot/scripts/strMovieName.groovy' }

If you were to use a relative path in strMovieName.groovy then it'll be resolved against your $HOME folder, exactly as if you had typed that code into the Format Editor instead of pasting it from a file, because that's what's happening internally:

Groovy: Select all

{ include 'strMovieName.groovy' }

@lines are notably unaware of Groovy code, they just copy & paste text, and Groovy code is unaware of how it might have been copy & pasted together:

Console Output: Select all

$ cat 1.groovy
include(
$ cat 2.groovy
'strMovieName'
$ cat 3.groovy
+ '.groovy')
$ cat format.groovy
{
@1.groovy
@2.groovy
@3.groovy
}
$ cat 1.groovy 2.groovy 3.groovy
include(
'strMovieName'
+ '.groovy')

Hence @lines can be nested relative to parent @lines, and include() can be nested relative to parent include(), but it's two separate independent systems that are completely unaware of each other. So Groovy include() relative to parent @line is unfortunately fundamentally not possible.

[shadowsoul]

That makes sense.

I'm happy as is now at least, easy enough to hard code the full path in just the format files and then have all the script files be relative (not that it was that hard to have absolute paths everywhere, but a lot easier now should I ever have to move it somewhere else in the future).