Issue when including partials

[devster]

I'm trying to split my format into partials and I'm getting the following issue.
If I use the simple @filepath.groovy format the partial works as intended:

Code: Select all

$ cat partialTest.groovy
@/path-to-filebot-scripts/partials/audio.groovy

$ cat audio.groovy
{ // audio map, some of these are probably not needed anymore
    def mCFP = [
        "FLAC": "FLAC",
        "PCM": "PCM",
        "MPEG Audio Layer 3": "MP3",
        "AAC LC": "AAC LC",
        "AAC LC SBR": "HE-AAC", // HE-AACv1
        "AAC LC SBR PS": "HE-AACv2",
        "AC-3 Dep": "E-AC-3+Dep",
        "AC-3 Blu-ray Disc Dep": "E-AC-3+Dep",
        "E-AC-3 Blu-ray Disc Dep": "E-AC-3+Dep",
        "E-AC-3 Dep": "E-AC-3+Dep",
        "E-AC-3 JOC": "E-AC-3 JOC",
        "DTS XBR": "DTS-HD HRA", // needs review
        "DTS ES": "DTS-ES Matrix",
        "DTS ES XBR": "DTS-HD HRA",
        "DTS ES XXCH XBR": "DTS-HD HRA", // needs review
        "DTS ES XXCH": "DTS-ES Discrete",
        "DTS ES XXCH XLL": "DTS-HD MA", // needs review
        "DTS XLL": "DTS-HD MA",
        /* "DTS XLL X": "DTS\u02D0X", // IPA triangular colon */
        "DTS XLL X": "DTS-X",
        "MLP FBA": "TrueHD",
        "MLP FBA 16-ch": "TrueHD",
        "DTS 96/24": "DTS 96-24", // needs review
    ]

        audio.collect { au ->
          /* Format seems to be consistently defined and identical to Format/String
             Format_Profile and Format_AdditionalFeatures instead
             seem to be usually mutually exclusive
             Format_Commercial (and _If_Any variant) seem to be defined
             mainly for Dolby/DTS formats */
          String _ac = any
                        { allOf
                            { any { au["Format/String"] } { au["Format"] } }
                            { au["Format_Profile"] }
                            { au["Format_AdditionalFeatures"] }
                          .collect{ it.tokenize() }.flatten().unique().join(" ") }
                        { au["Format_Commercial"] }
          /* original _aco_ binding uses "Codec_Profile", "Format_Profile", "Format_Commercial" */
          String _aco = any { au["Codec_Profile"] } { au["Format_Profile"] } { au["Format_Commercial"] }
          /* def atmos = (_aco =~ /(?i:atmos)/) ? "Atmos" : null */
          Boolean fAtmos = any { au.FormatCommercial =~ /(?i)atmos/ } { false }
          Boolean oAtmos = any { au.NumberOfDynamicObjects } { false }
          String isAtmos = (fAtmos || oAtmos) ? "Atmos" : null
          /* _channels_ uses "ChannelPositions/String2", "Channel(s)_Original", "Channel(s)"
             compared to _af_ which uses "Channel(s)_Original", "Channel(s)"
             local _channels uses the same variables as {channels} but calculates
             the result for each audio stream */
          String    _channels = any
                                  { au["ChannelPositions/String2"] }
                                  { au["Channel(s)_Original"] }
                                  { au["Channel(s)"] }
          String    _ch
          /* _channels can contain no numbers */
          Object    splitCh = _channels =~ /^(?i)object.based$/ ? "Object Based" :
                              _channels.tokenize("\\/\\.")
                              /* the below may be needed for 3/2/0.2.1/3/2/0.1 files */
                              // _channels.tokenize("\\/").take(3)*.tokenize("\\.")
                              //          .flatten()*.toInteger()

          String    chSimple = any { au["Channel(s)"] } { au["Channel(s)/String"].replaceAll("channels", "") }

          switch (splitCh) {
            case { it instanceof String }:
              _ch = allOf { splitCh } { chSimple + "ch" }.join(" ")
              break

            case { it.size > 4 }:
              def wide = splitCh.takeRight(1)
              Double main = splitCh.take(4)*.toDouble().inject(0, { a, b -> a + b })
              Double sub = Double.parseDouble("0." + wide.last())
              _ch = (main + sub).toBigDecimal().setScale(1, java.math.RoundingMode.HALF_UP).toString()
              break

            case { it.size > 1 }:
              /* original logic is _mostly_ unchanged if format is like 3/2/0.1 */
              Double sub = Double.parseDouble(splitCh.takeRight(2).join("."))
              _ch = splitCh.take(2)*.toDouble().plus(sub).inject(0, { a, b -> a + b })
                           .toBigDecimal().setScale(1, java.math.RoundingMode.HALF_UP).toString()
              break

            default:
              _ch = splitCh.first().toDouble()
          }

          /* UNUSED - possible fix for mistakes in ChannelPositions/String2 */
          String _channelPositions = any{au["ChannelPositions"]}{null}
          String channelParse
          if ( chSimple.toInteger() != _ch.tokenize(".")*.toInteger().sum() &&
               _channelPositions != null ) {
            List   channelsPos = _channelPositions.tokenize(",")
            String mainFix = channelsPos.take(3).inject(0) { acc, p ->
              Integer parsedCh = p.tokenize(":").takeRight(1).first().trim().tokenize(" ").size()
              acc + parsedCh
            }
            String subFix = channelsPos.takeRight(1).first().trim().tokenize(" ").size()
            channelParse = "${mainFix}.${subFix}"
          }

          String _chFix
          if (channelParse != null && Float.parseFloat(_ch).compareTo(Float.parseFloat(channelParse)) != 0)  {
              _chFix = channelParse
          } else {
              _chFix = _ch
          }
            // { allOf{ _chFix }{ au["NumberOfDynamicObjects"] + "obj" }.join("+") }
        String _lang = any { au["Language"] } { video.first()["Language"] }
          def stream = allOf
            { allOf { _ch } { au["NumberOfDynamicObjects"].concat("obj") }.join("+") }
            { allOf { mCFP.get(_ac, _ac) } {isAtmos/* atmos */}.join("+") }
            /* { allOf{ mCFP.get(combined, _aco) }{atmos}.join("+") } /* bit risky keeping _aco as default */
            { net.filebot.Language.findLanguage(_lang).ISO3.upperInitial() }
            /* _cf_ not being used > "Codec/Extensions", "Format" */
          def ret = [:]
          /* this is done to retain stream order */
          ret.id = any{ au["StreamKindId"] }{ au["StreamKindPos"] }{ au["ID"] }
          ret.data = stream
          return ret
        }.toSorted{ it.id }.collect{ it.data }*.join(" ").join(", ")
}

Please forgive the indentation (if you have a formatter I'd be happy to use it).
However if I use the { include 'path-to-file.groovy' } expression I get either:

Code: Select all

Expression yields empty value: startup failed:
/path-to-filebot-scripts/partials/audio.groovy: 1: Ambiguous expression could be either a parameterless closure expression or an isolated open code block;
   solution: Add an explicit closure parameter list, e.g. {it -> ...}, or force it to be treated as an open block by giving it a label, e.g. L:{...} @ line 1, column 1.
   { // audio map, some of these are probably not needed anymore
   ^

1 error

Or I can change it with an it -> at the beginning in which case I get:

Code: Select all

Expression yields empty value: Cannot invoke method tokenize() on null object

Any tips please?

[rednoah]

Keep in mind that FileBot format expressions and groovy code are conceptually two entirely different and unrelated things. We we just so happen to use *.groovy extension for both for syntax highlighting reasons. FileBot format code consistent of literal parts and dynamic {...} parts where ... is interpreted as groovy code.


FileBot format code can be split into multiple files that are combined at "compile time" by FileBot, while Groovy code can run other Groovy code by merit of being a scripting language, i.e. FileBot format code can include format code from other files, groovy code can include groovy code from other files, but groovy code cannot include FileBot format code from other files.


Exhibit A:

This is both a valid format expression and valid groovy code, but the meaning is completely different depending on how interpret the code:

Code: Select all

{ plex }

Interpretation F: format code that prints the plex path
Interpretation G: groovy code


Exhibit B:

This is both a valid format expression and valid groovy code, but the meaning is completely different depending on how interpret the code:

Code: Select all

plex

Interpretation F: format code that literally just prints "plex"
Interpretation G: groovy code that references a variable and then prints the value of that variable


Exhibit C:

This is a valid format expression, but not valid groovy code:

Code: Select all

/path/to/{ plex }

Interpretation F: format code that prints "/path/to/" followed by whatever the groovy code in {...} prints
Interpretation G: <syntax error>




I recommend using splitting FileBot format code via FileBot format code @includes for simplicity and performance:




[DOCS] Split code into external *.groovy script files

[devster]

In short I'm including groovy code, not format?
But then I'm not seeing where it breaks.

Including this:

Code: Select all

// audio map, some of these are probably not needed anymore
HashMap codecMap = [
    "FLAC": "FLAC",
    "PCM": "PCM",
    "MPEG Audio Layer 3": "MP3",
    "AAC LC": "AAC LC",
    "AAC LC SBR": "HE-AAC", // HE-AACv1
    "AAC LC SBR PS": "HE-AACv2",
    "AC-3 Dep": "E-AC-3+Dep",
    "AC-3 Blu-ray Disc Dep": "E-AC-3+Dep",
    "E-AC-3 Blu-ray Disc Dep": "E-AC-3+Dep",
    "E-AC-3 Dep": "E-AC-3+Dep",
    "E-AC-3 JOC": "E-AC-3 JOC",
    "DTS XBR": "DTS-HD HRA", // needs review
    "DTS ES": "DTS-ES Matrix",
    "DTS ES XBR": "DTS-HD HRA",
    "DTS ES XXCH XBR": "DTS-HD HRA", // needs review
    "DTS ES XXCH": "DTS-ES Discrete",
    "DTS ES XXCH XLL": "DTS-HD MA", // needs review
    "DTS XLL": "DTS-HD MA",
    /* "DTS XLL X": "DTS\u02D0X", // IPA triangular colon */
    "DTS XLL X": "DTS-X",
    "MLP FBA": "TrueHD",
    "MLP FBA 16-ch": "TrueHD",
    "DTS 96/24": "DTS 96-24", // needs review
]

audio.collect { au ->
    String _ac = any
                { allOf
                    { any { au["Format/String"] } { au["Format"] } }
                    { au["Format_Profile"] }
                    { au["Format_AdditionalFeatures"] }
                    .collectMany{ it.tokenize() }.unique().join(" ") }
                { au["Format_Commercial"] }
    String _aco = any { au["Codec_Profile"] } { au["Format_Profile"] } { au["Format_Commercial"] }
    Boolean fAtmos = any { au.FormatCommercial =~ /(?i)atmos/ } { false }
    Boolean oAtmos = any { au.NumberOfDynamicObjects } { false }
    String isAtmos = (fAtmos || oAtmos) ? "Atmos" : null
    String _channels = any
                { au["ChannelPositions/String2"] }
                { au["Channel(s)_Original"] }
                { au["Channel(s)"] }
    String _ch
    Object splitCh = _channels =~ /^(?i)object.based$/ ? "Object Based" :
                    _channels.tokenize("\\/\\.")

    String chSimple = any { au["Channel(s)"] } { au["Channel(s)/String"].replaceAll("channels", "") }

    switch (splitCh) {
        case { it instanceof String }:
            _ch = allOf { splitCh } { chSimple + "ch" }.join(" ")
            break

        case { it.size > 4 }:
            def wide = splitCh.takeRight(1)
            Double main = splitCh.take(4)*.toDouble().inject(0, { a, b -> a + b })
            Double sub = Double.parseDouble("0." + wide.last())
            _ch = (main + sub).toBigDecimal().setScale(1, java.math.RoundingMode.HALF_UP).toString()
            break

        case { it.size > 1 }:
            Double sub = Double.parseDouble(splitCh.takeRight(2).join("."))
            _ch = splitCh.take(2)*.toDouble().plus(sub).inject(0, { a, b -> a + b })
                        .toBigDecimal().setScale(1, java.math.RoundingMode.HALF_UP).toString()
            break

        default:
            _ch = splitCh.first().toDouble()
    }

    String _channelPositions = any{au["ChannelPositions"]}{null}
    String channelParse
    if ( _channelPositions != null && chSimple.toInteger() != _ch.tokenize(".")*.toInteger().sum() ) {
        List   channelsPos = _channelPositions.tokenize(",")
        String mainFix = channelsPos.take(3).inject(0) { acc, p ->
            Integer parsedCh = p.tokenize(":").takeRight(1).first().trim().tokenize(" ").size()
            acc + parsedCh
        }
        String subFix = channelsPos.takeRight(1).first().trim().tokenize(" ").size()
        channelParse = "${mainFix}.${subFix}"
    }

    String _chFix
    if (channelParse != null && Float.parseFloat(_ch) <=> Float.parseFloat(channelParse))  {
        _chFix = channelParse
    } else {
        _chFix = _ch.replaceAll(/(?i)Object Based/, '')
    }

    String _lang = any { au["Language"] } { video.first()["Language"] }
    List stream = allOf
        { allOf { _chFix } { au["NumberOfDynamicObjects"].concat("obj") }.join("+") }
        { allOf { codecMap.get(_ac, _ac) } {isAtmos/* atmos */}.join("+") }
        /* { allOf{ codecMap.get(combined, _aco) }{atmos}.join("+") } /* bit risky keeping _aco as default */
        { net.filebot.Language.findLanguage(_lang).ISO3.upperInitial() }
    /* _cf_ not being used > "Codec/Extensions", "Format" */
    HashMap ret = [:]
    ret.id = any{ au["StreamKindId"] }{ au["StreamKindPos"] }{ au["ID"] }
    ret.data = stream
    return ret
}.toSorted{ it.id }.collect{ it.data }*.join(" ").join(", ")

yields:

Code: Select all

Expression yields empty value: Cannot invoke method tokenize() on null object

How should I modify the code for inclusion?

I'm not splitting it into format bits as I'm also using these pieces in allOf blocks and there's many.

[rednoah]

1.

Code: Select all

Ambiguous expression could be either a parameterless closure expression or an isolated open code block

This error message suggests that you are interpreting FileBot format code as if it was Groovy code.



2.

Code: Select all

Cannot invoke method tokenize() on null object

This error message just means that your Groovy code crashes because some value is null. This is unrelated to splitting code into external files.


e.g. _channels could be null if none of these closures yield a value:

Code: Select all

String _channels = any
                { au["ChannelPositions/String2"] }
                { au["Channel(s)_Original"] }
                { au["Channel(s)"] }

You could add a fallback value at the end so that the result is never no value:

Code: Select all

String _channels = any
                { au["ChannelPositions/String2"] }
                { au["Channel(s)_Original"] }
                { au["Channel(s)"] }
                { "NO_AUDIO" }

[devster]

Thanks, first error is pretty straightforward, I solved it by removing {} from beginning and end, which is verbatim what I'm including from the previous post.
The second is weird, because the exact same code is not null when used as @/file.groovy or inside a format block.
I believe something breaks in the variables the included file uses, it seems the au variable is not an audio stream when including the file, but I can't figure out why.

example, adding assert au == "" right after au -> results in:

Code: Select all

java.util.concurrent.ExecutionException: Assertion failed: 

assert au == ''
       |  |
       |  false
       class audio

when included, instead of:

Code: Select all

java.util.concurrent.ExecutionException: Assertion failed: 

assert au == ''
       |  |
       |  false
       ['Count':{Count}, 'StreamCount':{StreamCount}, 'StreamKind':{StreamKind}, 'StreamKind/String':{StreamKind/String}, 'StreamKindID':{StreamKindID}, 'StreamOrder':{StreamOrder}, 'ID':{ID}, 'ID/String':{ID/String}, 'UniqueID':{UniqueID}, 'Format':{Format}, 'Format/String':{Format/String}, 'Format/Info':{Format/Info}, 'Format/Url':{Format/Url}, 'Format_Commercial':{Format_Commercial}, 'Format_Commercial_IfAny':{Format_Commercial_IfAny}, 'Format_Profile':{Format_Profile}, 'Format_Settings_Endianness':{Format_Settings_Endianness}, 'Format_AdditionalFeatures':{Format_AdditionalFeatures}, 'InternetMediaType':{InternetMediaType}, 'CodecID':{CodecID}, 'Duration':{Duration}, 'Duration/String':{Duration/String}, 'Duration/String1':{Duration/String1}, 'Duration/String2':{Duration/String2}, 'Duration/String3':{Duration/String3}, 'Duration/String4':{Duration/String4}, 'Duration/String5':{Duration/String5}, 'BitRate_Mode':{BitRate_Mode}, 'BitRate_Mode/String':{BitRate_Mode/String}, 'BitRate':{BitRate}, 'BitRate/String':{BitRate/String}, 'Channel(s)':{Channel(s)}, 'Channel(s)/String':{Channel(s)/String}, 'ChannelPositions':{ChannelPositions}, 'ChannelPositions/String2':{ChannelPositions/String2}, 'ChannelLayout':{ChannelLayout}, 'SamplesPerFrame':{SamplesPerFrame}, 'SamplingRate':{SamplingRate}, 'SamplingRate/String':{SamplingRate/String}, 'SamplingCount':{SamplingCount}, 'FrameRate':{FrameRate}, 'FrameRate/String':{FrameRate/String}, 'FrameCount':{FrameCount}, 'Compression_Mode':{Compression_Mode}, 'Compression_Mode/String':{Compression_Mode/String}, 'Delay':{Delay}, 'Delay/String3':{Delay/String3}, 'Delay_Source':{Delay_Source}, 'Delay_Source/String':{Delay_Source/String}, 'Video_Delay':{Video_Delay}, 'Video_Delay/String3':{Video_Delay/String3}, 'StreamSize':{StreamSize}, 'StreamSize/String':{StreamSize/String}, 'StreamSize/String1':{StreamSize/String1}, 'StreamSize/String2':{StreamSize/String2}, 'StreamSize/String3':{StreamSize/String3}, 'StreamSize/String4':{StreamSize/String4}, 'StreamSize/String5':{StreamSize/String5}, 'StreamSize_Proportion':{StreamSize_Proportion}, 'Language':{Language}, 'Language/String':{Language/String}, 'Language/String1':{Language/String1}, 'Language/String2':{Language/String2}, 'Language/String3':{Language/String3}, 'Language/String4':{Language/String4}, 'ServiceKind':{ServiceKind}, 'ServiceKind/String':{ServiceKind/String}, 'Default':{Default}, 'Default/String':{Default/String}, 'Forced':{Forced}, 'Forced/String':{Forced/String}, 'bsid':{bsid}, 'dialnorm':{dialnorm}, 'dialnorm/String':{dialnorm/String}, 'compr':{compr}, 'compr/String':{compr/String}, 'acmod':{acmod}, 'lfeon':{lfeon}, 'dialnorm_Average':{dialnorm_Average}, 'dialnorm_Average/String':{dialnorm_Average/String}, 'dialnorm_Minimum':{dialnorm_Minimum}, 'dialnorm_Minimum/String':{dialnorm_Minimum/String}, 'dialnorm_Maximum':{dialnorm_Maximum}, 'dialnorm_Maximum/String':{dialnorm_Maximum/String}, 'dialnorm_Count':{dialnorm_Count}, 'compr_Average':{compr_Average}, 'compr_Average/String':{compr_Average/String}, 'compr_Minimum':{compr_Minimum}, 'compr_Minimum/String':{compr_Minimum/String}, 'compr_Maximum':{compr_Maximum}, 'compr_Maximum/String':{compr_Maximum/String}, 'compr_Count':{compr_Count}]

when used as format.

[rednoah]

I recommend using splitting FileBot format code via FileBot format code @includes for simplicity and performance:



That'll allow you to just not include Groovy code from Groovy code, and bypass the issue altogether. FileBot format code @includes are much simpler since they really just concatenate the code.

[devster]

Unfortunately these would be dynamic files, different versions will be included in different formats, I'd really like to be able to use the include option.
I'm going to include different audio versions for different languages, it's just one example, I'd also like to use the same pattern for video and file name.

[rednoah]

I see. After some trial and error testing...

Looks like include 'audio.groovy' doesn't work because audio will then refer to the script class itself, due to the file name, and not the audio binding, in the context of that specific script file:

Code: Select all

println audio
println this

Code: Select all

class audio
audio@467f77a5

Anything other than audio.groovy will work, e.g. Audio.groovy, audio123.groovy, etc.

[devster]

Yep, I tried and it looks right, I imagine this also applies to any other names used by bindings, right?

[rednoah]

Yep, I tried and it looks right, I imagine this also applies to any other names used by bindings, right?

Yep. Just stick to CamelCase file names and lowercase variables and you won't have any issues.

Unfortunately these would be dynamic files, different versions will be included in different formats, I'd really like to be able to use the include option.

Please paste the code when you're done.