Changes to ar binding - format no longer working

[cheaters]

Hey Red

TLDR I found a solution - bottom of this post

I had been changing the result of Aspect Ratio in my formula from this:

2.40.1

to this:

2.40∶1

Since colons are not allowed in filenames on mac I had been using the ratio character which is allowed, but that stopped working in my format sometime around Aug 25, 2021 and just noticed this week You provided the format below at some point and I had been using it for years - thanks again for that tip.

Code: Select all

{info:video[0].displayAspectRatioString.colon(":").replace('?', '∶')}

I tried this today but it also failed to change anything (FYI in code view the colon and ratio character appear the same)

Code: Select all

{ar.toString().replace(':' '∶')}

This begins to work, other than the ratio character is duplicated...

Code: Select all

{ar.toString().replace('1', '∶1')}

Results in:

2.40∶∶1

Other interesting and unexpected formatting results...

Code: Select all

ar.toString().replace('1', '-1')}

Produces...

2.40∶-1

Code: Select all

ar.toString().replace('∶1', '-1')

(with ratio character)

Produces

2.40-1

After trying many things... the following works - not sure what's going on behind the scenes or what's changed though

Code: Select all

{ar.toString().replace('∶1', '"∶"1')}

That is: "single quote, ratio, 1, single quote, comma, single quote, double quote, ratio, double quote, 1, single quote"

2.40∶1

That is: "2, period, 4, 0, ratio, 1"

[cheaters]

Also I want to rename all of the existing files in my catalog ~6000. At one point Red you were going to explain how to do this without re-querying but using existing data instead?

[rednoah]

The {ar} binding is already using the ∶ RATIO character:

Code: Select all

$ filebot -mediainfo *.mp4 --format "{ ar } | { ar.transliterate('name') }"
4∶3 | \N{DIGIT FOUR}\N{RATIO}\N{DIGIT THREE}

The {ar} binding notably replaces all non-digit characters, including . DOT, with ∶ RATIO, so if {video[0].DisplayAspectRatioString} is giving you 2.40:1 then {ar} will give you 2∶40∶1 but you claim different results. Please double check that the code you write is actually the code that is being used.



What does the MediaInfo Inspector say for the file at hand?

Open FileBot ➔ Edit Format ➔ Change Sample ➔ Open MediaInfo ➔ Copy to Clipboard

[rednoah]

Also I want to rename all of the existing files in my catalog ~6000. At one point Red you were going to explain how to do this without re-querying but using existing data instead?

Please watch How do I restore match information from locally stored xattr metadata?
Please read Re-organize previously organized files using local xattr metadata

[cheaters]

I explained above what I am seeing when using the GUI not the CLI.

While testing it to show evidence of my "claim" I see a different but possibly related issue. See video screenshot below

In the GUI using just this preset format on a movie file with a display aspect ratio of 16:9...
{info:video[0].displayAspectRatioString.colon(":").replace('?', '∶')}

I get this as a result in the GUI "New Names" panel

16:9

If I double click to "edit name" in the GUI and drag that to the Character Viewer on macOS it tells me it's a "colon" not a ratio character and when I hit "Copy" (since macOS file system can't handle colons) it is changed to a forward slash by the macOS filesystem.

https://dl.dropboxusercontent.com/s/jq9 ... .mov?raw=1

Another video showing more (75MB)
Using this formula

Code: Select all

{n} ({y}) {info:video[0].displayAspectRatioString.colon(":").replace('?', '∶')}

https://dl.dropboxusercontent.com/s/4kc ... .mov?raw=1


When using the following formula I see the original issue and results. Looks like it's caused by {plex.derive} in combination with {info:video[0].displayAspectRatioString.colon(":").replace('?', '∶')

Code: Select all

{plex.derive{' '}{info:video[0].displayAspectRatioString.colon("∶").replace('?', '∶')}}

[rednoah]

The GUI the and the CLI are interchangeable. The CLI is more suitable for copy & paste:

Code: Select all

$ filebot -mediainfo *.mp4 --format "{ ar } | { ar.transliterate('name') }"
4∶3 | \N{DIGIT FOUR}\N{RATIO}\N{DIGIT THREE}

Video 1/2 show you're not using the {ar} binding. Video 2 @ 10s confirms that the {ar} does use RATIO but incorrectly replaces DOTS with RATIO as well:

The {ar} binding notably replaces all non-digit characters, including . DOT, with ∶ RATIO, so if {video[0].DisplayAspectRatioString} is giving you 2.40:1 then {ar} will give you 2∶40∶1 but you claim different results.

What does the MediaInfo Inspector say for the file at hand? That would allow me to fix the {ar} binding.

Open FileBot ➔ Edit Format ➔ Change Sample ➔ Open MediaInfo ➔ Copy to Clipboard

EDIT: Nevermind. I found a sample file with 2.35:1 aspect ratio. {ar} has been fixed with FileBot r9320 and now yields 2.35∶1 instead of 2∶35∶1.



As for your code, it's just wrong: video[0].displayAspectRatioString.colon(":").replace('?', '∶') doesn't do anything, because first you replace colons with colons, i.e. do nothing, and then you replace question mark with ratio, i.e. do nothing because there are no question marks to replace.

e.g. replace COLON with RATIO using the String.colon() method:

Code: Select all

{ video[0].displayAspectRatioString.colon('∶') }

e.g. replace COLON with RATIO using the String.replace() method:

Code: Select all

{ video[0].displayAspectRatioString.replace(':', '∶') }

[rednoah]

Ah! I see. The code you posted isn't actually the code you are using. If you inject custom values into {plex} then {plex} will indeed apply it's normalisation steps not just to its own components but also to components you've added. That's probably a fine detail that might have changed over time. Best to keep it though because it'd be a problem if someone accidentally injected / and it wasn't taken care of. If you must have a ratio, then you can always write your own format entirely.

Code: Select all

{ plex.derive{ " : " }{ '1∶4' } }

If you don't have subtitles, then you don't need to inject anything into the file name (before the subtitle language extension) then things are easy:

Code: Select all

{ plex } { ar }

If you want to use {plex} and deal with subtitles, and bypass {plex} normalisation steps, then you'll have to take it apart like so:

Code: Select all

{ plex ^ null } { ar }{ subt }

[cheaters]

What does the MediaInfo Inspector say for the file at hand? That would allow me to fix the {ar} binding.

Code: Select all

# MediaInfoLib - v21.09

CompleteName               : /Volumes/PlexMedia/PlexServer_1/Movies/Clockwatchers (1997)/Clockwatchers (1997) {imdb-tt0118866} 
[Video]

Width                     : 1920
Width/String              : 1 920 pixels
Height                    : 1040
Height/String             : 1 040 pixels
Sampled_Width             : 1920
Sampled_Height            : 1040
PixelAspectRatio          : 1.000
DisplayAspectRatio        : 1.846
DisplayAspectRatio/String : 1.85:1

https://dl.dropboxusercontent.com/s/d5j ... .mov?raw=0

using {ar.toString.replace}

https://dl.dropboxusercontent.com/s/gw1 ... .mov?raw=0

[rednoah]

Haha. Looks like you found a way to trick the system into doing what you want, but keep in mind that this is purely by happy accident, and not by design.


The colon() normalisation step will do different kinds of replacement depending on the context, and I guess you can trick the code into not replacing a ratio character by surrounding it with non-word characters:

Code: Select all

{ '[∶] 1∶2 1:2 : C : R'.colon(".", " - ")}

Code: Select all

[∶] 1.2 1.2 - C - R

Any invalid character like " is stripped, and so that's why what you're doing in your latest video kinda accidentally returns the value you want. The " " prevents the ratio from being converted to a . and the " " is then stripped leaving only the ratio remaining:

Code: Select all

{ plex.derive{ ' - "x"y"z"'}.name }

Code: Select all

Firefly - S01E01 - Serenity - xyz

[cheaters]

I would rather the aspect ratios were all "Film" ratios and not intermixed with "TV" ratios but I guess this depends on the metadata in the file itself?

4:3 should be 1.33:1
5:3 should be 1.66:1

etc.

I even see some ratios in metadata where instead of a ":1" there is a "0"

1.370

Here is a list of all the different and weird ratios in my filenames: I am thinking that some are calculated from the dimensions since the real ratio was never entered for the file.

Code: Select all

0-zsh% find -E . -regex ".*2\.[0-9]{1,3}.*|.*1\.[0-9]{1,2}.*" > output2.txt 
0-zsh% egrep -o -e '2\.[0-9]{1,3}' -e '1\.[0-9]{1,3}'  output2.txt | sort -u   

Code: Select all

1.098
1.104
1.106
1.185
1.189
1.196
1.200
1.21
1.22
1.278
1.296
1.299
1.30
1.31
1.32
1.370
1.371
1.373
1.374
1.375
1.376
1.378
1.38
1.381
1.383
1.397
1.40
1.400
1.404
1.407
1.422
1.43
1.430
1.439
1.448
1.49
1.700
1.726
1.739
1.85
1.882
1.886
1.890
1.893
1.895
1.897
1.901
1.905
1.912
1.959
1.967
1.983
1.992
2.0
2.000
2.004
2.008
2.013
2.017
2.022
2.043
2.096
2.115
2.14
2.2
2.25
2.35
2.40
2.469
2.474
2.500
2.540
2.548
2.551
2.553
2.560
2.667
2.682
2.743
2.759
2.791
2.909

[rednoah]

FileBot just passes along whatever MediaInfo can read from the file content, so maybe DisplayAspectRatio (decimal) might work better than DisplayAspectRatio/String (fraction) for your needs, and you can always add the ∶1 at the end:

Code: Select all

{ video[0].DisplayAspectRatio + '∶1' }

[cheaters]

Also I want to rename all of the existing files in my catalog ~6000. At one point Red you were going to explain how to do this without re-querying but using existing data instead?

Please watch How do I restore match information from locally stored xattr metadata?
Please read Re-organize previously organized files using local xattr metadata

This is unexpected. Nearly all of the movie files I have were renamed with FileBot CLI using a called bash script. Looking through those files it's hit and miss for xattr data.
Whereas the files I have recently renamed using the GUI (while testing this new formula) do contain xattr data.
Why would xattr data disappear or have never been added if the files were renamed with filebot?
How can I correct this?

[rednoah]

1.
There is a large number of possible reasons, that generally fall into one of two categories:
* Files are copied to a drive that doesn't support xattr / Alternate Data Streams and are silently lost at that point, or files are copied by a tool that is not aware of xattr / Alternate Data Streams and so that tool might write the file content to the target location, but might forget to do the same for Alternate Data Streams.
* Files have xattr / Alternate Data Streams are readable / writable in one file system view but not another, e.g. if you write xattr on a local Linux server, and then access these files remotely via SMB from a remote Windows client, then it probably won't work, because the SMB protocol typically doesn't expose local Linux xattr as Alternate Data Streams to remote SMB clients.



2.
You have to process files with FileBot again - as usual - to write metadata again. FileBot stores xattr / Alternate Data Streams when you click the Rename button.


If you choose to process files again, then you may want to configure FileBot to store metadata into plain text files beforehand, because normal files don't silently get lost, and work better if you are using SMB shares:

Configure FileBot to store xattr metadata in plain text files instead of filesystem xattr streams:

Code: Select all

filebot -script fn:properties --def net.filebot.xattr.store=.xattr

[cheaters]

If you choose to process files again, then you may want to configure FileBot to store metadata into plain text files beforehand, because normal files don't silently get lost, and work better if you are using SMB shares:

Configure FileBot to store xattr metadata in plain text files instead of filesystem xattr streams:

Code: Select all

filebot -script fn:properties --def net.filebot.xattr.store=.xattr

I liked this idea since whatever process is causing the metadata do be removed is unknown to me. My system supports xattr metadata.

So I did configure to save xattr as sidecar text files and hoped to move that metadata to external text files... Unfortunately I had already used tmdb datasource to rename all of my files, which was an overnight process

As of now the xattr data is attached to each of the files I renamed, so I hoped I could "export" that existing xattr data instead of re-querying tmdb datasource. If this is possible it seems it should be faster.

Is that possible? How?

[rednoah]

You can Export .xattr folders for existing files via the Filter ➔ Attributes ➔ Apply ➔ Export .xattr folders action.


Make sure to reset FileBot to default settings first. If you configure FileBot to use .xattr folders first, then it won't see native file system xattr.

[islandrob]

I have a question in relation to your last post. Some time ago I configured FIleBot to use .xattr folders, using the properties script you mentioned. I've decided to go back to native file system xattr. How do I switch it back?

Also, is there a way to write it into the file, as well as have the .xattr folder? I'm just curious on this point.

Rob

[rednoah]

I've decided to go back to native file system xattr. How do I switch it back?

You can unset settings by setting them to empty:

Code: Select all

filebot -script fn:properties --def net.filebot.xattr.store=

You can also manually edit the configuration file. See console output of the command if you need help finding the configuration file.

Also, is there a way to write it into the file, as well as have the .xattr folder? I'm just curious on this point.

Yes. You can Export .xattr folders in addition to default behaviour via the Filter ➔ Attributes ➔ Apply ➔ Export .xattr folders post-processing action.

That will only export the .xattr folders as a separate independent behaviour. FileBot internally won't read/write them unless it's configured to do so. There is no "use native xattr if possible; but try .xattr folders to if the former doesn't work" feature because having things in one place or the other almost guarantees that one will get lost somehow at some point. That said, the Smart Mode: Attributes feature (at least in newer revisions) does check for internal embedded attributes and external .xattr folders.

[cheaters]

Can we revisit this

FileBot just passes along whatever MediaInfo can read from the file content, so maybe DisplayAspectRatio (decimal) might work better than DisplayAspectRatio/String (fraction) for your needs, and you can always add the ∶1 at the end:

Code: Select all

{ video[0].DisplayAspectRatio + '∶1' }

On the command line, using {plex.id.derive}, I can not get a Ratio sign to appear on macOS. With no special formatting every colon is still replaced by periods. Not just {ar} but {h} as well.

Using a preset in FB GUI I CAN still get the following to work, but it does not work on the CLI.

Code: Select all

{
plex.id.derive{' {edition-'+tags.toString().replace('[','').replace(']','')+'}' }
{'['}{ hours.replace('∶', '"∶"')+']' }
{'['}{ video[0].displayAspectRatio + '"∶"1'}{']'}
}

[rednoah]

Can you provide a CLI test case that doesn't work? If you have an issue with the CLI, then presumably there's a CLI command at play that you could easily copy & paste?


e.g. {hours} uses ∶ RATIO and thus works by default:

Code: Select all

$ filebot -mediainfo -r /input --format "{hours}"
0∶23
...

e.g. adding ∶1 to some value also works:

Code: Select all

$ filebot -mediainfo -r /input --format "{ video[0].displayAspectRatio + '∶1' }"
1.333∶1
...

e.g. combining all of the above to rename files shows no issues:

Code: Select all

$ filebot -rename -r /input --db file --format "{fn} [{hours}] [{ video[0].displayAspectRatio + '∶1' }]"
[MOVE] from [X.mkv] to [X [0∶23] [1.333∶1].mkv]
...

If it works in the GUI, but not in the CLI, then you're likely not actually passing the format value you think you are passing. The " in your format would be the usual suspect if you also use "..." to escape argument values. Please read Cmdline and Argument Passing for details.



EDIT:

Note that {plex} will not work because {plex} will automatically replace COLON and RATIO with appropriate replacement characters, either - or . depending on the context. Any additional values that you inject into the {plex} path is subject to cleanup and normalization operations, and so you cannot inject colon, ratio, slash, etc into the {plex} path. This behaviour is identical in both GUI and CLI, so if you do indeed get different behaviours in GUI and CLI, then it's not that, or we're missing something.



EDIT 2:

FileBot r9649 will now use RATIO instead of . when cleaning/normalizing 1:1 patterns. This will also allow you to inject RATIO into {plex} with either COLON or RATIO since either will be normalized to RATIO.