Recent build changes

[devster]

Hello, I'll start by saying that I know building from source is unsupported and I'm not asking for support.
There have also been questions about the legality of a private build (albeit not appropriately phrased in my opinion), but as far as I can see this is allowed by the license, provided I bought one, which I have (lifetime even).

The build has been difficult, but possible, api keys were absent but I could provide my own and I could fill in the url.data.

Recently, however, my CI pipeline broke down after a few changes, mainly commits d14af0ce479b85e310ec1a888abe7c3754a068e3 and fa9793f37f10522fa1eab4a72341bb14f542aa1f which made the process much harder.
From what i can gather it seems key bits of the build process (ivy.xml and jre management) were moved to another repo, which, however, I'm unable to find. I also realized some useful build files (makefile, makefile.variables, profile.properties, .project, .classpath, .settings, .externalToolBuilders) are not included in the main repo.

So I would like to know whether there is a way to retrieve them?
Personally I would like to still have the option of a personal build, given the software is currently hosted on an open source platform. Similar softwares do provide this option (Sonarr, Radarr, SickChill, CouchPotato, Kodi, etc.).

[rednoah]

Oh, sorry to break your build setup. I didn't know anybody was actually building from source. I've changed a lot on my build, but a lot of that isn't really compatible with sharing publicly on GitHub (without additional effort).

The best way going forward is using the binaries I started publishing every other day:
https://get.filebot.net/filebot/BETA/

I have a lot of extra files on my development machine, IDE configuration files, helper scripts, etc. But these are not convenient for sharing as they change every day and contain sensitive information (i.e. code signing, ssh access, api tokens, etc).

It seems you are primarily missing the filebot-lib project (i.e. all the 3rd party binaries and related scripts) which currently isn't available on GitHub. I could send you a copy (~100 MB) but going forward keeping your own builds going probably will require significant investment of time and effort on your end, so I recommend just using the latest official beta builds, which will always be kept on the latest working revision.

[devster]

No worries, I always knew this could happen even if I don't particularly like the approach.
I will move to published binaries, as I don't see useful having a one-off copy of evolving data, but I hope you will consider the points below.

For the sake of argument (and everything that follows is strictly personal opinion), I would say that the repo as is on github seems too incomplete, there are too many crucial bits missing.
I would consider helper scripts and IDE config files as part of the project, as they are used for development, and in this case building as well, but the biggest part now seem to be the dependencies.
As I mentioned before, similar software hosted on GitHub provides most/all the information required to build it; I don't pretend to know much about software development but ease of build/test seems pretty important to attract possible contributors and they seem to manage the separation of secrets from their build systems so I wouldn't see a specific reason for not doing it. This would also have the side effect that Travis and AppVeyor, for example, offer free plans for open source, which could be a solid advantage.

From my perspective, the main argument would be the possibility to keep using the software should you decide to stop development in the future, for any reason. Another reason was my attempt (with little success) to use BellSoft Liberica alpine images for much smaller builds (JDK for alpine is around 40 Mb), as well as trying to figure out how the thing works.

[rednoah]

For the sake of argument (and everything that follows is strictly personal opinion), I would say that the repo as is on github seems too incomplete, there are too many crucial bits missing.

Admittedly, the public GitHub repository is primarily intended for the purpose of documentation and education at this point.

I would consider helper scripts and IDE config files as part of the project, as they are used for development, and in this case building as well, but the biggest part now seem to be the dependencies.

If it was just ivy.xml dependencies I probably wouldn't have bothered, but I was quite unhappy with having 100+ MB of binaries checked in as well, and de-coupling specific versions of dependencies (Java, Groovy, MediaInfo, etc) from the build just makes things easier when doing lots of builds for lots of platforms.

As I mentioned before, similar software hosted on GitHub provides most/all the information required to build it; I don't pretend to know much about software development but ease of build/test seems pretty important to attract possible contributors and they seem to manage the separation of secrets from their build systems so I wouldn't see a specific reason for not doing it.

Easy reproducible builds that work out of the box for new developers are doable, just a lot of extra work, and definitely worth it if the project relies on attracting new committers. FileBot has never had any 3rd party contributors, even when it was completely free and open and building was just a matter of doing git clone && ant run (builds were easy when there was just a single filebot.jar)

From my perspective, the main argument would be the possibility to keep using the software should you decide to stop development in the future, for any reason.

Well, you'd always have access to the existing binaries and services. IMHO, in the case of my sudden death, being able to easily rebuild filebot.jar from source will be the least of your problems. If anything, I'd wish for someone to take over and provide packages for end-users on all platforms, and that would most certainly be a serious commitment of time and effort, with hunting down a few jars to make the code compile being fairly negligible.


I've spent 30 min rewriting this paragraph. I don't really have a good answer for you here. It's complicated. FileBot is highly dependent on online services and has its own set of online services (and not just for license validation either). Last week you may have had a false sense of security by being able to build the jar yourself. This week maybe less so. Either way, you were never really protected against the event of my untimely demise. You probably wouldn't notice for 1-2 years, but then things would break in strange unexpected ways. Keeping things going past that point would require someone get seriously intimate with the code and related services.

Another reason was my attempt (with little success) to use BellSoft Liberia alpine images for much smaller builds (JDK for alpine is around 40 Mb), as well as trying to figure out how the thing works.

I suspect the problem isn't building FileBot (i.e. the platform-independent Java byte code) for Alpine, but building all the native dependencies for Alpine. (which is hard, which is why I used to just dump Debian binaries into GitHub repository, instead of building from source)


My java-installer is using the BellSoft Liberica JDK for armv7, aarch64 and x86 Linux builds and I'm not away of any fundamental issues running Java code on these platforms.


GraalVM native-image builds are on the horizon, but making it work for FileBot for all platforms will be very difficult and time-consuming.

[devster]

Easy reproducible builds that work out of the box for new developers are doable, just a lot of extra work, and definitely worth it if the project relies on attracting new committers. FileBot has never had any 3rd party contributors, even when it was completely free and open and building was just a matter of doing git clone && ant run (builds were easy when there was just a single filebot.jar)

Easy and reproducible builds should be something useful for anyone, and again, useful also because there are services who provide building infrastructure for free. Until recently I could do ant resolve + ant portable, I don't really see a downside. I accept your conclusion, I'm just having a difficult time figuring out the reasoning behind it, especially since several commits lately are related to it.
Regarding contributors I think the landscape changed a bit since 2007. A major driver for Sonarr and Radarr usage for example seem to be https://linuxserver.io container images which make them fairly easily accessible (and their heavy focus on WebUI).
It seems a catch-22 scenario.

If anything, I'd wish for someone to take over and provide packages for end-users on all platforms, and that would most certainly be a serious commitment of time and effort, with hunting down a few jars to make the code compile being fairly negligible.

... You probably wouldn't notice for 1-2 years, but then things would break in strange unexpected ways. Keeping things going past that point would require someone get seriously intimate with the code and related services.

This is actually a good argument for having everything available, from what I can see they don't just build filebot.jar.
Regarding related services, excluding the databases (TVDB, TMDB, etc.), leaves out https://api.filebot.net/v5/data (replaceable by -Durl.<file> options even at runtime) and scripts, a compressed jar file. Is there more?

I suspect the problem isn't building FileBot (i.e. the platform-independent Java byte code) for Alpine, but building all the native dependencies for Alpine. (which is hard, which is why I used to just dump Debian binaries into GitHub repository, instead of building from source)

Actually the issue is only with platforms other than x86_64, alpine has OpenJDK 12 and OpenJFX 12 only on that arch, while for the others it only has OpenJDK 11 (as well as MediaInfo, chromaprint, ffmpeg, etc.), but without OpenJFX. At least this is what I gathered.

P.S. I was thinking more about the risk of you getting bored with it, nothing worse.

[rednoah]

This is actually a good argument for having everything available, from what I can see they don't just build filebot.jar.
Regarding related services, excluding the databases (TVDB, TMDB, etc.), leaves out https://api.filebot.net/v5/data (replaceable by -Durl.<file> options even at runtime) and scripts, a compressed jar file. Is there more?

I have a Synology NAS who's sole purpose (other than testing Synology NAS) to keep those updated. Incremental updates only take a few minutes, but building from scratch would take a few weeks. That's been running smoothly for years though, so I myself forget how things work. If debug logging is enabled, then you should have a pretty good idea what's going in and out. I think a notable recent addition is the thumbnail index. This one might also take a week or two to build from scratch. This stuff is all pretty hacky, but it works, so I don't touch it.

I forgot about the script jar. That one is code signed so you can't build a working one yourself, without also replacing the public key that's baked into the filebot jar. You can always clone the scripts from GitHub though. There's no build or dependencies so switching to local scripts would be easy.

Actually the issue is only with platforms other than x86_64, alpine has OpenJDK 12 and OpenJFX 12 only on that arch, while for the others it only has OpenJDK 11 (as well as MediaInfo, chromaprint, ffmpeg, etc.), but without OpenJFX. At least this is what I gathered.

The problem of having one big fat repository is that you can't build the CLI without also building the GUI, which means you can't build with a headless JRE because AWT / Swing / JavaFX / etc will be missing. The good thing is that since it's Java, and all compiled for Java 11, so you can take the jar build from any platform, and use it on any other platform. Currently, all builds are targeted at Java 11 for compatibility reasons since that's the LTS, except for the Java 8 backwards compatibility builds of course, i.e. the portable package should work out of the box, as far as FileBot and Java code is concerned, with the caveat of optional native dependencies (i.e. libmediainfo.so and friends) possibly not working.

I was thinking more about the risk of you getting bored with it, nothing worse.

If that were to happen, I could just dump everything on GitHub and let the community figure it out. But as long as I'm around to maintain things and pay bills, things will continue to run smoothly indefinitely, no worries there, especially with the new license system, which is aimed at steady long-term revenue.

[devster]

I have a Synology NAS who's sole purpose (other than testing Synology NAS) to keep those updated. Incremental updates only take a few minutes, but building from scratch would take a few weeks. That's been running smoothly for years though, so I myself forget how things work. [...] This stuff is all pretty hacky, but it works, so I don't touch it.

Now I'm even more convinced a proper pipeline would help. Also a build on a provider likely wouldn't take that long.
I'm still lost on the motivation, why not have a solid build pipeline, especially for the jar, which as you say is platform-independent?
I mean, it's not like it's not being executed every couple of days to create new beta versions.

The problem of having one big fat repository is that you can't build the CLI without also building the GUI, which means you can't build with a headless JRE because AWT / Swing / JavaFX / etc will be missing.

I think this was mentioned at some point in the past, but you also said that decoupling it would take massive amount of work. In that case I'd probably vote for a persistent process with an API / WebUI.

[rednoah]

Both of these things basically boil down to lots of not-so-fun work with no tangible benefits for end users. It's like destroying a puzzle, and then putting it back together, but better this time. If things break and require a serious make over, I'll think about doing it in a more reproducible way. I'm just pretty sure you'd be the only person in the world interested in that.


EDIT:

Notably, you are the only person who ever contributed useful code, and that was probably after your CI already didn't work anymore.

[devster]

Sorry, been out of commission for a few days.

Notably, you are the only person who ever contributed useful code, and that was probably after your CI already didn't work anymore.

That's probably the height of my abilities. Also something that can be decoupled from FileBot, I don't see people contributing to the main code.

Anyway, I'll adapt to the new state of things and hope it will never become a new Deluge situation.
Thank you.

[rednoah]

What's happening with Deluge?

[devster]

Single developer for the whole project, tried Patreon, had some issues and development of new v2 stalled for years.
Now he just released new versions, but many changed clients which is unfortunate, the daemon + thin client I think is a very good model.

[rednoah]

That's a sad story. I tried Patreon as well. It just doesn't really for software products. At least it's still around. Sickrage died from one day to the next when the developer moved on.

On the bright side, FileBot chose a different path forward. Since FileBot doesn't rely on donations anymore, and introduced a sustainable sales model instead, you don't have to worry about FileBot falling into disarray anytime soon.

[devster]

SickRage story is a bit more complex.

echel0n forked SickBeard and created SickRage, to which a lot of people were contributing. He then disappeared for a while and the other devs kept going.

Then he came back and tried a hostile takeover which triggered the revocation of his commit access. He forked again with the same name but adding bitcoin miners to the website and the web interface.
He also sent a DMCA request to GitHub citing copyright infringement (but it's GPL, so that got bounced).

Devs split off in 2 directions: Medusa and the renamed SickChill to avoid further meddling by echel0n.

Because of all the ruckus many switched to Sonarr, but alternatives didn't die as Sonarr doesn't yet do symlinks, which for torrents are very useful.

[rednoah]

Wow. I had no idea. I completely missed all that drama! I just noticed a few competitors have come and gone over the years, though I usually don't notice until years after the fact, cause I personally don't really use any of them. I occasionally check things out when people need help with FileBot integration.

The tree of evolution there is interesting though!

[Nonlo]

SickRage story is a bit more complex.

echel0n forked SickBeard and created SickRage, to which a lot of people were contributing. He then disappeared for a while and the other devs kept going.

Then he came back and tried a hostile takeover which triggered the revocation of his commit access. He forked again with the same name but adding bitcoin miners to the website and the web interface.
He also sent a DMCA request to GitHub citing copyright infringement (but it's GPL, so that got bounced).

Devs split off in 2 directions: Medusa and the renamed SickChill to avoid further meddling by echel0n.

Because of all the ruckus many switched to Sonarr, but alternatives didn't die as Sonarr doesn't yet do symlinks, which for torrents are very useful.

I did not even think that such interesting vicissitudes occur)

[kim]

Did you completely remove the source code now ?
(if so, I really don't like this move)

[rednoah]

Yes, I've refactored things into a couple of private repositories, so I stopped commiting into the public one a few weeks ago.

I don't mind sending you a source tarball thought if you want, but what do you need it for anyway? As documentation for internals?

I'm surprised somebody even noticed. Nobody seemed to contribute or comments on anything.

[devster]

Did you completely remove the source code now ?
(if so, I really don't like this move)

Neither do I, but it was just a matter of time.

I'm surprised somebody even noticed. Nobody seemed to contribute or comments on anything.

Maybe, but I routinely used it to check how bindings were defined for example, or which changes were made to APIs and scrapers.

[kim]

"Nobody seemed to contribute or comments on anything."
You are wrong here, I did many times, just not directly.
I/We use this forum for that.
If I was a better programmer I wound add directly to source code,
but it's your code and you are way better to make changes then I.

Pro source code:
e.g. how would I know that Filebot is only checking for 7 digit IMDb id and that this could be a problem in the future if no source code?
or that something is missing from "pattern.video.source" or "pattern.video.tags"
or when api's "downloaded" 0 bytes files, I compiled the source code to confirm the fix was to change http to https.
or missing/basic support for "{info.adult}", spokenLanguages or AlternativeTitles
or that Filebot is using old api info on themoviedb
and more...

Over the years I have looked at the source code MANY times to:
1. learn from it
2. To get access to internal stuff to "hack" and make it do what I need it to do,
without up to date source code, I can't do this.
3. I look at the bindings also
4. etc.

Why I use Filebot:
1. I can customize it to what I want/need with the help from e.g. source code.
2. To learn java/groovy e.g from source code.
3. I feel like I'm part of something to help make it better and can see the changes in the source code I helped with or requested.

So no source code = not happy

[rednoah]

1.
It's always a compromise. I'd be nice if it could just be open and free, but I do have a business to run, and unfortunately there's bad actors that don't use the source like you do, and do this or that instead.

A shoutout to Phoenix09 here because he actually respected the license change. A reason the source was available for so long despite going commercial and the obvious dangers that entails.


2.
The API docs include source attachments, so sources relevant to format and script development are available:
https://www.filebot.net/docs/api/

If you need help with "How to do A?" then ask so I know there's a use case, so that I can introduce well-defined standard API for scripting. That would be preferred over accessing internals which are subject to change.


3.
Users report bugs. Developers look into them and fix them. Indeed, users have contributed much in terms of bug reports and feature requests. And I'm committed to read and resolve all bug reports. User do what users do best. Developers do what developers do best.

If something doesn't work, report it, and I'll fix it and release an update, probably even on the same day. Same for new features. Same for additional logging. I'll do the work for you. That's what you pay me for.


4.

1. I can customize it to what I want/need with the help from e.g. source code.
2. To learn java/groovy e.g from source code.
3. I feel like I'm part of something to help make it better and can see the changes in the source code I helped with or requested.

That is indeed unfortunate, and I'm sorry to take this from you.

All supporting tools that enhance FileBot (rather than replace) will remain open of course:
https://github.com/filebot/scripts


5.
Let's try closed-source for a while and see what happens. Maybe there are many unforeseen disadvantages. I'd be happy to change my mind on this one if it makes sense from a business perspective.

[devster]

I don't believe there are disadvantages for you, the only disadvantage is for us.
We bought a license for an open source software which became closed source.
Nothing we can do either, despite its flaws, it's still one of the most comprehensive tools to rename media.

A small compromise could be to leave a non-updated version of the repo on GitHub, archived.

[rednoah]

Well, for people here just ask and I'll mail you a tarball of the latest.

In general, no source means no excuse for lack of documentation. So if there's anything you need, just ask and I'll write related documentation.

[ZGab]

I agree kim and devster.

We've paid a licence for the product to help you on your work on this stuff, you really deserve it. Much greetings on you've accomplished !
This licence was acquired on Patreon as a support for your hard development support, at this moment it was not question about closing the OpenSource resources.
Since few month you are about deconstruct the conditions on what we have contribute.
Please do not go on the way of the deconstruction of trust.

The visibility of the source and the possibility for developers to get, modify and build the code in complete respect of your original terms is in my own mind the warranty of a global understanding of what you are doing and an important value of your product.

I understand that some projects as you mentioned upper make you mad rage. It's shame and your licence must cover your ownership.
Note: I think you can report to Microsoft the first case as an usurpation, they are aware about such situation.

I've nothing to advice to you, but as a faithful user and supporter of FileBot I can just tell my disappointment and kind of frustration about what is coming...

Note: whatever, I will respect your choice, and I thank you again for the great job you are doing

[kim]

Why is this still up ?
"File Bot - Intelligent software For FFmpeg"
"Release date
6/5/2018"
btw: does not look like it works anyway
"Does not work at all I get a Malformed URL exception error..."

In general, no source means no excuse for lack of documentation.

This will take you a looong time make properly. Even if the info is on this forum, it can be very difficult find what you need.

Filebot has been open source 10+ years ?
If someone wanted to take/copy your hard work they would have done it already.
In your own words it would take at least 30+ min. to compile the first time (hours/days for a noob, if ever) + there was missing stuff from source code a long time AND don't forget the new license stuff (maybe hours to bypass this ?)
I think most people would just use an alternative software https://alternativeto.net/software/filebot/ if they want it free.

Anyway, I do not think the source code (program) is the product, it's way to complex to maintain/update without better documentation.
The product is YOU and the very good support you provide (this is what people will pay for), because it's so complex

Maybe go a other way, no license key, no support / no access to source code ?

[rednoah]

Writing lots of docs is easy. Writing docs that container neither too little nor too much information as to make them useful is the tricky part here. Lots of users have never seen a command-line before. Others are Groovy developers feel right at home with Groovy expressions.

Plus, 95% of FileBot code is useless even for the most advanced scripting needs. It makes sense to identify those 5% and refactor those into a stable API and open-source it.

Do you have your custom scripts on GitHub? What internal FileBot APIs do you use?

(Note that JPMS modularization coming in future releases might make it impossible to access internal APIs anyway unless explicitly declared; GraalVM ahead-of-time-compilation might make certain internal APIs nonexistent at runtime; but that's a greater Groovy / Java 11 / GraalVM topic for another day; but knowing what internal stuff is being used would be extremely helpful with ensuring future backwards compatibility)

Based on posts here, I can only identify 1 internal API that might benefit from a public API:
* FileBot Cache API

[devster]

Definitely docs on how the mediainfo used for bindings.
Possibly convenience methods for Language and multiple audios.
Right now I'm doing:

Code: Select all

{ def _lang = any{ au["Language"] }{ video.first()["Language"] }
  net.filebot.Language.findLanguage(_lang).ISO3.upperInitial() }

[rednoah]

1.
MediaInfo changes quite often, so the MediaInfo fields that various bindings are based on are subject to change. I'll see about better documentation, but it'll focus on purpose and possible values, rather than implementation details. That being said, implementation details are and remain available in the public API documentation.


2.
{audioLanguages} is available in recent releases, though it's based on MediaInfo audio stream languages only. Language is on my list of public API classes already, since Language objects are returned by various format bindings.

I've never seen a video stream with Language defined. Is that a thing? Different video streams for different locales?

[devster]

1.
MediaInfo changes quite often, so the MediaInfo fields that various bindings are based on are subject to change. I'll see about better documentation, but it'll focus on purpose and possible values, rather than implementation details. That being said, implementation details are and remain available in the public API documentation.

The link you provided is more than sufficient. That's how we noticed the changes in mediainfo from v17 to v18 so I'd say it's useful to have.

2.
{audioLanguages} is available in recent releases, though it's based on MediaInfo audio stream languages only. Language is on my list of public API classes already, since Language objects are returned by various format bindings.

{audioLanguages} does do Languages but not different audio formats. Most of my movies are BD rips, which means there are commentary tracks with lower audio quality, hence the "convenience". Kim already has a way to extract all audio streams and various info, it's doable.

I've never seen a video stream with Language defined. Is that a thing? Different video streams for different locales?

There are files for which a mediainfo language field is set for video but not for audio, for some reason. No idea why they do it, but it comes from the encoders most likely.

[rednoah]

2.
Strange. If you could make a separate thread and dump the raw MediaInfo tables for these files, that would be appreciated (or maybe even sample files, a few seconds will do). Then I can see about adding a filter to the built-in bindings to ignore commentary tracks.

I've added a new script to make it easy to dump MediaInfo tables:

Code: Select all

filebot -script fn:mediainfo .

https://github.com/filebot/scripts/blob ... nfo.groovy

Same as doing find . -type f -exec mediainfo {} + but works on Windows too.

[kim]

ALL of this a min. must have ( MUST be able to use any path (string), params, Locale and Cache)

net.filebot.web.OpenSubtitlesHasher.* ( computeHash() )
net.filebot.Cache.getCache()

TheTVDB.requestJson()
TheTVDB.getEpisodeList()
TheTVDB.getSeriesInfo()
TheTVDB.resolveImage()
TheTVDB.getArtwork()

TheMovieDB.properties.configuration.images.secure_base_url
TheMovieDB.getArtwork()
TheMovieDB.getAlternativeTitles()
TheMovieDB.request()
TheMovieDB.getMovieInfo()
TheMovieDB.resolveImage()

OMDb.request()

TVmaze.request()

FanartTV.getArtwork()

+ any already in a script
e.g. MediaInfo.snapshot()

and to be sure it still will work:
File.metadata
_args.order
instanceof MultiEpisode
.nameWithoutExtension

my script is 1600 lines atm so I maybe forgeting something

[rednoah]

Do you happen to have a GitHub repository? Then I could add it to my integration tests to make sure it "compiles" at the very least.

[kim]

No, I do not
but here is an old "cutdown" version
viewtopic.php?f=4&t=5451

I'm working on a new version with more stuff, but this takes a long time

[rednoah]

No worries. I'll at the TheTVDB / TheMovieDB / etc client classes to the public API list.

Fortunately, most of it is already public API and generally not subject to change:
* https://www.filebot.net/docs/api/net/fi ... a.io.File)
* https://www.filebot.net/docs/api/net/fi ... web.Movie)
* ...

[kim]

when will it be added ?

[rednoah]

You mean comprehensive docs in general, cause that will take time... or are you asking about something more specific?

What specific TheTVDB / TheMovieDB features do you need? Because I'd rather implement public API for whatever information you might need, via extension methods similar to Movie.alternativeTitles, rather than giving you direct access to the client classes (which are subject to change, and may become inaccessible in the future due to JMS module restrictions at runtime, or possibly ahead-of-time compilation if GraalVM works out).

i.e. you share the code where you use these generic API access methods?

Code: Select all

TheTVDB.requestJson()
TheTVDB.resolveImage()

Code: Select all

TheMovieDB.properties.configuration.images.secure_base_url
TheMovieDB.request()
TheMovieDB.resolveImage()

I'd rather implement code for all the requirements you (or others) might have, and then make it available via Movie class extension methods, so you don't need to do the plumbing yourself.



EDIT:

I've added the requested classes to the docs:
https://www.filebot.net/docs/api/net/fi ... lient.html
https://www.filebot.net/docs/api/net/fi ... lient.html

Since the methods aren't even public in the internal API, they won't show up in the docs, so just click on any public method to see the source, and then scroll to what you are interested in. Naturally, this is a bad solution. Best to write proper public API.

[kim]

That looks like most of it, but not all e.g.

net.filebot.web.OpenSubtitlesHasher.* ( computeHash() )
net.filebot.Cache.getCache()
MediaInfo.snapshot()

The problem with this is you have all the power, e.g. you will not support episode in amc/htpc viewtopic.php?f=6&t=3660#p20471

Can you give me some samples of "subject to change" ?
(to me "restrictions" sounds like I can't do all I need = less Filebot)

[rednoah]

The problem with this is you have all the power, e.g. you will not support episode in amc/htpc viewtopic.php?f=6&t=3660#p20471

Well, TheTVDB / TheMovieDB API is not infinite, and FileBot will implement as much as necessary. It makes sense to implement more of it, so that I can expose it as stable public API to you via scripting.


Note that I have already introduced public API for this particular use case, i.e. Episode.getInfo() allows you to get per-episode information in both format and scripting:

Code: Select all

filebot -list --q Firefly --format "{episode} => {episode.info.director} | {episode.info.writer} | {episode.info.overview}"

Using internal API is possible by nature of Groovy, not because it's wise, and maybe at the time no better option was available, and so now we're stuck with having lots of code that relies on internal APIs, and I'm stuck having to worry about unexpectedly breaking things whenever I refactor things (which means I will avoid refactoring code) because I have absolutely no idea what is being used and what isn't.


I once changed the name space from net.sf.filebot.* to just net.filebot.* and that broke all formats that used internal API. I accidentally broke lots of formats because constructs like {{vc} + {ac}} (an accidental feature I wasn't even aware of) stopped working. MediaInfo related code keeps breaking for corner cases all the time, since MediaInfo changes behaviour with each version.


TL;DR If you use internal API, do so at your own risk, don't blame me if things break. That's what internal API means. It's probably gonna be fine, but it might not. Since I don't know what you're doing, I cannot tell you what might break.

[rednoah]

Code: Select all

net.filebot.web.OpenSubtitlesHasher.* ( computeHash() )
net.filebot.Cache.getCache()
MediaInfo.snapshot()

I don't have public API for these because I don't have a use case yet. You can provide one.


There is File.getMediaCharacteristics() but that's either a MediaInfo object or a FFProbe object. (I've added implementing classes for MediaInfo and FFProbe to the docs, though I wouldn't strictly call them public API. I've just added them for your convenience. Use at your own risk.)


MediaInfo.snapshot() is meant for debugging if you don't know what raw MediaInfo properties there are. It is likely best to use getMediaInfo(file, "{vf}") if you need something specific so that you can use the usual format bindings (which typically try various MediaInfo fields to make it work regardless of underlying libmediainfo version).


IDK what you use Cache for. Depending on how you use it, a public Cache interface might make sense, or more API interfaces so you don't need to worry about the Cache yourself.


OpenSubtitlesHasher, AFAIK, it's only useful for OpenSubtitles search requests, and FileBot already implements those. I'd be curious what you use it for, so I can think about a good way of exposing it (or void the need to access it directly in the first place).




TL;DR These things should have been done properly from the beginning, with clearly defined public interfaces, rather than just accessing internal classes directly. That was my fault. Advanced custom scripting just wasn't a priority 3-4 years ago, and these kludgy solutions were just kinda there.




EDIT:

Note that the timing for these "restrictions" is not random. The whole point of Java 9 / 10 / 11 is modularization, which means properly defining and strictly enforcing at runtime all the dependencies within a system. Once Java 8 support is no longer necessary, it will be dropped, and with a proper Java 11 port comes proper modularization and clearly defined access restrictions between modules (which Groovy can't just ignore since the JVM natively enforces them). Things will break. If I have known use cases (i.e. test cases) then I will notice the issue and fix the issue. If not, then backwards compatibility issues will go unnoticed.

[kim]

If Filebot's "public API" does not provide access to EVERYTHING e.g. thetvdb's api does for series, episode, artwork...
I need an alternative like TheTVDB.requestJson, same goes for every other api/data.

Code: Select all

{episode}

e.g. cant use, because missing info:
filename (thumb)
imdbId
dvdSeason
dvdEpisodeNumber

this is the min. ref. https://kodi.wiki/view/NFO_files
I use this plus more in my nfo's

Code: Select all

MediaInfo.snapshot()

You do know you use it in the htpc ?
https://github.com/filebot/scripts/blob ... roovy#L256

Code: Select all

getMediaInfo(file, "{vf}")

Again I need access to EVERYTHING like MediaInfo.snapshot() does

Cache:
I use Cache for any resource that does not use Cache
e.g.

Code: Select all

net.filebot.Cache.getCache('IMDbData_JSON', net.filebot.CacheType.Weekly).computeIfAbsent(IMDbId) {}

OpenSubtitlesHasher:
I use

Code: Select all

computeHash(inputFile)

e.g. fast way to find duplicates files

I have no problem with things breaking, as long as it's possible for me to fix.
(also a good way to learn code)

[rednoah]

1.
I have to think about better EpisodeInfo API support, because ideally you shouldn't need to know TheTVDB / TheMovieDB / TVmaze / etc API details to use that. I'll look into it.


2.
In the meanwhile, I've added these extension methods. I've added GroovyObjectSupport to make it easier to navigate the MediaInfo data structure.

Code: Select all

def f = '/path/to/file' as File

// Get Format property of Video Stream #0
println f.mediaInfo.Video[0].Format

// Iterate audio streams
f.mediaInfo.Audio.each{ 
	println it.StreamKindID
	println it.Format
}

Code: Select all

f.hash 'moviehash'  // OpenSubtitles Movie Hash
f.hash 'crc32'
f.hash 'md5'
f.hash 'sha256'

EDIT:

Added artwork extension methods:

Code: Select all

{episode.seriesInfo.artwork}

(tentatively) Added extension methods for raw json objects:

Code: Select all

{episode.seriesInfo.raw}

Code: Select all

{episode.info.raw}

(currently only implemented for TheTVDB)