Recent build changes

Talk about the Ultimate Question of Life, The Universe, and Everything
Post Reply
devster
Posts: 275
Joined: 06 Jun 2017, 22:56

Recent build changes

Post by devster » 08 Jun 2019, 19:56

Hello, I'll start by saying that I know building from source is unsupported and I'm not asking for support.
There have also been questions about the legality of a private build (albeit not appropriately phrased in my opinion), but as far as I can see this is allowed by the license, provided I bought one, which I have (lifetime even).

The build has been difficult, but possible, api keys were absent but I could provide my own and I could fill in the url.data.

Recently, however, my CI pipeline broke down after a few changes, mainly commits d14af0ce479b85e310ec1a888abe7c3754a068e3 and fa9793f37f10522fa1eab4a72341bb14f542aa1f which made the process much harder.
From what i can gather it seems key bits of the build process (ivy.xml and jre management) were moved to another repo, which, however, I'm unable to find. I also realized some useful build files (makefile, makefile.variables, profile.properties, .project, .classpath, .settings, .externalToolBuilders) are not included in the main repo.

So I would like to know whether there is a way to retrieve them?
Personally I would like to still have the option of a personal build, given the software is currently hosted on an open source platform. Similar softwares do provide this option (Sonarr, Radarr, SickChill, CouchPotato, Kodi, etc.).
I only work in black and sometimes very, very dark grey. (Batman)

User avatar
rednoah
The Source
Posts: 15958
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Recent build changes

Post by rednoah » 09 Jun 2019, 04:37

Oh, sorry to break your build setup. I didn't know anybody was actually building from source. I've changed a lot on my build, but a lot of that isn't really compatible with sharing publicly on GitHub (without additional effort).

The best way going forward is using the binaries I started publishing every other day:
https://get.filebot.net/filebot/BETA/

I have a lot of extra files on my development machine, IDE configuration files, helper scripts, etc. But these are not convenient for sharing as they change every day and contain sensitive information (i.e. code signing, ssh access, api tokens, etc).

It seems you are primarily missing the filebot-lib project (i.e. all the 3rd party binaries and related scripts) which currently isn't available on GitHub. I could send you a copy (~100 MB) but going forward keeping your own builds going probably will require significant investment of time and effort on your end, so I recommend just using the latest official beta builds, which will always be kept on the latest working revision.
:idea: Please read the FAQ and How to Request Help.

devster
Posts: 275
Joined: 06 Jun 2017, 22:56

Re: Recent build changes

Post by devster » 09 Jun 2019, 08:23

No worries, I always knew this could happen even if I don't particularly like the approach.
I will move to published binaries, as I don't see useful having a one-off copy of evolving data, but I hope you will consider the points below.

For the sake of argument (and everything that follows is strictly personal opinion), I would say that the repo as is on github seems too incomplete, there are too many crucial bits missing.
I would consider helper scripts and IDE config files as part of the project, as they are used for development, and in this case building as well, but the biggest part now seem to be the dependencies.
As I mentioned before, similar software hosted on GitHub provides most/all the information required to build it; I don't pretend to know much about software development but ease of build/test seems pretty important to attract possible contributors and they seem to manage the separation of secrets from their build systems so I wouldn't see a specific reason for not doing it. This would also have the side effect that Travis and AppVeyor, for example, offer free plans for open source, which could be a solid advantage.

From my perspective, the main argument would be the possibility to keep using the software should you decide to stop development in the future, for any reason. Another reason was my attempt (with little success) to use BellSoft Liberica alpine images for much smaller builds (JDK for alpine is around 40 Mb), as well as trying to figure out how the thing works.
I only work in black and sometimes very, very dark grey. (Batman)

User avatar
rednoah
The Source
Posts: 15958
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Recent build changes

Post by rednoah » 09 Jun 2019, 11:31

devster wrote:
09 Jun 2019, 08:23
For the sake of argument (and everything that follows is strictly personal opinion), I would say that the repo as is on github seems too incomplete, there are too many crucial bits missing.
Admittedly, the public GitHub repository is primarily intended for the purpose of documentation and education at this point.

devster wrote:
09 Jun 2019, 08:23
I would consider helper scripts and IDE config files as part of the project, as they are used for development, and in this case building as well, but the biggest part now seem to be the dependencies.
If it was just ivy.xml dependencies I probably wouldn't have bothered, but I was quite unhappy with having 100+ MB of binaries checked in as well, and de-coupling specific versions of dependencies (Java, Groovy, MediaInfo, etc) from the build just makes things easier when doing lots of builds for lots of platforms.

devster wrote:
09 Jun 2019, 08:23
As I mentioned before, similar software hosted on GitHub provides most/all the information required to build it; I don't pretend to know much about software development but ease of build/test seems pretty important to attract possible contributors and they seem to manage the separation of secrets from their build systems so I wouldn't see a specific reason for not doing it.
Easy reproducible builds that work out of the box for new developers are doable, just a lot of extra work, and definitely worth it if the project relies on attracting new committers. FileBot has never had any 3rd party contributors, even when it was completely free and open and building was just a matter of doing git clone && ant run (builds were easy when there was just a single filebot.jar)

devster wrote:
09 Jun 2019, 08:23
From my perspective, the main argument would be the possibility to keep using the software should you decide to stop development in the future, for any reason.
Well, you'd always have access to the existing binaries and services. IMHO, in the case of my sudden death, being able to easily rebuild filebot.jar from source will be the least of your problems. If anything, I'd wish for someone to take over and provide packages for end-users on all platforms, and that would most certainly be a serious commitment of time and effort, with hunting down a few jars to make the code compile being fairly negligible.


:idea: I've spent 30 min rewriting this paragraph. I don't really have a good answer for you here. It's complicated. FileBot is highly dependent on online services and has its own set of online services (and not just for license validation either). Last week you may have had a false sense of security by being able to build the jar yourself. This week maybe less so. Either way, you were never really protected against the event of my untimely demise. You probably wouldn't notice for 1-2 years, but then things would break in strange unexpected ways. Keeping things going past that point would require someone get seriously intimate with the code and related services.


devster wrote:
09 Jun 2019, 08:23
Another reason was my attempt (with little success) to use BellSoft Liberia alpine images for much smaller builds (JDK for alpine is around 40 Mb), as well as trying to figure out how the thing works.
I suspect the problem isn't building FileBot (i.e. the platform-independent Java byte code) for Alpine, but building all the native dependencies for Alpine. (which is hard, which is why I used to just dump Debian binaries into GitHub repository, instead of building from source)


:idea: My java-installer is using the BellSoft Liberica JDK for armv7, aarch64 and x86 Linux builds and I'm not away of any fundamental issues running Java code on these platforms.


:idea: GraalVM native-image builds are on the horizon, but making it work for FileBot for all platforms will be very difficult and time-consuming.
:idea: Please read the FAQ and How to Request Help.

devster
Posts: 275
Joined: 06 Jun 2017, 22:56

Re: Recent build changes

Post by devster » 09 Jun 2019, 12:57

rednoah wrote:
09 Jun 2019, 11:31
Easy reproducible builds that work out of the box for new developers are doable, just a lot of extra work, and definitely worth it if the project relies on attracting new committers. FileBot has never had any 3rd party contributors, even when it was completely free and open and building was just a matter of doing git clone && ant run (builds were easy when there was just a single filebot.jar)
Easy and reproducible builds should be something useful for anyone, and again, useful also because there are services who provide building infrastructure for free. Until recently I could do ant resolve + ant portable, I don't really see a downside. I accept your conclusion, I'm just having a difficult time figuring out the reasoning behind it, especially since several commits lately are related to it.
Regarding contributors I think the landscape changed a bit since 2007. A major driver for Sonarr and Radarr usage for example seem to be https://linuxserver.io container images which make them fairly easily accessible (and their heavy focus on WebUI).
It seems a catch-22 scenario.
rednoah wrote:
09 Jun 2019, 11:31
If anything, I'd wish for someone to take over and provide packages for end-users on all platforms, and that would most certainly be a serious commitment of time and effort, with hunting down a few jars to make the code compile being fairly negligible.

... You probably wouldn't notice for 1-2 years, but then things would break in strange unexpected ways. Keeping things going past that point would require someone get seriously intimate with the code and related services.
This is actually a good argument for having everything available, from what I can see they don't just build filebot.jar.
Regarding related services, excluding the databases (TVDB, TMDB, etc.), leaves out https://api.filebot.net/v5/data (replaceable by -Durl.<file> options even at runtime) and scripts, a compressed jar file. Is there more?
rednoah wrote:
09 Jun 2019, 11:31
I suspect the problem isn't building FileBot (i.e. the platform-independent Java byte code) for Alpine, but building all the native dependencies for Alpine. (which is hard, which is why I used to just dump Debian binaries into GitHub repository, instead of building from source)
Actually the issue is only with platforms other than x86_64, alpine has OpenJDK 12 and OpenJFX 12 only on that arch, while for the others it only has OpenJDK 11 (as well as MediaInfo, chromaprint, ffmpeg, etc.), but without OpenJFX. At least this is what I gathered.

P.S. I was thinking more about the risk of you getting bored with it, nothing worse.
I only work in black and sometimes very, very dark grey. (Batman)

User avatar
rednoah
The Source
Posts: 15958
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Recent build changes

Post by rednoah » 09 Jun 2019, 16:30

devster wrote:
09 Jun 2019, 12:57
This is actually a good argument for having everything available, from what I can see they don't just build filebot.jar.
Regarding related services, excluding the databases (TVDB, TMDB, etc.), leaves out https://api.filebot.net/v5/data (replaceable by -Durl.<file> options even at runtime) and scripts, a compressed jar file. Is there more?
I have a Synology NAS who's sole purpose (other than testing Synology NAS) to keep those updated. Incremental updates only take a few minutes, but building from scratch would take a few weeks. That's been running smoothly for years though, so I myself forget how things work. If debug logging is enabled, then you should have a pretty good idea what's going in and out. I think a notable recent addition is the thumbnail index. This one might also take a week or two to build from scratch. This stuff is all pretty hacky, but it works, so I don't touch it.

:!: I forgot about the script jar. That one is code signed so you can't build a working one yourself, without also replacing the public key that's baked into the filebot jar. You can always clone the scripts from GitHub though. There's no build or dependencies so switching to local scripts would be easy.

devster wrote:
09 Jun 2019, 12:57
Actually the issue is only with platforms other than x86_64, alpine has OpenJDK 12 and OpenJFX 12 only on that arch, while for the others it only has OpenJDK 11 (as well as MediaInfo, chromaprint, ffmpeg, etc.), but without OpenJFX. At least this is what I gathered.
The problem of having one big fat repository is that you can't build the CLI without also building the GUI, which means you can't build with a headless JRE because AWT / Swing / JavaFX / etc will be missing. The good thing is that since it's Java, and all compiled for Java 11, so you can take the jar build from any platform, and use it on any other platform. Currently, all builds are targeted at Java 11 for compatibility reasons since that's the LTS, except for the Java 8 backwards compatibility builds of course, i.e. the portable package should work out of the box, as far as FileBot and Java code is concerned, with the caveat of optional native dependencies (i.e. libmediainfo.so and friends) possibly not working.

devster wrote:
09 Jun 2019, 12:57
I was thinking more about the risk of you getting bored with it, nothing worse.
If that were to happen, I could just dump everything on GitHub and let the community figure it out. But as long as I'm around to maintain things and pay bills, things will continue to run smoothly indefinitely, no worries there, especially with the new license system, which is aimed at steady long-term revenue.
:idea: Please read the FAQ and How to Request Help.

devster
Posts: 275
Joined: 06 Jun 2017, 22:56

Re: Recent build changes

Post by devster » 09 Jun 2019, 17:17

rednoah wrote:
09 Jun 2019, 16:30
I have a Synology NAS who's sole purpose (other than testing Synology NAS) to keep those updated. Incremental updates only take a few minutes, but building from scratch would take a few weeks. That's been running smoothly for years though, so I myself forget how things work. [...] This stuff is all pretty hacky, but it works, so I don't touch it.
Now I'm even more convinced a proper pipeline would help. Also a build on a provider likely wouldn't take that long.
I'm still lost on the motivation, why not have a solid build pipeline, especially for the jar, which as you say is platform-independent?
I mean, it's not like it's not being executed every couple of days to create new beta versions.
rednoah wrote:
09 Jun 2019, 16:30
The problem of having one big fat repository is that you can't build the CLI without also building the GUI, which means you can't build with a headless JRE because AWT / Swing / JavaFX / etc will be missing.
I think this was mentioned at some point in the past, but you also said that decoupling it would take massive amount of work. In that case I'd probably vote for a persistent process with an API / WebUI.
I only work in black and sometimes very, very dark grey. (Batman)

User avatar
rednoah
The Source
Posts: 15958
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Recent build changes

Post by rednoah » 09 Jun 2019, 17:29

Both of these things basically boil down to lots of not-so-fun work with no tangible benefits for end users. It's like destroying a puzzle, and then putting it back together, but better this time. If things break and require a serious make over, I'll think about doing it in a more reproducible way. I'm just pretty sure you'd be the only person in the world interested in that. :lol:


EDIT:

Notably, you are the only person who ever contributed useful code, and that was probably after your CI already didn't work anymore. :lol:
:idea: Please read the FAQ and How to Request Help.

devster
Posts: 275
Joined: 06 Jun 2017, 22:56

Re: Recent build changes

Post by devster » 14 Jun 2019, 18:58

Sorry, been out of commission for a few days.
rednoah wrote:
09 Jun 2019, 17:29
Notably, you are the only person who ever contributed useful code, and that was probably after your CI already didn't work anymore. :lol:
That's probably the height of my abilities. Also something that can be decoupled from FileBot, I don't see people contributing to the main code.

Anyway, I'll adapt to the new state of things and hope it will never become a new Deluge situation.
Thank you.
I only work in black and sometimes very, very dark grey. (Batman)

User avatar
rednoah
The Source
Posts: 15958
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Recent build changes

Post by rednoah » 14 Jun 2019, 19:03

What's happening with Deluge?
:idea: Please read the FAQ and How to Request Help.

devster
Posts: 275
Joined: 06 Jun 2017, 22:56

Re: Recent build changes

Post by devster » 15 Jun 2019, 09:37

Single developer for the whole project, tried Patreon, had some issues and development of new v2 stalled for years.
Now he just released new versions, but many changed clients which is unfortunate, the daemon + thin client I think is a very good model.
I only work in black and sometimes very, very dark grey. (Batman)

User avatar
rednoah
The Source
Posts: 15958
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Recent build changes

Post by rednoah » 15 Jun 2019, 11:52

That's a sad story. I tried Patreon as well. It just doesn't really for software products. At least it's still around. Sickrage died from one day to the next when the developer moved on.

On the bright side, FileBot chose a different path forward. Since FileBot doesn't rely on donations anymore, and introduced a sustainable sales model instead, you don't have to worry about FileBot falling into disarray anytime soon. ;)
:idea: Please read the FAQ and How to Request Help.

devster
Posts: 275
Joined: 06 Jun 2017, 22:56

Re: Recent build changes

Post by devster » 15 Jun 2019, 19:18

SickRage story is a bit more complex.

echel0n forked SickBeard and created SickRage, to which a lot of people were contributing. He then disappeared for a while and the other devs kept going.

Then he came back and tried a hostile takeover which triggered the revocation of his commit access. He forked again with the same name but adding bitcoin miners to the website and the web interface.
He also sent a DMCA request to GitHub citing copyright infringement (but it's GPL, so that got bounced).

Devs split off in 2 directions: Medusa and the renamed SickChill to avoid further meddling by echel0n.

Because of all the ruckus many switched to Sonarr, but alternatives didn't die as Sonarr doesn't yet do symlinks, which for torrents are very useful.
I only work in black and sometimes very, very dark grey. (Batman)

User avatar
rednoah
The Source
Posts: 15958
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Recent build changes

Post by rednoah » 15 Jun 2019, 23:38

Wow. I had no idea. I completely missed all that drama! I just noticed a few competitors have come and gone over the years, though I usually don't notice until years after the fact, cause I personally don't really use any of them. I occasionally check things out when people need help with FileBot integration.

The tree of evolution there is interesting though!
:idea: Please read the FAQ and How to Request Help.

Post Reply