Extend FileBot or FileBot-Node into full API

All your suggestions, requests and ideas for future development
devster
Posts: 417
Joined: 06 Jun 2017, 22:56

Extend FileBot or FileBot-Node into full API

Post by devster »

This feature request is pretty niche, and probably involving a lot of work, so no issues should it be denied.

I'm a container user, mostly because the simplicity of running a single command and having a production-ready application deployed is very appealing.
From my limited experience working with HTTP requests with containers is by far the easiest approach to integrate multiple applications.
Based on this and the existence of the FileBot-Node application I was wondering whether it'd be possible to have a full API for FileBot for containerized use.
Based on how flexible groovy scripts are it may even be possible to write a script to do this but I'm unable to judge the feasibility.
In the meantime I'm trying to write a simple app to expose the FileBot command via an API, but not being a good coder it's probably going to take me a long time.
I only work in black and sometimes very, very dark grey. (Batman)
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Extend FileBot or FileBot-Node into full API

Post by rednoah »

I feel like having a node.js script that just accepts HTTP requests and then executes arbitrary commands in return would be quite simple, i.e. less than 10 lines, and so generic that there's plenty of copy & paste solutions already.

Do docker containers still not have a good standardized way to talk to each other? Maybe via SSH?
:idea: Please read the FAQ and How to Request Help.
devster
Posts: 417
Joined: 06 Jun 2017, 22:56

Re: Extend FileBot or FileBot-Node into full API

Post by devster »

I wouldn't really be able to, but fair enough, it is possible to stitch something together with https://github.com/adnanh/webhook or similar.

There is a good standardized way to talk to each other, it's the network.
As far as I know, a container is otherwise as isolated as possible, so using docker inside containers to send commands to another is feasible but not really encouraged (docker/podman).
SSH is doable (it works on the network) but it requires a daemon, which goes slightly against the principle of docker images (1 image, 1 logical program - this includes nginx workers in an nginx container but not an SSH daemon in the same container). Another option used by the Jenkins image is jnlp.
In short it's feasible, but network seem the easiest.
This is further facilitated in the case of docker or kubernetes, by the provided DNS resolution for services, for example from a container I can call other containers in the same network via their service name http://redis:<port>.
I only work in black and sometimes very, very dark grey. (Batman)
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Extend FileBot or FileBot-Node into full API

Post by rednoah »

Fair enough. I'll look into adnanh/webhook and add an official filebot docker container that extends rednoah/filebot and adds a running webhook service.


:?: Would you prefer something like adnanh/webhook where you have to write a configuration file, mapping HTTP endpoints to script calls?

:?: Or something more generic, where you simply call /execute?command=filebot -version, or any other command?


I like using other tools, because then there'll be documentation and examples right there. But adnanh/webhook seems like it adds unnecessary work in writing configuration files (precisely so that you can't just run any command and take over the machine remotely) which may or may not be what we want here.
:idea: Please read the FAQ and How to Request Help.
devster
Posts: 417
Joined: 06 Jun 2017, 22:56

Re: Extend FileBot or FileBot-Node into full API

Post by devster »

I think the easiest route would be slightly tweaking filebot-node, it already does its job.
Two big bits missing, I believe, are the option to run without a web client, server only, and slight tweaks to the API to allow, for example, complex formats, which may get a bit messy with queries.
Personally I'd change the execute to a POST route

Code: Select all

/**
 * apologies, this is ExpressJs which is the only thing I know.
 */
const app = require('express')()
const bodyParser = require('body-parser')

app.post('/execute/:id', bodyParser.json(), function (req, res, next) {
  opts = req.query
  defs = req.body.defs
  if (req.params.id === 'amc') {
  [...] // basically getCommandArguments() here or similar
  res.json(<stuff>)
})

expected request would be 
curl -X POST http://<servicename>:<port>/execute/amc?channel=fn&action=test -d '{"input": "fileinputpath", "defs": {"seriesFormat": "mylongseriesformat"}}'
I think this'd be the least amount of work and what I'm experimenting with.
Unfortunately I didn't realize it was ExtJS in the client and started with ExpressJS which seemed the simplest option.
I also skipped Syno and Qnap stuff and used the process UID and GID as these can be customized with --user option with containers, without requiring environment variables.
I can post my messy stuff in the weekend.
I only work in black and sometimes very, very dark grey. (Batman)
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Extend FileBot or FileBot-Node into full API

Post by rednoah »

Well, the client is served by the server, if you open that page, so not having the client and having the client but not accessing it, is pretty much the same.

If we go that way, and if we only care about the amc script, then I'm not quite sure what you might need that isn't already there.

:?: So the main issue is to use POST instead of GET parameters because because curl doesn't have an easy way to pass in complex GET parameters?

e.g. You could easily have a groovy or python script make the HTTP request, so you don't have to query encode all the GET parameters yourself. Having your amc update request in a separate python script might make things much more readable as well compared to curl commands.
:idea: Please read the FAQ and How to Request Help.
devster
Posts: 417
Joined: 06 Jun 2017, 22:56

Re: Extend FileBot or FileBot-Node into full API

Post by devster »

Curl was just an example, I'm very likely going to use python for this, as suggested.

I had more than a few issues trying to send requests to the filebot-node docker image with curl, the main issue was it kept returning 401.

Code: Select all

$ docker run --rm -it -e FILEBOT_NODE_AUTH=NONE -e FILEBOT_NODE_HTTP_PORT=5455 -p 5455:5455 rednoah/filebot:node

$ curl 'http://<domain>:5455/filebot/execute?fn=sysinfo' -i
HTTP/1.1 401 Unauthorized
Access-Control-Allow-Origin: *
Date: Fri, 22 Mar 2019 20:02:40 GMT
Connection: keep-alive
Content-Length: 0
The issue isn't complex arguments however, it's length, query strings usually are limited to 1024 or 2048 characters, my seriesFormat alone is over 4000

Code: Select all

$ wc -m seriesFormat.groovy
    4488 seriesFormat.groovy
and contains stuff like

Code: Select all

it.replaceAll(/[`´‘’ʻ""“”]/, "'")
and possibly easier to escape in JSON than urls (?).
I only work in black and sometimes very, very dark grey. (Batman)
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Extend FileBot or FileBot-Node into full API

Post by rednoah »

1.
Is it though? It's supposed to be 80*1024 chars:
https://stackoverflow.com/a/32763588/1514467


2.
You don't escape anything. You just copy & paste your format into a text file verbatim and then read that in your script to get the format as String value. That's how I'd do it.

Code: Select all

def format = new File('format.groovy').getText('UTF-8')

3.
What is the difference between the requests sent by you and the ones sent by the FileBot Node WebUI? You can check in your browser what requests FileBot Node is sending.

In this case, I guess maybe the endpoint is just wrong? You'll always get "unauthorized" if you access WebUI resources that don't exist, or aren't known resources:
https://github.com/filebot/filebot-node ... pp.js#L290

Best to capture the requests FileBot Node is sending in your browser so you can see where the endpoints are. I think it's /execute and not /filebot/execute:
https://github.com/filebot/filebot-node ... pp.js#L336
:idea: Please read the FAQ and How to Request Help.
devster
Posts: 417
Joined: 06 Jun 2017, 22:56

Re: Extend FileBot or FileBot-Node into full API

Post by devster »

1. I stand corrected, I didn't realize they increased it this much. I do believe a POST and JSON would be cleaner.
The use of a "/script/execute/:id" route would also allow for matching of default scripts and custom ones, for example, assuming /data as the mounted folder, one could call /script/execute/fn:amc or /script/execute/local:myscript which would try to run a myscript.groovy from /data.
I'll see what I can throw together, maybe formalize more what I'm thinking.

3. It seems the request from the client contain more information, they use _dc= and other parameters.
I'll test some more, see if I can figure out what's happening.
I only work in black and sometimes very, very dark grey. (Batman)
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Extend FileBot or FileBot-Node into full API

Post by rednoah »

1.
I agree. But switching between GET and POST is just a matter changing "POST" to "GET" in higher-level languages. Complex GET request are tricky with curl though it seems.


2.
"/script/execute/:id" is a pretty good idea actually, since FileBot Node already creates @files for :id commands. There's just no handler for calling them via FileBot Node, since currently that's just used by Synology Task Manager and QNAP cron. This could actually be handy to call completely arbitrary filebot commands, by mapping a folder with :id.args files into the container. I don't work much on FileBot Node, but if there's a list of TODOs, then it makes sense to check out the repository again and do some development and testing.


3.
AFAIK, _dc is something that is automatically added by ExtJS to make sure the browser doesn't cache the GET request. FileBot Node doesn't care about it.
:idea: Please read the FAQ and How to Request Help.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Extend FileBot or FileBot-Node into full API

Post by rednoah »

I'm adding a new /task?id=0 endpoint to call prepared tasks. But I don't have test builds for that right now.
:idea: Please read the FAQ and How to Request Help.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Extend FileBot or FileBot-Node into full API

Post by rednoah »

The new release includes that API method now:
https://github.com/filebot/filebot-node/releases/tag/0.2.8.3

Docker build is incoming.
:idea: Please read the FAQ and How to Request Help.
devster
Posts: 417
Joined: 06 Jun 2017, 22:56

Re: Extend FileBot or FileBot-Node into full API

Post by devster »

Thanks, how does that work?
Is it simply a file containing filebot arguments?
I only work in black and sometimes very, very dark grey. (Batman)
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Extend FileBot or FileBot-Node into full API

Post by rednoah »

You have to use FileBot Node first time create a scheduled task.

After that you can have FileBot Node run a previous schedule task with the /task request.
:idea: Please read the FAQ and How to Request Help.
frankw
Posts: 21
Joined: 15 Feb 2016, 16:29

Re: Extend FileBot or FileBot-Node into full API

Post by frankw »

Hi guys,

I would also like to express interest in a feature like this, focused around calling the AMC script via an API.

I have recently dockerized everything on my home server except for Transmission and Filebot, as up until today I have not been able to figure out how to get Transmission to call Filebot AMC when the torrent is complete. I'd like to not have to build a custom docker image and inject Filebot into an existing Transmission container like some have done, and if possible pass parameters from Transmission to Filebot-node via an API of some kind. I'd also like this to be triggered by Transmission, not to wait for a scheduled task of some kind to kick in.

My current call to amc is as follows:

Code: Select all

filebot -script fn:amc --log-file "/home/frank/.filebot/amc.log" --def clean=y --output "/tmp" --action copy --conflict override -non-strict --def artwork=n "ut_dir=$TR_TORRENT_DIR/$TR_TORRENT_NAME" "ut_kind=multi" "ut_title=$TR_TORRENT_NAME" --def "seriesFormat=/storage/TV/{n}/{'S'+s.pad(2)}/{n.replaceAll(/[!?.]+$/).space('.')}.{'s'+s.pad(2)}e{e.pad(2)}.{vf}.{source}.{vc}.{ac}" "movieFormat=/storage/Movies/{n.upperInitial().replaceAll(/[!?.]+$/).space('.')}.{y}.{vf}.{source}.{vc}.{ac}" --def pushbullet=xxx --def reportError=y
Anyway, I've been having a play around and hacked the following (ugly) Python 2 (version of Python currently in the linuxserver/transmission container) script together that I can append on the end of a GET to the filebot-node.

Code: Select all

import urllib

# For testing - apparently defined by Transmission
TR_TORRENT_DIR="/downloads"
TR_TORRENT_NAME="Chicago.PD.S06E22.1080p.WEB.H264-AMCON"

OUTPUT = urllib.quote_plus("/tmp")
INPUT = urllib.quote_plus(TR_TORRENT_DIR + "/" + TR_TORRENT_NAME)
LANG = "en"
SERIESFORMAT = urllib.quote_plus("/storage/TV/{n}/{'S'+s.pad(2)}/{n.replaceAll(/[!?.]+$/).space('.')}.{'s'+s.pad(2)}e{e.pad(2)}.{vf}.{source}.{vc}.{ac}")
MOVIEFORMAT = urllib.quote_plus("/storage/Movies/{n.upperInitial().replaceAll(/[!?.]+$/).space('.')}.{y}.{vf}.{source}.{vc}.{ac}")
PUSHBULLET = "xxx"

CMD = "action=COPY&fn=amc&input=" + INPUT + "&output=" + OUTPUT + "&lang=" + LANG + "&conflict=override&movieFormat=" + MOVIEFORMAT + "&seriesFormat=" + SERIESFORMAT + "&clean=on&pushbullet=" + PUSHBULLET + "&log=all"

print (CMD)
The creates a call like:

Code: Select all

https://filebot.myserver.net/execute?action=copy&fn=amc&input=%2Fdownloads%2FChicago.PD.S06E22.1080p.WEB.H264-AMCON&output=%2Ftmp&lang=en&conflict=override&movieFormat=%2Fstorage%2FMovies%2F%7Bn.upperInitial%28%29.replaceAll%28%2F%5B%21%3F.%5D%2B%24%2F%29.space%28%27.%27%29%7D.%7By%7D.%7Bvf%7D.%7Bsource%7D.%7Bvc%7D.%7Bac%7D&seriesFormat=%2Fstorage%2FTV%2F%7Bn%7D%2F%7B%27S%27%2Bs.pad%282%29%7D%2F%7Bn.replaceAll%28%2F%5B%21%3F.%5D%2B%24%2F%29.space%28%27.%27%29%7D.%7B%27s%27%2Bs.pad%282%29%7De%7Be.pad%282%29%7D.%7Bvf%7D.%7Bsource%7D.%7Bvc%7D.%7Bac%7D&clean=on&pushbullet=xxxx&log=all
This does seem to work, the only challenge I have now is the permissions of the resulting file (EP22) are very restrictive:

Code: Select all

-rw-rw-rw- 1 debian-transmission debian-transmission 2910137810 May 16 09:04 Chicago.P.D.s06e21.1080p..AVC.EAC3.mkv
-rw-r--r-- 1 root                root                3097842311 May 23 05:04 Chicago.P.D.s06e22.1080p..AVC.EAC3.mkv
Docker compose file:

Code: Select all

  filebot-node:
    image: rednoah/filebot:node
    container_name: filebot-node
    restart: always
    volumes:
      - ${USERDIR}/docker/filebot-node:/data
      - ${USERDIR}/TransmissionDL:/downloads
      - /storage:/storage
    environment:
      - PUID=${PUID}
      - PGID=${PGID}
      - TZ=${TZ}
    networks:
      - traefik_proxy
    labels:
      - "traefik.enable=true"
      - "traefik.backend=filebot-node"
      - "traefik.frontend.rule=Host:filebot.${DOMAINNAME}"  
      - "traefik.port=5452"
      - "traefik.docker.network=traefik_proxy"
It would be great if passing in the UID and GID would work, so other containers not running as root could handle those files (e.g. deleting media in Emby) :)

Next step is I will try and connect this up to Transmission and see if I can get it to work
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Extend FileBot or FileBot-Node into full API

Post by rednoah »

1.
You can execute previously created Scheduled Tasks via the /task handler with the current release of FileBot Node.

e.g.

Code: Select all

/task/0

So you can prototype and test your amc call in the FileBot Node UI first, and then click the Schedule button. You can copy the task id from the popup and then do invoke calls via curl.


2.
Permissions are tricky if you're using docker. Best to ask docker experts for help. IDK.

:idea: You can use the --def exec to have filebot call chmod / chown but since that's gonna be running within the docker, it might not do much.
:idea: Please read the FAQ and How to Request Help.
devster
Posts: 417
Joined: 06 Jun 2017, 22:56

Re: Extend FileBot or FileBot-Node into full API

Post by devster »

The user in a docker container is root by default (not the host root, the container root), it seems this is the case based on the owner of the EP22 you processed. The environment variables are common for linuxserver images and they control under which user s6-overlay runs the main process. I don't believe rednoah's image uses s6-overlay.
You might try with

Code: Select all

  user:
  - ${PUID}:${PGID}
to change the user it's running as, and add an --exec part to the filebot command in order to extend the perms of the processed file.
I'd also suggest using systemd unit files instead of restart: always to better manage dependencies, for example mounts.
I only work in black and sometimes very, very dark grey. (Batman)
frankw
Posts: 21
Joined: 15 Feb 2016, 16:29

Re: Extend FileBot or FileBot-Node into full API

Post by frankw »

Thanks guys for the replies.

@rednoah, I am not sure how I would use a previously created scheduled task and pass in variables from a torrent client at the same time. If this was possible, it would certainly simplify what is passed over a GET req.

@devster, thanks for the docker compose suggestion, I tried that and the container would not start properly, just saw the following error repeatedly in the logs:

Code: Select all

  return binding.mkdir(pathModule._makeLong(path),
                 ^
Error: EACCES: permission denied, mkdir '/usr/local/filebot-node/data'
    at Object.fs.mkdirSync (fs.js:885:18)
    at Object.<anonymous> (/usr/local/filebot-node/server/app.js:50:8)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)
    at Function.Module.runMain (module.js:693:10)
    at startup (bootstrap_node.js:191:16)
    at bootstrap_node.js:612:3
Not sure how to get filebot to write any files without root ownership on the host without an s6-overlay? I'm new to docker 1 week, so not sure how I would approach fixing this problem.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Extend FileBot or FileBot-Node into full API

Post by rednoah »

You indeed can't pass variables to a Scheduled Task. You'd set your Scheduled Task to process your Completed Downloads folder, where your torrent client moves files after completion. Meanwhile, the amc exclude list file will ensure that only new files are processed.

:!: Variables that mean something to the torrent client container (e.g. file paths) are meaningless in other containers (e.g. filebot container) anyway. So relying on torrent client variables for input file paths is probably not a good idea if things aren't running in the same container.
:idea: Please read the FAQ and How to Request Help.
devster
Posts: 417
Joined: 06 Jun 2017, 22:56

Re: Extend FileBot or FileBot-Node into full API

Post by devster »

If you have already started the container and it has persistent or mounted volumes the current permissions are root, it makes sense that changing user to a non-privileged one wouldn't work.
s6-overlay is convenience and most definitely not required.
I would suggest checking permissions on your docker host (the machine, not the container) and changing them as appropriate if necessary, in this case ${USERDIR}/docker/filebot-node, ${USERDIR}/TransmissionDL and /storage should be owned by whichever user you want (recursively).
I only work in black and sometimes very, very dark grey. (Batman)
frankw
Posts: 21
Joined: 15 Feb 2016, 16:29

Re: Extend FileBot or FileBot-Node into full API

Post by frankw »

devster wrote: 25 May 2019, 10:52 If you have already started the container and it has persistent or mounted volumes the current permissions are root, it makes sense that changing user to a non-privileged one wouldn't work.
s6-overlay is convenience and most definitely not required.
I would suggest checking permissions on your docker host (the machine, not the container) and changing them as appropriate if necessary, in this case ${USERDIR}/docker/filebot-node, ${USERDIR}/TransmissionDL and /storage should be owned by whichever user you want (recursively).
I know, I removed all the files created by root, but it still does not work. ${USERDIR}/docker/filebot-node is owned by UID/GID and has perms the same as all the other container data dirs:

Code: Select all

drwxrwxr-x+  2 frank docker 4096 May 24 23:24 filebot-node/
Last edited by frankw on 25 May 2019, 15:43, edited 1 time in total.
frankw
Posts: 21
Joined: 15 Feb 2016, 16:29

Re: Extend FileBot or FileBot-Node into full API

Post by frankw »

rednoah wrote: 25 May 2019, 04:37 You indeed can't pass variables to a Scheduled Task. You'd set your Scheduled Task to process your Completed Downloads folder, where your torrent client moves files after completion. Meanwhile, the amc exclude list file will ensure that only new files are processed.
Ah ok, I'll give that a try, thanks.
devster
Posts: 417
Joined: 06 Jun 2017, 22:56

Re: Extend FileBot or FileBot-Node into full API

Post by devster »

Sorry, I'm a bit out of ideas.
Maybe you're trying to mount ${USERDIR}/docker/filebot-node into /data inside the container but it seems the container tries writing to /usr/local/filebot-node/data, not sure.
I only work in black and sometimes very, very dark grey. (Batman)
mrmeeper
Posts: 1
Joined: 31 Jul 2019, 22:18

Re: Extend FileBot or FileBot-Node into full API

Post by mrmeeper »

I got pretty excited when I saw this post as it almost did exactly what I was looking for!
I don't have enough flexibility when using the scheduled tasks and the best way to communicate with docker containers is via http(s).

Can you implement the plan as you first envisioned it rednoah?:
--> :?: Or something more generic, where you simply call /execute?command=filebot -version, or any other command?

This way I can call the amc script with programatic parameters. Let me know if this is possible? Thanks!
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Extend FileBot or FileBot-Node into full API

Post by rednoah »

1.
The most recent versions of FileBot Node should already include the /task interface which allows you to call prepared tasks via curl.

Caveat is that you'd need to create a prepared task first, so you can't call it with arbitrary arguments. That being said, if you expose the FileBot Node docker container configuration folder, then you can create prepared tasks easily simply by generating the necessary *.args configuration files, and then issue the curl call, which in turn will call filebot with your newly created *.args files.


2.
As for remotely calling completely arbitrary commands, a netcat | bash one-liner will probably do:
https://en.wikipedia.org/wiki/Netcat#Ma ... s_a_server
:idea: Please read the FAQ and How to Request Help.
Post Reply