Web Sources

Running FileBot from the console, Groovy scripting, shell scripts, etc
Post Reply
devster
Posts: 417
Joined: 06 Jun 2017, 22:56

Web Sources

Post by devster »

More and more streaming services exist. More and more releases come from them.
The current {source} binding takes care of identifying the broad "WEBDL" or "WEBRip" and similar, but does not include where exactly from.
I'll share my solution to this particularly pointless problem.

Code: Select all

      { def file = new File('/scripts/websources.txt')
        def websources = file.exists() ? readLines(file).join("|") : null // to avoid non-existing paths failing the whole block
        def isWeb = (source ==~ /WEB.*/)
        // or def isWeb = source.matches(/WEB.*/) don't know which one is preferrable
        def lfr = { if (isWeb) fn.match(/($websources)\.(?i)WEB/) } // logo free release
        return allOf{fn.match(/(?i)(UHD).$source/).upper()}{lfr}{source}.join(".") } // it also specifies that source is downscaled from a 4K stream 
the magic happens in the websources.txt file which contains a fairly comprehensive list of shortcode for websources.

Code: Select all

9NOW
AE
AUBC
AMBC
AS
AJAZ
AMZN
AMC
ATK
ANPL
ANLB
AOL
ARD
iP
BRAV
CNLP
CN
CBC
CBS
4OD
CHGD
CMAX
CNBC
CC
CCGC
COOK
CMT
CRKL
CR
CSPN
CTV
CUR
CWS
DSKI
DHF
DEST
DDY
DTV
DISC
DSNY
DIY
DOCC
DPLY
ETV
ETTV
EPIX
ESPN
ESQ
FAM
FJR
FOOD
FOX
FREE
FYI
GLBL
GLOB
GO90
GC
HLMK
HBO
HGTV
HIDI
HIST
HULU
TOU
IFC
ID
iT
ITV
KNOW
LIFE
LN
MNBC
MTOD
MTV
NATG
NBA
NBC
NF
NFLN
NFL
NICK
NRK
PLUZ
PBS
PBSK
PSN
RSTR
RTE
SBS
SESO
SHMI
SPIK
SNET
SPRT
STAN
STZ
SVT
SWER
SYFY
TBS
TEN
TFOU
TLC
TRVL
TUBI
TV3
TV4
TVL
VH1
VICE
VMEO
UFC
UKTV
UNIV
VLCT
VIAP
VRV
WNET
WME
WWEN
XBOX
YHOO
RED
ZDF
I only work in black and sometimes very, very dark grey. (Batman)
orcoboi
Posts: 7
Joined: 18 Oct 2018, 20:11

Re: Web Sources

Post by orcoboi »

Aren't those release groups?
I don't know all of those, but AMZN and ETTV are release groups.
devster
Posts: 417
Joined: 06 Jun 2017, 22:56

Re: Web Sources

Post by devster »

Never heard about them.
In any case, they're also commonly used to identify the source of WEB-DL and WEBRip releases:
Roswell.New.Mexico.S01.1080p.AMZN.WEBRip.DDP5.1.x264-NTb
NTb = release group, AMZN.WEBRip = websource

Family.Food.Fight.AU.S01.720p.9NOW.WEBRip.AAC2.0.x264-RTN
RTN = release group, 9NOW.WEBRip = websource

Fangbone.S01.1080p.NF.WEBRip.DD5.1.x264-RCVR
RCVR = release group, NF.WEBRip = websource

Club.de.Cuervos.Presents.The.Ballad.of.Hugo.Sanchez.S01.SPANISH.1080p.NF.WEB-DL.DD5.1.x264-NTb
NTb = release group, NF.WEB-DL = websource

just the first few on google search.
If the above don't work then my format has an issue.
I only work in black and sometimes very, very dark grey. (Batman)
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Web Sources

Post by rednoah »

I'm considering add the more popular patterns to the {source} list. At least patterns like AMZN.WEBRip make sense to me.

:?: Your full list is too large for me though, can you recommend a small section of post popular once where it makes sense that {source} includes that?
:idea: Please read the FAQ and How to Request Help.
devster
Posts: 417
Joined: 06 Jun 2017, 22:56

Re: Web Sources

Post by devster »

I'm not sure I'd include it in {source}, I would consider this an additional, optional detail.
Besides the possibility of this info not always being present (which could fit into the allOf{b1}{b2} pattern), I see this information more akin to the {group} binding, and potentially implementable in a similar manner (remote, updatable file from which to retrieve a pattern).

The reasoning is that there may be new providers (e.g. Disney this year) with massive amount of viewers and the list would need to be updated to match that.
In addition, I believe the list was also created to distinguish releases of shows (and increasingly movies) which were released on a popular provider first and were subsequently re-released by a proprietary streaming service.

Anyway, I guess the most common ones would correspond to the most used streaming services (Amazon, Netflix, Hulu, Google PLAY, Crunchy Roll, iTunes, Vimeo, YouTube RED [yes, season 2 of Cobra Kai was released as Cobra.Kai.S02.1080p.RED.WEB-DL...]); however many acronyms also correspond to tv networks (HBO, STRZ [Starz], SYFY, CC [Comedy Central], CSPN [Cspan, who the hell releases these??], DSNY [Disney]) and it may be worth adding some of the biggest ones (I'm thinking about CW for the Arrow/Flash/Supergirl/etc. or HBO for example).
I only work in black and sometimes very, very dark grey. (Batman)
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Web Sources

Post by rednoah »

Alright, lets not make any premature decisions then, see how things develop, collect some sample data, and then see what makes the most sense.
:idea: Please read the FAQ and How to Request Help.
AbedlaPaille
Posts: 107
Joined: 12 Apr 2020, 04:02

Re: Web Sources

Post by AbedlaPaille »

g6045476
Posts: 21
Joined: 23 Oct 2018, 11:33

Re: Web Sources

Post by g6045476 »

devster wrote: 15 Apr 2019, 23:41 More and more streaming services exist. More and more releases come from them.
The current {source} binding takes care of identifying the broad "WEBDL" or "WEBRip" and similar, but does not include where exactly from.
I'll share my solution to this particularly pointless problem.

Code: Select all

      { def file = new File('/scripts/websources.txt')
        def websources = file.exists() ? readLines(file).join("|") : null // to avoid non-existing paths failing the whole block
        def isWeb = (source ==~ /WEB.*/)
        // or def isWeb = source.matches(/WEB.*/) don't know which one is preferrable
        def lfr = { if (isWeb) fn.match(/($websources)\.(?i)WEB/) } // logo free release
        return allOf{fn.match(/(?i)(UHD).$source/).upper()}{lfr}{source}.join(".") } // it also specifies that source is downscaled from a 4K stream 
the magic happens in the websources.txt file which contains a fairly comprehensive list of shortcode for websources.

Code: Select all

9NOW
AE
AUBC
AMBC
AS
AJAZ
AMZN
AMC
ATK
ANPL
ANLB
AOL
ARD
iP
BRAV
CNLP
CN
CBC
CBS
4OD
CHGD
CMAX
CNBC
CC
CCGC
COOK
CMT
CRKL
CR
CSPN
CTV
CUR
CWS
DSKI
DHF
DEST
DDY
DTV
DISC
DSNY
DIY
DOCC
DPLY
ETV
ETTV
EPIX
ESPN
ESQ
FAM
FJR
FOOD
FOX
FREE
FYI
GLBL
GLOB
GO90
GC
HLMK
HBO
HGTV
HIDI
HIST
HULU
TOU
IFC
ID
iT
ITV
KNOW
LIFE
LN
MNBC
MTOD
MTV
NATG
NBA
NBC
NF
NFLN
NFL
NICK
NRK
PLUZ
PBS
PBSK
PSN
RSTR
RTE
SBS
SESO
SHMI
SPIK
SNET
SPRT
STAN
STZ
SVT
SWER
SYFY
TBS
TEN
TFOU
TLC
TRVL
TUBI
TV3
TV4
TVL
VH1
VICE
VMEO
UFC
UKTV
UNIV
VLCT
VIAP
VRV
WNET
WME
WWEN
XBOX
YHOO
RED
ZDF
Hello,
I am testing your function to add the video source and it fails me. Has anything changed?

http://prntscr.com/srtyse

The txt is well imported, if I do a return websources it gives me the complete list.

Greetings and thank you!
devster
Posts: 417
Joined: 06 Jun 2017, 22:56

Re: Web Sources

Post by devster »

Yes, new versions highlighted some issues in my code, you need to change the following line:

Code: Select all

        def lfr = { if (isWeb) fn.match(/($websources)\.(?i)WEB/) } // logo free release
with something like:

Code: Select all

def lfr
if (isWeb) {
        lfr = fn.match(/($websources)\.(?i)WEB/)
}
or similar.
I only work in black and sometimes very, very dark grey. (Batman)
g6045476
Posts: 21
Joined: 23 Oct 2018, 11:33

Re: Web Sources

Post by g6045476 »

devster wrote: 01 Jun 2020, 18:34 Yes, new versions highlighted some issues in my code, you need to change the following line:

Code: Select all

        def lfr = { if (isWeb) fn.match(/($websources)\.(?i)WEB/) } // logo free release
with something like:

Code: Select all

def lfr
if (isWeb) {
        lfr = fn.match(/($websources)\.(?i)WEB/)
}
or similar.
Hello, thanks for answering. I have this and it doesn't work

Code: Select all

{
	def file = new File('C:/websources.txt')
	def websources = file.exists() ? readLines(file).join("|") : null 
	def isWeb = (source ==~ /WEB.*/)
	def lfr
	if (isWeb) {
		lfr = fn.match(/($websources)\.(?i)WEB/)
	}
	return allOf{fn.match(/(?i)(UHD).$source/).upper()}{lfr}{source}.join(".")
}
Now the return doesn't work either. I have tried return "hi" and it returns nothing.
devster
Posts: 417
Joined: 06 Jun 2017, 22:56

Re: Web Sources

Post by devster »

Sorry, no clue as to why, you'll probably have to debug on your machine.
The following:

Code: Select all

{
	def isWeb = true
	def lfr
	if (isWeb) {
		lfr = 'hi'
	}
}
works, so it's likely something with the file or the regex or something else.
As a suggestion you can use

Code: Select all

lfr = any{fn.match(/($websources)\.(?i)WEB/)}{null}
as it unwinds gracefully.
I only work in black and sometimes very, very dark grey. (Batman)
kim
Power User
Posts: 1251
Joined: 15 May 2014, 16:17

Re: Web Sources

Post by kim »

Code: Select all

{
	any{ def file = new File("C:/websources.txt")
	def websources = file.exists() ? lines(file).join('|') : null 
	source ==~ /WEB.*/ ? lfr = fn.match(/($websources)\.(?i)WEB/) : ''

	return allOf{fn.match(/(?i)(UHD).$source/).upper()}{lfr}{source}.join('.')
	}{source}
}
output e.g.
NF.WEB-DL
g6045476
Posts: 21
Joined: 23 Oct 2018, 11:33

Re: Web Sources

Post by g6045476 »

Solved! I have removed the parentheses and it works correctly.

Code: Select all

def lfr = fn.match(/$websources\.(?i)WEB/)
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Web Sources

Post by rednoah »

I've updated the WEB-DL {source} pattern to make this work more easily out-of-the-box:
https://github.com/filebot/data/blob/ma ... ources.txt
:idea: Please read the FAQ and How to Request Help.
ZeroByDivide
Posts: 170
Joined: 16 Dec 2014, 01:39

Re: Web Sources

Post by ZeroByDivide »

Is there a way I can either revert back to an older "{source}" scheme or something, recent changes to the {source} has botched my naming scheme up and I see no way to fix it other than removing my source stuff and also writing up a bunch of "replaceall" stuff to fix various things or something ...

1. Way I used to have it when something was missing from the {source} I would have it name the {source} just "WEB-DL" with

Code: Select all

{"${self.source ?: 'WEB-DL'}"}
this was mostly when the download had the source as just "web", now it no longer seems to do this and just puts the source as "web" when I much prefer it to say "WEB-DL".. That'd require a "replaceall" for each source that wasn't there before that I'd prefer to have just set as "WEB-DL" ontop of trying to make sure that the name of the actual show or episode didn't have that bit from the "replaceall" in there as well..

2. Way I used to have it correlates with the above, any "webdl" that I had downloaded I actually want to have the "-" in between the "web" and the 'dl" part, this requires another "replaceall" making at least a minimum of two to bring back my naming scheme a little bit.

3. Way I used to have it when something had a websource like AMZN or any of the others there was a space between the AMZN bit and the source bit, now is no longer the case as they are now seemingly squished together, once again requiring a "replaceall".


This was my current websource / sources bit from my script that was working perfectly fine until this recent change in {source}

Code: Select all

{ def websources = new File('path/to/websource.txt')
                    def websource = websources.exists() ? lines(websources).join("|") : null // to avoid non-existing paths failing the whole block

                    def lfr = { fn.match(/\.($websource)\./) }
                    return allOf{fn.match(/(?i)(UHD).$source/).upper()}{hc}{lfr}{"${self.source ?: 'WEB-DL'}"}.join(" ") }
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Web Sources

Post by rednoah »

ZeroByDivide wrote: 10 Jun 2020, 06:06 when the download had the source as just "web", now it no longer seems to do this and just puts the source as "web" when I much prefer it to say "WEB-DL"
Have you tried the recently introduced {vs} source class binding yet?
viewtopic.php?t=11265

ZeroByDivide wrote: 10 Jun 2020, 06:06 Way I used to have it when something had a websource like AMZN or any of the others there was a space between the AMZN bit and the source bit, now is no longer the case as they are now seemingly squished together
The updated pattern requires a separator between AMZN and WEB.DL and so if things are seemingly squished together now, then it's most likely your code that's making it so. ;)
:idea: Please read the FAQ and How to Request Help.
ZeroByDivide
Posts: 170
Joined: 16 Dec 2014, 01:39

Re: Web Sources

Post by ZeroByDivide »

rednoah wrote: 10 Jun 2020, 06:41
ZeroByDivide wrote: 10 Jun 2020, 06:06 when the download had the source as just "web", now it no longer seems to do this and just puts the source as "web" when I much prefer it to say "WEB-DL"
Have you tried the recently introduced {vs} source class binding yet?
viewtopic.php?t=11265
I actually just now found that, as I was looking through naming schemes again over at http://www.filebot.net/naming.html and so far it seems to at least be working how I prefer. Not sure how I missed it though all this time since it seems it was introduced back in November of last year xD

rednoah wrote: 10 Jun 2020, 06:41
ZeroByDivide wrote: 10 Jun 2020, 06:06 Way I used to have it when something had a websource like AMZN or any of the others there was a space between the AMZN bit and the source bit, now is no longer the case as they are now seemingly squished together
The updated pattern requires a separator between AMZN and WEB.DL and so if things are seemingly squished together now, then it's most likely your code that's making it so. ;)
even doing just plain old {source} in the filebot GUI editor produces something like

Code: Select all

AMZNWEBDL
even though the file name would be something like

Code: Select all

Random.File.Name.720p.AMZN.WEB-DL.DD+5.1.H.264.mkv
so filebot would go about producing something like

Code: Select all

Random Name AMZNWEBDL.mkv
for example: https://i.imgur.com/jVUyOiX.png
ZeroByDivide
Posts: 170
Joined: 16 Dec 2014, 01:39

Re: Web Sources

Post by ZeroByDivide »

well I guess there are a few caveats to {vs} vs using {source}.. is there a way I could make it so that it only uses {vs} for any releases that just have web/web-dl for the source, while everything else uses {source}...

I'm tinkering around with if then else conditions a bit now, but if you can come up with something possibly that would work nicely then I would be very happy :D
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Web Sources

Post by rednoah »

{vs} is definitely more appropriate if you want to check for specific values, because {vs} is limited to specific values, while {source} could yield anything that matches the patterns.


{source} works correctly as far as I can tell:

Code: Select all

$ filebot -mediainfo *.mkv --format "{source} | {fn}"
AMZN.WEB-DL | Random.File.Name.720p.AMZN.WEB-DL.DD+5.1.H.264

What specific version / revision are you using?

Code: Select all

$ filebot -version
FileBot 4.9.1 (r7372) / OpenJDK Runtime Environment 14 / Windows 10 10.0 (amd64)
:idea: Please read the FAQ and How to Request Help.
ZeroByDivide
Posts: 170
Joined: 16 Dec 2014, 01:39

Re: Web Sources

Post by ZeroByDivide »

rednoah wrote: 10 Jun 2020, 07:54 {source} works correctly as far as I can tell:

Code: Select all

$ filebot -mediainfo *.mkv --format "{source} | {fn}"
AMZN.WEB-DL | Random.File.Name.720p.AMZN.WEB-DL.DD+5.1.H.264

What specific version / revision are you using?

Code: Select all

$ filebot -version
FileBot 4.9.1 (r7372) / OpenJDK Runtime Environment 14 / Windows 10 10.0 (amd64)
Well that would explain it, for some reason my filebot was still on version 4.9.0, I had to go into the microsoft store and manually update it since for some reason or another microsoft store doesn't want to auto update stuff -.- anyways that fixed that, tbh now that that issue I can just replace the "." with a space and from what I can tell things will hopefully be back to how they used to be for my naming scheme.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Web Sources

Post by rednoah »

Another problem solved by just updating. My favourite kind. :lol:
:idea: Please read the FAQ and How to Request Help.
FunnyFunk
Posts: 2
Joined: 20 Jan 2021, 14:14

Re: Web Sources

Post by FunnyFunk »

Hi,
Quick question: is it possible to only get the websource abbreviation, so I could name my files something like xxx{“.$websource”}{“.$vf”}{“.vs”}.xxx, only thing I found is to use source.vf.vs, but that gives: xxx.amzn.web-dL.1080p.web-dl.xxx and it would be a little exhausting to use a lot of replacealls. Finally this would also give me files like: xxx.Bluray.1080p.bluray.xxx
Thanks in advance,
FunnyFunk
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Web Sources

Post by rednoah »

You mean {vs} source class as opposed to {source} source match?
viewtopic.php?t=11265
:idea: Please read the FAQ and How to Request Help.
FunnyFunk
Posts: 2
Joined: 20 Jan 2021, 14:14

Re: Web Sources

Post by FunnyFunk »

First of all, thanks for your quick reply.
My problem actually is at the beginning if I put source I get a file like this: xxx.amzn.web-dl.1080p.web-dl.xxx and I don’t want that first web-dl. Like I said previously the problem gets worse if it isn’t a web release, than I have something like xxx.bluray.1080p.bluray.xxx or xxx.hdtv.1080p.hdtv.xxx
FunnyFunk
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Web Sources

Post by rednoah »

Use only {vs} or {source} at the position where you want to insert one or the other, but don't do it twice in two different positions. You may need to use fn.match() with a custom pattern if you need something more specific.
:idea: Please read the FAQ and How to Request Help.
Post Reply