Is there a way to add to bindings? (i.e. normalized {source} values)

Support for Windows users
Post Reply
DevXen
Power User
Posts: 164
Joined: 12 Oct 2014, 21:15

Is there a way to add to bindings? (i.e. normalized {source} values)

Post by DevXen »

Several of my shows just have 'web' in the filename. not web-dl or webrip. and i noticed that web isn't in the {model.source}. i did setup a replaceALL(/.web./, "WEB-DL") which works for some of it. but not the part of my script that checks the source of each file in the season. if a file just has 'web' in the filename. it breaks that part of the script.

So is there a way for me to map/add to the source binding? - to solve this and any other possible future issues that may arise?

Thanks,
-Dev
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Is there a way to add to bindings?

Post by rednoah »

Well, modifying how {source} works is not supported, so {model.fn} is the only solution I see, and then match it from there for each file. The code will get convoluted though, for these kinda things that don't work out of the box.

:idea: A few sample file paths would be appreciated.

:idea: I'd do a pre-rename step with plain file mode, where you do your WEB ➔ WEB-DL replacement, so it'll say WEB-DL in the filename when you process files for real as usual.

:!: Adding web is probably a bad idea, Charlottes Web, Web of Lies, etc. You're setting yourself up for mismatches galore.
:idea: Please read the FAQ and How to Request Help.
devster
Posts: 417
Joined: 06 Jun 2017, 22:56

Re: Is there a way to add to bindings?

Post by devster »

Files may have a specific order in which to put information:
Movie.Title.Year.Resolution.source.Audio.VideoFormat-Group.extension
A partial but fairly safe match could be {resolution}./WEB/.{audio} but both resolution and audio may have different formats (DD+5.1 vs EAC3-5.1).
I only work in black and sometimes very, very dark grey. (Batman)
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Is there a way to add to bindings?

Post by rednoah »

For this approach, I would need a large number of sample file paths for training and testing.


EDIT:

As easy solution, I think you can just assume WEB if {source} is undefined:

Code: Select all

any{source}{/WEB/}
Presumably, you're using {model}, so in this case we have to replace null values with our preferred default value:

Code: Select all

{model.source.collect{ it ?: 'WEB' }}
:idea: Please read the FAQ and How to Request Help.
DevXen
Power User
Posts: 164
Joined: 12 Oct 2014, 21:15

Re: Is there a way to add to bindings?

Post by DevXen »

That solution probably won't work for me directly. But I think I can work with it to figure something out. Make it so if it has 'web' in the filename. And has no {source} defined. That may be a valid solution for my use. I'll work on it when I get home and see what I can do. Thank you.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Is there a way to add to bindings?

Post by rednoah »

I've started a new list for Pattern ➔ Label rules:
https://github.com/filebot/data/blob/ma ... ources.txt
:idea: Please read the FAQ and How to Request Help.
devster
Posts: 417
Joined: 06 Jun 2017, 22:56

Re: Is there a way to add to bindings?

Post by devster »

I only work in black and sometimes very, very dark grey. (Batman)
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Is there a way to add to bindings?

Post by rednoah »

Perhaps. But right now it doesn't do anything. FileBot doesn't use that file yet. I'm inclined to add a new binding that yields the normalized value, vs one that gives the actual match from the file name.
:idea: Please read the FAQ and How to Request Help.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Is there a way to add to bindings?

Post by rednoah »

The latest FileBot r6884 now implements the media-sources.txt approach and normalizes {source} accordingly:
https://github.com/filebot/data/blob/ma ... ources.txt

:idea: Only the standard value from the list is returned, not the pattern that was matched, so WEBDL / WEB.DL / AmazonHD / NetflixHD / etc all now normalize to WEB-DL, same same for everything on the list. This is a major change. But it does make {source} consistent which is desirable for most I presume.

:idea: Return standard value WEB-DL for different kinds of WEB patterns (but only within the context of 720p.WEB or WEB.h264 to limit false positives)
:idea: Please read the FAQ and How to Request Help.
kim
Power User
Posts: 1251
Joined: 15 May 2014, 16:17

Re: Is there a way to add to bindings? (i.e. normalized {source} values)

Post by kim »

this is wrong...
It would be great to separate the quality profiles for WEB-DL and WEBRip. WEB-DL are significantly superior to WEBRip and should not be lumped together.

WEBRIP:
Re-encoded --> Recorded/captured from web video stream
Network logos & on-screen ads
Sometimes include abrupt transitions (due to commercial breaks)
Higher occurrence of artifacts, frame skips, audio sync, and picture
issues (due to captured source & commercial splits.)

WEB-DL:
Untouched --> Source file from online distribution website (remuxed/not re-encoded)
No logos or ads
No abrupt transitions (no commercial breaks like blu-ray)
Low to no occurrence of quality issues (similar in quality to blu-ray)
https://forums.sonarr.tv/t/web-dl-webri ... rated/4473

Web Download
WEB-DL (P2P) refers to a file losslessly ripped from a streaming service, such as Netflix, Amazon Video, Hulu, Crunchyroll, Discovery GO, BBC iPlayer, etc., or downloaded via an online distribution website such as iTunes. The quality is quite good, since they are not reencoded. The video (H.264 or H.265) and audio (AC3/AAC) streams are usually extracted from the iTunes or Amazon Video and remuxed into a MKV container without sacrificing quality. An advantage with these releases is that, like BD/DVDRips, they usually have no onscreen network logos. HDRips are typically transcoded versions of HDTV or WEB-DL source files, but may be any type of HD transcode.

Web Rip
In a WEB-Rip (P2P), the file is often extracted using the HLS or RTMP/E protocols and remuxed from a TS, MP4 or FLV container to MKV.

Web Capture
A WEBCap is a rip created by capturing video from a DRM-enabled streaming service, such as Amazon Instant or Netflix. Quality can range from mediocre (comparable with low quality XVID encodes) to excellent (comparable with high quality BR encodes). Essentially, the quality of the image obtained depends on internet connection speed and the specifications of the recording machine. WEBCaps are often labeled as WEBRips, with a note attached by the encoder to let others know it was capped & encoded instead of losslessly remuxed.
https://en.wikipedia.org/wiki/Pirated_m ... ease_types
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Is there a way to add to bindings? (i.e. normalized {source} values)

Post by rednoah »

Suggestions are welcome.

How would you split this line and the pattern up into distinct source values?

Code: Select all

WEB-DL	WEB.?DL|WEB.?DLRip|WEB.?Cap|WEB.?Rip|HC|HD.?Rip|VODR|VODRip|PPV|PPVRip|iTunesHD|ithd|AmazonHD|NetflixHD|NetflixUHD|(?<=\d{3,4}[p].)WEB|WEB(?=.[hx]\d{3})
:idea: Please read the FAQ and How to Request Help.
kim
Power User
Posts: 1251
Joined: 15 May 2014, 16:17

Re: Is there a way to add to bindings? (i.e. normalized {source} values)

Post by kim »

What was wrong with the old way ?

Do you have the old list ?

Also this is not what he was asking for (I think)
viewtopic.php?f=4&t=10690

Something like this is answer to add NOT replace:

Code: Select all

WEB-DL	[0-9a-zA-Z]{2,4}.?WEB.?DL
WEBRip	[0-9a-zA-Z]{2,4}.?WEB.?Rip
or

Code: Select all

WEB-DL	\w{2,4}.?WEB.?DL
WEBRip	\w{2,4}.?WEB.?Rip
remember "source" is a quality index, which can be used to tell what file is better

EDIT: For now I would add/change this:

Code: Select all

WEB-DL	WEB.?DL
WEBRip	(?<=\d{3,4}[p].)WEB|WEB(?=.[hx]\d{3})
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Is there a way to add to bindings? (i.e. normalized {source} values)

Post by rednoah »

The patterns are the same. But the old "list" was just a single pattern, so it couldn't normalize different matches to the same standard value:

Code: Select all

CAMRip|CAM|TS|TELESYNC|PDVD|PTVD|PPVRip|Screener|SCR|SCREENER|DVDSCR|DVDSCREENER|BDSCR|R4|R5|R5LINE|R5.LINE|DVD|DVD5|DVD9|DVDRip|DVDR|TVRip|DSR|PDTV|SDTV|HDTV|HDTVRip|DVB|DVBRip|DTHRip|VODRip|VODR|BDRip|BRRip|BR.Rip|BluRay|Blu.Ray|BD|BDR|BD25|BD50|3D.BluRay|3DBluRay|3DBD|BR.Scr|BR.Screener|HDDVD|HDRip|WorkPrint|VHS|VCD|TELECINE|WEBRip|WEB.Rip|WEBDL|WEB.DL|WEBCap|WEB.Cap|ithd|iTunesHD|Laserdisc|AmazonHD|NetflixHD|NetflixUHD|VHSRip|LaserRip|URip|UnknownRip|MicroHD
:idea: This was long over-due for an overhaul, since it makes scripting impossible, since you can get all kinds of different values, even for WEB.DL that would match WEB-DL / WEB.DL / WEB_DL / etc and give back that value, plus WEB (which is not uncommon) was hard to support well.

:idea: BDRip|BRRip|BR.Rip|BluRay|Blu.Ray (regex pattern) should all give the same {source} value to make things consistent. For that we need a map of Name ➔ Pattern values, rather than just a list of pattern.
:idea: Please read the FAQ and How to Request Help.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Is there a way to add to bindings? (i.e. normalized {source} values)

Post by rednoah »

:?: So WEB refers to WEBRip rather than WEB-DL? Or is it an entirely separate thing? I'm not sure about that one. But I don't know either.

Let's see if @DevXen and @devster have input on that.
:idea: Please read the FAQ and How to Request Help.
kim
Power User
Posts: 1251
Joined: 15 May 2014, 16:17

Re: Is there a way to add to bindings? (i.e. normalized {source} values)

Post by kim »

yes
WEB (Scene)
https://en.wikipedia.org/wiki/Pirated_m ... es#Web_Rip

= if someone has named a file .WEB. when it's not WebRip that is an error by that person
( if logo = 100% WebRip )

if {source} is not changed to respect the real source from filename, then it's useless.

if you want to make Plex/Kodi friendly add it to e.g. {Kodi.source}
because (I believe) will e.g. only show "BluRay" if any of this in filename
Blu-Ray
BluRay
BLURAY
BDRip
BRip
BRRip
BDMV
BDR
BD25
BD50
BD5
BD9
DVD, if the filename contains DVD.
Bluray, if the filename contains BLURAY, BRRIP, BD25, or BD50.
HDDVD, if the filename contains HDDVD.
TV, if the filename contains HDTV, PDTV, or DSR.
VHS, if the filename contains VHS.
https://kodi.wiki/view/Media_flags

maybe you can find better list from the source code?
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Is there a way to add to bindings? (i.e. normalized {source} values)

Post by rednoah »

Define "real source" because BRRIP is as much a BluRayRip.

If BD25 and BD50 should be classified separately, that can be done if requested.

The media-sources file is online and subject to automatic updates. Like the group lists. It can evolve over time to fit the needs of the many.


EDIT:

I'll add {vs} ... video source as new binding with the new behaviour, and keep {source} to just match the pattern and return the matched String value as is.
:idea: Please read the FAQ and How to Request Help.
kim
Power User
Posts: 1251
Joined: 15 May 2014, 16:17

Re: Is there a way to add to bindings? (i.e. normalized {source} values)

Post by kim »

"real source" = is from the org. filename (hopefully following the Standard rules)
https://en.wikipedia.org/wiki/Standard_(warez)

thx about {source} / {vs}
but where can we find the {source} "list" ?

btw: I still think the {vs} list has errors
kim
Power User
Posts: 1251
Joined: 15 May 2014, 16:17

Re: Is there a way to add to bindings? (i.e. normalized {source} values)

Post by kim »

to differentiate between {source} and {vs}
I think change {vs} to something like this:

Code: Select all

BluRay	Blu.?Ray|BDRip|BRip|BR.?Rip|BDMV|BD|BDR|BD25|BD50|BD5|BD9|3D.?BluRay|3DBD|MicroHD
CAM	CAM|CAMRip|CAM.?Rip|TELECINE|TC|HDTC|TELESYNC|TS|HDTS|PDVD|PTVD|PreDVDRip
DVD	DVD.?R|DVD.?Full|Full.?Rip|DVD|DVD.?[59]|DVD.?Rip|DVD.?Mux|R5|R5.?LINE|VCD|HDDVD|LaserRip|Laserdisc
VHS	VHS|VHSRip
HDTV	HDTV|DVB|DVBRip|DTVRip|HDTVRip|PPV|PPVRip|HD.?Rip
SCREENER	SCREENER|SCR|DVDSCR|DVDSCREENER|BDSCR|BR.?Scr|BR.?Screener|WORKPRINT|WP
SDTV	SDTV|PDTV|DSR|DSRip|SATRip|DTHRip|TVRip
WEB	WEB.?DL|WEB.?DLRip|WEB.?Cap|WEB.?Rip|VODR|VODRip|(?<=\d{3,4}[p].)WEB|WEB(?=.[hx]\d{3})
UnknownRip	UnknownRip|URip

and add support for e.g. "NF.WebRip" to {source} and {vs} or a new {websource}
maybe like

Code: Select all

(\w{2,4}.?(?:WEB.?DL|WEB.?DLRip|WEB.?Cap|WEB.?Rip|VODR|VODRip|WEB(?=.[hx]\d{3})))
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Is there a way to add to bindings? (i.e. normalized {source} values)

Post by rednoah »

1.
What does "NF.WebRip" mean? What does NF stand for?


2.
{vs} and {source} use the same list. {vs} will give you the mapped value for a match. {source} will give you the matched substring:
viewtopic.php?f=5&t=11265


3.
I don't think there's a clear standard for any of this. I've made this an online file so we can evolve it as needed, to and make things work according to user expectations.
:idea: Please read the FAQ and How to Request Help.
kim
Power User
Posts: 1251
Joined: 15 May 2014, 16:17

Re: Is there a way to add to bindings? (i.e. normalized {source} values)

Post by kim »

NF = Netflix
filenames never use full name like "Netflix" only 2-4 e.g. NF
see list:
viewtopic.php?f=4&t=10690
DevXen
Power User
Posts: 164
Joined: 12 Oct 2014, 21:15

Re: Is there a way to add to bindings? (i.e. normalized {source} values)

Post by DevXen »

sorry i didn't get notifications of these replies.

the new {vs} seems useful. but yeah i agree WEB-DL and WEBRip should be different.
but in my case i'm okay with assuming WEB is WEB-DL

apart from that. reading this i realized i could probably just create a CSV file with a mapping from WEB:WEB-DL
then I could also force BluRay:BDRip

So imma work on seeing if i can get that to play nice with the rest of my script.
Post Reply