FEATURE SUGGESTION: Notication when the parsing egg timer has finished

[stephen147]

See here.



As you can see that's a lot of timers.

It would be useful to have a notification pushed like the top of the screen like when the rename is finished (with a sound) OR,
for windows push the notifications altogether to the notification bar should for anyone running win 10.

This process of parsing can take up to 5mins for me so this would be a nice little reminder.

[rednoah]

The timer is there while it's evaluating your format.

Checking your format for bottlenecks might be worth a try.

For the majority of users, this process is rather instant. So there's decisions to make if and when there should be a notification, definitely not every time, so lots of additional code to track progress and timing would be required.

[stephen147]

For me it takes longer for the format evaluation than the rename. So maybe a user setting to display a notification after x seconds should work?

My format is complex but I can't shorten it much more.

This is also true for the filter. There's no notification for this and this can be time consuming also.

The notification after rename is less useful for me anyway as the files get cleared automatically from the UI so it's easily noticed. Whereas the filter timer and the evaluation mean either checking the timer for the filtering to be complete or scrolling to the bottom of the list to see if the files are ready to be renamed.

[rednoah]

I'll see if it's something that can be added easily. I won't take on much technical debt for this particular feature though.

[rednoah]

Is your format slow because it's:

CPU heavy?
IO heavy?
NETWORK heavy?

All of the above?

[stephen147]

It's just complex. I'll post the example tomorrow. Mostly the audio portion of it is.

The files are stored on a NAS in raid 5 over 10Gbe. Speed tests are around 1,200 MB/s read, and 550 MB/s write. 500 files, each in a separate folder.

I suppose it could work once the process of evaluation has ended. When your for loop argument reaches a value of 0.

Two processes for me take the longest are loading files in the filter (slowest, 500 files around 8mins) followed by evaluation of the format (2nd slowest, 500 files around 5mins).

Like:

Code: Select all

if (files == 0) (timer > setTime) {
pushNofication.Evaluation()
timer.End()
} 

[rednoah]

There's no loop. It's all event based, and no overarching part of FileBot does any book keeping which tasks are open or is notified about completed tasks.

All doable, but out not without significant code changes.

The thing that is very possible though, and just disabled for the sake of simplicity (and making it look cool), is allowing arbitrary parallelism for all the format evaluation tasks.

Presumably, the bottleneck is network IO and network latency when MediaInfo is parsing files over the network. Ideally, MediaInfo should only need to read a few MB for that, but there's known cases where it'll read the whole file due to bugs, which then makes things take much much longer than expected for certain files.

Accessing MediaInfo via filter and then via format would lend itself to caching. Which may or may not be happening already. Untested IO wise. Worth looking into. Avoiding unnecessary network IO can speed things up immensely.

[rednoah]

I've changed the MediaInfo cache from 5 minutes to 20 minutes. So the first round of reading MediaInfo during the --filter phase should be unaffected, but --format phase would be infinitely faster if MediaInfo is accessed from the in-memory cache rather than over the network.

[stephen147]

Thank you.. I'll be sure to test.

BTW, here's my format so you can see how badly written it is and how it affects the speed

Code: Select all

{
	//////////////////////////////////////////////
	// MOVIE BINDING
	//////////////////////////////////////////////
	// Posted here: https://www.filebot.net/forums/viewtopic.php?f=4&t=10766&p=43842#p43842
	//////////////////////////////////////////////
	def space = call{' '};
	// Root Directory
	def dir_root = 'Z:\\Movies & TV\\'+
	call{hd.matches(/(?i)SD/) ? '1. SD\\' : ' '}+
	call{hd.matches(/(?i)HD/) ? '2. HD\\' : ' '}+
	call{hd.matches(/(?i)UHD/) ? '3. UHD\\' : ' '};
	// Main Title e.g.
	// 1408 (2007) (Director's Cut) 1080p HD Blu-ray non-HDR x264 DTS5.1ch_SiNNERS
	// 300 (2007) 720p HD BRRip non-HDR x265 DTS 5.1ch_ESiR
	// Deadpool (2016) 2160p UHD WEB-DL non-HDR AVC DTS-HD MA 7.1ch_DDR
	// Deadpool 2 (2018) (Super Duper Cut) 2160p UHD Blu-ray REMUX HDR10bit ATEME TrueHD Atmos 13Obj 7.1ch_EPSiLON
	def main_title = call{n}+
	space + '(' + call{y} + ')'+
	space + call{fn.matches(/(?i).+\b25th.+?anniv.+/) ? '(25th Anniv. Edition)' : ' '}+
	space + call{fn.matches(/(?i).+\b\(limited\b.*?\).+/) ? '(Limited Edition)' : ' '}+
	space + call{fn.matches(/(?i).+\b\(uncut\b.*?\).+/) ? '(Uncut)' : ' '}+
	space + call{fn.matches(/(?i).+\bcollector.+?s.+?edition\b.+/) ? '(Collector\'s Edition)' : ' '}+
	space + call{fn.matches(/(?i).+\bdirect.+?cut\b.+/) ? '(Director\'s Cut)' : ' '}+
	space + call{fn.matches(/(?i).+\bextended.+?\b.+/) ? '(Extended)' : ' '}+
	space + call{fn.matches(/(?i).+\bextended.+?edit\b.+/) ? '(Extended Edition)' : ' '}+
	space + call{fn.matches(/(?i).+\bimax\b.+/) ? '(IMAX Edition)' : ' '}+
	space + call{fn.matches(/(?i).+\blimited\b.+/) ? '(Limited)' : ' '}+
	space + call{fn.matches(/(?i).+\bremastered\b.+/) ? '(Remastered)' : ' '}+
	space + call{fn.matches(/(?i).+\bsuper.+duper.+cut\b.+/) ? '(Super Duper Cut)' : ' '}+
	space + call{fn.matches(/(?i).+\btheatrical\b.+/) ? '(Theatrical)' : ' '}+
	space + call{fn.matches(/(?i).+\bunrated\b.+/) ? '(Unrated)' : ' '}+
	space + call{any{fn.match(/\([^\()+?[^\d]+?\)\s*/)} {' '}{' '}}+
	space + call{self.vf ? self.vf : self.hpi}+
	space + call{hd}+
	space + call{source.matches(/(?i)blu.*ray/) ? 'Blu-ray' : {source} ?: 'WEB-DL'}+
	space + call{fn.matches(/(?i).+\bremux\b.+/) ? 'REMUX' : ' '}+
	// Only calls {hdr} if it's not SD else non-HDR
	space + call{if (hd =~ 'HD') {any{hdr + "-" + bitdepth + 'bit'}{'non-HDR'}}}+
	space + call{vc}+
	space +
	// Call audio
	// Thread here where I got the base code from: https://www.filebot.net/forums/viewtopic.php?f=5&t=5285
	call {
		def mCFP =
		[
		'AAC LC SBR PS' : 'AAC',
		'AAC LC SBR' : 'AAC',
		'AAC LC' : 'AAC',
		'AC 3 Dep' : 'E-AC3',
		'AC 3' : 'AC3',
		'DTS 96 24' : 'DTS 96-24',
		'DTS ES XBR' : 'DTS-HD HRA',
		'DTS ES XLL' : 'DTS-HD MA',
		'DTS ES XXCH XBR' : 'DTS-HD HRA',
		'DTS ES XXCH XLL' : 'DTS-HD MA',
		'DTS ES XXCH' : 'DTS-ES',
		'DTS ES' : 'DTS-ES',
		'DTS XBR' : 'DTS-HD HRA',
		'DTS XLL X' : 'DTS X',
		'DTS XLL' : 'DTS-HD MA',
		'DTS' : 'DTS',
		'E AC 3 JOC' : 'EAC3 Atmos',
		'E AC 3' : 'EAC3',
		'MLP FBA 16 ch' : 'TrueHD Atmos',
		'MLP FBA' : 'TrueHD',
		'MP3' : 'MP3',
		'MPEG Audio' : 'MP2',
		'PCM' : 'PCM'
		];
		def audioClean = { it.replaceAll(/[\p{Pd}\p{Space}]/, ' ').replaceAll(/\p{Space}{2,}/, ' ').slash(' ') };
		def channelClean = { it.replaceAll(/Debug.+|Object\sBased\s?\/?|(\d+)?\sobjects\s\/\s|0.(?=\d.\d)|20/).replaceAll(/6/,'5.1').replaceAll(/8/,'7.1') };
		def audioCollection = audio.collect
		{ au ->
			def channels = any{ channelClean(au['ChannelPositionsString2'])}{ channelClean(au['ChannelsOriginal'])}{ channelClean(au['Channels']) };
			def dynChannel = {au['NumberOfDynamicObjects'] + 'Obj'};
			def ch = channels.tokenize('\\/').take(3)*.toDouble().inject(0, { a, b -> a + b }).findAll { it > 0 }.max().toString() + 'ch';
			def codec = audioClean(any{ au['CodecID/Hint'] }{ au['Format'] });
			def format_profile = { ( au['Format_AdditionalFeatures'] != null) ? audioClean(au['Format_AdditionalFeatures']) : '' };
			def combined = allOf{codec}{format_profile}.join(' ');
			def stream = allOf { mCFP.get(combined, 'UNKNOWN_FORMAT--'+combined+'--') } { dynChannel } { ch };
		};
		return audioCollection[0].join( ' ' )
		} +
		'_'+
		// Group
		call{ any{ fn.match(/(?<=[_-])[^\s_-]+?$/)} {group} {'_NA'} };
		// Language
		def lang = call{ any{'.'+subt} + {fn.matches(/(?i).+sdh.+/) ? '_SDH' : ''} {any{fn.match(/(?i)\(foreignpartsonly\)/)} '' } };
		// Extension
		// def ext = call{'.'+ext};
		// call all the bindings to create the result
		// call(dir_root) +
		(call(main_title).replace(':', ';').replace('*', '') + call(lang)).replaceAll(/null/,'')
	}

[rednoah]

Looks like it should be pretty instant. If you're testing in the Format Editor, and you get back the result immediately for each change, then it means the format itself is pretty instant (since MediaInfo will already have been cached for your sample file).

[stephen147]

Not sure why but it's faster to filter using the latest daily from today. Like 5x faster it seems. Did you rework that?

BTW, I love the fact that the movies category got by the filter is just that. Exact same count as what's in the folder.

The format is still a little slow though.

STILL, thanks for whatever you did to the filter.

[rednoah]

Wait, are we talking about the Filter Panel / Types Tab?

Haha. I thought we're talking about --file-filter and --format.

For the Types Tab, I definitely made some changes some time ago, like last week, that would exclude non-video files from certain categories right away, and so that'll save many CPU cycles if you had many text / audio / etc files before.

I tried your format on a few files. It's pretty fast, but not quite instant, so I guess it'll need a minute or two for a few hundred files. Not sure if there's a specific bottleneck somewhere. Kinda hard to test, since it's just blink of an eye speed for each individual item.

[stephen147]

Well it certainly seems snappier!

I was pretty sure my 1st post had a screenshot of the UI!

So in that filter, it can filter the movie files, but can it also show the folders of those as well so both files and folders together can be sent to the renamer.

SC of what I propose

[rednoah]

That was a different thread though. This one is about Rename Panel.

The newer revisions also allow you to set a system property for parallel format evaluation:

Code: Select all

-Dparallelism=6

Code: Select all

-Dparallelism=32

Would be interesting to see if throwing lots of threads at the problems makes it go noticeably faster or not.

[stephen147]

Okay, I've got a TR1950X. Let's see how that performs!!!

[rednoah]

Movie Folders being like DVD / BD Disk Folders? Or folders that contain movie files? Or folders that are named like movie files with Name (Year) pattern?


EDIT:

Note: Only format evaluation in the Rename panel (the hourglass) is parallel. The Filter / Types isn't parallelized at all AFAIK, but nobody ever asked for performance improvements there.


EDIT 2:

[r6384] Parallelize CPU heavy operations in Filter Panel / Types Tools

[stephen147]

Okay, I'm trying to get -Dparallelism=32 to work. Please help

Oh, I think the time-zone threw me of here: https://get.filebot.net/filebot/BETA/ and I grabed the r6383 version!

Downloading r6384 right meow!!

Movie Folders being like DVD / BD Disk Folders? Or folders that contain movie files? Or folders that are named like movie files with Name (Year) pattern?

Well, folders that have the pattern > Movie Name/Movie Name.mkv | 1 file and one folder.

Note: Only format evaluation in the Rename panel (the hourglass) is parallel. The Filter / Types isn't parallelized at all AFAIK, but nobody ever asked for performance improvements there.

Filter is faster now so as long as I'm going to be parallelized in the rename I don't care!

[rednoah]

You can permanently set these system properties for FileBot like so:

Code: Select all

filebot -script fn:properties --def parallelism=32

That's rather new. You'll find most docs here will be setting these options via environment variables.

[stephen147]

Just got it as I refreshed the page!

32 threads in testing!

[stephen147]

I did notice something strange and this happened with r6384.

1. Added files to filter.
2. Waiting until they appeared in Types tab.
3. RMB>Send to rename
4. Auto switched to rename
5. No files appeared
6. Back to filter
7. I then had to repeat step 3 for the files to appear

[stephen147]

I've tested -Dparallelism=32 and I cannot see any noticeable difference.

I don't know, maybe it prefers clock speed as opposed to thread count!

[rednoah]

You can use a format such as this for testing, to make sure each format requires at least 2 seconds to finish:

Code: Select all

{sleep(2000); fn}

Parallelism is working if you get the green arrow hour glass for many items at the same time:



But the bottleneck could very well be the network drive, and so many network IO requests don't necessarily make things faster.

[stephen147]

Yip, it's taking 2 seconds. Re the Parallelism, it's not working by getting the green arrows in a bunch. Just one by one atm.

Checked my settings:

Code: Select all

#FileBot System Properties
#Wed May 22 22:45:39 BST 2019
Dparallelism=32

[rednoah]

Please run this command verbatim:

Code: Select all

filebot -script fn:properties --def parallelism=32

-Dname=value is a Java thing and only necessary if you set Java options on startup, so just drop the -D part.

[kim]

A quick test:

Code: Select all

-Dparallelism=32

works

Getting MediaInfo via network = slow

e.g. 1 x 4GB file = ~55MB download
1 x 1 GB file = ~14MB download
bottleneck = network

if the read MediaInfo can be done local (at same place as file)
maybe using something like
https://mediaarea.net/MediaInfoOnline
it will be way faster