Filesize limit?

All your suggestions, requests and ideas for future development
Post Reply
DavidRTurner
Power User
Posts: 85
Joined: 01 Feb 2014, 16:59

Filesize limit?

Post by DavidRTurner »

I've just had to rename all my media files, and now find that FB will not touch files under about 150mb in size. Is there a system- or programmed- limit to the size of files that FB will process in the current version?

Specifically, I use one format for movies, and one for the movies' DVD extras, etc. - while movies are typically 700mb+, the extras can be small. I'm finding that FB will not look at anything under ~150mb. I've cleared the cache (I use the GUI), but can't see any way to force it to act on these small files.

I've used FB for years, doing this fairly consistently; this is the first time I've encountered this limitation, and it would be with the most recent version only, that I've seen it, as I haven't done the DVD extras in a while, and wanted to do something new.

No issues with any files over 150mb.

Any thoughts? Is there a setting I can tweak somewhere?
User avatar
rednoah
The Source
Posts: 19015
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Filesize limit?

Post by rednoah »

There is a concept of "clutter file" which includes all the extras like *.nfo and *.jpg files, but also small video files. But I don't really know what you're doing, so I don't know how this could affect your particular use case. There's nothing you can tweak, and the 150 MB clutter file limit has been around since forever, and doesn't really affect what you can and cannot process.

But as far as processing small files is concerned, it works just fine:

Code: Select all

filebot -rename Avatar.2009.mkv --db TheMovieDB --format "{ny} [{megabytes} MB]" --log INFO
[MOVE] from [Avatar.2009.mkv] to [Avatar (2009) [41 MB].mkv]

:idea: If you're using the GUI, please add screenshots. If you're using the CLI, please add commands and logs. Either way, please include file paths too.
:idea: Please read the FAQ and How to Request Help.
DavidRTurner
Power User
Posts: 85
Joined: 01 Feb 2014, 16:59

Re: Filesize limit?

Post by DavidRTurner »

Hey, red - I'm simply doing what I've been doing for years; renaming TV & movies, but also DVD 'extras'.
The 'extras' format is simply shorter than the regular format.

i.e. Backdraft (1991) [137 min] (1A,1S) (SE) [Kurt Russell, William Baldwin, Robert De Niro] [Action, Crime, Drama].mp4
and Backdraft (1991) (1A,1S) =The Explosive Stunts=.mp4
For the extras, I use the "=" symbol to mark start/finish of text I don't want it to touch. So when I re-name this small extras file for the movie Backdraft, FB will never overwrite it as just the movie name; it always renames it as this short name.

I have just done a big purging of my collection, and after running through the movies & shows, I'm doing the extras (which I haven't changed in a couple of FB updates), and FB now, definitely will not process anything smaller than 150 mb.
File duration isn't an issue - a two-hour-long movie, highly compressed to 125mb, isn't picked up.
But a 2-minute bit, de-compressed so it's 160mb, IS picked up. See image:
https://snipboard.io/OUHqAZ.jpg

So it seems very clear that it's treating everything under 150mb as a 'clutter' file - yet, for years, I've never had any 'clutter files' ignored in the GUI - just in the current version.

I've never used a script, or ever wanted to; I've always had to test the accuracy of the naming suggestions, manually, and that will continue.
User avatar
rednoah
The Source
Posts: 19015
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Filesize limit?

Post by rednoah »

FileBot fundamentally doesn't support extras, bonus materials, artwork files, etc, at least not "consciously" so. FileBot doesn't know it's processing extras, but it will happily process "badly named" movie files, and then your custom format fixes things up, and then it appear to work for your special case.


Notably, the 150 MB limit is not the thing that prevents you from processing these files, it's the thing that allows you to process files larger than 150 MB in the first place that would otherwise be ignored. FileBot simply assumes that large files are definitely not clutter, just to make sure we don't accidentally misclassify valid movie files or episode files as clutter.


:!: Processing extras is dangerous for normal users, because FileBot doesn't support extras, and so extras will likely get matched to the same movie / episode object as the video file, so common formats such as {plex} will yield the same file path for the main video file and all the extras, leading files getting either skipped or overridden.


You still didn't include file paths, so I can't point out any specific patterns for your specific case, but my educated guess is that excluded file paths contain keywords such as Extras, Trailers, Featurettes, Shorts, etc, which means they get ignored by default unless it's large files.


EDIT:

These things have been unchanged for the better part of a decade, perhaps with slight updates to the specific patterns and numbers here and there.

Since we can generally assume that MediaInfo works on all platforms these days, perhaps we could consider using a video length limit, though the goal would still be to automatically exclude extra and bonus material, just more accurately.
:idea: Please read the FAQ and How to Request Help.
DavidRTurner
Power User
Posts: 85
Joined: 01 Feb 2014, 16:59

Re: Filesize limit?

Post by DavidRTurner »

OK - I believe there's no follow-up at this time, red - if there's no FB feature in place that covers this; then this post will simply serve as a note in case something is found later...

I don't use scripts or plex or other media library, so there is no auto-cleanup of clutter or auto-filtering; I download files, pull them into the GUI & process them manually. Sonarr will download & move some TV files to folders, but that's the limit of the automation - I pull them into the FB GUI manually. So I can't reference file paths or plex settings to check - there simply aren't any.

I process everything relating to FB, manually through the GUI; there is no logic in my formats that ignores small files or anything named like a 'clutter' file; I manually choose each file to be processed, so I know exactly what FB is doing. I verify every rename function for every file.
If a new file's name is "broken" or unrecognizable, I will fix it manually, before pulling it into FB - so there should be no confusion on FB's side in reading the filename; it should always be pretty good before FB ever sees it.
My formats have worked fundamentally the same way all the time I've used FB.

The GUI has always worked before now - no matter what size or duration that video file was.
But, the FB GUI is now ignoring* every file under 145mb** in size. **145.25mb apparently is the limit**
A 2-hr movie compressed down to 140mb, is ignored; but a 30-second trailer expanded to 150mb, is processed just fine.

*When I say "ignoring"; I mean, if I drag files of 200mb, 175mb, 150mb, 125mb file into the GUI, and run a movie format, it brings results for the 3 larger files and ignores the 125mb file. If the 125mb file is the only one I pull into the GUI, FB doesn't even check it - it appears to do nothing at all.

Clearly, this is a new issue. I assumed it must be a FB issue, but now I don't know.
After further digging, it seems to be happening only where I'm pulling from the movie databases (regardless if it's a full movie or an extra or anything else).
Is it possible that FB is telling IMDB the filesize and name, and IMDB is refusing to answer if the filesize is under 145mb?

I'll keep testing to see if I can narrow it down any further, and report back.
Thanks.
kim
Power User
Posts: 1027
Joined: 15 May 2014, 16:17

Re: Filesize limit?

Post by kim »

User avatar
rednoah
The Source
Posts: 19015
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Filesize limit?

Post by rednoah »

DavidRTurner wrote: 04 May 2020, 14:37 *When I say "ignoring"; I mean, if I drag files of 200mb, 175mb, 150mb, 125mb file into the GUI, and run a movie format, it brings results for the 3 larger files and ignores the 125mb file. If the 125mb file is the only one I pull into the GUI, FB doesn't even check it - it appears to do nothing at all.
Can you provide one full file path for one of the ignored files for testing?

Because basic test cases such as this one work regardless of file size as far as I can tell:

Code: Select all

X:/Movies/Avatar.2009.mp4
:idea: Please read the FAQ and How to Request Help.
DavidRTurner
Power User
Posts: 85
Joined: 01 Feb 2014, 16:59

Re: Filesize limit?

Post by DavidRTurner »

rednoah wrote: 04 May 2020, 15:23 Can you provide one full file path for one of the ignored files for testing?

Because basic test cases such as this one work regardless of file size as far as I can tell:

Code: Select all

X:/Movies/Avatar.2009.mp4
OK, further testing results after checking paths: it is the Synology NAS that's the problem.
When my Drobo NAS broke down in November, I replaced it with this Synology DS1819+.
I only JUST got everything restored & re-organized, and then had to do a renaming, which is where I am now.

On all my HDs and SSD, FB works fine as it used to on all the test files, of any size.
But on the Synology, under either a network name or a drive letter substitution, it hits that 145mb limit.
Strangely, though - this issue exists with movies dbase calls only; the TV dbase calls work fine.

I have had some issues with certain programs not recognizing the new substitute drive letter; but they will recognize the network pathname... so this is the first time I've experienced an issue with the NAS itself (or, I assume, the NAS' operating software).


I'll go to some NAS user groups & see if anyone has any info there - or that someone with a Synology DS model here, finds this thread & suggests a fix.

Thanks!
User avatar
rednoah
The Source
Posts: 19015
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Filesize limit?

Post by rednoah »

1.
DavidRTurner wrote: 04 May 2020, 23:12
rednoah wrote: 04 May 2020, 15:23 Can you provide one full file path for one of the ignored files for testing?
If the 150 MB size limit changes behaviour, then the file path most likely has something to do with it. Please copy & paste one of the file paths that don't work. You can PM me if you don't want to publicly post file paths.


2.
FileBot r7515 changes the file size limit to 180 MB (larger files are never clutter) and adds an additional video length limit of 40 min (longer videos are never clutter).
:idea: Please read the FAQ and How to Request Help.
DavidRTurner
Power User
Posts: 85
Joined: 01 Feb 2014, 16:59

Re: Filesize limit?

Post by DavidRTurner »

<groans internally...>
I should have trusted your gut a little earlier, red...
None of my FILES have the 'clutter' words in the filenames, but the FOLDERS do.

That is,
Z:\Movies\C\C=Extras - typical path for the bits & pieces that aren't working now.
I just tested some sample files in different folders, and it seems that the word 'Extras' may well be the issue.
If I change this folder to
Z:\Movies\C\C-Misc - it works as it used to.

NOTE: on the Drobo, and with FB versions at the time, there was never an issue; so this issue might be the Synology or the current version of FB - I have files dated April 16.

That is; could the current version of FB be analyzing the folder name for clutter words and making decisions about the folder contents as a result (a new feature)? Even while I'm dropping files (not the folder) into the GUI?
User avatar
rednoah
The Source
Posts: 19015
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Filesize limit?

Post by rednoah »

1.
Yep, that would do it.

The logic fundamentally hasn't changed in years though, but I think the file size limit might have been lower. Due to common code base, some other change may slightly indirectly change / effect / fix behaviour elsewhere. There's a reason why worked for you before, but finding it is probably not worth our time and effort. :lol:

If you process files from some original path into "Extras" folders with FileBot, then it would work the first time around with no "Extras" folders, but then be excluded the second time around now that we have "Extras" folders.


:idea: If you are processing files that have already been processed with FileBot before, then you'll want to use Xattr Mode. That way FileBot will look at nothing but Xattr Metadata. Please read Metadata and Extended Attributes for details.


2.
rednoah wrote: 05 May 2020, 07:31 FileBot r7515 changes the file size limit to 180 MB (larger files are never clutter) and adds an additional video length limit of 40 min (longer videos are never clutter).
Do you have any thoughts on the new limits? What file size / video length should we use for our "this file is definitely not an extra" litmus test?
:idea: Please read the FAQ and How to Request Help.
DavidRTurner
Power User
Posts: 85
Joined: 01 Feb 2014, 16:59

Re: Filesize limit?

Post by DavidRTurner »

I use the GUI for these files, drag & drop - I do not use folders in the paths for movies or extras.
FB should NOT be looking at folders when I drop files directly into the GUI. This is a case example of WHY it should not look at folders. Just because I have the word 'extras' in a foldername, does not mean MY definition of an extra should match what FB is using.
I have alternate versions of full movies, for example, in these 'extras' folders. I name them as full movies, but keep them in that extras folder. FB should not use the foldernames to guide its operation, unless I drop a folder into the GUI.
I have never seen any indication of these 180mb and 40min limits; if they are script-based checks, I wouldn't see them since I only use the GUI.
Perhaps a toggle to turn on/off that sort of functionality; to let the user have more control over these sorts of options?


Extra vs NotExtra is tough - as above, I have full-length alternate versions of movies that I consider 'extras'.
I keep some TV documentary programs that I've custom-named to be part of my extras collection for that movie...
As well, some independent videos I've found online may only be 5-10 minutes, but they're recognized as unique films by IMDB. But their 'making of' video might be an hour long... which would FB identify as 'the movie' if it uses a duration logic?
If it's possible to let the user choose their own duration settings to suit their collecting habits, that would be great. Everyone will, at some point, run into a scenario that makes an exception, so rules can never work for everyone...
Post Reply