HOW TO START A RENAMING JOB ON A HUGE DOWNLOAD FOLDER?

Any questions? Need some help?
Post Reply
fmuaddib
Posts: 3
Joined: 27 Jun 2012, 11:04

HOW TO START A RENAMING JOB ON A HUGE DOWNLOAD FOLDER?

Post by fmuaddib »

Hi!

I've downloaded FileBot and I can say that it seems amazing.
I need some guidance to do the first steps, because I cannot find the right way yo start my first renaming job.

My current situation:

- MacMini with Plex Media installed and connected to the HDTV
- ReadyNAS Pro 6TB connected via LAN/WIFi, with Plex Media Server for Linux Installed
- A HUGE Download folder on the NAS ( > 5 TB files in chaos: many subfolders, thousands of video files of all kinds, movies, anime, tv episodes, documentaries, subtitles, etc)

To start using Plex (or any other media server) I need to rename, move and refolder all files and subtitles in a clean and organized way.
At the beginning I was desperate (too many files to even think about doing it manually), but lucky for me I found FileBot!

But here is the problems I encountered:

- After installing FileBot (v2.64) I dragged and dropped the Download folder on the "Rename" section of FileBot, but there are movies and documentaries too in it. Is it ok or there is a specific way for those?

- After having waited a long time for FileBot to load all file names in the list, I pushed the "Match" button. There were many choices, but I didn't know what to choose. I read that OpenSubtitles has hashing and that is more reliable to recognize video files, so I pushed it. Was opensubtitles the best way? I don't want to do this work twice, because is a huge renaming job. There is a way to check the files against ALL databases, and later choosing between ALL the proposed choices? For example I know that all the MVGroup documentaries fail to be identified by OpensSubtitles, even if they are all provided with subtitles and even a wiki with cover art (http://forums.mvgroup.org/) so I wanted to use some other database to integrate the metadata results and avoid that many documentaries are tagged as movies or anime.

- Before chosing the id provider, I tried to find a good renaming string for the Plex required folder structure ( http://wiki.plexapp.com/index.php/PlexN ... amingGuide ), but I didn't find anything good enough for a mix like mine (anime episodes, oav, tv episodes, specials, documentaries, music videos, multiple subtitles files, and so on) that also moved the files and create the right folders with the right subtitles (I've many subtitles, always with the same name of the videofile and a .language.srt or .language.group.srt extension added).

- After some time FileBot started asking me questions about the identity of each file, like he didn't found a precise match. I had to select the right answer, but I didn't have any info on the original file, not even a full file and path name (the file name was truncated by the window). Is this right, or there is a way to automatically improve the recognition and to narrow down the number of candidates? There is a way to cross check the results between ALL the databases, and then choosing the candidates with the most results? Or I'm doing all wrong?

- And here is the big problem that forced me to stop: the dialogs asking for the identifications where TOO MANY to handle, because my folder contains billions of files! After forcing me clicking for a couple of hours, I gave up. That is surely not the way to use FileBot, because it was taking me more time than to renaming the files myself manually! So I stopped.

- All the files already recognized where listed, so I looked at those to check for the id to be right. But I didn't find any way to change to alternatives identifications AFTER the match process. I expected to see files with PROPOSED Identifications listed, and then a button to eventually CHANGE those from a drop down menu. Is there a way to do this? Or Filebot discard all the proposed alternatives collected from the databases after the matching job and require you to resubmit all files to the matching metadata databases online?

- Another problem I encountered was the fact that if a file isn't recognized at all by the databases (like when FileBot asks "Enter series video name:" or similar) it doesn't appear in the list, not even with a warning symbol, like a huge "?" button near the name, for example. I was expecting to see how many files were still in doubt/uncertain/unidentified and eventually doing the renaming of those manually some other day, saving the unidentified file list in FileBot and executing the rename job only on those with a secure identification. But I cannot find a way to do this either, so I must doing all wrong.

So any help is appreciated!
Please, I just need a step by step procedure to order my Download folder automatically with FileBot, but I cannot find a way to do it in a single pass, even after having read many tutorials and docs about FileBot. :?

Thanks in advance! :)
Emanuele
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: HOW TO START A RENAMING JOB ON A HUGE DOWNLOAD FOLDER?

Post by rednoah »

Main problem is that FileBot will not let you mix movies and tv shows. By selecting a tvshow database or movie database you also tell filebot to treat everything as movies or tv shows. The GUI is designed to be used semi-automatically that way. It's hard to tell Anime VS Series VS Movie. So that's way the easiest way is do things in smaller batches because you should check through proposed matches anyway. Handling an entire show like Stargate will take just a few seconds so it's kind reasonable for some manual interaction.

Now that being said the scripting supports everything you wanna do but you'll need figure out some logic so that everything just magically works. The sortivo.groovy script is my novel approach to separate tv shows from movies in a general way.

Basically we'll need to come up with a new filebot/groovy script for heavy duty usecases like this. So far nobody bothered to since, well, you'll only use it once right? :D
:idea: Please read the FAQ and How to Request Help.
fmuaddib
Posts: 3
Joined: 27 Jun 2012, 11:04

Re: HOW TO START A RENAMING JOB ON A HUGE DOWNLOAD FOLDER?

Post by fmuaddib »

rednoah wrote:Main problem is that FileBot will not let you mix movies and tv shows. By selecting a tvshow database or movie database you also tell filebot to treat everything as movies or tv shows. The GUI is designed to be used semi-automatically that way. It's hard to tell Anime VS Series VS Movie.
It is not so hard. For example: series have episode numbers in the filenames, while movies don't. There are exceptions, but there you propose all best candidates for each match, and later the user will choose those names manually. But in the meantime the 90% of files would be processable.
One of the most important design flaw of the lack of an option to rename partially the lists. Like:

A - Rename Only file matches approved by user (all matches are approved by default, but the user can unapprove them)
B - Rename all files matched with a STRONG match rating (i.e. > 2 databases or hash match)
C - Rename all matched files, leaving unmatched files in a list for a manual scrutiny later
D - Rename all files, both matched and unmatched files
rednoah wrote: So that's way the easiest way is do things in smaller batches because you should check through proposed matches anyway. Handling an entire show like Stargate will take just a few seconds so it's kind reasonable for some manual interaction.
For smaller batches I think that you are right. But for big download folders (and they are pretty common) forcing to stay there while the app downloads the ids from the databases and checking all matches is too much. There should be an option to deferring the choice between the found matches later.
rednoah wrote:Now that being said the scripting supports everything you wanna do but you'll need figure out some logic so that everything just magically works. The sortivo.groovy script is my novel approach to separate tv shows from movies in a general way. Basically we'll need to come up with a new filebot/groovy script for heavy duty usecases like this.
Yes, that would be great. Heavy duty cases like this are just what make your app shine. All that automation you built will be a requirement for many heavy downloaders. Keep me updated about that new groovy script then. I'll wait before making the renaming work then.
So far nobody bothered to since, well, you'll only use it once right? :D
Well, I've I couple of HD somewhere also filled with video files, and your app just made me wanna take those back online to give some order to them. And you app is also great for realtime renaming when downloading new files. I'm just searching for a way to use it on my ReadyNAS Pro (no luck till now..).

Thanks,
Emanuele
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: HOW TO START A RENAMING JOB ON A HUGE DOWNLOAD FOLDER?

Post by rednoah »

Send me a snapshot of your paths that I can test with:

Code: Select all

filebot -script "script:args.getFiles().each{println(it.canonicalFile)}" /path/to/media > index.txt
(PM or send me email if you're worried about privacy)
:idea: Please read the FAQ and How to Request Help.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: HOW TO START A RENAMING JOB ON A HUGE DOWNLOAD FOLDER?

Post by rednoah »

You can try this and see what happens:

Code: Select all

// all media files
def files = args.getFiles{ it.isVideo() || it.isSubtitle() }

// exclude clutter
files = files.findAll{ !(it.path =~ /\b(?i:sample|trailer|extras|deleted.scenes|music.video|scrapbook)\b/) }

// filter by SxE
def episodes = args.getFiles{ parseEpisodeNumber(it.name, false) }
episodes.each{ println "episode: $it"}

// filter by date
def dailys = args.getFiles{ parseDate(it.name) }
dailys.each{ println "daily: $it"}

// find possible movie matches in the other files
def movies = (files - episodes - dailys).findAll{
	def m = net.sourceforge.filebot.media.MediaDetection.detectMovie(it, null, null, null, false) // match movie via from local index
	println "matchMovie: $it.name => $m"
	return !m.empty
}
movies.each{ println "movie: $it"}

// files that are neither episodes nor dailys nor movies
def remainder = files - episodes - dailys - movies
remainder.each{ println "remainder: $it"}
You'll probably find that it's really not that easy to seperate movies from episodes. ;)
:idea: Please read the FAQ and How to Request Help.
fmuaddib
Posts: 3
Joined: 27 Jun 2012, 11:04

Re: HOW TO START A RENAMING JOB ON A HUGE DOWNLOAD FOLDER?

Post by fmuaddib »

Ok, I'll try, thanks!
I'll let you know!
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: HOW TO START A RENAMING JOB ON A HUGE DOWNLOAD FOLDER?

Post by rednoah »

Here's a start for that script:
http://pastebin.com/GcgHPX99

Just play with that and see what happens. Since you know some C# this shouldn't be hard to work with for you. Feel free to tinker around and see if you can find a way to differentiate between tvshows/movies/anime/docus. The main problem will be to exclude the right false positives and figure out the correct match.

Didn't build in anything special for documentaries yet but if you need a list of docu names just use this:

Code: Select all

def docuIndex = new URL('http://docuwiki.net/index.php?title=Category:Name').html.depthFirst().find{it['@id'] == "mw-pages"}.depthFirst().A.findResults{ it.text() }
docuIndex.each{ println it }
:idea: Please read the FAQ and How to Request Help.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: HOW TO START A RENAMING JOB ON A HUGE DOWNLOAD FOLDER?

Post by rednoah »

Here's some useful stuff I got from JumpinS at forums.mvgroup.org:

List documentaries as simple text file (to replace the page scraping I posted above)
http://docuwiki.net/postbot/getList.php?subject=Name

List/Filter documentaries as xml:
http://docuwiki.net/postbot/getFile.php?name=
http://docuwiki.net/postbot/getFile.php?name=bbc
:idea: Please read the FAQ and How to Request Help.
JumpinS
Posts: 2
Joined: 04 Jul 2012, 12:49
Contact:

Re: HOW TO START A RENAMING JOB ON A HUGE DOWNLOAD FOLDER?

Post by JumpinS »

I'm very eager to learn more about the progress on this. fmuaddib please continue posting about it. ;)

I'll try to provide any means necessary on the docuwiki side to make this work out.
Hopefully you can provide some kind of guide on how to organize your library of documentaries.

Some threads I've found on the same topic:
http://forums.mvgroup.org/index.php?showtopic=43417
http://forums.mvgroup.org/index.php?showtopic=39773

Also the integration with Plex is very interesting. Please keep on posting.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: HOW TO START A RENAMING JOB ON A HUGE DOWNLOAD FOLDER?

Post by rednoah »

Here's a new version that's actually doing something and should mostly work. Still needs lots of testing of course:
http://pastebin.com/uH9s5VJH

Running with or without -non-strict will make a big difference (for good and for worse).
While testing you can safely run it with --action test that'll just do nothing instead of renaming/moving files.

Might need the latest jar from here:
http://sourceforge.net/projects/filebot ... ebot/HEAD/
:idea: Please read the FAQ and How to Request Help.
JumpinS
Posts: 2
Joined: 04 Jul 2012, 12:49
Contact:

Re: HOW TO START A RENAMING JOB ON A HUGE DOWNLOAD FOLDER?

Post by JumpinS »

I must admit I'm still far from understanding what actually happening there so will have to do some more reading on the basics.
Meanwhile I hope fmuaddib also has some progress and decides to share it :)
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: HOW TO START A RENAMING JOB ON A HUGE DOWNLOAD FOLDER?

Post by rednoah »

Just try:

Code: Select all

filebot -script "http://pastebin.com/raw.php?i=uH9s5VJH" --action test -non-strict /path/to/media
It'll need r1035 cause I tweaked some internals a bit.
:idea: Please read the FAQ and How to Request Help.
ByteJunk
Posts: 5
Joined: 17 Jan 2013, 12:58

Re: HOW TO START A RENAMING JOB ON A HUGE DOWNLOAD FOLDER?

Post by ByteJunk »

Hi, let me just say, this is an amazing piece of software. Thank you for all your hard work! :D

I've been playing a bit with this script (the massrename r2), as I also have a huge, chaotic media folder, and ran into a problem:

Code: Select all

$ filebot -version
FileBot 3.1 (r1296) / OpenJDK Runtime Environment 1.6.0_24 (headless)

$ filebot -script "http://pastebin.com/raw.php?i=uH9s5VJH" --action test -non-strict /media/Tisco/Unsorted
MissingMethodException: No signature of method: static net.sourceforge.filebot.media.MediaDetection.detectMovie() is applicable for argument types: (java.io.File, null, java.util.Locale, net.sourceforge.filebot.web.TMDbClient, java.lang.Boolean) values:  <snipped>
I haven't played with it much yet, hope it's not something I did wrong. :roll:[/i]

Right, strike that. It seems that somewhere along the line the argument order got shuffled, calling detectMovie(f, false, TheMovieDB, null, Locale.ENGLISH) instead works flawlessly. *Crawls under a rock*
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: HOW TO START A RENAMING JOB ON A HUGE DOWNLOAD FOLDER?

Post by rednoah »

Yeah. That one never made it to maturity. If it works for you nice but basically this is one for the digital nirvana.

I'm pretty sure utorrent-postprocess can do anything and more than this prototype did.
:idea: Please read the FAQ and How to Request Help.
ByteJunk
Posts: 5
Joined: 17 Jan 2013, 12:58

Re: HOW TO START A RENAMING JOB ON A HUGE DOWNLOAD FOLDER?

Post by ByteJunk »

Yeah, I've been playing with utorrent-postprocess now, but it seems that anime detection in this one is better. I only glanced at it so far, but although there's something in there for anidb lookup, it never gets used. I'll have a more thorough look latter and page you if I can figure it out. :D

Thanks!
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: HOW TO START A RENAMING JOB ON A HUGE DOWNLOAD FOLDER?

Post by rednoah »

Anime-Mode is in there but it can only be activated by forcing it via --def ut_label=anime ;)

Anime<->TV Series differentiation is pretty much impossible for obvious reasons. Good solution anyway. You can probably make it work for most cases just be checking if there is a CRC32 in the filename. :D

Having a quick glance as this old script here, I guess I was just doing: IF (is in AniDB) && NOT(is in TheTVDB) THEN process as anime... works for the Romanji names but not the popular One Piece, Naruto, etc
:idea: Please read the FAQ and How to Request Help.
ByteJunk
Posts: 5
Joined: 17 Jan 2013, 12:58

Re: HOW TO START A RENAMING JOB ON A HUGE DOWNLOAD FOLDER?

Post by ByteJunk »

Yeah, I haven't bothered much with labels. I call filebot through a shell script that I have running as a daemon, monitoring new files moved to a "completed" folder by transmission-daemon. It's working quite well, except for the odd anime file.

Matching ed2k hashes with anidb would be overkill? It would eliminate all false positives, and since it's a rather complete DB, the false negatives would be quite low as well. MD4 hashing all those chunks would imply a sizable performance hit though...
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: HOW TO START A RENAMING JOB ON A HUGE DOWNLOAD FOLDER?

Post by rednoah »

Actually I just meant checking the filename for [12345678] crc32 checksum patterns. Filename with CRC32 => Anime, without => TV Series. Everything else would be overkill. :P

e.g.

Code: Select all

def forceAnime(f) {
	tryQuietly{ ut_label } =~ /^(?i:Anime)/ || f.name =~ "[\\(\\[]\\p{XDigit}{8}[\\]\\)]"
}
EDIT: That part of the AniDB API only works with authenticated users, so that'd be crappy for the works-out-of-the-box emphasis i've put on these scripts.
:idea: Please read the FAQ and How to Request Help.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: HOW TO START A RENAMING JOB ON A HUGE DOWNLOAD FOLDER?

Post by rednoah »

Better Idea:

Code: Select all

def forceAnime(f) {
	tryQuietly{ ut_label } =~ /^(?i:Anime)/ || (f.isVideo() && (f.name =~ "[\\(\\[]\\p{XDigit}{8}[\\]\\)]" || getMediaInfo(file:f, format:'''{media.AudioLanguageList} {media.TextCodecList}''').tokenize().containsAll(['Japanese', 'ASS'])))
}
If it has a Japanese audio track and an Advanced Substation Alpha subtitle track one can be pretty sure it's Anime. Gonna add it this way to utorrent-postprocess.
:idea: Please read the FAQ and How to Request Help.
Post Reply