Scrape metadata and build NFO

[frobberio]

New to Filebot here and a bit overwhelmed by all the features !

I would like to do the following:

Given a folder that contains (1) a single movie (MKV, AVI, or VIDEO_TS subfolder), and (2) a stub NFO file that just contains the IMDB or TMDB id, e.g.:

Code: Select all

<?xml version="1.0" encoding="UTF-8" ?>
<movie>
  <id>tt00000000</id>
  <original_filename>MyMovie.mkv</original_filename>
</movie>

...scrape all the metadata from IMDB or TMDB, fetch artwork, extract mediainfo, and then overwrite the stub with a complete NFO file.

The folder doesn't necessarily need to be renamed.

Is it possible to do some or all of this?

TIA !!

[rednoah]

FileBot does what you want by default, for the purpose of organising files that is. FileBot cannot not rename / move / etc files, so you will want to organise your files into a new file structure as usual and just tick the desired post-processing features along the way.


FileBot does parse and use IDs from nearby *.nfo files by default.


FileBot can generate fetch artwork, generate *.nfo files, etc as a post-processing step.





You can only select video files for processing by only selecting video files. See How do I process only specific kinds of files? on how to do that.


Let me now how it goes and if you have any questions along the way.

[frobberio]

Thanks !!

Actually, I'd prefer not to rename the movie file, though moving it into a new folder would be very helpful. I guess(?) there's some way to define the pattern for a new folder name?

Also, I didn't express myself well, but really I'd like to do all of this from the command line, so I can make it part of a script.

Any tips on doing that?

[rednoah]

I'd prefer not to rename the movie file, though moving it into a new folder would be very helpful. I guess(?) there's some way to define the pattern for a new folder name?

What are the file names you have? Why do you want to keep them? Why not create a custom format the generates the file name pattern you want?

The target file path is entirely up to your custom format. You can create a custom format that generates a new file path based on the old file path, i.e. keep the file name. This is generally not a good idea, but it's up to you.

Format: Select all

Movies/{ ny }/{ fn }

You'll definitely want to use the Format Editor in the GUI first if you're not yet familiar with custom formats.

I'd like to do all of this from the command line, so I can make it part of a script.

The same concept applies:


Use the CLI to process files as usual:

Shell: Select all

filebot -rename -r "/input" --db TheMovieDB -non-strict --action duplicate --output "/output" --format "{plex.id}"

Add Apply Post-Processing Features via the --apply option from Terminal to your command-line call as desired:

Shell: Select all

--apply cover artwork nfo

You can only select video files via Start FileBot with --file-filter from Terminal if you want to exclude the existing *.nfo files from processing:

Shell: Select all

--file-filter f.video

[frobberio]

Thanks for kindly pointing the way. OK, I bought a license so I can proceed.

A couple of questions, if you don't mind...

(1) I really prefer not to change the original filename of the actual movie files. Instead, I want to just copy the contents of the source folder (i.e., including an original MKV file or VIDEO_TS folder, possibly an /Extras folder, etc.) into a newly-created output movie folder, and then give that new folder a custom name (e.g., --format "{n} ({y}) -- {director} [{vf} {lang}]"). If possible, the NFO and artwork files should just follow the same pattern as the original file name, if this makes sense.

I guess my first question, then, is how to do this in the command line. I see there are bindings like source and original but I don't yet understand which one I want, or how to get Filebot to just copy the contents of the existing folder into a new folder with a custom name.

(2) Another thing I'm wondering about is how to get mention of subtitles into the NFO file. I guess if there are subs embedded in the MKV file, that Filebot will find them via the mediainfo (is that right?), but there may also be SRT subs in the same folder, and maybe hardsubs are mentioned in the stub NFO — can all of those be copied/listed in the output NFO file ?

Thanks again !

[rednoah]

e.g.

Console Output: Select all

$ filebot -rename -r "INPUT" --db TheMovieDB -non-strict --action duplicate --output "OUTPUT" --format "Movies/{ ny }/{ fn }" --apply cover artwork nfo import
Rename movies using [TheMovieDB]
Auto-detect movie from context [INPUT/avatar.mp4]
Auto-detect movie from context [INPUT/avatar.nfo]
[DUPLICATE] from [INPUT/avatar.mp4] to [OUTPUT/Movies/Avatar (2009)/avatar.mp4]
[DUPLICATE] from [INPUT/avatar.eng.srt] to [OUTPUT/Movies/Avatar (2009)/avatar.eng.srt]
[DUPLICATE] from [INPUT/avatar.nfo] to [OUTPUT/Movies/Avatar (2009)/avatar.nfo]
[COVER] https://image.tmdb.org/t/p/original/kyeqWdyUXW608qlYkRqosgbbJyK.jpg (OUTPUT/Movies/Avatar (2009)/folder.jpg)
[ARTWORK] https://image.tmdb.org/t/p/original/6EiRUJpuoeQPghrs3YNktfnqOVh.jpg (OUTPUT/Movies/Avatar (2009)/poster.jpg)
[ARTWORK] https://image.tmdb.org/t/p/original/vL5LR6WdxWPjLPFRLe133jXWsh5.jpg (OUTPUT/Movies/Avatar (2009)/backdrop.jpg)
[ARTWORK] https://assets.fanart.tv/fanart/movies/19995/hdmovieclearart/avatar-51a3a817f375b.png (OUTPUT/Movies/Avatar (2009)/clearart.png)
[ARTWORK] https://assets.fanart.tv/fanart/movies/19995/hdmovielogo/avatar-51290d0678d4e.png (OUTPUT/Movies/Avatar (2009)/logo.png)
[ARTWORK] https://assets.fanart.tv/fanart/movies/19995/moviedisc/avatar-520968c2dde7b.png (OUTPUT/Movies/Avatar (2009)/disc.png)
[NFO] 19995 (OUTPUT/Movies/Avatar (2009)/movie.nfo)
[IMPORT] Duplicating Behind the Scenes.mp4 (OUTPUT/Movies/Avatar (2009)/Extras/Behind the Scenes.mp4)
Processed 3 files

e.g. Movies/Avatar (2009)/movie.nfo





I've added --apply import because you want to import companion files such as Extras folders as well:

Shell: Select all

--apply import

Console Output: Select all

[IMPORT] Duplicating Behind the Scenes.mp4 (OUTPUT/Movies/Avatar (2009)/Extras/Behind the Scenes.mp4)

--apply nfo generates nfo files based on movie information and mediainfo information only meaning that external *.srt files will not be listed in the NFO file.

[frobberio]

Thanks. Getting closer !!

When I try this, I now get two NFO files: (1) my stub NFO, e.g., Avatar (2009).nfo, and (2) the newly-generated movie.nfo. Is there a way to output a single, new NFO file called Avatar (2009).nfo that contains all the newly-scraped info?

Next, you mention that external *.srt files won't be listed in the NFO file — so that means I should list them in my stub NFO, to get them copied into the final NFO? Or, is there a way for Filebot to detect SRT files and list them in the final NFO?

I think this is really going to help a LOT to tame my media library.

[rednoah]

When I try this, I now get two NFO files: (1) my stub NFO, e.g., Avatar (2009).nfo, and (2) the newly-generated movie.nfo. Is there a way to output a single, new NFO file called Avatar (2009).nfo that contains all the newly-scraped info?

You can use --file-filter to only process video files:

Shell: Select all

--file-filter "f.video"

You can use --file-filter to only process video files and subtitle files:

Shell: Select all

--file-filter "f.video || f.subtitle"

Unfortunately, in your specific case, you are also doing --apply import so that will probably copy along the *.nfo file that would otherwise have been left behind. If you were to rename all your nfo files to movie.nfo then --apply import will not copy them along because --apply import does not overwrite existing files, I suppose. You could also just delete all the *.nfo files that were not generated by FileBot in a second step.

Next, you mention that external *.srt files won't be listed in the NFO file — so that means I should list them in my stub NFO, to get them copied into the final NFO?

--apply nfo generates nfo files based on movie information and mediainfo information only also meaning that your stub NFO file has no effect whatsoever on the newly generated NFO file.

Or, is there a way for Filebot to detect SRT files and list them in the final NFO?

I cannot think of an easy way to do that.

What is your use case? We include mediainfo in the NFO file because reading an NFO is much much easier / faster than reading mediainfo from some mediafile. Listing *.srt files in a folder is even more easier / faster than than reading a NFO file. Suppose you could include external subtitle information (Example Code?) in your NFO file, would that information actually be read / used by anyone or anything?

[frobberio]

Thank you for kindly explaining these Filebot features. I'm getting a better understanding now.

To try to speak to your question: my use case is that I have a collection of movies in a Kodi media library (various formats, e.g.: MKV, AVI, VIDEO_TS, ISO, TS), and I want complete info in NFO files, especially about the subs already in these movie files/folders. The subs are either inside the MKV files, or VobSub, or external SRT/ASS files, and some are hardsubbed. I have a spreadsheet with IMDB/TMDB ids for 90% of these, and notes for those with hardsubs. For this, I can generate NFO stubs.

Why is this necessary? Because: Kodi itself won't parse VIDEO_TS or ISO (too slow), so it cannot "know" what all the subs are. So, I need another app to do that, probably using NFOs, and this is how I started looking at Filebot. If I can build complete info in NFOs, I can import it to Kodi.

I can make a script to rename/move the NFO files as you suggest, but I am still unclear on a couple of points.

--apply nfo generates nfo files based on movie information and mediainfo information only also meaning that your stub NFO file has no effect whatsoever on the newly generated NFO file.

Okay, maybe I was on the wrong track here. Is there a situation in which Filebot can read a stub NFO file, or do you mean that it will never ever read a NFO? How is this handled?

For example, let's say I have Chaos (2005), directed by David DeFalco, and not Chaos (2005), directed by Tony Giglio. Same title/year, but different films — so how does Filebot know which film to scrape from IMDB?

I was thinking to use a stub NFO to provide the IMDB id (e.g., tt0405977) to disambiguate the film.

Regarding your next question about the external subtitle information, the answer is: if I can get that into a NFO file, then yes, I have a Kodi skin that will report it. If Filebot has no provision for noticing the SRT files, I think I could work around that, by adding the info to the stub NFO (tho again, this depends on whether Filebot will read/process NFOs, and not simply write them).

Hope these questions make sense — thanks !

[rednoah]

Why is this necessary? Because: Kodi itself won't parse VIDEO_TS or ISO (too slow), so it cannot "know" what all the subs are.

MediaInfo may work on the video files inside a VIDEO_TS folder but I'm pretty sure that MediaInfo does not work for ISO files. You definitely want to extract disk images into disk folders. FileBot (both GUI and CLI) can help you batch extract ISO images.

Is there a situation in which Filebot can read a stub NFO file, or do you mean that it will never ever read a NFO? How is this handled?

FileBot will read nearby *.nfo files for the sake of identifying a movie. So FileBot will read the *.nfo file, check the TMDB / IMDB ID, and if that ID is valid, match the movie file to that movie ID.

At a much later stage, after renaming, the --apply nfo post-process is given the movie details / media info table and generates an NFO file from that. The --apply nfo post-process does not know or care how the movie file at hand was originally identified and does not take into account any *.nfo files that may or may not be around in the source folder or target folder.

For example, let's say I have Chaos (2005), directed by David DeFalco, and not Chaos (2005), directed by Tony Giglio. Same title/year, but different films — so how does Filebot know which film to scrape from IMDB?

FileBot does not use or scrape IMDB. If FileBot cannot reliably identify a movie, then FileBot will make a best guess as per the -non-strict option. The GUI will likely ask you to confirm one or the other. If the nearby *.nfo file already identifies the movie, then FileBot will probably just go with that.

Regarding your next question about the external subtitle information, the answer is: if I can get that into a NFO file, then yes, I have a Kodi skin that will report it. If Filebot has no provision for noticing the SRT files, I think I could work around that, by adding the info to the stub NFO (tho again, this depends on whether Filebot will read/process NFOs, and not simply write them).

Which Kodi skin? Do Kodi skins have access to the NFO file but not file listings? Maybe the skin could check NFO and *.srt files and then display that information?

FileBot, by design, does not include external subtitle information in NFO files. If your skins supports multiple NFO files that would make things easier. Then you can just generate an extra custom NFO file. If you need to merge the FileBot-generated NFO and your stub NFO that is probably possible, but you'd need to write the code yourself, or at least find a command-line tool to do it for you.

[frobberio]

That all sounds good, thanks !

You're right about converting ISO files. I have been using 7x for that, but I should do all of them.

When you say FileBot doesn't use/scrape IMDB, do you mean it primarily uses TMDB instead?

For the Kodi skin, I have been trying several. I've hacked Estuary before, so I may do that again to get the functionality that I want. AFAIK, Kodi skins do not read NFOs directly, but rather I must use the NFO to import the correct data into Kodi's own SQLite database. I need to investigate the specifics, but the main thing is that Filebot can help to generate these NFO files.

Finally, I will look at the custom NFO file option you mention, but if it is necessary merge NFOs to incorporate info on external SRT subs or else hardsubs, I can write code to do that.

[rednoah]

FileBot does indeed use TMDB only, even for looking up IMDB IDs.



I made a Custom Post-Processing Script for splicing extra information into the NFO file:

Shell: Select all

--apply /path/to/extend-nfo-file.groovy

Groovy: Select all

def nfo = args.dir.files.find{ it.name == 'movie.nfo' }
if (nfo) {
	def subtitles = nfo.dir.files.findAll{ it.subtitle }
	if (subtitles) {
		def dom = new groovy.xml.XmlSlurper().parse(nfo)
		subtitles.each{ f ->
			dom << {
				fileinfo {
					name(f.name)
					streamdetails {
						subtitle {
							codec(f.name.tokenize('.')[-1])
							language(f.name.tokenize('.')[-2])
						}
					}
				}
			}
		}
		groovy.xml.XmlUtil.serialize(dom).saveAs(nfo)
	}
}

xml: Select all

<movie>
  ...
  <fileinfo>
    <name>Avatar (2009).eng.srt</name>
    <streamdetails>
      <subtitle>
        <codec>srt</codec>
        <language>eng</language>
      </subtitle>
    </streamdetails>
  </fileinfo>
</movie>

[frobberio]

Thank for kindly preparing this !

I will add it as the last parameter of --apply and try it out.

[rednoah]

I did some prototyping for this use case and added a few API extensions to make things easier. The following code requires r10496 or higher:

Groovy: Select all

args.dir.getFiles{ it.name == 'movie.nfo' }.each{ nfoFile ->
	def dom = nfoFile.xml
	nfoFile.dir.getFiles{ it.subtitle }.each{ subtitleFile ->
		dom.fileinfo.streamdetails << {
			subtitle(file: subtitleFile.name) {
				codec(subtitleFile '{ext}')
				language(subtitleFile '{lang}')
			}
		}
	}
	dom.saveAs(nfoFile)
}

[frobberio]

OK, I will try to upgrade to that version, thanks!

Meanwhile, I am confused about something related — evidently I am doing something wrong.

As you mentioned, ISO files should be unpacked into VIDEO_TS folders. Agree this is a good idea.

After unpacking, I have a folder, e.g., /Latest/MyMovie (2021) -- Some Director that contains a VIDEO_TS subfolder, and I put a stub NFO into the VIDEO_TS folder. The stub NFO is XML that just contains a TMDB id.

And run it through Filebot, like this:

Code: Select all

filebot -rename -r "/Latest/MyMovie (2021) -- Some Director" --db TheMovieDB -non-strict --action duplicate --output "/Movies" --format "{ ny } -- {director} [{vf}]/{ fn }" --apply cover artwork nfo import

Filebot says: Auto-detect movie from context ... for each file in the VIDEO_TS folder, but then also Stripping invalid characters from the output file path ... but with a different and seemingly-random movie name, and then it copies the video files into three(1) different folders, randomly splitting the original movie into three different new folders, and it doesn't copy the folder structure, i.e., the video files just get copied to the new folder, and are no longer in a VIDEO_TS sub-folder.

I may not be describing this very well, but clearly I'm doing something wrong.

What pattern of parameters should I use to copy a movie folder that contains a VIDEO_TS sub-folder?

[rednoah]

The expected folder structure looks like this:

Code: Select all

Movie.mkv <-- movie file
Movie.nfo

Code: Select all

Movie.iso <-- disk folder as image file
Movie.nfo

Code: Select all

Movie <-- disk folder as folder structure (must contain a VIDEO_TS subfolder)
Movie.nfo

EDIT: I lied. This is wrong. The *.nfo file is expected to be inside the folder in this particular case.

FileBot likely doesn't search for identifying *.nfo files inside a disk folder structure, just like it doesn't search for *.nfo files inside *.iso images or *.mkv files. Few people have this particular use case so I'd need to run tests myself to see what is and isn't happening.



EDIT:

I stand corrected. If you are processing disk folders then FileBot will expect the *.nfo file inside the disk folder top-level folder next to the VIDEO_TS subfolder:

Code: Select all

Avatar/VIDEO_TS
Avatar/*.nfo

EDIT 2:

Actually, putting the *.nfo file into the VIDEO_TS folder works as well:

Code: Select all

Avatar/VIDEO_TS/*.nfo

The issue at hand may be unrelated. Please paste console output so that we can see what is actually happening. Please provide sample file paths as text so that we can reproduce the issue.

[frobberio]

OK, I just tried that (i.e., putting the NFO file where you suggest), but I'm getting "Movie not found" errors for each file in the VIDEO_TS folder.

My Filebot command line looks like this:

Code: Select all

filebot -rename -r "/Latest/MyMovie (2021) -- Some Director" --db TheMovieDB -non-strict --action duplicate --output "/Movies" --format "{ ny } -- {director} [{vf}]/{ fn }" --apply cover artwork nfo import

...so I must still be doing something wrong.

Output looks like this:

Rename movies using [TheMovieDB]
Auto-detect movie from context [/Users/me/Latest/MyMovie (2021) -- Some Director/VIDEO_TS/VTS_01_0.VOB]
Movie not found: /Users/me/Latest/MyMovie (2021) -- Some Director/VIDEO_TS/VTS_01_0.VOB
Auto-detect movie from context [/Users/me/Latest/MyMovie (2021) -- Some Director/VIDEO_TS/VTS_01_1.VOB]
Movie not found: /Users/me/Latest/MyMovie (2021) -- Some Director/VIDEO_TS/VTS_01_1.VOB
Auto-detect movie from context [/Users/me/Latest/MyMovie (2021) -- Some Director/VIDEO_TS/VTS_02_0.VOB]
Movie not found: /Users/me/Latest/MyMovie (2021) -- Some Director/VIDEO_TS/VTS_02_0.VOB
...

[rednoah]

Please paste the output verbatim so that I can copy and paste the file path and run tests. If you obfuscate the output with "MyMovie" then I cannot run tests. If you have privacy concerns about copy & pasting console output verbatim, then you can also PM me the console output.


Lets use the following real file paths for testing:

Console Output: Select all

$ filebot -find .
Avatar/VIDEO_TS/VTS_01_0.VOB
Avatar/VIDEO_TS/VTS_01_1.VOB
Avatar/VIDEO_TS/VTS_02_0.VOB
Avatar/Avatar.nfo

That fails immediately:

Console Output: Select all

$ filebot -rename -r . --db TheMovieDB --output "/Movies" --format "{ ny } -- {director} [{vf}]/{ fn }" --action TEST
Rename movies using [TheMovieDB]
Auto-detect movie from context [Avatar/VIDEO_TS/VTS_01_0.VOB]
Name (Year) movie pattern not found: Avatar/VIDEO_TS/VTS_01_0.VOB
* Consider using -non-strict to enable opportunistic matching
Auto-detect movie from context [Avatar/VIDEO_TS/VTS_01_1.VOB]
Name (Year) movie pattern not found: Avatar/VIDEO_TS/VTS_01_1.VOB
* Consider using -non-strict to enable opportunistic matching
Auto-detect movie from context [Avatar/VIDEO_TS/VTS_02_0.VOB]
Name (Year) movie pattern not found: Avatar/VIDEO_TS/VTS_02_0.VOB
* Consider using -non-strict to enable opportunistic matching
Stripping invalid characters from the output file path: /Movies/Avatar: The Way of Water (2022) -- James Cameron []/Avatar.nfo
[TEST] from [Avatar/Avatar.nfo] to [/Movies/Avatar The Way of Water (2022) -- James Cameron []/Avatar.nfo]
Processed 1 file

Looks like -r is working against you in this case. You process the *.VOB files but you want to process the Avatar disk folder rater than the files inside.


If you use the GUI then FileBot will add Avatar as a single item by default rather than load all the files:




The -d option allows you to process a folder as if it was a file which is what we want to do when processing a disk folder as a single unit:

Console Output: Select all

$ filebot -rename -d * --db TheMovieDB --output "/Movies" --format "{ ny } -- {director} [{vf}]/{ fn }" --action TEST
* Consider using --mode interactive to enable interactive mode
Rename movies using [TheMovieDB]
Stripping invalid characters from the output file path: /Movies/Avatar: The Way of Water (2022) -- James Cameron [2160p]/Avatar
[TEST] from [Avatar] to [/Movies/Avatar The Way of Water (2022) -- James Cameron [2160p]/Avatar]
Processed 1 file

NOTE: You cannot (easily) combine -d and -r for obvious reasons, e.g. -r -d --file-filter "f.children.name =~ /VIDEO_TS/" would select the parent folders of the VIDEO_TS folders. But it's getting pretty complicated.



If your disk folders are all in the same folder, then it's done. However, if everything is mixed, then you may need to find a way to call different commands for different files. That is one of the things that the amc script does:

Console Output: Select all

$ filebot -script fn:amc . --output "/" --def movieFormat="Movies/{ ny } -- {director} [{vf}]/{ fn }" --action TEST
...
Disk Folder: Movies/Avatar
...
[TEST] from [Movies/Avatar] to [/Movies/Avatar The Way of Water (2022) -- James Cameron [2160p]/Avatar]
...

NOTE: The amc script adds a lot of complexity for all kinds of things. Disk folder detection is only a small part. The additional complexity may or may not be worth it depending on your specific use case.






Unrelated to any of the above, the "Stripping invalid characters from the output file path" message just means that FileBot is removing the : colon characters from the file path. See [DOCS] Target Unix Filesystem for details.

Code: Select all

Stripping invalid characters from the output file path: /Movies/Avatar: The Way of Water (2022) -- James Cameron [2160p]/Avatar

[frobberio]

Sorry, I'm getting all confused by the . and * inputs.

It's becoming too complicated. Tbh, all I need is something to generate the NFO file.

I'm using Kodi, not Plex, so I don't think the amc script is going to help.

For the input, can I just specify a full path to a folder, like a usual Linux program ? E.g.:

Code: Select all

ls "/Users/me/Latest/MyMovie (2021) -- Some Director"

...and then inside this folder, there is a VIDEO_TS subfolder.

Thanks.

[rednoah]

Sorry, I'm getting all confused by the . and * inputs. It's becoming too complicated.

. is the current working directory. * is all the files / folders in the current working directory. That's a shell thing, not a FileBot thing. You can always paste the actual file paths instead.

e.g. all of these commands do the same thing:

Shell: Select all

filebot -rename -d "/Users/me/Latest/MyMovie (2021) -- Some Director" --db TheMovieDB

Shell: Select all

cd "/Users/me/Latest/MyMovie (2021) -- Some Director"
filebot -rename -d . --db TheMovieDB

Shell: Select all

cd "/Users/me/Latest/MyMovie (2021) -- Some Director/VIDEO_TS"
filebot -rename -d .. --db TheMovieDB

Shell: Select all

cd "/Users/me/Latest"
filebot -rename -d * --db TheMovieDB

NOTE: * does the exact same thing if and only if "Latest" contains only "MyMovie (2021) -- Some Director" and no other files / folders. Otherwise it does a bit more.

I'm using Kodi, so I don't think the amc script is going to work.

Per se, one has nothing to do with the other. Many people use the amc script to organize files for Kodi. The amc script merely scripts different filebot -rename commands for different sets of files (which is handy when you need to do different things for different files) and lots of things you might want to do afterwards, such as sending Kodi a request to rescan its library for new files.