Media Index CSV

Running FileBot from the console, Groovy scripting, shell scripts, etc
Post Reply
seequcn
Posts: 17
Joined: 25 May 2012, 01:13

Media Index CSV

Post by seequcn »

I tried running this Groovy script over a directory with 1000 SD avi movies (from http://filebot.sourceforge.net/forums/v ... =4&t=5#p53:
rednoah wrote:Description:
  1. Write media info of all video files in the given folder and subfolders to an easily readable CSV file (e.g. open in Excel)
Usage:

Code: Select all

filebot -script "http://filebot.sf.net/scripts/mi.groovy" -trust-script /path/to/media/ "MediaIndex.csv"
Download Script:
http://filebot.sf.net/scripts/mi.groovy
Unfortunately the only fields that were printed in the csv were the filename and the directory path. All the others were blankety-blank.

Does the script not work with .avi files?

Also, I thought it would be wonderful to have a script to obtain movie information from IMDb. As I'm endeavoring to find a way to have an automated 100% fool-proof file-renaming workflow, having the duration of the movie in IMDb + the actual movie file duration in the file would be a great reference point to find errors.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Media Index CSV

Post by rednoah »

Does MediaInfo work in your setup? If u just added the jar you need to add the MediaInfo native lib somewhere in the library path. Since you can't install the libmediainfo package you have to just get the .so and then set jna.library.path when running java. Check out the filebot.sh startup script for the Debian package in SVN as reference.
:idea: Please read the FAQ and How to Request Help.
seequcn
Posts: 17
Joined: 25 May 2012, 01:13

Re: Media Index CSV

Post by seequcn »

This is the original filebot.sh file:-

Code: Select all

#!/bin/bash
SOURCE="${BASH_SOURCE[0]}"
while [ -h "$SOURCE" ] ; do SOURCE="$(readlink "$SOURCE")"; done
dir_bin="$( cd -P "$( dirname "$SOURCE" )" && pwd )"

# WARNING: NOT TESTED / HERE THERE BE DRAGONS
java -Dunixfs=false -Xmx256m -Dapplication.deployment=portable "-Dapplication.dir=$dir_bin" "-Duser.home=$dir_bin" "-Djna.library.path=$dir_bin" "-Djava.library.path=$dir_bin" -Djava.util.prefs.PreferencesFactory=net.sourceforge.tuned.prefs.FilePreferencesFactory -Dnet.sourceforge.tuned.prefs.file=prefs.properties -jar "$dir_app/FileBot.jar" "$@"
I changed it to:

Code: Select all

#!/bin/bash
SOURCE="${BASH_SOURCE[0]}"
while [ -h "$SOURCE" ] ; do SOURCE="$(readlink "$SOURCE")"; done
dir_bin="$( cd -P "$( dirname "$SOURCE" )" && pwd )"

# WARNING: NOT TESTED / HERE THERE BE DRAGONS
java -Dunixfs=false -Xmx256m -Dapplication.deployment=portable "-Dapplication.dir=/media/sde1/home/seequcn/filebot/" "-Duser.home=/media/sde1/home/seequcn/filebot/" "-Djna.library.path=/media/sde1/home/seequcn/filebot/" "-Djava.library.path=/media/sde1/home/seequcn/filebot/" -Djava.util.prefs.PreferencesFactory=net.sourceforge.tuned.prefs.FilePreferencesFactory -Dnet.sourceforge.tuned.prefs.file=prefs.properties -jar "/media/sde1/home/seequcn/filebot/FileBot.jar" "$@"
but when running the script I still get the following messages / error:

Code: Select all

May 28, 2012 12:18:18 PM net.sourceforge.tuned.prefs.FilePreferencesFactory getPreferencesFile
INFO: Preferences file is /media/sde1/home/seequcn/filebot/prefs.properties
May 28, 2012 12:18:21 PM net.sourceforge.filebot.mediainfo.MediaInfo <clinit>
WARNING: Failed to preload libzen
I compiled libzen.so according to the instructions from Eterimos at http://filebot.sourceforge.net/forums/v ... ainfo#p886

Am I configuring the filebot.sh file incorrectly? Or is there something else that I should configure?

I placed all the libraries that were compiled into the same directory as FileBot.jar, namely:

-rw------- 1 seequcn seequcn 915 May 28 13:12 libmediainfo.la
-rw------- 1 seequcn seequcn 916 May 28 13:12 libmediainfo.lai
-rwx------ 1 seequcn seequcn 6.6M May 28 13:12 libmediainfo.so
-rwx------ 1 seequcn seequcn 6.6M May 28 13:12 libmediainfo.so.0
-rwx------ 1 seequcn seequcn 6.6M May 28 13:12 libmediainfo.so.0.0.0
-rw------- 1 seequcn seequcn 5.1M May 28 13:08 libzen.a
-rw------- 1 seequcn seequcn 794 May 28 13:08 libzen.la
-rw------- 1 seequcn seequcn 795 May 28 13:08 libzen.lai
-rwx------ 1 seequcn seequcn 2.1M May 28 13:08 libzen.so
-rwx------ 1 seequcn seequcn 2.1M May 28 13:08 libzen.so.0
-rwx------ 1 seequcn seequcn 2.1M May 28 13:08 libzen.so.0.0.0

Help!
seequcn
Posts: 17
Joined: 25 May 2012, 01:13

Re: Media Index CSV

Post by seequcn »

Installing mediainfo manually on a seedbox without root access or any support from the seedbox provider is proving too difficult for me. I give up! Especially when everything works out of the box perfectly on my own computer.

The Media Index CSV script is very very handy, would it be possible to add some more info from the IMBb database?

I would so love to have the IMDb recorded movie duration time to compare to the movie in my collection to have a numeric benchmark to check for accuracy. Any possibility of adding such info to the script?

Thanks!!
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Media Index CSV

Post by rednoah »

jna.library.path points at the folder with the .so files. Should work. Also you compiled yourself so arch shouldn't be a problem. Well, maybe java is running in 32bit compability mode but the native lib is 64bit? That's a common Problem on Windows.

That script doesn't do any auto-detection for the movie so the script is simple. I can do that using internal filebot stuff quite easily. But detectMovie() and tmdb.getMovieInfo(movie) should already do most of the work.
:idea: Please read the FAQ and How to Request Help.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Media Index CSV

Post by rednoah »

This will grab all the movie info. Should give you all the metainfo you'll need.

Script:

Code: Select all

args.getFiles{ it.isVideo() }.each{
	def movie = detectMovie(it)
	println movie
	def info = TheMovieDB.getMovieInfo(movie, Locale.ENGLISH)
	println info
}
Output:

Code: Select all

Avatar (2009)
{translated=true, adult=false, language=en, original_name=Avatar, name=Avatar, type=movie, id=19995, imdb_id=tt0499549, url=http://www.themoviedb.org/movie/19995, overview=Set in the mid-22nd century, at a time when humans have depleted Earth's resources, and are mining lucrative unobtanium on Pandora, a lush moon in the Alpha Centauri star system. When his twin brother dies, disabled Marine Jake Sully travels to Pandora in his place to become an Avatar, an alien-human hybrid body electronically linked to a genetically matched human, to integrate himself with the native Na'vi and help the mining effort. But he finds himself caught in between an interstellar conflict after falling for Na'vi warrior Neytiri., votes=500, rating=7.9, tagline=Enter the World of Pandora., certification=PG-13, released=2009-12-18, runtime=162}
:idea: Please read the FAQ and How to Request Help.
kiloback
Posts: 10
Joined: 11 Nov 2012, 05:57

Re: Media Index CSV

Post by kiloback »

hi about media index csv how to get the last date modified to display in this format "MM/dd/YYYY HH:mm", because the file.LastModified() only give integers
should i use date function? a sample code could really help
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Media Index CSV

Post by rednoah »

Should be something like this, just from the back of my head:

Code: Select all

new Date(file.lastModified()).dateTimeString
@see
http://docs.oracle.com/javase/7/docs/ap ... /Date.html
http://groovy.codehaus.org/groovy-jdk/j ... /Date.html
:idea: Please read the FAQ and How to Request Help.
kiloback
Posts: 10
Joined: 11 Nov 2012, 05:57

Re: Media Index CSV

Post by kiloback »

thanks rednoah i got it working with this

Code: Select all

new Date(file.lastModified()).format("MM/dd/yyyy hh:mm:ss a")
but encountered a little bug.I have a media index csv script within my utorrent-postprocess script and it seems getMediaInfo(file,format) doesn't work great with Utorrent because it wont return Container,Resolution,Video Codec,Video Format,Audio Codec,Audio Format,Audio Language(s),Duration of the file though it works perfectly fine in cmd.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Media Index CSV

Post by rednoah »

Do you have 32 and 64 bit Java installed at the same time? If you installed 64 bit FileBot but for some reason run it with 32 bit Java, then native code can't work.
:idea: Please read the FAQ and How to Request Help.
kiloback
Posts: 10
Joined: 11 Nov 2012, 05:57

Re: Media Index CSV

Post by kiloback »

hey is there a run around with this

Code: Select all

16:25:49.302 Updating List...
16:27:17.738 OutOfMemoryError: PermGen space
16:27:17.738 NullPointerException: java.lang.NullPointerException
16:27:18.534 java.lang.NullPointerException
16:27:18.534 	at Script3$_run_closure25_closure61.doCall(Script3.groovy:296)
16:27:18.534 	at Script3$_run_closure25.doCall(Script3.groovy:290)
16:27:18.534 	at Script3.run(Script3.groovy:287)
16:27:18.534 	at net.sourceforge.filebot.cli.ScriptShell.evaluate(Unknown Source)
16:27:18.534 	at net.sourceforge.filebot.cli.ScriptShell.runScript(Unknown Source)
16:27:18.534 	at net.sourceforge.filebot.cli.ArgumentProcessor.process(Unknown Source)
16:27:18.534 	at net.sourceforge.filebot.Main.main(Unknown Source)
16:27:18.534 Failure (°_°)
16:27:19.985 Exception in thread "Thread-5" java.lang.OutOfMemoryError: PermGen space
16:27:19.985 	at com.sun.xml.internal.bind.v2.ContextFactory.createContext(Unknown Source)
16:27:19.985 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
16:27:19.985 	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
16:27:19.985 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
16:27:19.985 	at java.lang.reflect.Method.invoke(Unknown Source)
16:27:19.985 	at javax.xml.bind.ContextFinder.newInstance(Unknown Source)
16:27:19.985 	at javax.xml.bind.ContextFinder.newInstance(Unknown Source)
16:27:19.985 	at javax.xml.bind.ContextFinder.find(Unknown Source)
16:27:19.985 	at javax.xml.bind.JAXBContext.newInstance(Unknown Source)
16:27:19.985 	at javax.xml.bind.JAXBContext.newInstance(Unknown Source)
16:27:19.985 	at net.sourceforge.filebot.History.importHistory(Unknown Source)
16:27:19.985 	at net.sourceforge.filebot.HistorySpooler.getCompleteHistory(Unknown Source)
16:27:19.985 	at net.sourceforge.filebot.HistorySpooler.commit(Unknown Source)
16:27:19.985 	at net.sourceforge.filebot.HistorySpooler$1.run(Unknown Source)
16:27:19.985 	at java.lang.Thread.run(Unknown Source)
it doesn't print all media files

and here is the code

Code: Select all

	//*/
 //* Print media info of all video files to CSV file  
 //*/
 def f = "E:/Some Videos"
 def h = "E:/Some Movies"
 def e = new File('E:/series list.csv')
 def gg = new File('E:/movie list.csv')
 def filesize = { attrs ->
labels = [ ' bytes', 'KB', 'MB', 'GB', 'TB' ]
size = attrs.size
label = labels.find {
if( size < 1024 ) {
true
}
else {
size /= 1024  
false
}
}
"${new java.text.DecimalFormat( attrs.format ?: '0.##' ).format( size )}$label"
}

def model = 'Filename;Filesize;Last Date Modified;Location;Container;Resolution;Video Codec;Video Format;Audio Codec;Audio Format;Audio Language(s);Duration'
def template = '{file.name};'
def template2 = ';{new Date(file.lastModified()).format("MM/dd/yyyy hh:mm:ss a")};{file.getParent()};{cf};{resolution};{vc};{vf};{ac};{af};{media.AudioLanguageList};{media.DurationString3}'

e.withWriter{ output ->
	output.writeLine(model)
	
	f.getFiles{ it.isVideo() }.sort{ a, b -> a.name.compareToIgnoreCase(b.name) }.each{
		def fz = (filesize(size:it.length()))
		def mi = getMediaInfo(file:it, format:template+fz+template2)
		
		
		// append to file
		output.writeLine(mi)
	}
}
gg.withWriter{ output ->
	output.writeLine(model)
	
	h.getFiles{ it.isVideo() }.sort{ a, b -> a.name.compareToIgnoreCase(b.name) }.each{
		def fz = (filesize(size:it.length()))
		def mi = getMediaInfo(file:it, format:template+fz+template2)
		
		
		// append to file
		output.writeLine(mi)
	}

	
}
running on:
win 7 64 bit
filebot 3.3 64 bit
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Media Index CSV

Post by rednoah »

Something is somehow causing PermGen memory leak which can happen somehow through Groovy meta programming.

Is my default script affected by this?

Try setting these options for the Java runtime:
-XX:MaxPermSize=128m
-XX:+UseConcMarkSweepGC
-XX:+CMSPermGenSweepingEnabled
-XX:+CMSClassUnloadingEnabled

EDIT:
Are you using the latest Java 7?
:idea: Please read the FAQ and How to Request Help.
kiloback
Posts: 10
Joined: 11 Nov 2012, 05:57

Re: Media Index CSV

Post by kiloback »

tried the default from the online repository and has the out of memory error as well.

will try the settings as soon i could figure out how to access it,
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Media Index CSV

Post by rednoah »

:idea: Please read the FAQ and How to Request Help.
kiloback
Posts: 10
Joined: 11 Nov 2012, 05:57

Re: Media Index CSV

Post by kiloback »

yes!much better now thanks.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Media Index CSV

Post by rednoah »

:idea: Please read the FAQ and How to Request Help.
Post Reply