Patchwork code for XBMC automation

Running FileBot from the console, Groovy / FileBot scripting, shell scripts, etc
Post Reply
Maxxodd
Posts: 22
Joined: 27 May 2012, 21:01

Patchwork code for XBMC automation

Post by Maxxodd » 03 Jul 2012, 22:39

I made a patchwork of the code kindly provided by Rednoah and many others here on this board that allows utorrent to:

1) launch a groovy script that
2) extracts rar files if necessary
3) ignore unfinished files (if more than one download is happening at the same time)
4) delete "sample" files so that they aren't processed
5) determine if the file is a movie or tv show
6) collect artwork and get subtitles from theMovieDB or theTVDB respectively
7) rename shows according to XBMC convention
8) create the appropriate directory if necessary
9) Send me a text message when finished to notify me of its completion and what was downloaded

Some of the things that I learned along the way:
- You should troubleshoot by looking at the log (tab at the bottom) from utorrent and cut and paste that into a cmd window. You can also troubleshoot groovy script with the "Groovy Console". I found the later less helpful because you often need the variables passed from utorrent to troubleshoot.
- Java for me (Windows 7 64bit) needed to be moved to the program files (x64) directory and permissions changed.
- sendemail would not work from the suggested directory (system32). I need to place it in the standard windows directory. It would work fine from the command line, but for some reason when called from the groovy script with the same command, it would fail until I moved the program.

I use a separate program "Belvedere" that keeps my downloads folder clean. After appropriate seeding time, it deletes the original downloaded torrent.

Huge thanks to Rednoah! I take no credit for any of the script. I felt like a child trying to build a rocket ship most of the time. He was very patient and helpful in constructing this hodgepodge of other people's work. I hope that some of this information can save others some frustration and time.

Code: Select all

// filebot -script "fn:utorrent-postprocess" --output "X:/media" --action copy --conflict override -non-strict -trust-script -Xxbmc=localhost "-Xut_dir=%D" "-Xut_file=%F" "-Xut_label=%L" "-Xut_state=%S" "-Xut_kind=%K"
def input = []

// print input parameters
_args.parameters.each{ k, v -> println "Parameter: $k = $v" }

if (ut_kind == "multi") {
  input += new File(ut_dir).getFiles() // multi-file torrent
} else {
  input += new File(ut_dir, ut_file) // single-file torrent
}

// extract archives if necessary
input += extract(file:input, output:".", conflict:"override")

// process only media files
input = input.findAll{ it.isVideo() || it.isSubtitle() }

// ignore chunk, part, par and hidden files
def incomplete(f) { f.name =~ /[.]incomplete|[.]chunk|[.]par$|[.]dat$/ || f.isHidden() }
[ut_dir].getFolders{ !it.hasFile{ incomplete(it) } && it.hasFile{ it.isVideo() } }.each{ dir ->
   println "Deleting Sample Video Processing $dir"
   def files = dir.listFiles{ it.isVideo() }

// if name contains "sample" and filesize is less than 50 MB
//def selection = args.getFiles{ it.name =~ "sample" && it.length() < 20 * 1024 * 1024}
   def selection = dir.getFiles{ (it.name =~ "sample" || it.name =~ "Sample") && it.length() < 50 * 1024 * 1024}
   selection.each{ println "delete $it" }

// delete all
   selection*.delete()
}

// ignore clutter files
input = input.findAll{ !(it.name =~ /(?i:sample)/) }

// print input fileset
input.each{ println "Input: $it" }

// xbmc artwork/nfo utility
include("fn:lib/xbmc")


// group episodes/movies and rename according to XBMC standards
def groups = input.groupBy{
  def tvs = detectSeriesName(it)
  def mov = detectMovie(it, false)
  println "$it.name [series: $tvs, movie: $mov]"

  // DECIDE EPISODE VS MOVIE (IF NOT CLEAR)
  if (tvs && mov) {
    if (it.name =~ "(?i:$tvs - .+)" || parseEpisodeNumber(it.name) || parseDate(it.name)) {
      println "Exclude Movie: $mov"
      mov = null
    } else if (detectMovie(it, true)) {
      println "Exclude Series: $tvs"
      tvs = null
    }
  }
  return [tvs:tvs, mov:mov]
}

groups.each{ group, files ->
  // fetch subtitles
  def subs = getMissingSubtitles(file:files, output:"srt", encoding:"utf-8")
  if (subs) files += subs

  // EPISODE MODE
  if (group.tvs && !group.mov) {
    def dest = rename(file:files, format:'TV Shows/{n}/{episode.special ? "Special" : "Season "+s}/{n} - {episode.special ? "S00E"+special.pad(2) : s00e00} - {t}', db:'TheTVDB')

    dest.mapByFolder().keySet().each{ dir ->
      println "Fetching artwork for $dir from TheTVDB"
      def query = group.tvs
      def sxe = dest.findResult{ parseEpisodeNumber(it) }
      def options = TheTVDB.search(query)
      if (options.isEmpty()) {
        println "TV Series not found: $query"
        return
      }

   dest.findAll{ it.isVideo() }.each{
   def msg = "${it.name} has been added to the library"
   execute("c:/windows/sendEmail.exe", "-f", "[email protected]", "-t", "[email protected]", "-s", "smtp.gmail.com:587", "-o", "tls=yes", "-xu", "[email protected]", "-xp", "password", "-m", msg)
   }
      options = options.sortBySimilarity(query, { it.name })
      fetchSeriesArtworkAndNfo(dir.dir, dir, options[0], sxe && sxe.season > 0 ? sxe.season : 1)
    }
  }

  // MOVIE MODE
  if (group.mov && !group.tvs) {
    def dest = rename(file:files, format:'Movies/{n} ({y})/{n} ({y}){" CD$pi"}', db:'TheMovieDB')

    dest.findAll{ it.isVideo() }.each{
   def msg = "${it.name} has been added to the library"
   execute("c:/windows/sendEmail.exe", "-f", "[email protected]", "-t", "[email protected]", "-s", "smtp.gmail.com:587", "-o", "tls=yes", "-xu", "[email protected]", "-xp", "password", "-m", msg)
   }

    dest.mapByFolder().keySet().each{ dir ->
      println "Fetching artwork for $dir from TheMovieDB"
      fetchMovieArtworkAndNfo(dir, group.mov)
    }
  }
}



// make XBMC scan for new content
xbmc.split(/[\s,|]+/).each{
  println "Notify XBMC: $it"
  invokeScanVideoLibrary(it)
}

User avatar
rednoah
The Source
Posts: 16978
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Patchwork code for XBMC automation

Post by rednoah » 04 Jul 2012, 03:32

You might wanna rethink how you handles samples. If you delete those doesn't that cause the torrent to be stopped? Or redownloaded, refinished and this script rerun?

You don't have to delete anything, just ignore those files. Exclude them from the list of files that you're processing like this:

Code: Select all

input = input.findAll{ !(it.path =~ /\b(?i:sample|trailer|extras|deleted.scenes|music.video|scrapbook)\b/) || it.length() > 20 * 1024 * 1024}
:idea: Please read the FAQ and How to Request Help.

Maxxodd
Posts: 22
Joined: 27 May 2012, 21:01

Re: Patchwork code for XBMC automation

Post by Maxxodd » 04 Jul 2012, 16:08

Hunh, I haven't had any problems with that an I have tested it on some files that contained sample files. That is part of the reason it took me this long to write this; I wanted to make sure that there were no major errors. I do seem to sometimes get an extra page if I donwload a season pack. There may be one episode that I get a couple pages for. Other than that, it has been working great. Having said that, I may go ahead and change my script to your suggested code. Once again, much appreciation!

Maxxodd
Posts: 22
Joined: 27 May 2012, 21:01

Re: Patchwork code for XBMC automation

Post by Maxxodd » 14 Jul 2012, 22:54

Everything seems to be working great, but it does not appear to be fetching clear logos (for Neon skin in XBMC) nor the Fanart. The thumb gets fetched just fine. Is this normal? It's not a huge deal, I end up just refreshing the video file and pulling it in from XBMC, but would be great if the script did this.

User avatar
rednoah
The Source
Posts: 16978
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Patchwork code for XBMC automation

Post by rednoah » 15 Jul 2012, 03:33

That's what it's doing right now:
http://filebot.sourceforge.net/forums/v ... ?f=4&t=181

Are you talking about series artwork or movie artwork?
:idea: Please read the FAQ and How to Request Help.

Maxxodd
Posts: 22
Joined: 27 May 2012, 21:01

Re: Patchwork code for XBMC automation

Post by Maxxodd » 15 Jul 2012, 06:48

rednoah wrote:That's what it's doing right now:
http://filebot.sourceforge.net/forums/v ... ?f=4&t=181

Are you talking about series artwork or movie artwork?
I was talking about Movie fanart and TV clear logos. Sorry for not being more specific.

User avatar
rednoah
The Source
Posts: 16978
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Patchwork code for XBMC automation

Post by rednoah » 15 Jul 2012, 06:54

As far as I can tell TheMovieDB doesn't have fanart and I don't know what "clear logos" are, but I don't think TheTVDB has them. Anyway if you run the artwork.* scripts you'll get lots of output as to what artwork is available. If on there I can add support for that.
:idea: Please read the FAQ and How to Request Help.

Maxxodd
Posts: 22
Joined: 27 May 2012, 21:01

Re: Patchwork code for XBMC automation

Post by Maxxodd » 15 Jul 2012, 21:48

I guess TheMovieDB calls it "backdrops" as you referenced previously. Anyway, they don't seem to load for me unless I do a refresh. As far as the clear logos. I believe they come from here: http://fanart.tv/tv-fanart/

User avatar
rednoah
The Source
Posts: 16978
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Patchwork code for XBMC automation

Post by rednoah » 16 Jul 2012, 00:59

1. artwork.tmdb already downloads backdrop:original as backdrop.jpg. You can modify the script to grab extra types. Let me know which is the one you want.

Code: Select all

fetchArtwork(movieDir['backdrop-cover.jpg'], movieInfo, 'backdrop', 'cover')
fetchArtwork(movieDir['backdrop-mid.jpg'], movieInfo, 'backdrop', 'mid')
fetchArtwork(movieDir['backdrop-thumb.jpg'], movieInfo, 'backdrop', 'thumb')
2. I'll look into their API
:idea: Please read the FAQ and How to Request Help.

User avatar
rednoah
The Source
Posts: 16978
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Patchwork code for XBMC automation

Post by rednoah » 16 Jul 2012, 02:44

It's pretty easy to access fanart.tv so I'll added some support for that with r1092.

e.g.

Code: Select all

import static net.sourceforge.filebot.WebServices.*

FanartTV.getArtwork(70327).each {
	println it
	println it.url.saveAs("artwork/$it.type/$it.name")
}
But I'll only add code for fanart.tv in my scripts some time after the next release. But you can play with that yourself for the time being.
:idea: Please read the FAQ and How to Request Help.

Maxxodd
Posts: 22
Joined: 27 May 2012, 21:01

Re: Patchwork code for XBMC automation

Post by Maxxodd » 17 Jul 2012, 04:38

Hmm, it seems to be downloading the backdrops, but they don't show up in XMBC. Not sure what's going on. The XBMC library is getting updated because the new movie shows up along with the right thumb, but the backdrop doesn't. I've tried to exit and restart XMBC as well as "update library", but neither work. The only thing that helps is if I refresh the file and then It just re-downloads everything (my default scraper is different). Maybe there's something wrong with my installation. Not sure, but its not that huge a deal to me considering how automated it currently is.

User avatar
rednoah
The Source
Posts: 16978
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Patchwork code for XBMC automation

Post by rednoah » 17 Jul 2012, 05:22

I don't use XBMC so maybe the script is not saving things in the proper filenames that XBMC will recognise.
:idea: Please read the FAQ and How to Request Help.

Maxxodd
Posts: 22
Joined: 27 May 2012, 21:01

Re: Patchwork code for XBMC automation

Post by Maxxodd » 29 Jul 2012, 20:13

Thanks again for all your help. Initially, I realized that the newer version of XBMC needs the backdrops to be named "wallpaper" instead, so I changed that in the script. Then I realized that XBMC will automatically bring in the backdrops, episode information, etc. Having them in the folder overrides XBMC. The only thing that it doesn't do by default is bring in subtitles. So, I ended up stripping out the script information for adding artwork and episode or movie information. Here are the updated scripts to accomplish this:

Code: Select all

// filebot -script "fn:utorrent-postprocess" --output "X:/media" --action copy --conflict override -non-strict -trust-script -Xxbmc=localhost "-Xut_dir=%D" "-Xut_file=%F" "-Xut_label=%L" "-Xut_state=%S" "-Xut_kind=%K"
def input = []

// print input parameters
_args.parameters.each{ k, v -> println "Parameter: $k = $v" }

if (ut_kind == "multi") {
  input += new File(ut_dir).getFiles() // multi-file torrent
} else {
  input += new File(ut_dir, ut_file) // single-file torrent
}

// extract archives if necessary
input += extract(file:input, output:".", conflict:"override")

// process only media files
input = input.findAll{ it.isVideo() || it.isSubtitle() }

// ignore chunk, part, par and hidden files
def incomplete(f) { f.name =~ /[.]incomplete|[.]chunk|[.]par$|[.]dat$/ || f.isHidden() }
[ut_dir].getFolders{ !it.hasFile{ incomplete(it) } && it.hasFile{ it.isVideo() } }.each{ dir ->
   println "Deleting Sample Video Processing $dir"
   def files = dir.listFiles{ it.isVideo() }

// ignore samples
   input = input.findAll{ !(it.path =~ /\b(?i:sample|trailer|extras|deleted.scenes|music.video|scrapbook)\b/) || it.length() > 50 * 1024 * 1024}

}
// ignore clutter files
input = input.findAll{ !(it.name =~ /(?i:sample)/) }

// print input fileset
input.each{ println "Input: $it" }

// xbmc artwork/nfo utility - ***This refers to the second script below.  Make sure you change to the correct location***
include("X:/Users/YourDirectoryLocation/XBMC.groovy")


// group episodes/movies and rename according to XBMC standards
def groups = input.groupBy{
  def tvs = detectSeriesName(it)
  def mov = detectMovie(it, false)
  println "$it.name [series: $tvs, movie: $mov]"

  // DECIDE EPISODE VS MOVIE (IF NOT CLEAR)
  if (tvs && mov) {
    if (it.name =~ "(?i:$tvs - .+)" || parseEpisodeNumber(it.name) || parseDate(it.name)) {
      println "Exclude Movie: $mov"
      mov = null
    } else if (detectMovie(it, true)) {
      println "Exclude Series: $tvs"
      tvs = null
    }
  }
  return [tvs:tvs, mov:mov]
}

groups.each{ group, files ->
  // fetch subtitles
  def subs = getMissingSubtitles(file:files, output:"srt", encoding:"utf-8")
  if (subs) files += subs

  // EPISODE MODE
  if (group.tvs && !group.mov) {
    def dest = rename(file:files, format:'TV Shows/{n}/{episode.special ? "Special" : "Season "+s}/{n} - {episode.special ? "S00E"+special.pad(2) : s00e00} - {t}', db:'TheTVDB')

   dest.findAll{ it.isVideo() }.each{
   def msg = "${it.name} has been added to the library"
   execute("c:/windows/sendEmail.exe", "-f", "[email protected]", "-t", "[email protected]", "-s", "smtp.gmail.com:587", "-o", "tls=yes", "-xu", "[email protected]", "-xp", "password", "-m", msg)
   }
  }

  // MOVIE MODE
  if (group.mov && !group.tvs) {
    def dest = rename(file:files, format:'Movies/{n} ({y})/{n} ({y}){" CD$pi"}', db:'TheMovieDB')

    dest.findAll{ it.isVideo() }.each{
   def msg = "${it.name} has been added to the library"
   execute("c:/windows/sendEmail.exe", "-f", "[email protected]", "-t", "[email protected]", "-s", "smtp.gmail.com:587", "-o", "tls=yes", "-xu", "[email protected]", "-xp", "password", "-m", msg)
   }


  }
}



// make XBMC scan for new content
xbmc.split(/[\s,|]+/).each{
  println "Notify XBMC: $it"
  invokeScanVideoLibrary(it)
}
and:

Code: Select all

// xbmc functions
def invokeScanVideoLibrary(host, port = 9090) {
  try {
    telnet(host, port) { writer, reader ->
      writer.println('{"id":1,"method":"VideoLibrary.Scan","params":[],"jsonrpc":"2.0"}') // API call for latest XBMC release
    }
    return true
  } catch(e) {
    println "${e.class.simpleName}: ${e.message}"
    return false
  }
}

part timer
Posts: 181
Joined: 09 May 2012, 23:35

Re: Patchwork code for XBMC automation

Post by part timer » 30 Jul 2012, 05:23

Actually it really should be fanart.jpg or <moviename>-fanart.jpg.

Check here for a pretty good guide (harder to find something official on this surprisingly):
http://forum.xbmc.org/showthread.php?tid=59281

User avatar
rednoah
The Source
Posts: 16978
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Patchwork code for XBMC automation

Post by rednoah » 30 Jul 2012, 06:03

Code: Select all

// fetch series banner, fanart, posters, etc
		fetchMovieArtwork(movieDir['folder.jpg'], movieInfo, 'posters', locale.language)
		fetchMovieArtwork(movieDir['wallpaper.jpg'], movieInfo, 'backdrops', locale.language)
		
		fetchMovieFanart(movieDir['logo.png'], movieInfo, 'movielogo', null, locale)
		fetchMovieFanart(movieDir['fanart.png'], movieInfo, 'movieart', null, locale)
		['bluray', 'dvd', null].findResult { diskType -> fetchMovieFanart(movieDir['disc.png'], movieInfo, 'moviedisc', diskType, locale) }
So what should I do?

I fetch:
tmdb poster as folder.jpg
tmdb backdrop as wallpaper.jpg (correct?)
fanarttv movielogo as logo.png
fanarttv movieart as fanart.png
fanarttv moviedisc as disk.png

What do I have to change so it matches XBMC better?

This is the latest SVN rev btw, bit different from what you had since i added support for the FanartTV API.
:idea: Please read the FAQ and How to Request Help.

part timer
Posts: 181
Joined: 09 May 2012, 23:35

Re: Patchwork code for XBMC automation

Post by part timer » 30 Jul 2012, 06:12

Actually I just started a new post with a request for changes to artwork here:
http://filebot.sourceforge.net/forums/v ... ?f=4&t=207
rednoah wrote:tmdb poster as folder.jpg
tmdb backdrop as wallpaper.jpg (correct?)
fanarttv movielogo as logo.png
fanarttv movieart as fanart.png
fanarttv moviedisc as disk.png
1.I say folder.jpg and movie.tbn although personally I also have poster.jpg as well, might really be overkill for some people. Doesn't hurt me to have 1 or 2 extra files in there
2.I've never seen anyone talk about using wallpaper.jpg, should be fanart.jpg or <moviename>-fanart.jpg for ultimate compatibility
3,4,5. I haven't touched my TV stuff yet, still sorting out movies.

You can also check another reference from an xbmc member who has an app just to download artwork for xbmc:
http://wiki.xbmc.org/index.php?title=Ad ... rk_sources:

Maxxodd
Posts: 22
Joined: 27 May 2012, 21:01

Re: Patchwork code for XBMC automation

Post by Maxxodd » 30 Jul 2012, 06:34

part timer wrote:Actually it really should be fanart.jpg or <moviename>-fanart.jpg.

Check here for a pretty good guide (harder to find something official on this surprisingly):
http://forum.xbmc.org/showthread.php?tid=59281
Sorry, that is correct. It's been a week or so since I was playing with it. I knew it wasn't backdrop, just got fanart and wallpaper mixed up. I know there are some advantages to having the artwork files manually placed in the folders, but I'm fine with XBMC doing it automatically for me.

part timer
Posts: 181
Joined: 09 May 2012, 23:35

Re: Patchwork code for XBMC automation

Post by part timer » 30 Jul 2012, 06:37

This is true. XBMC can do all the heavy lifting of getting all nfo info and artwork and everything, but then it takes a while to do it. My main reason for wanting it locally in the folder with the movie was if xbmc ever crashes which only happened once so far for me, then you have to wait days for it to reload everything.

If it's all right there in the folders though, it takes way less time to grab it and show it. Plus it gives me that chance to confirm as I go that I'm getting the right data for the right movie.

I hate sitting there and waiting for it to scan the library of thousands of files, personally.

Post Reply