Fully automated media downloading

[F12]

Hi,

I've just discovered FileBot and I'm really impressed by how powerful it is compared to the other media renamers around. I've been having a look at the documentation and playing around with it a little but it is quite complex, but I am eager to support the project.

What I would like to achieve is being able to automatically copy and rename any movies and episodes downloaded via uTorrent to the appropriate directory upon completion of the download, and then ideally update XBMC library. The files need to be copied rather than moved so that the original can be seeded.

The event to trigger the Filebot operations could be either monitoring of the download folder, or alternatively using uTorrent's setting to launch a program upon download completion. Ideally, Filebot would then distinguish between movies and episodes and copy and rename accordingly. (Would other files in the download folder which are not movies or episodes be ignored at this point, or would they need to be filtered out from the directory beforehand?)

Any help on achieving this would be much appreciated. I am eager to get into it and make use of this project!

Cheers

[rednoah]

The basic command-line should be very simple to use. Things like notifying xbmc and seperating movie from tv shows and complicated and really depends on what you need.

The pre-defined example scripts already do what you need. You can either use those in sequence or copy things into one big scripts that does everything you need the way you want.

btw move/rename is only the standard action, there is also --action move | copy | keeplink | symlink | hardlink and test (which does nothing, so it's useful for testing).

This post tells you where to find the code needed for each task. Should be pretty much copy&paste:
http://filebot.sourceforge.net/forums/v ... &t=86#p583

If you want a folder monitor or start things yourself via utorrent is up to you. Both is possible. Maybe the latter is easier for you.

[F12]

So I've knocked something together using code mostly from the examples. All seems to work well for now, except the XBMC notification.



Not sure what the problem is here.

Also- Is there a way to prevent Filebot from moving all the files already in my downloads folder when I direct it there properly? All the files have been renamed properly at the moment, I only want Filebot to deal with any new incoming files from now.

For triggering within uTorrent, would I just be writing a batch file with the command to run the Groovy script, and running that on completion of all torrents?

Code: Select all

// filebot -script "rename.groovy" -trust-script <folder> -non-strict --action keeplink

// PERSONALISED SETTINGS
def episodeFormat = "C:/Videos/{n}/Season {s}/{n} - {sxe} - {t}"
def movieFormat   = 'C:/Movies/{n} ({y})/{n} ({y}){" CD$pi"}'

def xbmc = ['Apple-TV.lan', 'localhost']

/*
 * Move/Rename a mix of episodes and movies that are all in the same folder.
 */
def groups = args.getFiles().groupBy{
	def tvs = detectSeriesName(it)
	def mov = detectMovie(it, false)
	println "$it.name [series: $tvs, movie: $mov]"
	
	// DECIDE EPISODE VS MOVIE (IF NOT CLEAR)
	if (tvs && mov) {
		if (it.name =~ "(?i:$tvs - .+)" || parseEpisodeNumber(it.name) || parseDate(it.name)) {
			println "Exclude Movie: $mov"
			mov = null
		} else if (detectMovie(it, true)) {
			println "Exclude Series: $tvs"
			tvs = null
		}
	}
	return [tvs:tvs, mov:mov]
}

groups.each{ group, files ->
	// EPISODE MODE
	if (group.tvs && !group.mov) {
		return rename(file:files, format:episodeFormat, db:'TheTVDB')
	}
	
	// MOVIE MODE
	if (group.mov && !group.tvs) {
		return rename(file:files, format:movieFormat, db:'TheMovieDB')
	}
}

// make XBMC scan for new content
xbmc.each { host ->
	telnet(host, 9090) { writer, reader ->
		writer.println('{"jsonrpc": "2.0", "method": "VideoLibrary.ScanForContent", "id": 1}')
	}
}

[rednoah]

Code: Select all

filebot -script "rename.groovy" -trust-script <folder> -non-strict --action keeplink

This only explains how to call filebot from the cmdline. "-non-strict --action keeplink" doesn't work in the script obviously, it's cmdline arguments not groovy script.

By default filebot won't override things, it'll process things and then skip if it already exists. If utorrent runs filebot on download complete then it shouldn't be problem.

[F12]

Code: Select all

filebot -script "rename.groovy" -trust-script <folder> -non-strict --action keeplink

This only explains how to call filebot from the cmdline. "-non-strict --action keeplink" doesn't work in the script obviously, it's cmdline arguments not groovy script.

Yeah, I know. I guess I could just paste that directly for uTorrent to run on download completion rather than using a batch.

By default filebot won't override things, it'll process things and then skip if it already exists. If utorrent runs filebot on download complete then it shouldn't be problem.

The movies in the destination directory won't necessarily be named identically to how Filebot will name them, though. How will this be handled?

Any ideas on what is happening with the telnet calls to XBMC?

[rednoah]

I'm not sure what you mean. The --conflict skip setting only applies what happens if the destination already exists. If it doesn't it'll just continue normally.

Is that telnet call not working? Maybe you need to activate the telnet api in xbmc first. Never tested that since I don't use XBMC. But it should work according to the specs.

[F12]

What I mean is that currently the script is just checking the whole download directory for files to rename. Let's say I have the movie Movie.2011.720p.BluRay.x264-REFiNED\refined-movie-720p.mkv in the downloads directory already (prior to Filebot) and the corresponding Movie (2008)\Movie (2008) 720p.mkv in my movies directory. Now when I first direct my Groovy script to the downloads directory, it is going to come across the movie and send it to my movies directory, with the name Movie (2008)\Movie (2008).mkv. It isn't going to realise that the movie is already in the directory because the file name convention isn't identical and there won't be any file name conflict, so the end result will be a duplicate. What I would need is a way to ignore anything that is present when I first run the script and from then on only rename new files in the downloads directory each time the script is run.

Alternatively, it might make more sense to just feed the directories/files to the script directly from uTorrent as those arguments can be passed when the download complete program is run (rather than feeding it the downloads directory each time). That way I don't have to worry about anything else that might be in the input directory or making sure that only new files are renamed.

It gets a bit hairier though because the Groovy script as is only works with directories and won't work with single-file torrents. I would need to pass directory and name for single-file torrents, and I guess %D and %N for multi-file torrents, and then interpret those somehow to arrive at which files actually need to be renamed. I'm not sure I really understand which arguments mean what in terms of passing on the actual file reference to a script.

Code: Select all

You can use the following parameters:

%F - Name of downloaded file (for single file torrents)
%D - Directory where files are saved
%N - Title of torrent
%P - Previous state of torrent
%L - Label
%T - Tracker
%M - Status message string (same as status column)
%I - hex encoded info-hash
%S - State of torrent
%K - kind of torrent (single|multi)

[rednoah]

1. It's all in the scripting. Of course you can check if a folder/file already exists somewhere. But you have to came up with the logic that makes sense yourself. Easier to just enable a utorrent process.

2. The cmdline interface has no problem with files/folders. rename(folder:path) will accept a folder whereas rename(file:set) will work with one or more files directly.

Now my example assumes input is a folder. But most of the script works with a set of files.

Think like this:
def f = args[0]
def files = f.isDirectory() ? f.listFiles() : [f]

As for how to use utorrent I also dont know. Just try a test script for various parameter. e.g. args.toString().saveAs("c:/output.txt") and see what the values are for these variables.

EDIT:
Basically you just have to change the code that initally collects the files that will be processed instead of using args.getFiles().

[F12]

Ok, I've abandoned the idea of only submitting to Filebot the files that have only just been downloaded because I just can't get my head around it. I am just going to use the code I posted most recently and have Filebot triggered to scan that folder on each download complete event, and move the files while keeping a symbolic link at the download location to continue seeding.

Now, how do I get Filebot to run under administrator privileges when it is automatically triggered? The only way KEEPLINK works is if the command-prompt is run under administrator privileges.

My plan at the moment would be to add to uTorrent Run Program, rename.bat:

Code: Select all

filebot -script "rename.groovy" -trust-script <folder> -non-strict --action keeplink

[rednoah]

I guess you'll need to find some kinda "sudo" for Windows. 

Using utorrent to kick off a cmd on complete shouldn't be a problem. Just needs a of testing what these variables really mean. Maybe you can find better documentation or examples online. 

[F12]

Odd... seems like the privileges isn't a problem when running it from a batch file. Anyway, I have the script running almost perfectly when run from batch. For any other readers, the new JSON-RPC API for XBMC Eden has some slight changes, so that the correct command is now "VideoLibrary.Scan" rather than the command from the example. XBMC needs to be set to receive commands from other programs.

Now I have an odd problem when running the batch file from uTorrent on download completion -- the batch file executes fine, but an error message comes up that Windows can't find the file "rename.groovy" (my Groovy script). At first I thought it might be because the relative file path is changed due to the batch being launched by a different program, but the same error occurs even when I change the path to an absolute one. Any ideas? I am baffled here.

At the moment I am getting the script to read all the files in the folder (as above), which I have set uTorrent to move completed downloads to. This may get to be an issue when more files are downloaded as there will be a lot of needless looking up of files before coming to a conflict at the renaming stage and aborting that file. Is there a way to mark the files as already having been scanned so that only new files will be acted upon by Filebot? From memory, theRenamer does this using the "Archived" attribute.

[rednoah]

1. Gonna update the samples for XBMC11 with "VideoLibrary.Scan"

2. Did you mess up the cmdline somehow?

Doing this with the latest rev:

Code: Select all

filebot -script "D:/test.groovy" -trust-script "-Xut_file=%F" "-Xut_dir=%D" "-Xut_name=%N" "-Xut_previous_torrent_state=%P" "-Xut_label=%L" "-Xut_tracker=%T" "-Xut_status_message=%M" "-Xut_infohash=%I" "-Xut_torrent_state=%S" "-Xut_torrent_kind=%K"

Where the script is:

Code: Select all

_args.parameters.toString().saveAs('D:/out.txt')

So here is some data I got from playing with this:

single-file torrent:

Code: Select all

ut_infohash:9622C779B10BE9A7F1F0A4D21D420EB57B0D1422
ut_previous_torrent_state:10
ut_status_message:Seeding
ut_tracker:https://tracker2.tvtorrents.com/TrackerServlet/...
ut_label:tvshows
ut_dir:E:\Storage
ut_torrent_state:5
ut_torrent_kind:single
ut_name:The.Big.Bang.Theory.It.All.Started.With.A.Big.Bang.720p.HDTV.X264-TLA.mkv
ut_file:The.Big.Bang.Theory.It.All.Started.With.A.Big.Bang.720p.HDTV.X264-TLA.mkv

multi-file torrent:

Code: Select all

ut_infohash:5BE429483B294E59AE7A643CDD41154C884879BC
ut_label:
ut_previous_torrent_state:12
ut_status_message:Downloading
ut_tracker:udp://tracker.1337x.org:80/announce
ut_dir:E:\Storage
ut_torrent_state:6
ut_torrent_kind:multi
ut_name:Sherlock Holmes A Game of Shadows (2011) DVDRip XviD-MAXSPEED
ut_file:

So just use the variables utorrent provides and just handle files that have been changed. As direct filebot -rename call from utorrent or as filebot -script is up to you depending on what you wanna do.

Try this:

Code: Select all

filebot -rename "%D\%N" -non-strict --action copy --format "..." --conflict override

Using a a filesystem property like "Archived" is a really ugly hack. Not to mentioned limited to Windows. But it can be done via scripting if you want to do it that way. If you need to "remember" what has been handled and what hasn't been handled it's probably better to keep a text file around to track that. Using filebots internal rename history might be an idea as well.

[F12]

No, the command-line seems to be all normal. Executing the batch file directly works fine, but when that same batch file is run from uTorrent, Windows can't find rename.groovy. It's bizarre.

I guess the command-line interface works just fine for this, I didn't take into account that the TV/movies differentiation wouldn't be an issue when it is just fed the path to what has been downloaded directly. The only functionality missing would be the XBMC library update (which I guess could be done with a separate Groovy script, provided that issue is sorted out).

I wondered whether "Archived" was Windows-specific. In any case I suppose it isn't necessary if it is just the downloaded file/s directly fed to Filebot rather than the entire downloads directory, though it might be for other situations.

[rednoah]

No, the command-line seems to be all normal. Executing the batch file directly works fine, but when that same batch file is run from uTorrent, Windows can't find rename.groovy. It's bizarre.

The stuff I posted above is what I got when running filebot -script from utorrent. It definitely works. Make sure you're escaping everything properly.

[rednoah]

For me --action symlink/hardlink all don't work because of the permissions problem. Otherwise this command works fine:

utorrent "Run Program":

Code: Select all

filebot -rename "%D\%N" -non-strict --action copy --format "..." --conflict override

PS: This will work for folders and files and it'll try some basic auto-detection as to handle the input as a set of episodes or as a movie (keep in mind that the given format will be applied to episode and movie data so it would have to be compatible with both).

[F12]

Didn't realise that the uTorrent "Run Program" worked like the command line, I have added the script line from before and it works fine. Running uTorrent as administrator seems to solve the permissions problem, but I am not sure as something else I did might have completely solved that. On a related note, though, KEEPLINK does not appear to permit uTorrent to keep seeding unfortunately ("Error: Files missing from job. Please recheck.") Perhaps HARDLINK would work if the destination and downloads directory were on the same volume, but they are not in my case so I might have to resort to using COPY.

(keep in mind that the given format will be applied to episode and movie data so it would have to be compatible with both)

Not sure how I would do this with seasons for episodes, CD numbers for movies and so on?

How can the XBMC telnet call be integrated into what we have currently?

[rednoah]

1. keeplink is two operations. First move, then symlink back. The move part breaks utorrent if it has the file open. utorrent just needs to close the original file and then reopen the symlink. So just restart utorrent and/or do stop/resume download. That should work. hardlink/symlink/copy work because the original file is not touched.

2. Nope, cmdline can't take care of more complicated processes. That's what scripting is for.

Working on something for that right now:
http://filebot.sourceforge.net/scripts/ ... ess.groovy

[rednoah]

You can try this now:

* Replace FileBot.jar with the latest revision r985:
http://sourceforge.net/projects/filebot ... ebot/HEAD/

* Use this utorrent "Run Command" action:

Code: Select all

filebot -script "http://filebot.sf.net/scripts/utorrent-postprocess.groovy" "%D\%N" --output "X:/media" --action copy --conflict override -non-strict -trust-script -Xxbmc=localhost

[F12]

1. keeplink is two operations. First move, then symlink back. The move part breaks utorrent if it has the file open. utorrent just needs to close the original file and then reopen the symlink. So just restart utorrent and/or do stop/resume download. That should work. hardlink/symlink/copy work because the original file is not touched.

That was my first thought as well, but after forcing a recheck the same error occurs. I am sure symlink would be more functional but I want the actual files residing in my media folder, not the seeding folder so it looks like I will have to use copy for the time being, at least until I make some changes to my filesystem structure some time down the road. The advantage offered by keeplink is great though, so I'll try to work out what's going on there and why uTorrent doesn't seem to be following the link.

2. Nope, cmdline can't take care of more complicated processes. That's what scripting is for.

Yeah, I was thinking maybe combining the command-line and a script just for the XBMC notification but it seemed ungainly. The solution you've put together is much better.

Working on something for that right now:
http://filebot.sourceforge.net/scripts/ ... ess.groovy

Looks perfect. I'm not home at the moment but I will check it when I get a chance. The only thing I can think of is that the original example allowed for multiple XBMC hostnames through the use of the array. Is that still allowed for in your example or does it only take a single string? The try-catch statement is good though, previously it would error if any of the XBMC instances were closed (or not set up correctly). Also, what is the reason for adding --conflict override?

This would be worthwhile putting on XBMC forums once it's done, it is far superior to most of the automation solutions used currently.

[rednoah]

Looks perfect. I'm not home at the moment but I will check it when I get a chance. The only thing I can think of is that the original example allowed for multiple XBMC hostnames through the use of the array. Is that still allowed for in your example or does it only take a single string? The try-catch statement is good though, previously it would error if any of the XBMC instances were closed (or not set up correctly). Also, what is the reason for adding --conflict override?

It's all in the script. I only provide a standard example then people can edit/modify it later to fit their needs.

If you download the same episode again later (maybe a better quality version) then you can decide what happens via --conflict skip|error|override.

[rednoah]

The utorrent variables really don't make sense. Added the logic for single-vs-multi torrent in the script now:

Code: Select all

filebot -script "http://filebot.sf.net/scripts/utorrent-postprocess.groovy" --output "X:/media" --action copy --conflict override -non-strict -trust-script -Xxbmc=localhost "-Xut_dir=%D" "-Xut_file=%F" "-Xut_label=%L" "-Xut_state=%S" "-Xut_kind=%K"

Also you can do -Xxbmc=localhost,xbmc,etc now.

[rednoah]

btw here's a link to utorrent api interface, might be useful if you need to script stopping/starting a torrent:
http://www.utorrent.com/community/devel ... bapi#devs1

[F12]

Turns out the symlink issue is a known bug in uTorrent. Given that it was reported in 2009 there is not much chance of it being fixed anytime in the near future. http://forum.utorrent.com/viewtopic.php?pid=478896

I think copy is really the only solution for me, although hardlink would work for users who have their seeding directory and the media directory on the same volume.

[F12]

Ok, really odd problem now. The script works perfectly when pointing it to the online version (straight copy-paste of the command line you posted) but when I use the exact same script locally (C:/Batches/utorrent-postprocess.groovy), it doesn't work. No idea what's happening but it is really frustrating. The script does run momentarily because I can see the command prompt and a few of the first lines (parameters etc.) but then it closes without doing all the look-up etc. I have no idea what is happening - literally the only change is that the script is local.

Another thing that is worth noting (although not really a problem) is that for multi-file torrents, if the files are downloaded one at at time there will be a torrent completion event for each file (which is desirable, obviously), but the path will still refer to the entire directory. That means that the lookup process will all happen for every file in that torrent each time a new file is downloaded. Again, not really a problem but worth noting.

[rednoah]

1. Check the utorrent log and see what command is called exactly. Local files work. That's what I work with obviously.

2. Oh? Then it's "multi" but also has "file" set? Can you help me do some testing and see what's happening? Then we can figure out the best way to integrate with utorrent. Is there a way to check if the torrent is completed?