Hardlinking with "Fully automated media center" script

[jkaberg]

Noticed the script supports hardlinks, which would be very nice for me

Using --action symlink|keeplink|hardlink requires at least Java 7 and (on Windows) also admin priviledges so you'll need to run µTorrent as admin

But how does it work? Will it extract files that need extracting, and hardlink media files (.mkv, .avi etc?) or will it hardlink everything including .rar files?

Im currently running

filebot -script fn:utorrent-postprocess --output "F:/Downloaded/_automated/_postprocess" --action copy --conflict override -non-strict --def "seriesFormat=tv/%N/{fn}" "animeFormat=anime/%N/{fn}" "movieFormat=movie/%N/{fn}" subtitles=n artwork=n "ut_dir=%D" "ut_file=%F" "ut_kind=%K" "ut_title=%N" "ut_label=%L" "ut_state=%S"

But with the hardlink option I guess it would look like

filebot -script fn:utorrent-postprocess --output "F:/Downloaded/_automated/_postprocess" --action copy hardlink --conflict override -non-strict --def "seriesFormat=tv/%N/{fn}" "animeFormat=anime/%N/{fn}" "movieFormat=movie/%N/{fn}" subtitles=n artwork=n "ut_dir=%D" "ut_file=%F" "ut_kind=%K" "ut_title=%N" "ut_label=%L" "ut_state=%S"

[jkaberg]

Ok so I tried this

Downloaded a torrent wich only contained a file (a series) with file ending .mkv (which got hardlinked), then I downloaded another series with a .rar set inside which ended up in extracting the series to <rarset dir><torrentname>/<torrentname>.mkv and then hardlinked that file

Shouldnt this just extract it directly to destination without unpacking in the rarset dir first?

My point with using this is not ending up with 2 copys of the .mkv file (less diskspace used) but still beeing able to seed the original files


I used this with uTorrent

Code: Select all

filebot -script fn:utorrent-postprocess --output "F:/Downloaded/_automated/_postprocess" --action hardlink --conflict override -non-strict --def "seriesFormat=tv/%N/{fn}" "animeFormat=anime/%N/{fn}" "movieFormat=movie/%N/{fn}" subtitles=n artwork=n "ut_dir=%D" "ut_file=%F" "ut_kind=%K" "ut_title=%N" "ut_label=%L" "ut_state=%S"

PS: I also tried using clean=y but no avail,

filebot -script fn:utorrent-postprocess --output "F:/Downloaded/_automated/_postprocess" --action hardlink --conflict override -non-strict --def "seriesFormat=tv/%N/{fn}" "animeFormat=anime/%N/{fn}" "movieFormat=movie/%N/{fn}" subtitles=n artwork=n clean=y "ut_dir=%D" "ut_file=%F" "ut_kind=%K" "ut_title=%N" "ut_label=%L" "ut_state=%S"

[rednoah]

Yeah, in that case it'd have to be kinda a mix between copy and hardlink on a case by case basis. You can add that yourself. I won't add that to the default utorrent-postprocess though for the sake of simplicity.

[jkaberg]

But shouldnt the clean variable remove the extracted content from the "seeded" dir automaticly, atleast thats how I interpreted it

--def clean=y Automatically empty folders and clutter files that may be left behind after moving the video files or temporary extracted files after copying

If thats not the cases, how would one proceed, eg what would need to be altered? Im guessting the extraction process but how do I specify it to extract to destination instead of the "seed dir" and then not hardlink itself in the destination dir (eg hardlink destionation to destinatioul as it would be)

Because as it is now, the only thing I really would need it to do is remove the extracted files in the seed dir (becasuse since they are hardlinked the other "linked file" in the destination dir will still exist and everything will be fine) after makeing the hardlink. And I've tried the clean variable several times, eg clean=y | clean=yes | clean=true with no success, it dosnt remove anything

[rednoah]

Yep, that makes sense. Maybe I'll add it next time. Just look the end of the script. "clean" has different meanings depending on --action.

[jkaberg]

Thanks for pointing that out! Have now edited the clean function as following

Code: Select all

if (clean) {
	if ('COPY'.equalsIgnoreCase(_args.action) && extractedFiles?.size() > 0) {
		println 'Clean temporary extracted files'
		// delete extracted files
		extractedFiles.each{
			if(it.isFile()) {
				println "Delete $it"
				it.delete()
			}
		}
		// delete remaining empty folders
		extractedFiles*.dir.unique().findAll{ it.listFiles().length == 0 }.each{
			if(it.isDirectory()) {
				println "Delete $it"
				it.delete()
			}
		}
	}
	
	if ('HARDLINK'.equalsIgnoreCase(_args.action) && extractedFiles?.size() > 0) {
		println 'Clean temporary extracted files'
		// delete extracted files
		extractedFiles.each{
			if(it.isFile()) {
				println "Delete $it"
				it.delete()
			}
		}
		// delete remaining empty folders
		extractedFiles*.dir.unique().findAll{ it.listFiles().length == 0 }.each{
			if(it.isDirectory()) {
				println "Delete $it"
				it.delete()
			}
		}
	}
	
	// deleting remaining files only makes sense after moving files
	if ('MOVE'.equalsIgnoreCase(_args.action)) {
		println 'Clean clutter files and empty folders'
		include('fn:cleaner', [:], !args.empty ? args : ut_kind == 'multi' && ut_dir ? [ut_dir as File] : [])
	}
}

And it does indeed remove the extracted files in the seed directory, does hardlink but when I try to process the file in the "postprocess" folder (where its hardlinked or extracted to) I cant access the file without haveing to shutdown uTorrent, do you know why? If i search for the file in Process Explorer I can see that uTorrent is accessing the counterpart in my seed directory, but no process is accessing it in my "postprocess" folder :O

[rednoah]

So it's extracted to to the current directory and then hardlinked to the new organized structure. And you can't remove the file from the downloads folder where the archives are as well?

In anycase I have no idea how Windows deals with file handles and hardlinks. Maybe adding a wait somewhere helps? Is the script failing or is it that later you can't access files?

[jkaberg]

Appreciate your help and knowledge rednoah! The hardlinking problem was related to uTorrent, resolved with setting diskio.flush_files to false in uTorrent (advanced settings)

diskio.flush_files: Enabling this option causes µTorrent to close file handles every minute. It helps to reduce the effect of Windows managing the system cache badly for some people and causing apparent “memory leaks.”

Seems everything works as intended now

[rednoah]

Google just told me about your new thread on Couchpotato:
https://couchpota.to/forum/thread-1454-post-6270.html

You're completely abusing the utorrent-postprocess script:
* by setting label 'movie' you disable any anime/series/movie auto-detection
* your movie format --def movieFormat=movie/%N/{fn} is completely oblivious of whatever movie-detection comes up with. Now that's really just wasting TheMovieDB bandwith for no reason.

So you've disabled any and all processing... What's the point? I guess you need extracting and hardlinking, and nothing else.

e.g. After deleting almost everything from my utorrent-postprocess you should come up with something like this:

Code: Select all

filebot -script "C:/fbcpto.groovy" --output "C:/hl" --def "ut_dir=%D" "ut_file=%F" "ut_kind=%K" "ut_title=%N" "ut_label=%L"

Code: Select all

def files = []
if (ut_kind == 'single') { files += new File(ut_dir, ut_file) }
if (ut_kind == 'multi')  { files += new File(ut_dir).getFiles() }

// extract archives (zip, rar, etc) that contain at least one video file
def extractedFiles = extract(file: files.findAll{ it.isArchive() }, output: '_extract', conflict: 'override', filter: { it.isVideo() })
files += extractedFiles

files.findAll{ it.isVideo() }.each{ f ->
	println net.sourceforge.filebot.StandardRenameAction.HARDLINK.rename(f, new File(_args.output, "${ut_label}/${ut_title}/${f.name}")) 
}

// delete extracted files later
extractedFiles*.delete()

[jkaberg]

Im sorry if I miss used your script rednoah, wasnt my intention to "void" anything. But my lack of knowledge in scripting got me to where I am with the "Fully automated media center script" (I've actully tried to do what you did with the script you posted one up)

Anyhow I'll edit my post at couchpotato forums with credits towards you


Thanks again

[rednoah]

Yeah, the one I posted should to pretty much exactly what you want. Except in alot less lines of code (10 lines instead of 300).

Let me know if you have any issues.

[jkaberg]

Yeah there was one thing I would wish to add, did I get this right? I would like the script to quit if the label isnt movie or tv

Code: Select all

if (ut_label != 'movie' || ut_label != 'tv' ) { return null }

eg full code

Code: Select all

// Author of this script is Rednoah at Filebot Forums, http://filebot.sourceforge.net
// Usage: filebot -script "C:/Downloaders/fbcpto.groovy" --output "C:/Downloaded/_postprocess" --def "ut_dir=%D" "ut_file=%F" "ut_kind=%K" "ut_title=%N" "ut_label=%L"

def files = []
if (ut_label != 'movie' || ut_label != 'tv' ) { return null }
if (ut_kind == 'single') { files += new File(ut_dir, ut_file) }
if (ut_kind == 'multi')  { files += new File(ut_dir).getFiles() }

// extract archives (zip, rar, etc) that contain at least one video file
def extractedFiles = extract(file: files.findAll{ it.isArchive() }, output: '_extract', conflict: 'override', filter: { it.isVideo() })
files += extractedFiles

files.findAll{ it.isVideo() }.each{ f ->
   println net.sourceforge.filebot.StandardRenameAction.HARDLINK.rename(f, new File(_args.output, "${ut_label}/${ut_title}/${f.name}")) 
}

// delete extracted files later
extractedFiles*.delete()

[rednoah]

Yes, that'll work. Though I'd make it crash with an exception so the process returns with an error exit code. Also fixed your boolean logic as you came up with an tautology

Code: Select all

if (ut_label != 'movie' && ut_label != 'tv' ) {
	throw new Exception("Invalid label: $ut_label")
}

[jkaberg]

Thanks! Updated Couchpotato post accordingly

This will be very good for Couchpotato and torrents

[jkaberg]

Ok so one more hehe, seems I cant bother you enough eh?

After I downloaded a few movies to test it, I found myself in need of filtering out the "sample/trailer" media files as I dont want them and the script was hardlinking thoes when it had extracted a movie

This is my attempt to fix it (by borrowing code from utorrent processor)

Code: Select all

// Author of this script is Rednoah at Filebot Forums, http://filebot.sourceforge.net
// Usage: filebot -script "C:/Downloaders/fbcpto.groovy" --output "C:/Downloaded/_postprocess" --def "ut_dir=%D" "ut_file=%F" "ut_kind=%K" "ut_title=%N" "ut_label=%L"

def files = []
if (ut_label != 'movie' && ut_label != 'tv' ) {
   throw new Exception("Invalid label: $ut_label")
}

if (ut_kind == 'single') { files += new File(ut_dir, ut_file) }
if (ut_kind == 'multi')  { files += new File(ut_dir).getFiles() }


// flatten nested file structure
files = files.flatten()

// extract archives (zip, rar, etc) that contain at least one video file
def extractedFiles = extract(file: files.findAll{ it.isArchive() }, output: '_extract', conflict: 'override', filter: { it.isVideo() })
files += extractedFiles

// sanitize input
files = files.findAll{ it?.exists() }.collect{ it.canonicalFile }.unique()

// ignore clutter files
files = files.findAll{ !(it.path =~ /\b(?i:sample|trailer|extras|deleted.scenes|music.video|scrapbook|behind.the.scenes)\b/) }

// print input fileset
files.each{ f -> _log.finest("Input: $f") }

// process only media files
files.findAll{ it.isVideo() }.each{ f ->
   println net.sourceforge.filebot.StandardRenameAction.HARDLINK.rename(f, new File(_args.output, "${ut_label}/${ut_title}/${f.name}")) 
}

// delete extracted files later
extractedFiles*.delete()

But it didnt fly to well (read: the script did nothing ), can you spot my error?

[rednoah]

There was some bug with forceExtractAll. Must be set to true as a workaround for now:

Code: Select all

def extractedFiles = extract(file: files.findAll{ it.isArchive() }, output: '_extract', conflict: 'override', filter: { it.isVideo() }, forceExtractAll: true)

[jkaberg]

Thanks red, I hope you would like to awnser one more So the file permission issue is still there but I've found a program which unlockes "locked" files

So I made this but it dosnt work very well, maybe you could point me in the right direction?

Code: Select all

// process only media files
files.findAll{ it.isVideo() }.each{ f ->

	outputThis = _args.output "${ut_label}/${ut_title}/${f.name}"
	
	//adding function to unlock files locked by uTorrent
	outputCatcher = new StringBuffer()
	errorCatcher = new StringBuffer()
	proc = "C:\Program Files (x86)\Unlocker\Unlocker.exe " outputThis " -s".execute()
	proc.consumeProcessOutput(outputCatcher, errorCatcher)
	proc.waitFor()
	
	println net.sourceforge.filebot.StandardRenameAction.HARDLINK.rename(f, new File(outputThis))
	println outputCatcher
}

Thanks again

[rednoah]

Actually I added some extras with FileBot for that:

Code: Select all

execute("C:\Program Files (x86)\Unlocker\Unlocker.exe", outputThis, "-s")

Though I guess this point outputThis as not been created. Do you mean f instead?

Also I think you mean this:

Code: Select all

outputThis = new File(_args.output, "${ut_label}/${ut_title}/${f.name}")

[jkaberg]

Actually I added some extras with FileBot for that:

Code: Select all

execute("C:\Program Files (x86)\Unlocker\Unlocker.exe", outputThis, "-s")

Though I guess this point outputThis as not been created. Do you mean f instead?

Also I think you mean this:

Code: Select all

outputThis = new File(_args.output, "${ut_label}/${ut_title}/${f.name}")

Urr, Im not entirely sure about f

This is what I got

Code: Select all

// process only media files
files.findAll{ it.isVideo() }.each{ f ->

	outputThis = new File(_args.output, "${ut_label}/${ut_title}/${f.name}")
	
	println net.sourceforge.filebot.StandardRenameAction.HARDLINK.rename(f, outputThis)
	execute("C:\Program Files (x86)\Unlocker\Unlocker.exe", outputThis, "-s")
}

But I failed since this dosnt work yet.. What I want it to do is "unlock" every file that the script trys to hardlink (because afterwards a postprocessor will try and move/rename the files in the postprocess directory)

An normal "unlock" would be like

Code: Select all

C:\Program Files (x86)\Unlocker\Unlocker.exe F:\Downloaded\_automated\_postprocess\movie\A.Royal.Affair.2012.1080p.BluRay.x264-PFA\A.Royal.Affair.2012.1080p.BluRay.x264-PFA.mkv -s

[rednoah]

Ah, so first you create a hardlink and then you have other scripts moving those hardlinks? Then first hardlink and then run the unlocker.

Also you can't use \ backslash in "" string expression. Try:

Code: Select all

"C:/Program Files (x86)/Unlocker/Unlocker.exe"