Hardlinking with "Fully automated media center" script

Running FileBot from the console, Groovy scripting, shell scripts, etc
Post Reply
jkaberg
Posts: 18
Joined: 30 Dec 2012, 20:21

Hardlinking with "Fully automated media center" script

Post by jkaberg »

Noticed the script supports hardlinks, which would be very nice for me :)
Using --action symlink|keeplink|hardlink requires at least Java 7 and (on Windows) also admin priviledges so you'll need to run µTorrent as admin
But how does it work? Will it extract files that need extracting, and hardlink media files (.mkv, .avi etc?) or will it hardlink everything including .rar files?

Im currently running
filebot -script fn:utorrent-postprocess --output "F:/Downloaded/_automated/_postprocess" --action copy --conflict override -non-strict --def "seriesFormat=tv/%N/{fn}" "animeFormat=anime/%N/{fn}" "movieFormat=movie/%N/{fn}" subtitles=n artwork=n "ut_dir=%D" "ut_file=%F" "ut_kind=%K" "ut_title=%N" "ut_label=%L" "ut_state=%S"
But with the hardlink option I guess it would look like
filebot -script fn:utorrent-postprocess --output "F:/Downloaded/_automated/_postprocess" --action copy hardlink --conflict override -non-strict --def "seriesFormat=tv/%N/{fn}" "animeFormat=anime/%N/{fn}" "movieFormat=movie/%N/{fn}" subtitles=n artwork=n "ut_dir=%D" "ut_file=%F" "ut_kind=%K" "ut_title=%N" "ut_label=%L" "ut_state=%S"
jkaberg
Posts: 18
Joined: 30 Dec 2012, 20:21

Re: Hardlinking with "Fully automated media center" script

Post by jkaberg »

Ok so I tried this

Downloaded a torrent wich only contained a file (a series) with file ending .mkv (which got hardlinked), then I downloaded another series with a .rar set inside which ended up in extracting the series to <rarset dir><torrentname>/<torrentname>.mkv and then hardlinked that file

Shouldnt this just extract it directly to destination without unpacking in the rarset dir first?

My point with using this is not ending up with 2 copys of the .mkv file (less diskspace used) but still beeing able to seed the original files


I used this with uTorrent

Code: Select all

filebot -script fn:utorrent-postprocess --output "F:/Downloaded/_automated/_postprocess" --action hardlink --conflict override -non-strict --def "seriesFormat=tv/%N/{fn}" "animeFormat=anime/%N/{fn}" "movieFormat=movie/%N/{fn}" subtitles=n artwork=n "ut_dir=%D" "ut_file=%F" "ut_kind=%K" "ut_title=%N" "ut_label=%L" "ut_state=%S"
PS: I also tried using clean=y but no avail,
filebot -script fn:utorrent-postprocess --output "F:/Downloaded/_automated/_postprocess" --action hardlink --conflict override -non-strict --def "seriesFormat=tv/%N/{fn}" "animeFormat=anime/%N/{fn}" "movieFormat=movie/%N/{fn}" subtitles=n artwork=n clean=y "ut_dir=%D" "ut_file=%F" "ut_kind=%K" "ut_title=%N" "ut_label=%L" "ut_state=%S"
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Hardlinking with "Fully automated media center" script

Post by rednoah »

Yeah, in that case it'd have to be kinda a mix between copy and hardlink on a case by case basis. You can add that yourself. I won't add that to the default utorrent-postprocess though for the sake of simplicity.
:idea: Please read the FAQ and How to Request Help.
jkaberg
Posts: 18
Joined: 30 Dec 2012, 20:21

Re: Hardlinking with "Fully automated media center" script

Post by jkaberg »

But shouldnt the clean variable remove the extracted content from the "seeded" dir automaticly, atleast thats how I interpreted it
--def clean=y Automatically empty folders and clutter files that may be left behind after moving the video files or temporary extracted files after copying
If thats not the cases, how would one proceed, eg what would need to be altered? Im guessting the extraction process but how do I specify it to extract to destination instead of the "seed dir" and then not hardlink itself in the destination dir (eg hardlink destionation to destinatioul as it would be)

Because as it is now, the only thing I really would need it to do is remove the extracted files in the seed dir (becasuse since they are hardlinked the other "linked file" in the destination dir will still exist and everything will be fine) after makeing the hardlink. And I've tried the clean variable several times, eg clean=y | clean=yes | clean=true with no success, it dosnt remove anything
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Hardlinking with "Fully automated media center" script

Post by rednoah »

Yep, that makes sense. Maybe I'll add it next time. Just look the end of the script. "clean" has different meanings depending on --action.
:idea: Please read the FAQ and How to Request Help.
jkaberg
Posts: 18
Joined: 30 Dec 2012, 20:21

Re: Hardlinking with "Fully automated media center" script

Post by jkaberg »

Thanks for pointing that out! Have now edited the clean function as following

Code: Select all

if (clean) {
	if ('COPY'.equalsIgnoreCase(_args.action) && extractedFiles?.size() > 0) {
		println 'Clean temporary extracted files'
		// delete extracted files
		extractedFiles.each{
			if(it.isFile()) {
				println "Delete $it"
				it.delete()
			}
		}
		// delete remaining empty folders
		extractedFiles*.dir.unique().findAll{ it.listFiles().length == 0 }.each{
			if(it.isDirectory()) {
				println "Delete $it"
				it.delete()
			}
		}
	}
	
	if ('HARDLINK'.equalsIgnoreCase(_args.action) && extractedFiles?.size() > 0) {
		println 'Clean temporary extracted files'
		// delete extracted files
		extractedFiles.each{
			if(it.isFile()) {
				println "Delete $it"
				it.delete()
			}
		}
		// delete remaining empty folders
		extractedFiles*.dir.unique().findAll{ it.listFiles().length == 0 }.each{
			if(it.isDirectory()) {
				println "Delete $it"
				it.delete()
			}
		}
	}
	
	// deleting remaining files only makes sense after moving files
	if ('MOVE'.equalsIgnoreCase(_args.action)) {
		println 'Clean clutter files and empty folders'
		include('fn:cleaner', [:], !args.empty ? args : ut_kind == 'multi' && ut_dir ? [ut_dir as File] : [])
	}
}
And it does indeed remove the extracted files in the seed directory, does hardlink but when I try to process the file in the "postprocess" folder (where its hardlinked or extracted to) I cant access the file without haveing to shutdown uTorrent, do you know why? If i search for the file in Process Explorer I can see that uTorrent is accessing the counterpart in my seed directory, but no process is accessing it in my "postprocess" folder :O
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Hardlinking with "Fully automated media center" script

Post by rednoah »

So it's extracted to to the current directory and then hardlinked to the new organized structure. And you can't remove the file from the downloads folder where the archives are as well?

In anycase I have no idea how Windows deals with file handles and hardlinks. Maybe adding a wait somewhere helps? Is the script failing or is it that later you can't access files?
:idea: Please read the FAQ and How to Request Help.
jkaberg
Posts: 18
Joined: 30 Dec 2012, 20:21

Re: Hardlinking with "Fully automated media center" script

Post by jkaberg »

Appreciate your help and knowledge rednoah! The hardlinking problem was related to uTorrent, resolved with setting diskio.flush_files to false in uTorrent (advanced settings)
diskio.flush_files: Enabling this option causes µTorrent to close file handles every minute. It helps to reduce the effect of Windows managing the system cache badly for some people and causing apparent “memory leaks.”
Seems everything works as intended now :)
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Hardlinking with "Fully automated media center" script

Post by rednoah »

Google just told me about your new thread on Couchpotato:
https://couchpota.to/forum/thread-1454-post-6270.html

You're completely abusing the utorrent-postprocess script:
* by setting label 'movie' you disable any anime/series/movie auto-detection
* your movie format --def movieFormat=movie/%N/{fn} is completely oblivious of whatever movie-detection comes up with. Now that's really just wasting TheMovieDB bandwith for no reason.

So you've disabled any and all processing... What's the point? I guess you need extracting and hardlinking, and nothing else.

e.g. After deleting almost everything from my utorrent-postprocess you should come up with something like this:

Code: Select all

filebot -script "C:/fbcpto.groovy" --output "C:/hl" --def "ut_dir=%D" "ut_file=%F" "ut_kind=%K" "ut_title=%N" "ut_label=%L"

Code: Select all

def files = []
if (ut_kind == 'single') { files += new File(ut_dir, ut_file) }
if (ut_kind == 'multi')  { files += new File(ut_dir).getFiles() }

// extract archives (zip, rar, etc) that contain at least one video file
def extractedFiles = extract(file: files.findAll{ it.isArchive() }, output: '_extract', conflict: 'override', filter: { it.isVideo() })
files += extractedFiles

files.findAll{ it.isVideo() }.each{ f ->
	println net.sourceforge.filebot.StandardRenameAction.HARDLINK.rename(f, new File(_args.output, "${ut_label}/${ut_title}/${f.name}")) 
}

// delete extracted files later
extractedFiles*.delete()
:idea: Please read the FAQ and How to Request Help.
jkaberg
Posts: 18
Joined: 30 Dec 2012, 20:21

Re: Hardlinking with "Fully automated media center" script

Post by jkaberg »

Im sorry if I miss used your script rednoah, wasnt my intention to "void" anything. But my lack of knowledge in scripting got me to where I am with the "Fully automated media center script" (I've actully tried to do what you did with the script you posted one up)

Anyhow I'll edit my post at couchpotato forums with credits towards you


Thanks again
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Hardlinking with "Fully automated media center" script

Post by rednoah »

Yeah, the one I posted should to pretty much exactly what you want. Except in alot less lines of code (10 lines instead of 300). ;)

Let me know if you have any issues.
:idea: Please read the FAQ and How to Request Help.
jkaberg
Posts: 18
Joined: 30 Dec 2012, 20:21

Re: Hardlinking with "Fully automated media center" script

Post by jkaberg »

Yeah there was one thing I would wish to add, did I get this right? I would like the script to quit if the label isnt movie or tv :)

Code: Select all

if (ut_label != 'movie' || ut_label != 'tv' ) { return null }
eg full code

Code: Select all

// Author of this script is Rednoah at Filebot Forums, http://filebot.sourceforge.net
// Usage: filebot -script "C:/Downloaders/fbcpto.groovy" --output "C:/Downloaded/_postprocess" --def "ut_dir=%D" "ut_file=%F" "ut_kind=%K" "ut_title=%N" "ut_label=%L"

def files = []
if (ut_label != 'movie' || ut_label != 'tv' ) { return null }
if (ut_kind == 'single') { files += new File(ut_dir, ut_file) }
if (ut_kind == 'multi')  { files += new File(ut_dir).getFiles() }

// extract archives (zip, rar, etc) that contain at least one video file
def extractedFiles = extract(file: files.findAll{ it.isArchive() }, output: '_extract', conflict: 'override', filter: { it.isVideo() })
files += extractedFiles

files.findAll{ it.isVideo() }.each{ f ->
   println net.sourceforge.filebot.StandardRenameAction.HARDLINK.rename(f, new File(_args.output, "${ut_label}/${ut_title}/${f.name}")) 
}

// delete extracted files later
extractedFiles*.delete()
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Hardlinking with "Fully automated media center" script

Post by rednoah »

Yes, that'll work. Though I'd make it crash with an exception so the process returns with an error exit code. Also fixed your boolean logic as you came up with an tautology :P

Code: Select all

if (ut_label != 'movie' && ut_label != 'tv' ) {
	throw new Exception("Invalid label: $ut_label")
}
:idea: Please read the FAQ and How to Request Help.
jkaberg
Posts: 18
Joined: 30 Dec 2012, 20:21

Re: Hardlinking with "Fully automated media center" script

Post by jkaberg »

Thanks! Updated Couchpotato post accordingly

This will be very good for Couchpotato and torrents :)
jkaberg
Posts: 18
Joined: 30 Dec 2012, 20:21

Re: Hardlinking with "Fully automated media center" script

Post by jkaberg »

Ok so one more hehe, seems I cant bother you enough eh? :)

After I downloaded a few movies to test it, I found myself in need of filtering out the "sample/trailer" media files as I dont want them and the script was hardlinking thoes when it had extracted a movie

This is my attempt to fix it (by borrowing code from utorrent processor)

Code: Select all

// Author of this script is Rednoah at Filebot Forums, http://filebot.sourceforge.net
// Usage: filebot -script "C:/Downloaders/fbcpto.groovy" --output "C:/Downloaded/_postprocess" --def "ut_dir=%D" "ut_file=%F" "ut_kind=%K" "ut_title=%N" "ut_label=%L"

def files = []
if (ut_label != 'movie' && ut_label != 'tv' ) {
   throw new Exception("Invalid label: $ut_label")
}

if (ut_kind == 'single') { files += new File(ut_dir, ut_file) }
if (ut_kind == 'multi')  { files += new File(ut_dir).getFiles() }


// flatten nested file structure
files = files.flatten()

// extract archives (zip, rar, etc) that contain at least one video file
def extractedFiles = extract(file: files.findAll{ it.isArchive() }, output: '_extract', conflict: 'override', filter: { it.isVideo() })
files += extractedFiles

// sanitize input
files = files.findAll{ it?.exists() }.collect{ it.canonicalFile }.unique()

// ignore clutter files
files = files.findAll{ !(it.path =~ /\b(?i:sample|trailer|extras|deleted.scenes|music.video|scrapbook|behind.the.scenes)\b/) }

// print input fileset
files.each{ f -> _log.finest("Input: $f") }

// process only media files
files.findAll{ it.isVideo() }.each{ f ->
   println net.sourceforge.filebot.StandardRenameAction.HARDLINK.rename(f, new File(_args.output, "${ut_label}/${ut_title}/${f.name}")) 
}

// delete extracted files later
extractedFiles*.delete()
But it didnt fly to well (read: the script did nothing :P), can you spot my error? :)
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Hardlinking with "Fully automated media center" script

Post by rednoah »

There was some bug with forceExtractAll. Must be set to true as a workaround for now:

Code: Select all

def extractedFiles = extract(file: files.findAll{ it.isArchive() }, output: '_extract', conflict: 'override', filter: { it.isVideo() }, forceExtractAll: true)
:idea: Please read the FAQ and How to Request Help.
jkaberg
Posts: 18
Joined: 30 Dec 2012, 20:21

Re: Hardlinking with "Fully automated media center" script

Post by jkaberg »

Thanks red, I hope you would like to awnser one more :) So the file permission issue is still there but I've found a program which unlockes "locked" files

So I made this but it dosnt work very well, maybe you could point me in the right direction? :)

Code: Select all

// process only media files
files.findAll{ it.isVideo() }.each{ f ->

	outputThis = _args.output "${ut_label}/${ut_title}/${f.name}"
	
	//adding function to unlock files locked by uTorrent
	outputCatcher = new StringBuffer()
	errorCatcher = new StringBuffer()
	proc = "C:\Program Files (x86)\Unlocker\Unlocker.exe " outputThis " -s".execute()
	proc.consumeProcessOutput(outputCatcher, errorCatcher)
	proc.waitFor()
	
	println net.sourceforge.filebot.StandardRenameAction.HARDLINK.rename(f, new File(outputThis))
	println outputCatcher
}
Thanks again
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Hardlinking with "Fully automated media center" script

Post by rednoah »

Actually I added some extras with FileBot for that:

Code: Select all

execute("C:\Program Files (x86)\Unlocker\Unlocker.exe", outputThis, "-s")
Though I guess this point outputThis as not been created. Do you mean f instead?

Also I think you mean this:

Code: Select all

outputThis = new File(_args.output, "${ut_label}/${ut_title}/${f.name}")
:idea: Please read the FAQ and How to Request Help.
jkaberg
Posts: 18
Joined: 30 Dec 2012, 20:21

Re: Hardlinking with "Fully automated media center" script

Post by jkaberg »

rednoah wrote:Actually I added some extras with FileBot for that:

Code: Select all

execute("C:\Program Files (x86)\Unlocker\Unlocker.exe", outputThis, "-s")
Though I guess this point outputThis as not been created. Do you mean f instead?

Also I think you mean this:

Code: Select all

outputThis = new File(_args.output, "${ut_label}/${ut_title}/${f.name}")

Urr, Im not entirely sure about f

This is what I got

Code: Select all

// process only media files
files.findAll{ it.isVideo() }.each{ f ->

	outputThis = new File(_args.output, "${ut_label}/${ut_title}/${f.name}")
	
	println net.sourceforge.filebot.StandardRenameAction.HARDLINK.rename(f, outputThis)
	execute("C:\Program Files (x86)\Unlocker\Unlocker.exe", outputThis, "-s")
}
But I failed since this dosnt work yet.. What I want it to do is "unlock" every file that the script trys to hardlink (because afterwards a postprocessor will try and move/rename the files in the postprocess directory)

An normal "unlock" would be like

Code: Select all

C:\Program Files (x86)\Unlocker\Unlocker.exe F:\Downloaded\_automated\_postprocess\movie\A.Royal.Affair.2012.1080p.BluRay.x264-PFA\A.Royal.Affair.2012.1080p.BluRay.x264-PFA.mkv -s
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Hardlinking with "Fully automated media center" script

Post by rednoah »

Ah, so first you create a hardlink and then you have other scripts moving those hardlinks? Then first hardlink and then run the unlocker.

Also you can't use \ backslash in "" string expression. Try:

Code: Select all

"C:/Program Files (x86)/Unlocker/Unlocker.exe"
:idea: Please read the FAQ and How to Request Help.
Post Reply