2 Questions: 20+ minutes to copy, move to destination automatically

Running FileBot from the console, Groovy scripting, shell scripts, etc
Post Reply
Tanyac
Posts: 20
Joined: 17 Nov 2014, 05:28

2 Questions: 20+ minutes to copy, move to destination automatically

Post by Tanyac »

(1). Sometime ago I posted here about files taking as much as 20 minutes to copy. During that time the PC literally stopped dead. The second part of that statement is fixed, but large files can still take 20 minutes or more to copy. Eg a 2160p 60gb file.

The AMC command line I use is;

Code: Select all

filebot -script fn:amc --action hardlink --output "E:/Media" --action copy --conflict skip -non-strict --log-file amc.log --def unsorted=y reporterror=y storereport=y "ut_label=%L" "ut_state=%S" "ut_title=%N" "ut_kind=%K" "ut_file=%F" "ut_dir=%D"
Is there any way to fix this?

(2). I have a PC dedicated to Kodi and has a RSS feed set up. It currently moves the downloaded files to a location on my file server so I can access them from anywhere and move them to the correct final location. But given that AMC is so powerful and versatile, it can move the files fore me. I just can't seem to get the correct AMC command line.

I'm currently using;

Code: Select all

filebot -script fn:amc --action copy --conflict skip -non-strict --log-file amc.log --def artwork=y unsorted=y reporterror=y storereport=y seriesFormat="T:/{n}/{s}}/{n} - {s00e00} - {t}" excludelist=amc.txt --filter "age < 10" "ut_label=%L" "ut_state=%S" "ut_title=%N" "ut_kind=%K" "ut_file=%F" "ut_dir=%D"
I removed --output as if I understand it, that is the output folder where the file will end up, and I want the location controlled by the seriesFormat. Obviously I'm on the wrong track because it doesn't work.

I've read several posts here and tried a few combinations, and a post regarding the seriesFormat seemed the most positive going by the comments.. What am I doing wrong?

Thanks for your help.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: 2 Questions: 20+ minutes to copy, move to destination automatically

Post by rednoah »

1.
Since you're using --action hardlink that operation should instantly succeed or fail, and I don't see any other reason why it would be slow for large files, as FileBot typically doesn't read / write any files.

Can you narrow down specific test cases that take unusually long? Can you post the console output for such a particularly long run?


2.
What does the console output say? Your format is wrong, i.e. {s}}, so it'll probably just print an error message telling you your format is wrong on startup and then quit.

The --output option is also a required option which defaults to the CWD if left unspecified. It's used for a few things, so you can't not specify it even if you use absolute formats.

i.e. do this:

Code: Select all

--output "E:/Media" --def seriesFormat="T:/{plex}"

3.
Use --log-file E:/amc.log so you can find the log file easily, and check up on what FileBot is telling you when something is not working.
:idea: Please read the FAQ and How to Request Help.
Tanyac
Posts: 20
Joined: 17 Nov 2014, 05:28

Re: 2 Questions: 20+ minutes to copy, move to destination automatically

Post by Tanyac »

Thanks for your response. Sorry, I should have mentioned this is on two different PCs. The Kodi PC is the one that uses the series format and has the RSS feed that I'm trying to automate. So, Ill put back the --output and fix the }} and retest.

The }} is embarrassing. I have been starting at the line for hours... Can't see the forest for the trees :)

My PC is the one that has the long delays. next time it occurs I will copy the console output and post.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: 2 Questions: 20+ minutes to copy, move to destination automatically

Post by rednoah »

Tanyac wrote: 19 Oct 2018, 01:12 The }} is embarrassing. I have been starting at the line for hours... Can't see the forest for the trees :)
You're not a computer. Don't look at the input. Look at the output the computer gives back to you. In this case I'm sure it'd say something about } being unexpected. ;)
:idea: Please read the FAQ and How to Request Help.
Tanyac
Posts: 20
Joined: 17 Nov 2014, 05:28

Re: 2 Questions: 20+ minutes to copy, move to destination automatically

Post by Tanyac »

I've fixed the silly mistake and added --output back into the command.

Code: Select all

filebot -script fn:amc --action copy --conflict skip -non-strict --log-file amc.log --def artwork=y unsorted=y reporterror=y storereport=y seriesFormat="T:/{n}/{s}/{n} - {s00e00} - {t}" excludelist=amc.txt --output "O:/KodiMedia" --filter "age < 10" "ut_label=%L" "ut_state=%S" "ut_title=%N" "ut_kind=%K" "ut_file=%F" "ut_dir=%D"
This is what the log shows...

Code: Select all

Illegal Argument: java.io.IOException: The filename, directory name, or volume label syntax is incorrect (ut_dir=D:\uTorrent\Marvels.Daredevil.S03E01.720p.WEB.x264-STRiFE[rarbg])
Run script [fn:amc] at [Fri Oct 19 20:32:49 AEDT 2018]
Parameter: artwork = y
Parameter: unsorted = y
Parameter: reporterror = y
Parameter: storereport = y
Parameter: seriesFormat = T:/{n}/{s}/{n} - {s00e00} - {t}
Parameter: excludelist = amc.txt
Argument[0]: File does not exist: C:\Tools\ut_label=ShowRSS
Argument[1]: File does not exist: C:\Tools\ut_state=11
Argument[2]: File does not exist: C:\Tools\ut_title=Marvels.Daredevil.S03E01.720p.WEB.x264-STRiFE[rarbg]
Argument[3]: File does not exist: C:\Tools\ut_kind=multi
Argument[4]: File does not exist: C:\Tools\ut_file=RARBG.txt
Argument[5]: File does not exist: ut_dir=D:\uTorrent\Marvels.Daredevil.S03E01.720p.WEB.x264-STRiFE[rarbg]
Illegal char <:> at index 8: ut_dir=D:\uTorrent\Marvels.Daredevil.S03E01.720p.WEB.x264-STRiFE[rarbg]
java.nio.file.InvalidPathException: Illegal char <:> at index 8: ut_dir=D:\uTorrent\Marvels.Daredevil.S03E01.720p.WEB.x264-STRiFE[rarbg]
	at net.filebot.util.FileSet.getPath(FileSet.java:90)
	at net.filebot.util.FileSet.contains(FileSet.java:82)
	at java_util_Set$contains.call(Unknown Source)
	at Script1.acceptFile(Script1.groovy:267)
	at Script1$_run_closure47.doCall(Script1.groovy:330)
	at Script1.run(Script1.groovy:330)
	at net.filebot.cli.ScriptShell.evaluate(ScriptShell.java:64)
	at net.filebot.cli.ScriptShell.runScript(ScriptShell.java:74)
	at net.filebot.cli.ArgumentProcessor.runScript(ArgumentProcessor.java:113)
	at net.filebot.cli.ArgumentProcessor.run(ArgumentProcessor.java:30)
	at net.filebot.Main.main(Main.java:120)

Failure (°_°)
I don't know what this C:\Tools\ rubbish is. I do have such a folder, but why it's suddenly included in the output I don't know.

The folder as specified in ut_dir does exist.

Sorry.. Forgot to add this..

Code: Select all

C:\Program Files\FileBot>filebot -script fn:sysinfo
FileBot 4.7.5 (r4600)
JNA Native: 4.0.1
MediaInfo: 0.7.88
7-Zip-JBinding: 9.20
Chromaprint: 1.1.0
Extended Attributes: OK
Unicode Filesystem: OK
Script Bundle: 2018-03-16 (r516)
Groovy: 2.4.7
JRE: Java(TM) SE Runtime Environment 1.8.0_181
JVM: 64-bit Java HotSpot(TM) 64-Bit Server VM
CPU/MEM: 20 Core / 7 GB Max Memory / 47 MB Used Memory
OS: Windows 10 (amd64)
Package: MSI
Last edited by Tanyac on 19 Oct 2018, 09:55, edited 1 time in total.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: 2 Questions: 20+ minutes to copy, move to destination automatically

Post by rednoah »

Just means that small errors in your command-line can screw up everything. Position matters. In this case, you just added some normal --options in between a --def name=value name=value option sequence, breaking the sequence, and so everything following your --filter option is interpreted as input file argument.

BAD:

Code: Select all

filebot ... "ut_label=%L" "ut_state=%S" "ut_title=%N" "ut_kind=%K" "ut_file=%F" "ut_dir=%D"
GOOD:

Code: Select all

filebot ... --def "ut_label=%L" "ut_state=%S" "ut_title=%N" "ut_kind=%K" "ut_file=%F" "ut_dir=%D"
GOOD:

Code: Select all

filebot ... --def "ut_label=%L" --def "ut_state=%S" --def "ut_title=%N" --def "ut_kind=%K" --def "ut_file=%F" --def "ut_dir=%D"
:idea: Please read the FAQ and How to Request Help.
Tanyac
Posts: 20
Joined: 17 Nov 2014, 05:28

Re: 2 Questions: 20+ minutes to copy, move to destination automatically

Post by Tanyac »

Ok, If I'm reading you right.. this is better..

Code: Select all

filebot -script fn:amc --action copy --conflict skip -non-strict --log-file amc.log --output "O:/KodiMedia" --filter "age < 10" --def artwork=y unsorted=y reporterror=y storereport=y seriesFormat="T:/{n}/{s}/{n} - {s00e00} - {t}" excludelist=amc.txt "ut_label=%L" "ut_state=%S" "ut_title=%N" "ut_kind=%K" "ut_file=%F" "ut_dir=%D"
I'll give it a try now... Much better... Another omission.. Forgot the word "Season " before the {s}..

It's moving the files now.

Thanks for your help
Tanyac
Posts: 20
Joined: 17 Nov 2014, 05:28

Re: 2 Questions: 20+ minutes to copy, move to destination automatically

Post by Tanyac »

May be I spoke to soon. This is now the command line;

Code: Select all

filebot -script fn:amc --action copy --conflict skip -non-strict --log-file amc.log --output "O:/KodiMedia" --filter "age < 10" --def artwork=y unsorted=y reporterror=y storereport=y seriesFormat="T:/{n}/Season {s}/{n} - {s00e00} - {t}" excludelist=amc.txt "ut_label=%L" "ut_state=%S" "ut_title=%N" "ut_kind=%K" "ut_file=%F" "ut_dir=%D"
And this is the output;

Code: Select all

Run script [fn:amc] at [Sun Oct 21 12:08:15 AEDT 2018]
Parameter: artwork = y
Parameter: unsorted = y
Parameter: reporterror = y
Parameter: storereport = y
Parameter: seriesFormat = T:/{n}/Season {s}/{n} - {s00e00} - {t}
Parameter: excludelist = amc.txt
Parameter: ut_label = ShowRSS
Parameter: ut_state = 5
Parameter: ut_title = Chicago.Med.S04E04.PROPER.720p.AMZN.WEBRip.DDP5.1.x264-KiNGS[rarbg]
Parameter: ut_kind = multi
Parameter: ut_file = Chicago.Med.S04E04.PROPER.Backed.Against.the.Wall.720p.AMZN.WEB-DL.DDP5.1.H.264-KiNGS.mkv
Parameter: ut_dir = D:\uTorrent\Chicago.Med.S04E04.PROPER.720p.AMZN.WEBRip.DDP5.1.x264-KiNGS[rarbg]
Input: D:\uTorrent\Chicago.Med.S04E04.PROPER.720p.AMZN.WEBRip.DDP5.1.x264-KiNGS[rarbg]\Chicago.Med.S04E04.PROPER.Backed.Against.the.Wall.720p.AMZN.WEB-DL.DDP5.1.H.264-KiNGS.mkv
Group: [tvs:chicago med] => [Chicago.Med.S04E04.PROPER.Backed.Against.the.Wall.720p.AMZN.WEB-DL.DDP5.1.H.264-KiNGS.mkv]
Rename episodes using [TheTVDB]
Auto-detected query: [Chicago Med]
Fetching episode data for [Chicago Med]
Fetching episode data for [Chicago P.D.]
Fetching episode data for [The Chicago Code]
Fetching episode data for [Chicago Hope]
Fetching episode data for [Chicago Fire]
Apply Filter: {age < 10}
Include [Chicago Med - 4x04 - Backed Against the Wall]
Include [Chicago P.D. - 6x04 - Ride Along]
Include [Chicago Fire - 7x04 - This Isn't Charity]
[COPY] Rename [D:\uTorrent\Chicago.Med.S04E04.PROPER.720p.AMZN.WEBRip.DDP5.1.x264-KiNGS[rarbg]\Chicago.Med.S04E04.PROPER.Backed.Against.the.Wall.720p.AMZN.WEB-DL.DDP5.1.H.264-KiNGS.mkv] to [T:\Chicago Med\Season 4\Chicago Med - S04E04 - Backed Against the Wall.mkv]
[COPY] Failed to rename [D:\uTorrent\Chicago.Med.S04E04.PROPER.720p.AMZN.WEBRip.DDP5.1.x264-KiNGS[rarbg]\Chicago.Med.S04E04.PROPER.Backed.Against.the.Wall.720p.AMZN.WEB-DL.DDP5.1.H.264-KiNGS.mkv]
Processed 0 files
T:\Chicago Med\Season 4: Unable to determine if root directory exists
Processing 1 unsorted files
Illegal parameter: map
java.lang.IllegalArgumentException: Illegal parameter: map
	at net.filebot.cli.ScriptShellBaseClass.lambda$getArgumentBean$1(ScriptShellBaseClass.java:470)
	at net.filebot.cli.ScriptShellBaseClass.getArgumentBean(ScriptShellBaseClass.java:464)
	at net.filebot.cli.ScriptShellBaseClass.rename(ScriptShellBaseClass.java:346)
	at Script1.run(Script1.groovy:538)
	at net.filebot.cli.ScriptShell.evaluate(ScriptShell.java:64)
	at net.filebot.cli.ScriptShell.runScript(ScriptShell.java:74)
	at net.filebot.cli.ArgumentProcessor.runScript(ArgumentProcessor.java:113)
	at net.filebot.cli.ArgumentProcessor.run(ArgumentProcessor.java:30)
	at net.filebot.Main.main(Main.java:120)
Caused by: java.lang.NoSuchFieldException: map
	at net.filebot.cli.ScriptShellBaseClass.lambda$getArgumentBean$1(ScriptShellBaseClass.java:466)
	... 8 more

Failure (°_°)
I suspect this has something to do with the fact that mapped network drives are in a disconnected state when Windows 10 starts. To fix that I run a batch file at start up which essentially gives the mapped drives a kick start. It doesn't work - Unless I run the batch file from explorer. This has to do with the way that mapped drives are connected in the user vs administrator context.

What's curious is that the O:/ drive is a mapped drive. It startup state is disconnected like all other drives. However, prior to adding the season format filebot successfully moved media files without error. Perhaps it is doing something funky that "reconnects" the drive. Don't know the answer.

There is a registry value called "EnableLinkedConnections" which allows the connections to be used across both contexts when running in an elevated state. It doesn't appear to have solved the problem

Any ideas?

I've come up with one rather messy idea - which is to use a local drive in seriesFormat. Have a tasks that runs regularly and moves the files to the server. But something less combersom would be preferred.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: 2 Questions: 20+ minutes to copy, move to destination automatically

Post by rednoah »

1.
This error message tells us as much. A very unexpected error message I've never seen before breaks things:

Code: Select all

T:\Chicago Med\Season 4: Unable to determine if root directory exists
This is unrelated, probably and odd bug in a fall-back feature that you wouldn't normally hit, if everything else works. Most likely an issue limited to the extremely outdated version of FileBot you're using:

Code: Select all

java.lang.IllegalArgumentException: Illegal parameter: map
Unfortunately, not much you can do. It's more of a Windows issue. FileBot doesn't know or care what drive latter is local or remote.

However, you can try to use \\UNC\share paths and try that instead of the mapped drive, which might allow you to work around the problem.


2.
FileBot 4.7.5 (r4600) is extremely outdated, so I won't spend time on troubleshooting issues specific to this version. I recommend purchasing a license and using the latest version, as I can only provide support for the latest version.
:idea: Please read the FAQ and How to Request Help.
Tanyac
Posts: 20
Joined: 17 Nov 2014, 05:28

Re: 2 Questions: 20+ minutes to copy, move to destination automatically

Post by Tanyac »

rednoah wrote: 21 Oct 2018, 07:10 FileBot 4.7.5 (r4600) is extremely outdated, so I won't spend time on troubleshooting issues specific to this version. I recommend purchasing a license and using the latest version, as I can only provide support for the latest version.
Fair point. I will do as you suggest. I've just spent $400 on a new bluray player and some software so it will have to wait until my next pension payment comes in, which is about 2 weeks or so. In the mean time I gather I can update to the latest version for free. I don't generally update software unless it's broken. Endless updates are very annoying when you have dozens of programs all wanting to continually update. If you use Windows 10 you'll know what I mean.

The environment here is standardized for 10 PCs, and all built from a single image with post install automation. Sadly, using UNC paths is not an option at this stage due to the range of applications being used and the user base. However, that doesn't mean I can't test it. I have a PC set aside specifically for testing purposes, so I'll give that a try too.

I'll respond back once I've updated to the latest version.

**EDIT:** $70 AUD for a license? That's rather steep!
Tanyac
Posts: 20
Joined: 17 Nov 2014, 05:28

Re: 2 Questions: 20+ minutes to copy, move to destination automatically

Post by Tanyac »

Ok, tried the latest version and the issue was the same. It doesn't like disconnected drives.
However, I found a way around it by running a batch script from the user context when uTorrent starts so that the drives are connected when amc script runs. All working now.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: 2 Questions: 20+ minutes to copy, move to destination automatically

Post by rednoah »

1.
The affordable $6 per year option is generally recommended.


2.
Tanyac wrote: 26 Oct 2018, 04:44 However, I found a way around it by running a batch script from the user context when uTorrent starts so that the drives are connected when amc script runs. All working now.
Mind sharing what exactly you did there to make this work? Might be useful for other users with similar problems in the future.
:idea: Please read the FAQ and How to Request Help.
Tanyac
Posts: 20
Joined: 17 Nov 2014, 05:28

Re: 2 Questions: 20+ minutes to copy, move to destination automatically

Post by Tanyac »

I just found this notification. I don't know why I haven't seen this for two years?
I purchased the full license. I believe at the exchange rate at the time resulted in a $76 cost. All good.

Windows seems to have had an issue with reconnecting drives at a login since at least Windows Vista. What I ended up doing was;

1. Ensure this registry key is disabled (0);

Code: Select all

"HKLM\Software\Policies\Microsoft\Windows\WcmSvc\GroupPolicy\fSoftDisconnectConnections"
2. Ensure this registry key is disabled (0):

Code: Select all

"HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\NetworkProvider\RestoreConnection"
3. Create a task that is run from task scheduler, in the user context that calls a vbs script (so the window is hidden), which runs at user logon. This vbs script simply calls a script called login.cmd, which executes net use commands to re-establish the shared drives. The task runs every two hours.

The vbs script;

Code: Select all

Dim WinScriptHost
Set WinScriptHost = CreateObject("WScript.Shell")
WinScriptHost.Run "C:\Tools\login.cmd", 0
Set WinScriptHost = Nothing
The login.cmd file is now used for a multitude of tasks, but this is the relevant section;

Code: Select all

net use m: \\server\applications /persistent:yes 
net use n: \\server\games /persistent:yes 
net use o: \\server\images /persistent:yes 
net use p: \\server\music /persistent:yes 
net use q: \\server\develop /persistent:yes 
There are a lot more than the above, but you get the gist. The shares are password protected so an Enterprise credential has been created in windows when it was installed using the cmdkey command.
Post Reply