SSH renaming of multiple folders and files

magicacaca · Post by **magicacaca** » 02 Jan 2020, 21:15

Hi,

I have everything set up and have done some tests to ensure Filebot is working as expected via SSH from my Mac to a QNAP. However, as a noob I'm unsure of the script to get all folders and files into a consistent format. What I have is (for example):

/share/MD0_DATA/Multimedia/Movies\ \&\ TV/Movies/Alien/Alien.mkv
/share/MD0_DATA/Multimedia/Movies\ \&\ TV/Movies/Aliens/Aliens.mkv
/share/MD0_DATA/Multimedia/Movies\ \&\ TV/Movies/Alien\ \3\/Alien\ \3\.mkv
/share/MD0_DATA/Multimedia/Movies\ \&\ TV/Movies/Alien\ \Covenant\.mkv
/share/MD0_DATA/Multimedia/Movies\ \&\ TV/Movies/After\ \Hours\.mkv

What I want is to do the following in one go:

- File names to have the movie year appended
- Folder names to have the year appended
- Files that are in the /Movies/ folder to each have a folder created with the movie name and year and for the file to be moved into it's respective folder

So:

Movies/Alien/Alien.mkv
would become
Movies/Alien (1979)/Alien (1979).mkv

Movies/Alien\ \Covenant\.mkv
would become
Movies/Alien\ \Covenant\ $2017$/Alien\ \Covenant\ $2017$.mkv

What I'm struggling with is how to wildcard files so that every file is included, whether it's in the /Movies/ folder itself or already within it's own sub-folder.

Help greatly appreciated!

Post by **rednoah** » 02 Jan 2020, 21:59

Have you tried the amc script yet?

1.
Rename Movies & TV to Input. This is our input folder. Create a folder called Media to serve as output folder. Cleanly separating input / output is essential. You don't want to re-organize an existing structure. You want to generate a new structure from the old one.

2.
Do a test run with --action TEST as follows:

Code: Select all

filebot -script fn:amc "/share/MD0_DATA/Multimedia/Input" --output "/share/MD0_DATA/Multimedia/Media" -non-strict  --log-file amc.log --def excludeList=amc.txt --action TEST

3.
Read the output carefully. Make sure the amc script is working as expected.

4.
Do a real run with --action HARDLINK as follows. We now have a neatly organized file structure, as well as the original file structure. We want to keep both. Since we use --action HARDLINK there are no downsides whatsoever to keeping both file structures.

magicacaca · Post by **magicacaca** » 02 Jan 2020, 22:41

Thanks for the quick reply. AMC seems to be exactly what I need. I'm getting an error though:

Code: Select all

# filebot -script fn:amc "/share/MD0_DATA/Multimedia/INPUT" --output "/share/MD0_DATA/Multimedia/OUTPUT" -non-strict  --log-file amc.log --def excludeList=amc.txt --action TEST
Locking /share/MD0_DATA/.qpkg/filebot/data/admin/logs/amc.log
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.codehaus.groovy.vmplugin.v7.Java7$1 (file:/share/MD0_DATA/.qpkg/filebot/jar/groovy.jar) to constructor java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int)
WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.vmplugin.v7.Java7$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
BUG! exception in phase 'semantic analysis' in source unit 'Script1.groovy' Unsupported class file major version 57
BUG! exception in phase 'semantic analysis' in source unit 'Script1.groovy' Unsupported class file major version 57
	at net.filebot.cli.ScriptShell.evaluate(ScriptShell.java:64)
	at net.filebot.cli.ScriptShell.runScript(ScriptShell.java:74)
	at net.filebot.cli.ArgumentProcessor.runScript(ArgumentProcessor.java:163)
	at net.filebot.cli.ArgumentProcessor.run(ArgumentProcessor.java:37)
	at net.filebot.Main.main(Main.java:132)
Caused by: java.lang.IllegalArgumentException: Unsupported class file major version 57
	at groovyjarjarasm.asm.ClassReader.<init>(ClassReader.java:184)
	at groovyjarjarasm.asm.ClassReader.<init>(ClassReader.java:166)
	at groovyjarjarasm.asm.ClassReader.<init>(ClassReader.java:152)
	at groovyjarjarasm.asm.ClassReader.<init>(ClassReader.java:273)
	... 5 more

Error (o_O)

Post by **rednoah** » 02 Jan 2020, 23:37

You may need to upgrade to the latest beta as discussed here:
viewtopic.php?f=13&t=11099

magicacaca · Post by **magicacaca** » 04 Jan 2020, 13:07

That seems to have fixed it, thanks. Running the test now.

magicacaca · Post by **magicacaca** » 05 Jan 2020, 06:26

Confirmed, all working, thanks!

Shame the node version doesn't want to load so that I can manage the NAS remotely. I just get the "Service Unavailable" error.

Post by **rednoah** » 05 Jan 2020, 10:41

magicacaca wrote: ↑05 Jan 2020, 06:26 Shame the node version doesn't want to load so that I can manage the NAS remotely. I just get the "Service Unavailable" error.

What does the log say?

magicacaca · Post by **magicacaca** » 07 Jan 2020, 05:25

rednoah wrote: ↑05 Jan 2020, 10:41
magicacaca wrote: ↑05 Jan 2020, 06:26 Shame the node version doesn't want to load so that I can manage the NAS remotely. I just get the "Service Unavailable" error.
What does the log say?

Code: Select all

431,  0,2020-01-02,11:42:46,System,127.0.0.1,localhost,[App Center] Installed FileBot Node 0.2.8.7 in "/share/MD0_DATA/.qpkg/filebot-node"., 46,1577950966,A039,App Center,C001,App Installation,
432,  0,2020-01-02,11:42:51,System,127.0.0.1,localhost,[App Center] FileBot Node enabled., 47,1577950971,A039,App Center,C001,App Installation,
433,  0,2020-01-02,11:46:14,System,127.0.0.1,localhost,[App Center] Installed nodejsv8 8.9.4.0.1 in "/share/MD0_DATA/.qpkg/nodejsv8"., 46,1577951174,A039,App Center,C001,App Installation,
434,  0,2020-01-02,11:46:20,System,127.0.0.1,localhost,[App Center] nodejsv8 enabled., 47,1577951180,A039,App Center,C001,App Installation,
435,  0,2020-01-02,11:47:19,System,127.0.0.1,localhost,[App Center] Installed Java Installer 1.9 in "/share/MD0_DATA/.qpkg/java-installer"., 46,1577951239,A039,App Center,C001,App Installation,
436,  0,2020-01-02,11:49:05,System,127.0.0.1,localhost,[Java Installer] openjdk version "13.0.1" 2019-10-15OpenJDK Runtime Environment (build 13.0.1+9)OpenJDK 64-Bit Server VM (build 13.0.1+9, mixed mode, sharing),  0,1577951345,---,---,---,---,
437,  0,2020-01-02,11:49:10,System,127.0.0.1,localhost,[App Center] Java Installer enabled., 47,1577951350,A039,App Center,C001,App Installation,
438,  0,2020-01-02,11:49:24,System,127.0.0.1,localhost,[App Center] Installed FileBot 4.8.5 in "/share/MD0_DATA/.qpkg/filebot"., 46,1577951364,A039,App Center,C001,App Installation,
439,  0,2020-01-02,11:49:29,System,127.0.0.1,localhost,[App Center] FileBot enabled., 47,1577951369,A039,App Center,C001,App Installation,

Post by **rednoah** » 07 Jan 2020, 12:24

I'm referring to the FileBot Node log, which will tell us if FileBot Node is even running, or just crashed on startup for some reason.

Here's how you can find the log:

Code: Select all

$ find / -name filebot-node.log 2> /dev/null
/var/log/upstart/filebot-node.log

magicacaca · Post by **magicacaca** » 07 Jan 2020, 19:18

Thanks but not having any luck running that:

[~] # $ find / -name filebot-node.log 2> /dev/null
[~] # /var/log/upstart/filebot-node.log
-sh: /var/log/upstart/filebot-node.log: No such file or directory

Post by **rednoah** » 07 Jan 2020, 20:04

The command above was run on Synology. $ ... marks the command. The following line is the console output of the find command. On QNAP, the find command will likely find a different file path.

Run this command:

Code: Select all

find / -name filebot-node.log 2> /dev/null

If FileBot Node is installed, and was started (or started and immediately crashed) then there will be a log file somewhere. Unfortunately, I can't really help you until you find that log file, and check what it says.

magicacaca · Post by **magicacaca** » 07 Jan 2020, 20:17

# cat /share/MD0_DATA/.qpkg/filebot-node/filebot-node.log
/share/MD0_DATA/.qpkg/filebot-node/start: line 34: node: command not found

magicacaca · Post by **magicacaca** » 07 Jan 2020, 20:23

Ooh, just started working for some reason. Complete uninstall and reinstall seems to have fixed it. Thanks for all your help anyway

Post by **rednoah** » 07 Jan 2020, 20:24

magicacaca wrote: ↑07 Jan 2020, 20:17 # cat /share/MD0_DATA/.qpkg/filebot-node/filebot-node.log
/share/MD0_DATA/.qpkg/filebot-node/start: line 34: node: command not found

Node JS was not installed. Installing Node JS (as it happens when starting / stopping a QNAP service) fixed the problem.

magicacaca · Post by **magicacaca** » 08 Jan 2020, 05:48

Thanks. Did a dry run and all was good. The script is running now. The live script takes waaay longer than the dry run (I guess because it's actually creating, moving, renaming and deleting now instead of just calculating). 181 titles in after 8 hours. Happy so far though.

Just a quick note FYI - the only way to see progress is the new window that opens when I click "Execute script". However, the window seems to stop reporting progress after all the 'Input' analysis has been done. During the dry run the window continued to show 'Group', 'Subtitle download' and 'Move' activities. In the live run the window isn't showing this, even though 181 of the titles (so far) have been processed correctly.

Post by **rednoah** » 08 Jan 2020, 06:49

That's strange. Processing happening without logging? There mostly no difference between dry run and execute. The amc script is executed exactly the same way, except with --action TEST forced. There should be lots of logging before the final operation, which should be instant for MOVE / HARDLINK / etc as long as source and destination are on the same file system.

What does filebot -script fn:sysinfo say?

What is the exact command that is executed? (see first line of the log)

magicacaca · Post by **magicacaca** » 08 Jan 2020, 09:23

Yep, turns out I was wrong. It wasn't still running. It stopped after a 5 minutes with an error. Exactly the same script for the dry run and the live execution. It processed 181 folders (and their associated files) but then stopped. Now every time I try and run it I get the following (the below is just an example using a single file, but it's the same error I get when I try and run the script against the entire folder directory.

Is there a way to check "filebot -script fn:sysinfo" remotely, or do I have to be on the lan to execute the SSH (probably a noob question, but hey, I am):

Code: Select all

filebot -script 'fn:amc' '/share/MD0_DATA/Multimedia/Movies & TV/Movies/The Professor' --output /share/MD0_DATA/Multimedia/TV --action move -non-strict --order Airdate --conflict skip --lang en --def 'ut_label=Movie' 'music=y' 'unsorted=y' 'subtitles=eng' 'clean=y' 'skipExtract=y' 'excludeList=.excludes' --log all --log-file '/opt/filebot-node/data/filebot.log'

------------------------------------------

Run script [fn:amc] at [Wed Jan 08 13:17:56 GST 2020]
Parameter: ut_label = Movie
Parameter: music = y
Parameter: unsorted = y
Parameter: subtitles = eng
Parameter: clean = y
Parameter: skipExtract = y
Parameter: excludeList = .excludes
Argument[0]: /share/MD0_DATA/Multimedia/Movies & TV/Movies/The Professor
Use excludes: /share/MD0_DATA/Multimedia/TV/.excludes (781)
Ignore hidden: /share/MD0_DATA/Multimedia/Movies & TV/Movies/The Professor/.@__thumb
No files selected for processing
Done ¯\_(ツ)_/¯
------------------------------------------

[Process error]

Post by **rednoah** » 08 Jan 2020, 15:23

1.
The exclude list will keep track of files that have already been processed, so they don't get processed over and over. For testing, you may delete the exclude list between runs, so FileBot will see all files as newly added files, and process them accordingly.

Please read the amc script manual for details on how --def excludeList works.

2.
Yes, FileBot Node has a System Information button which allows you to run filebot -script fn:sysinfo remotely.

magicacaca · Post by **magicacaca** » 08 Jan 2020, 19:11

Thanks, but the file in the snippet above hasn't previously been processed. The script processed 181 folders/files out of about 700 and then stopped and then wouldn't process any of the remaining 500 or so.

I've changed the "--def excludeList" to "=amc.txt" instead of "=excludes" and it's picking all the files up now (presumably because it's ignoring the previously created excludes list in favour of the new one. Just two things:

1) any idea why it ignored the files on subsequent runs despite not having completed processing them on the first live run?
2) how do I delete the amc.tx or excludes files?

Thanks for all your help

Post by **rednoah** » 08 Jan 2020, 21:43

The exclude list is written before processing. The exclude list is there to prevent accidental infinite loops where you end up processing the same files over and over, and errors are an explicit cause of this. Otherwise you'd end up trying to process the same files over and over, and fail over and over.

The exclude list is wherever you want it to be. You can delete it like any other file.

e.g. your exclude list is here and contains 781 file paths to be excluded:

Code: Select all

Use excludes: /share/MD0_DATA/Multimedia/TV/.excludes (781)

SSH renaming of multiple folders and files

SSH renaming of multiple folders and files

Re: SSH renaming of multiple folders and files

Re: SSH renaming of multiple folders and files

Re: SSH renaming of multiple folders and files

Re: SSH renaming of multiple folders and files

Re: SSH renaming of multiple folders and files

Re: SSH renaming of multiple folders and files

Re: SSH renaming of multiple folders and files

Re: SSH renaming of multiple folders and files

Re: SSH renaming of multiple folders and files

Re: SSH renaming of multiple folders and files

Re: SSH renaming of multiple folders and files

Re: SSH renaming of multiple folders and files

Re: SSH renaming of multiple folders and files

Re: SSH renaming of multiple folders and files

Re: SSH renaming of multiple folders and files

Re: SSH renaming of multiple folders and files

Re: SSH renaming of multiple folders and files

Re: SSH renaming of multiple folders and files

Re: SSH renaming of multiple folders and files