Move Files per Filebot over CLI/SSH

Support for Synology NAS, QNAP NAS and other Embedded Linux systems
Post Reply
Predator2912
Posts: 64
Joined: 20 May 2017, 17:51

Move Files per Filebot over CLI/SSH

Post by Predator2912 »

Hello,

I have again a question about Filebot via cli/ssh.

I'm getting a new hard disk this week and then I have to move my TV series to this disk. But since there were always problems with file or xattr attributes in the past, so that many episodes via "filebot -script fn:miss" were no longer found or recognized when I moved the files via DSM, I wanted to ask if there is a way to move files via Filebot CLI/SSH WITHOUT renaming or changing them?

For series with few episodes I just run them through the "fn:amc" script again, but with about 9 TB that makes less sense.

So I would like, if it is possible, to move the files to the new hard drive via filebot.
And if so, what is the command line for this?

Thx for helping and
best regards from Germany
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Move Files per Filebot over CLI/SSH

Post by rednoah »

1.
You can use the rsync command to sync files from one Linux machine to another:
https://www.tecmint.com/rsync-local-rem ... -commands/

:idea: rsync can be configured to copy along xattr as well. Please run your own tests to confirm.




2.
You can configure FileBot to store xattr to plain/text files:
https://www.filebot.net/help/xattr.html

:idea: You can use -find --apply metadata to export file system xattr to plain/text files for existing files:
viewtopic.php?t=11079
:idea: Please read the FAQ and How to Request Help.
Predator2912
Posts: 64
Joined: 20 May 2017, 17:51

Re: Move Files per Filebot over CLI/SSH

Post by Predator2912 »

I tried this with the metadata before my complete DSM update to 6.2.4 but it also seems to have problems when the files are later moved via DSM Filestation.

Here is an example:

Code: Select all

NAS:/volume8/TV Serien_Complete/Navy CIS (2003)/Season 01/.xattr# ls -la
total 0
drwx------+ 1 admin users  3486 Apr 14  2020 .
drwx------+ 1 admin users 13992 Apr 14  2020 ..
drwx------+ 1 admin users    80 Apr 14  2020 Navy CIS (2003) - S01E01 - Air Force One_[1280x720p]_[720p_x264__2ch].mkv
drwx------+ 1 admin users    80 Apr 14  2020 Navy CIS (2003) - S01E02 - Sprung in den Tod_[1280x720p]_[720p_x264__2ch].mkv
drwx------+ 1 admin users    80 Apr 14  2020 Navy CIS (2003) - S01E03 - Seadog_[1280x720p]_[720p_AVC_ithd_2ch].mkv
.
.
and so on
.

Code: Select all

NAS:/volume8/TV Serien_Complete/Navy CIS (2003)/Season 01# ls -la
drwx------+ 1 admin users      13992 Apr 14  2020 .
drwx------+ 1 admin users       1276 Jan 30  2021 ..
-rwx------+ 1 admin users 1466446605 Apr  3  2011 Navy CIS (2003) - S01E01 - Air Force One_[1280x720p]_[720p_x264__2ch].mkv
-rwx------+ 1 admin users 1466694439 Apr  5  2011 Navy CIS (2003) - S01E02 - Sprung in den Tod_[1280x720p]_[720p_x264__2ch].mkv
-rwx------+ 1 admin users 1447720963 Jan 30  2012 Navy CIS (2003) - S01E03 - Seadog_[1280x720p]_[720p_AVC_ithd_2ch].mkv
-rwx------+ 1 admin users 1466225111 Apr 18  2011 Navy CIS (2003) - S01E04 - Die Unsterblichen_[1280x720p]_[720p_x264__2ch].mkv
.
.
and so on
.

Code: Select all

NAS:/# filebot -script fn:miss --log off /volume8/TV\ Serien_Complete/Navy\ CIS\ \(2003\)/
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by net.filebot.Main (file:/volume1/@appstore/filebot/jar/filebot.jar)
WARNING: Please consider reporting this to the maintainers of net.filebot.Main
WARNING: System::setSecurityManager will be removed in a future release
Navy CIS - 1x01 - Air Force One
Navy CIS - 1x02 - Sprung in den Tod
Navy CIS - 1x03 - Seadog
Navy CIS - 1x04 - Die Unsterblichen
Navy CIS - 1x05 - Der Fluch der Mumie
.
.
.
and so on
.
Here you can see that the episodes are not found as existing, although an xattr directory exists.
Or have I just misunderstood something in general?

I think I'll do some test runs with rsync and/or install the Synology community app "Ultimate Backup" and see what it can do.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Move Files per Filebot over CLI/SSH

Post by rednoah »

rednoah wrote: 12 Jan 2022, 01:08 You can configure FileBot to store xattr to plain/text files:
https://www.filebot.net/help/xattr.html
FileBot does not use .xattr folders by default, unless configured to do so, and once configured to use .xattr folders, FileBot will no longer look at native xattr at all:

Code: Select all

filebot -script fn:properties --def net.filebot.xattr.store=.xattr

:idea: You can use the xattr script to view xattr for each file, as seen by the currently configured xattr storage engine:

Code: Select all

filebot -script fn:xattr /path/to/files
:idea: Please read the FAQ and How to Request Help.
Predator2912
Posts: 64
Joined: 20 May 2017, 17:51

Re: Move Files per Filebot over CLI/SSH

Post by Predator2912 »

rednoah wrote: 13 Jan 2022, 00:27 FileBot does not use .xattr folders by default, unless configured to do so, and once configured to use .xattr folders, FileBot will no longer look at native xattr at all:

Code: Select all

filebot -script fn:properties --def net.filebot.xattr.store=.xattr
Is it possible that this option is reset to default after DSM and/or Filebot updates?
I'm pretty sure I had it enabled.
rednoah wrote: 13 Jan 2022, 00:27 :idea: You can use the xattr script to view xattr for each file, as seen by the currently configured xattr storage engine:

Code: Select all

filebot -script fn:xattr /path/to/files
I was also able to solve the moving of files via DSM in the meantime. After the move I have reset the files that were no longer found via fn:miss via xattr --action clear and recreate.

So:

Code: Select all

fn:xattr --action clear /path/to/files
fn:verify /path/to/files
fn:xattr /path/to/files
is the right Way to "fixed" it.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Move Files per Filebot over CLI/SSH

Post by rednoah »

1.
Predator2912 wrote: 15 Jan 2022, 20:59 Is it possible that this option is reset to default after DSM and/or Filebot updates?
:?: What does the console output say when you run the command?

Code: Select all

filebot -script fn:properties --def net.filebot.xattr.store=.xattr

EDIT:

Note that adding --def net.filebot.xattr.store=.xattr to an amc script command has no effect. The properties script is a dedicated script that will write a FileBot configuration file. On DSM, this configuration file is global and remain unmodified during updates, but will be deleted if you uninstall the FileBot package.
Predator2912 wrote: 15 Jan 2022, 22:03 This is my default DSM Task for TV series:

Code: Select all

...
--def
net.filebot.xattr.store=.xattr
...




2.
Predator2912 wrote: 15 Jan 2022, 20:59 I was also able to solve the moving of files via DSM in the meantime. After the move I have reset the files that were no longer found via fn:miss via xattr --action clear and recreate.

So:

Code: Select all

fn:xattr --action clear /path/to/files
fn:verify /path/to/files
fn:xattr /path/to/files
Note that the xattr script does not recreate xattr. It simply deletes them or prints them. If you first delete them, and then print them, then none will be printed because they've all been deleted already.

:idea: The verify script will add the CRC32 xattr field. This is unrelated to FileBot metadata which relies on the JSON object stored in in the net.filebot.metadata xattr field.
:idea: Please read the FAQ and How to Request Help.
Predator2912
Posts: 64
Joined: 20 May 2017, 17:51

Re: Move Files per Filebot over CLI/SSH

Post by Predator2912 »

1.
rednoah wrote: 16 Jan 2022, 02:09 :?: What does the console output say when you run the command?

Code: Select all

filebot -script fn:properties --def net.filebot.xattr.store=.xattr

Code: Select all

MadM4x-NAS:~# filebot -script fn:properties --def net.filebot.xattr.store=.xattr
/volume1/@appstore/filebot/filebot.sh must NOT run as root
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by net.filebot.Main (file:/volume1/@appstore/filebot/jar/filebot.jar)
WARNING: Please consider reporting this to the maintainers of net.filebot.Main
WARNING: System::setSecurityManager will be removed in a future release
Load user-defined System Properties
* Read /volume1/@appstore/filebot/data/0/system.properties
Update user-defined System Properties
* Set net.filebot.xattr.store = .xattr
Store user-defined System Properties
* Write /volume1/@appstore/filebot/data/0/system.properties
#FileBot System Properties
#Sun Jan 16 03:22:52 CET 2022
net.filebot.xattr.store=.xattr
net.filebot.cli.theme=BusinessMachine

Done ヾ(@⌒ー⌒@)ノ
2.
rednoah wrote: 16 Jan 2022, 02:09 Note that the xattr script does not recreate xattr. It simply deletes them or prints them. If you first delete them, and then print them, then none will be printed because they've all been deleted already.

:idea: The verify script will add the CRC32 xattr field. This is unrelated to FileBot metadata which relies on the JSON object stored in in the net.filebot.metadata xattr field.
Fortunately I only tested this on one season and didn't delete all of them right away.
Is there any way to recreate xattr?
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Move Files per Filebot over CLI/SSH

Post by rednoah »

1.

Code: Select all

Store user-defined System Properties
* Write /volume1/@appstore/filebot/data/0/system.properties
This path is user-specific and DSM may quite possibly remove the entire /volume1/@appstore/filebot folder on upgrade, so...
* that configuration file is user-specific (i.e. if you run as user other than User 0 / root, then that user will have it's own configuration file; which can empty or different)
* that configuration file might not survive upgrades (not sure on that one; haven't tested this one way or another; should work, but might not; might work differently on DSM6 and DSM7; you can confirm by checking the configuration file for each user after upgrade)


:arrow: So you might want to configure those properties right before you make your normal filebot call, just in case, because only there can we guarantee same user, same update:

Code: Select all

filebot -script fn:properties --def net.filebot.xattr.store=.xattr
filebot -script fn:amc ...


2.
There is no way to recreate xattr, other than just processing the files again as if they were new.
:idea: Please read the FAQ and How to Request Help.
Predator2912
Posts: 64
Joined: 20 May 2017, 17:51

Re: Move Files per Filebot over CLI/SSH

Post by Predator2912 »

rednoah wrote: 16 Jan 2022, 02:57 1.

Code: Select all

Store user-defined System Properties
* Write /volume1/@appstore/filebot/data/0/system.properties
This path is user-specific and DSM may quite possibly remove the entire /volume1/@appstore/filebot folder on upgrade, so...
* that configuration file is user-specific (i.e. if you run as user other than User 0 / root, then that user will have it's own configuration file; which can empty or different)
* that configuration file might not survive upgrades (not sure on that one; haven't tested this one way or another; should work, but might not; might work differently on DSM6 and DSM7; you can confirm by checking the configuration file for each user after upgrade)
Okay,

I looked at the contents of the folder and there is a history file for each user I used Filebot with.
These files could be copied before an update to be on the safe side, but I wonder if it is possible to change the user specific path... or better yet, if there is a Filebot configuration file where I can change the path?

Then at least the history files are preserved after each update and you would only have to change the Filebot File.

Do you know if this is possible?
Or is this rather a DSM system specific setting.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Move Files per Filebot over CLI/SSH

Post by rednoah »

1.
You'll want to always run filebot as the same user to avoid the complexities of a multi-user environment, just like on your Desktop PC. Sharing the same application data folder / files (though of course possible via manually created symlinks that just redirect file operations to wherever you want) will lead to permission issues, since files created by one user (e.g. root) will likely not be readable / writable by other users by default, and FileBot will break in strange and unexpected ways if it's own application data folder is not readable / writable.



2.
I checked the code. If you're using DSM6, then the FileBot package will archive and restore the application data folder during upgrade. If you're using DSM7, then FileBot will use the package-specific application data folder that is unaffected by upgrades. Either way, upgrades shouldn't be a problem. Having backups of important data off-site is always a good idea though, since the hardware itself could fail at any time. I'd just rsync to the data folder to a remote machine once a day via a scheduled task.
:idea: Please read the FAQ and How to Request Help.
Predator2912
Posts: 64
Joined: 20 May 2017, 17:51

Re: Move Files per Filebot over CLI/SSH

Post by Predator2912 »

rednoah wrote: 16 Jan 2022, 09:25 1.
You'll want to always run filebot as the same user to avoid the complexities of a multi-user environment, just like on your Desktop PC. Sharing the same application data folder / files (though of course possible via manually created symlinks that just redirect file operations to wherever you want) will lead to permission issues, since files created by one user (e.g. root) will likely not be readable / writable by other users by default, and FileBot will break in strange and unexpected ways if it's own application data folder is not readable / writable.
The Filebot task schedulers always run with the same user or are started by the same user. Only the tests under Filebot-Node GUI are run by a different one.

rednoah wrote: 16 Jan 2022, 09:25 2.
I checked the code. If you're using DSM6, then the FileBot package will archive and restore the application data folder during upgrade. If you're using DSM7, then FileBot will use the package-specific application data folder that is unaffected by upgrades. Either way, upgrades shouldn't be a problem. Having backups of important data off-site is always a good idea though, since the hardware itself could fail at any time. I'd just rsync to the data folder to a remote machine once a day via a scheduled task.
I let the whole thing now with rsync via DSM task scheduler once a day safe or better said only the newer files "backup / update". So if necessary, the corresponding files are at least available.
Post Reply