Move Files per Filebot over CLI/SSH

[Predator2912]

Hello,

I have again a question about Filebot via cli/ssh.

I'm getting a new hard disk this week and then I have to move my TV series to this disk. But since there were always problems with file or xattr attributes in the past, so that many episodes via "filebot -script fn:miss" were no longer found or recognized when I moved the files via DSM, I wanted to ask if there is a way to move files via Filebot CLI/SSH WITHOUT renaming or changing them?

For series with few episodes I just run them through the "fn:amc" script again, but with about 9 TB that makes less sense.

So I would like, if it is possible, to move the files to the new hard drive via filebot.
And if so, what is the command line for this?

Thx for helping and
best regards from Germany

[rednoah]

1.
You can use the rsync command to sync files from one Linux machine to another:
https://www.tecmint.com/rsync-local-rem ... -commands/

rsync can be configured to copy along xattr as well. Please run your own tests to confirm.




2.
You can configure FileBot to store xattr to plain/text files:
https://www.filebot.net/help/xattr.html

You can use -find --apply metadata to export file system xattr to plain/text files for existing files:
viewtopic.php?t=11079

[Predator2912]

I tried this with the metadata before my complete DSM update to 6.2.4 but it also seems to have problems when the files are later moved via DSM Filestation.

Here is an example:

Code: Select all

NAS:/volume8/TV Serien_Complete/Navy CIS (2003)/Season 01/.xattr# ls -la
total 0
drwx------+ 1 admin users  3486 Apr 14  2020 .
drwx------+ 1 admin users 13992 Apr 14  2020 ..
drwx------+ 1 admin users    80 Apr 14  2020 Navy CIS (2003) - S01E01 - Air Force One_[1280x720p]_[720p_x264__2ch].mkv
drwx------+ 1 admin users    80 Apr 14  2020 Navy CIS (2003) - S01E02 - Sprung in den Tod_[1280x720p]_[720p_x264__2ch].mkv
drwx------+ 1 admin users    80 Apr 14  2020 Navy CIS (2003) - S01E03 - Seadog_[1280x720p]_[720p_AVC_ithd_2ch].mkv
.
.
and so on
.

Code: Select all

NAS:/volume8/TV Serien_Complete/Navy CIS (2003)/Season 01# ls -la
drwx------+ 1 admin users      13992 Apr 14  2020 .
drwx------+ 1 admin users       1276 Jan 30  2021 ..
-rwx------+ 1 admin users 1466446605 Apr  3  2011 Navy CIS (2003) - S01E01 - Air Force One_[1280x720p]_[720p_x264__2ch].mkv
-rwx------+ 1 admin users 1466694439 Apr  5  2011 Navy CIS (2003) - S01E02 - Sprung in den Tod_[1280x720p]_[720p_x264__2ch].mkv
-rwx------+ 1 admin users 1447720963 Jan 30  2012 Navy CIS (2003) - S01E03 - Seadog_[1280x720p]_[720p_AVC_ithd_2ch].mkv
-rwx------+ 1 admin users 1466225111 Apr 18  2011 Navy CIS (2003) - S01E04 - Die Unsterblichen_[1280x720p]_[720p_x264__2ch].mkv
.
.
and so on
.

Code: Select all

NAS:/# filebot -script fn:miss --log off /volume8/TV\ Serien_Complete/Navy\ CIS\ \(2003\)/
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by net.filebot.Main (file:/volume1/@appstore/filebot/jar/filebot.jar)
WARNING: Please consider reporting this to the maintainers of net.filebot.Main
WARNING: System::setSecurityManager will be removed in a future release
Navy CIS - 1x01 - Air Force One
Navy CIS - 1x02 - Sprung in den Tod
Navy CIS - 1x03 - Seadog
Navy CIS - 1x04 - Die Unsterblichen
Navy CIS - 1x05 - Der Fluch der Mumie
.
.
.
and so on
.

Here you can see that the episodes are not found as existing, although an xattr directory exists.
Or have I just misunderstood something in general?

I think I'll do some test runs with rsync and/or install the Synology community app "Ultimate Backup" and see what it can do.

[rednoah]

You can configure FileBot to store xattr to plain/text files:
https://www.filebot.net/help/xattr.html

FileBot does not use .xattr folders by default, unless configured to do so, and once configured to use .xattr folders, FileBot will no longer look at native xattr at all:

Code: Select all

filebot -script fn:properties --def net.filebot.xattr.store=.xattr

You can use the xattr script to view xattr for each file, as seen by the currently configured xattr storage engine:

Code: Select all

filebot -script fn:xattr /path/to/files

[Predator2912]

FileBot does not use .xattr folders by default, unless configured to do so, and once configured to use .xattr folders, FileBot will no longer look at native xattr at all:

Code: Select all

filebot -script fn:properties --def net.filebot.xattr.store=.xattr

Is it possible that this option is reset to default after DSM and/or Filebot updates?
I'm pretty sure I had it enabled.

You can use the xattr script to view xattr for each file, as seen by the currently configured xattr storage engine:

Code: Select all

filebot -script fn:xattr /path/to/files

I was also able to solve the moving of files via DSM in the meantime. After the move I have reset the files that were no longer found via fn:miss via xattr --action clear and recreate.

So:

Code: Select all

fn:xattr --action clear /path/to/files
fn:verify /path/to/files
fn:xattr /path/to/files

is the right Way to "fixed" it.

[rednoah]

1.

Is it possible that this option is reset to default after DSM and/or Filebot updates?

What does the console output say when you run the command?

Code: Select all

filebot -script fn:properties --def net.filebot.xattr.store=.xattr

EDIT:

Note that adding --def net.filebot.xattr.store=.xattr to an amc script command has no effect. The properties script is a dedicated script that will write a FileBot configuration file. On DSM, this configuration file is global and remain unmodified during updates, but will be deleted if you uninstall the FileBot package.

This is my default DSM Task for TV series:

Code: Select all

...
--def
net.filebot.xattr.store=.xattr
...

2.

I was also able to solve the moving of files via DSM in the meantime. After the move I have reset the files that were no longer found via fn:miss via xattr --action clear and recreate.

So:

Code: Select all

fn:xattr --action clear /path/to/files
fn:verify /path/to/files
fn:xattr /path/to/files

Note that the xattr script does not recreate xattr. It simply deletes them or prints them. If you first delete them, and then print them, then none will be printed because they've all been deleted already.

The verify script will add the CRC32 xattr field. This is unrelated to FileBot metadata which relies on the JSON object stored in in the net.filebot.metadata xattr field.

[Predator2912]

1.

What does the console output say when you run the command?

Code: Select all

filebot -script fn:properties --def net.filebot.xattr.store=.xattr

Code: Select all

MadM4x-NAS:~# filebot -script fn:properties --def net.filebot.xattr.store=.xattr
/volume1/@appstore/filebot/filebot.sh must NOT run as root
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by net.filebot.Main (file:/volume1/@appstore/filebot/jar/filebot.jar)
WARNING: Please consider reporting this to the maintainers of net.filebot.Main
WARNING: System::setSecurityManager will be removed in a future release
Load user-defined System Properties
* Read /volume1/@appstore/filebot/data/0/system.properties
Update user-defined System Properties
* Set net.filebot.xattr.store = .xattr
Store user-defined System Properties
* Write /volume1/@appstore/filebot/data/0/system.properties
#FileBot System Properties
#Sun Jan 16 03:22:52 CET 2022
net.filebot.xattr.store=.xattr
net.filebot.cli.theme=BusinessMachine

Done ヾ(＠⌒ー⌒＠)ノ

2.

Note that the xattr script does not recreate xattr. It simply deletes them or prints them. If you first delete them, and then print them, then none will be printed because they've all been deleted already.

The verify script will add the CRC32 xattr field. This is unrelated to FileBot metadata which relies on the JSON object stored in in the net.filebot.metadata xattr field.

Fortunately I only tested this on one season and didn't delete all of them right away.
Is there any way to recreate xattr?

[rednoah]

1.

Code: Select all

Store user-defined System Properties
* Write /volume1/@appstore/filebot/data/0/system.properties

This path is user-specific and DSM may quite possibly remove the entire /volume1/@appstore/filebot folder on upgrade, so...
* that configuration file is user-specific (i.e. if you run as user other than User 0 / root, then that user will have it's own configuration file; which can empty or different)
* that configuration file might not survive upgrades (not sure on that one; haven't tested this one way or another; should work, but might not; might work differently on DSM6 and DSM7; you can confirm by checking the configuration file for each user after upgrade)


So you might want to configure those properties right before you make your normal filebot call, just in case, because only there can we guarantee same user, same update:

Code: Select all

filebot -script fn:properties --def net.filebot.xattr.store=.xattr
filebot -script fn:amc ...

2.
There is no way to recreate xattr, other than just processing the files again as if they were new.

[Predator2912]

1.

Code: Select all

Store user-defined System Properties
* Write /volume1/@appstore/filebot/data/0/system.properties

This path is user-specific and DSM may quite possibly remove the entire /volume1/@appstore/filebot folder on upgrade, so...
* that configuration file is user-specific (i.e. if you run as user other than User 0 / root, then that user will have it's own configuration file; which can empty or different)
* that configuration file might not survive upgrades (not sure on that one; haven't tested this one way or another; should work, but might not; might work differently on DSM6 and DSM7; you can confirm by checking the configuration file for each user after upgrade)

Okay,

I looked at the contents of the folder and there is a history file for each user I used Filebot with.
These files could be copied before an update to be on the safe side, but I wonder if it is possible to change the user specific path... or better yet, if there is a Filebot configuration file where I can change the path?

Then at least the history files are preserved after each update and you would only have to change the Filebot File.

Do you know if this is possible?
Or is this rather a DSM system specific setting.

[rednoah]

1.
You'll want to always run filebot as the same user to avoid the complexities of a multi-user environment, just like on your Desktop PC. Sharing the same application data folder / files (though of course possible via manually created symlinks that just redirect file operations to wherever you want) will lead to permission issues, since files created by one user (e.g. root) will likely not be readable / writable by other users by default, and FileBot will break in strange and unexpected ways if it's own application data folder is not readable / writable.



2.
I checked the code. If you're using DSM6, then the FileBot package will archive and restore the application data folder during upgrade. If you're using DSM7, then FileBot will use the package-specific application data folder that is unaffected by upgrades. Either way, upgrades shouldn't be a problem. Having backups of important data off-site is always a good idea though, since the hardware itself could fail at any time. I'd just rsync to the data folder to a remote machine once a day via a scheduled task.

[Predator2912]

1.
You'll want to always run filebot as the same user to avoid the complexities of a multi-user environment, just like on your Desktop PC. Sharing the same application data folder / files (though of course possible via manually created symlinks that just redirect file operations to wherever you want) will lead to permission issues, since files created by one user (e.g. root) will likely not be readable / writable by other users by default, and FileBot will break in strange and unexpected ways if it's own application data folder is not readable / writable.

The Filebot task schedulers always run with the same user or are started by the same user. Only the tests under Filebot-Node GUI are run by a different one.

2.
I checked the code. If you're using DSM6, then the FileBot package will archive and restore the application data folder during upgrade. If you're using DSM7, then FileBot will use the package-specific application data folder that is unaffected by upgrades. Either way, upgrades shouldn't be a problem. Having backups of important data off-site is always a good idea though, since the hardware itself could fail at any time. I'd just rsync to the data folder to a remote machine once a day via a scheduled task.

I let the whole thing now with rsync via DSM task scheduler once a day safe or better said only the newer files "backup / update". So if necessary, the corresponding files are at least available.