Move Operation on Synology SHR Volume

Support for Synology NAS, QNAP NAS and other Embedded Linux systems
Post Reply
minitoine
Posts: 6
Joined: 08 Jan 2019, 21:48

Move Operation on Synology SHR Volume

Post by minitoine » 08 Jan 2019, 21:54

Hi,

I use Filebot to move the movies from the shared folder 'Downloads/scan' to the shared folder 'Videos/Movies'.
It actually works, but, the transfer is not "instant", files are transferred like a copy (a refresh in FileStation and you can see the size growing).
I'm using a 918+, with 3 drives forming a SHR Volume (volume1) and all the shared folders are on that volume.

Moving files with FileStation is instant.
Moving files with SSH (cmd mv) is instant.
Moving files with Filebot (And so, through java ?), the transfer is not instant.
Which could be problematic for 60Go+ files.

Edit:
If I create for example, a temporary folder in the same shared folder (e.g. Videos), moving with Filebot is instant.


Thx in advance,

minitoine

User avatar
rednoah
The Source
Posts: 16433
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Move Operation on Synology SHR Volume

Post by rednoah » 09 Jan 2019, 03:51

FileBot is running physically on the Synology device, correct? Not via network shares remotely, right?
:idea: Please read the FAQ and How to Request Help.

minitoine
Posts: 6
Joined: 08 Jan 2019, 21:48

Re: Move Operation on Synology SHR Volume

Post by minitoine » 09 Jan 2019, 08:21

Nope, installed via package center with your java installer.
I forgot to tell that I am using BRTFS for the file format.
Maybe it is not actually supported by Filebot ?

User avatar
rednoah
The Source
Posts: 16433
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Move Operation on Synology SHR Volume

Post by rednoah » 09 Jan 2019, 08:51

Well, FileBot just tells the OS to "move A to B" and then OS decides how to do that physically on disk, so if the mv command works, then so should the filebot command.


Please try --action clone and see if that works instantly:

Code: Select all

--action clone
The --action clone action literally just calls cp --reflink so it can't not work really. Since you're using BTRFS, you should be able to create CoW clones.


Alternatively, you can pass in your own shell script that moves files, so if for some odd reason you really need to use the mv system executable, you can:
viewtopic.php?f=4&t=4915
:idea: Please read the FAQ and How to Request Help.

minitoine
Posts: 6
Joined: 08 Jan 2019, 21:48

Re: Move Operation on Synology SHR Volume

Post by minitoine » 09 Jan 2019, 14:03

So, as you said, using --action clone worked instantly, but.. cloned the file. So I need to code a clean-up phase after cloning, can be very quickly complicated.

Retried after that a move action (don't know why, but I gave a try), and the file is still copied.

Sure that there isn't any problems with Java and multi-disks volumes ?

Edit: I also tried with --action hardlink and got the error : Invalid cross-device link
Could be useful.

Edit2: Tried using --action /bin/mv and worked as expected. There is definitely something wrong somewhere. I am opened to help if interested.

User avatar
rednoah
The Source
Posts: 16433
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Move Operation on Synology SHR Volume

Post by rednoah » 09 Jan 2019, 17:05

Maybe an issue with the Java implementation then, because if move isn't instant, then hardlink shouldn't work either, so at least it's consistent. :lol:

Code: Select all

Invalid cross-device link
Though I somewhat doubt that the Java implementation has any Linux filesystem specific code, so probably some odd bug somewhere deep down somewhere, file system driver, system libs, etc.

Out of curiosity, what does the stat command say for different files in different folders? Does it return the same or different device ids?

If you're using Java 8, then using Java 11 is definitely worth a try, and vice versa.
:idea: Please read the FAQ and How to Request Help.

minitoine
Posts: 6
Joined: 08 Jan 2019, 21:48

Re: Move Operation on Synology SHR Volume

Post by minitoine » 09 Jan 2019, 17:19

For a same SHR Volume :

In Downloads shared folder :

File: ‘toto’
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 2ch/44d Inode: 268 Links: 1
Access: (0777/-rwxrwxrwx) Uid: ( 1024/ admin) Gid: ( 100/ users)
Access: 2019-01-09 18:17:21.659796309 +0100
Modify: 2019-01-09 18:17:21.659796309 +0100
Change: 2019-01-09 18:17:21.659796309 +0100
Birth: -

In Videos shared folder:
File: ‘toto’
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 25h/37d Inode: 2270 Links: 1
Access: (0777/-rwxrwxrwx) Uid: ( 1024/ admin) Gid: ( 100/ users)
Access: 2019-01-09 18:18:07.395365384 +0100
Modify: 2019-01-09 18:18:07.395365384 +0100
Change: 2019-01-09 18:18:07.395365384 +0100
Birth: -


As you can see, Device is different for both the shared folders.

User avatar
rednoah
The Source
Posts: 16433
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Move Operation on Synology SHR Volume

Post by rednoah » 09 Jan 2019, 17:37

Alright, so even the OS on some level thinks these files are on different devices / file systems. There's a good chance Synology has their on modified versions of the mv and cp tools, that somehow do Synology SHR specific magic. :lol:
:idea: Please read the FAQ and How to Request Help.

minitoine
Posts: 6
Joined: 08 Jan 2019, 21:48

Re: Move Operation on Synology SHR Volume

Post by minitoine » 09 Jan 2019, 17:42

What about 'classic' RAID volume ? Device IDS for folders are the same ?

You said earlier that Filebot use system commands, this should overcome the issue of different IDs then, no ?
Unless, there is no explicit call to 'mv' command right ?

User avatar
rednoah
The Source
Posts: 16433
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Move Operation on Synology SHR Volume

Post by rednoah » 10 Jan 2019, 04:16

AFAIK, yes, on a classic RAID volume, the hardware does take care of presenting the array as a single volume / device to the operating system.

I have no idea how SHR works, but it seems like it might require or benefit from customizations to mv / cp / SMB server / AFP server / etc.

From my experience, I know that mv / cp definitely have some customizations to make sure that Synology file index always gets updated when files are moved / copied. Same thing for SMB / AFP services. So likey additional customization for SHR for some reason.

This is a long shot, but have you tried the Java package provided by Synology? Maybe that one comes compiled with the same customizations that somehow make their mv / cp work too?
:idea: Please read the FAQ and How to Request Help.

Kopernikus
Posts: 5
Joined: 03 Feb 2019, 20:07

Re: Move Operation on Synology SHR Volume

Post by Kopernikus » 03 Feb 2019, 21:12

@minitoine

Hi,

Looking at the same problem, have a Synology DS916+ SHR in BTFRS.
Have you found a solution yet?
For the moment I let a plugin in Deluge copy the complete file to a directory where Filbotnode then auto moves and renames it.
Btw @rednoah is there a way the folder get's deleted after the file has moved?

minitoine
Posts: 6
Joined: 08 Jan 2019, 21:48

Re: Move Operation on Synology SHR Volume

Post by minitoine » 03 Feb 2019, 21:23

@Kopernikus

I have 2 shared folders in my case to handle my media : Downloads and Videos.
As you might guess, my downloaded media files are stored into the Downloads folder, and then, Sonarr and Filebot scan this folder to move files to Videos (Tv and Movies subfolders).
To overcome the move pb, I created a scan folder inside the Videos shared folder, and wrote a little script to move (using the system copy cmd 'cp' of Synology) downloaded files to the scan folder.
Once files are moved to the scan folder I call the Complete Download Handling command of Sonarr and Radarr/Filebot to scan.
In that way, Sonarr and filebot/Radarr move files from Videos/scan to Videos/TV and Videos/Movies instantly.

Didn't had any other ideas..

LightningRider
Posts: 2
Joined: 08 Aug 2019, 09:40

Re: Move Operation on Synology SHR Volume

Post by LightningRider » 08 Aug 2019, 10:07

I know this is an old topic, but thought I'd also add that this isn't specific to SHR Volumes. But rather to BTRFS on Synology
I am using a Synology 1515+ with RAID 5 volume and can confirm the same behaviour as the OP.
Up until recencly I have been using EXT4 as the filesystem for my media and --action move behaved as expected (instant). However, as soon as I changed filesystem to BTRFS --action move began to behave as a copy. I have switched to using --action /bin/mv as suggested above, as synology may have inserted some magic sauce to make this work as expected. I am using Oracle JRE 8u221 (linux x64), AFAIK there isn't a Synology Java package (only a Java installer).

FWIW: stat and btrfs subvolume list command outputs (for my from and to directories)
-- OUTPUT stat --
File: ‘/volume2/downloads/’
Size: 0 Blocks: 0 IO Block: 4096 directory
Device: 32h/50d Inode: 256 Links: 1

File: ‘/volume2/TV Shows/’
Size: 4498 Blocks: 0 IO Block: 4096 directory
Device: 25h/37d Inode: 256 Links: 1

-- OUTPUT btrfs subvolume list --
ID 259 gen 14260 top level 257 path /volume2/TV Shows
ID 2661 gen 14259 top level 257 path /volume2/downloads

User avatar
rednoah
The Source
Posts: 16433
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Move Operation on Synology SHR Volume

Post by rednoah » 08 Aug 2019, 15:31

Looks like mv does have a special case, where if instant move, doesn't work, it'll try reflink + delete first, and if that doesn't work, it'll try copy + delete:
https://superuser.com/questions/948808/ ... 380_948808

mv will try a reflink before falling back to a standard copy, which is
more efficient when moving files across BTRFS subvolume boundaries.
:arrow: https://github.com/coreutils/coreutils/blob/master/NEWS


:idea: Presumably, the JRE implementation does move and if that doesn't work fallback to copy + delete.

:idea: You could try using --action clone which allows you to do reflink via FileBot.

:?: Have you tried the latest OpenJDK 12.0.2 to see if Java move works better on the latest release? If not, then I'll make a bug report.
:idea: Please read the FAQ and How to Request Help.

LightningRider
Posts: 2
Joined: 08 Aug 2019, 09:40

Re: Move Operation on Synology SHR Volume

Post by LightningRider » 09 Aug 2019, 05:26

rednoah wrote:
08 Aug 2019, 15:31
:idea: Presumably, the JRE implementation does move and if that doesn't work fallback to copy + delete.

:?: Have you tried the latest OpenJDK 12.0.2 to see if Java move works better on the latest release? If not, then I'll make a bug report.
Upon trying OpenJDK 12.0.2 I notice the same behaviour as JRE 8u221; i.e. copy + delete.
However I also get the following warnings:

Code: Select all

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.codehaus.groovy.vmplugin.v7.Java7$1 (file:/volume1/@appstore/filebot/jar/groovy.jar) to constructor java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int)
WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.vmplugin.v7.Java7$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
rednoah wrote:
08 Aug 2019, 15:31
:idea: You could try using --action clone which allows you to do reflink via FileBot.
I changed my default behaviour from --action hardlink to --action move as I started experiencing "Invalid cross-device link". --action clone seems like the correct replacement in my case, as I only switched to move once hardlink broke (as I was unaware of clone).

User avatar
rednoah
The Source
Posts: 16433
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Move Operation on Synology SHR Volume

Post by rednoah » 09 Aug 2019, 10:28

--action clone is a recent addition on only tested on macOS / APFS (via the clonefile syscall) and Linux / btrfs (via the reflink syscall).


:idea: --action duplicate is supposed to be smarter, and pick the best copy operation. Though it may or may not try to reflink depending on the revision, since I've reverted things due to issues: viewtopic.php?f=11&t=10822
:idea: Please read the FAQ and How to Request Help.

Post Reply