Page 1 of 1

Failed to hardlink when --action=duplicate

Posted: 19 May 2019, 21:53
by queimadus
I think I may have found a bug when the action is set to duplicate and the underlying OS doesn't fully support for the first approach.

After digging through the code, I saw that when the action is set to "duplicate" it will try to "clone", if that fails fallback to "duplicate" and if that fails, fallback to "copy".

The "clone" option makes use of "cp --reflink" which apparently my OS does not support (Inappropriate ioctl for device). That alone logs an error but instead of fallbacking to "duplicate" it goes straight to a "copy". (I know duplicate works because that's how I fixed my setup, and it now correctly hardlinks files)

I didn't get much into why it skipped the second fallback, I think when the "cp --reflink" fails, it still creates an empty target file which may interfere with the duplicate fallback.

PS: Running "Linux vero 3.14.29-143-osmc #1 SMP osmc-ccachefix aarch64" while using the amc script

Re: Failed to hardlink when --action=duplicate

Posted: 20 May 2019, 08:06
by rednoah
Here's how it works:
1. Try clone / reflink
2. Try hardlink
3. Try copy

If clone doesn't work, then it will hardlink. If hardlink doesn't work, it will copy.

:idea: Debug logging will give you more information: viewtopic.php?f=3&t=3913

queimadus wrote: 19 May 2019, 21:53 I think when the "cp --reflink" fails, it still creates an empty target file which may interfere with the
Please confirm that cp --reflink indeed creates an empty target file. That would be pretty stupid cp behaviour. But if that's indeed the case, then I'll remove the clone attempt altogether.

Re: Failed to hardlink when --action=duplicate

Posted: 20 May 2019, 11:32
by queimadus
Seems like it does.

Code: Select all

[ osmc@vero /tmp/test ]$ ls                                                                                                                                                                                                                                          
[ osmc@vero /tmp/test ]$ echo hello > foo                                                                                                                                                                                                                            
[ osmc@vero /tmp/test ]$ cp --reflink foo bar                                                                                                                                                                                                                        
cp: failed to clone 'bar' from 'foo': Inappropriate ioctl for device
[ osmc@vero /tmp/test ]$ ls                                                                                                                                                                                                                                          
bar  foo
[ osmc@vero /tmp/test ]$ cat bar                                                                                                                                                                                                                                     
[ osmc@vero /tmp/test ]$ cat foo                                                                                                                                                                                                                                     
hello

Re: Failed to hardlink when --action=duplicate

Posted: 20 May 2019, 11:35
by rednoah
Noted. Will remove clone attempt from the duplicate action.

Re: Failed to hardlink when --action=duplicate

Posted: 09 Aug 2019, 11:20
by rednoah
I've updated the code to:

Code: Select all

1. Try hardlink
2. Try clonefile (via cp -c) / reflink (cp --reflink=auto)
3. Try physical copy
:idea: To make sure we don't create empty files, 2. Try clonefile / reflink now uses cp --reflink=auto, so on Linux the cp call will implicitly take care of copying files if necessary.

:idea: @since r6569

Re: Failed to hardlink when --action=duplicate

Posted: 16 Nov 2019, 22:45
by ghostserverd
I think I'm seeing something similar with r6224.

Code: Select all

FileBot 4.8.5 (r6224)
JNA Native: 5.2.0
MediaInfo: 17.12
p7zip: p7zip Version 16.02 (locale=C.UTF-8,Utf16=on,HugeFiles=on,64 bits,8 CPUs Intel(R) Core(TM) i7-4770R CPU @ 3.20GHz (40661),ASM,AES-NI)
unrar: UNRAR 5.50 freeware
Chromaprint: fpcalc version 1.4.3
Extended Attributes: OK
Unicode Filesystem: OK
Script Bundle: 2019-05-15 (r565)
Groovy: 2.5.6
JRE: OpenJDK Runtime Environment 11.0.4
JVM: 64-bit OpenJDK 64-Bit Server VM
CPU/MEM: 8 Core / 1 GB Max Memory / 32 MB Used Memory
OS: Linux (amd64)
HW: Linux 25436ff19aab 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
DATA: /data
Package: DOCKER
License: UNREGISTERED
Done ヾ(@⌒ー⌒@)ノ
when I force it to hardlink, it's successful

Code: Select all

Locking /config/amc.log
Run script [fn:amc] at [Sat Nov 16 22:21:02 UTC 2019]
Parameter: unsorted = y
Parameter: clean = y
Parameter: music = y
Parameter: artwork = y
Parameter: subtitles = en
Parameter: minLengthMS = 0
Parameter: movieFormat = {plex} - {hd}
Parameter: seriesFormat = {plex} - {hd}
Parameter: animeFormat = {plex} - {hd}
Parameter: musicFormat = {plex} - {hd}
Parameter: ut_dir = /share/downloads/complete/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO
Parameter: ut_kind = multi
Parameter: ut_title = The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO
Parameter: ut_label = N/A
Input: /share/downloads/complete/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO.mkv
Group: [Series:the mandalorian] => [The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO.mkv]
Get [English] subtitles for 1 files
CmdlineException: OpenSubtitles: Please enter your login details by calling `filebot -script fn:configure`
Rename episodes using [TheTVDB]
Auto-detected query: [the mandalorian]
Fetching episode data for [The Mandalorian]
[HARDLINK] from [/share/downloads/complete/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO.mkv] to [/share/media/TV Shows/The Mandalorian/Season 01/The Mandalorian - S01E02 - Chapter 2 - The Child - HD.mkv]
Processed 1 files
Done ヾ(@⌒ー⌒@)ノ
and it's very quick (5.005s)

but when I set it to duplicate, it fails to clone and then takes way longer (46.387s), which I believe is performing an actual copy

Code: Select all

Locking /config/amc.log
Run script [fn:amc] at [Sat Nov 16 22:35:36 UTC 2019]
Parameter: unsorted = y
Parameter: clean = y
Parameter: music = y
Parameter: artwork = y
Parameter: subtitles = en
Parameter: minLengthMS = 0
Parameter: movieFormat = {plex} - {hd}
Parameter: seriesFormat = {plex} - {hd}
Parameter: animeFormat = {plex} - {hd}
Parameter: musicFormat = {plex} - {hd}
Parameter: ut_dir = /share/downloads/complete/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO
Parameter: ut_kind = multi
Parameter: ut_title = The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO
Parameter: ut_label = N/A
Input: /share/downloads/complete/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO.mkv
Group: [Series:the mandalorian] => [The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO.mkv]
Get [English] subtitles for 1 files
CmdlineException: OpenSubtitles: Please enter your login details by calling `filebot -script fn:configure`
Rename episodes using [TheTVDB]
Auto-detected query: [the mandalorian]
Fetching episode data for [The Mandalorian]
[DUPLICATE] from [/share/downloads/complete/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO.mkv] to [/share/media/TV Shows/The Mandalorian/Season 01/The Mandalorian - S01E02 - Chapter 2 - The Child - HD.mkv]
cp: failed to clone '/share/media/TV Shows/The Mandalorian/Season 01/The Mandalorian - S01E02 - Chapter 2 - The Child - HD.mkv' from '/share/downloads/complete/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO.mkv': Operation not supported
Processed 1 files
Done ヾ(@⌒ー⌒@)ノ
I'd like to be able to maintain the fallback to copy if necessary (I'm building a container, and I want to be resilient to different volumes / block device setups) but I also want to be able to hardlink when possible.

Is there a different version I should be using that will have the updated code you described? Thanks!

Re: Failed to hardlink when --action=duplicate

Posted: 17 Nov 2019, 05:58
by rednoah
The changes mentioned in the previous post have been implemented with r6569 and previous revisions such as r6224 use a different implementation.

You can find the latest revision here:
viewtopic.php?f=7&t=1609

Re: Failed to hardlink when --action=duplicate

Posted: 17 Nov 2019, 07:06
by ghostserverd
Oh wow I was totally dyslexic reading that revision number. I’ll take a look at the other builds. Thanks!