Page 1 of 1
Failed to hardlink when --action=duplicate
Posted: 19 May 2019, 21:53
by queimadus
I think I may have found a bug when the action is set to duplicate and the underlying OS doesn't fully support for the first approach.
After digging through the code, I saw that when the action is set to "duplicate" it will try to "clone", if that fails fallback to "duplicate" and if that fails, fallback to "copy".
The "clone" option makes use of "cp --reflink" which apparently my OS does not support (Inappropriate ioctl for device). That alone logs an error but instead of fallbacking to "duplicate" it goes straight to a "copy". (I know duplicate works because that's how I fixed my setup, and it now correctly hardlinks files)
I didn't get much into why it skipped the second fallback, I think when the "cp --reflink" fails, it still creates an empty target file which may interfere with the duplicate fallback.
PS: Running "Linux vero 3.14.29-143-osmc #1 SMP osmc-ccachefix aarch64" while using the amc script
Re: Failed to hardlink when --action=duplicate
Posted: 20 May 2019, 08:06
by rednoah
Here's how it works:
1. Try clone / reflink
2. Try hardlink
3. Try copy
If clone doesn't work, then it will hardlink. If hardlink doesn't work, it will copy.
Debug logging will give you more information:
viewtopic.php?f=3&t=3913
queimadus wrote: ↑19 May 2019, 21:53
I think when the "cp --reflink" fails, it still creates an empty target file which may interfere with the
Please confirm that
cp --reflink indeed creates an empty target file. That would be pretty stupid cp behaviour. But if that's indeed the case, then I'll remove the clone attempt altogether.
Re: Failed to hardlink when --action=duplicate
Posted: 20 May 2019, 11:32
by queimadus
Seems like it does.
Code: Select all
[ osmc@vero /tmp/test ]$ ls
[ osmc@vero /tmp/test ]$ echo hello > foo
[ osmc@vero /tmp/test ]$ cp --reflink foo bar
cp: failed to clone 'bar' from 'foo': Inappropriate ioctl for device
[ osmc@vero /tmp/test ]$ ls
bar foo
[ osmc@vero /tmp/test ]$ cat bar
[ osmc@vero /tmp/test ]$ cat foo
hello
Re: Failed to hardlink when --action=duplicate
Posted: 20 May 2019, 11:35
by rednoah
Noted. Will remove clone attempt from the duplicate action.
Re: Failed to hardlink when --action=duplicate
Posted: 09 Aug 2019, 11:20
by rednoah
I've updated the code to:
Code: Select all
1. Try hardlink
2. Try clonefile (via cp -c) / reflink (cp --reflink=auto)
3. Try physical copy
To make sure we don't create empty files,
2. Try clonefile / reflink now uses
cp --reflink=auto, so on Linux the
cp call will implicitly take care of copying files if necessary.
@since
r6569
Re: Failed to hardlink when --action=duplicate
Posted: 16 Nov 2019, 22:45
by ghostserverd
I think I'm seeing something similar with r6224.
Code: Select all
FileBot 4.8.5 (r6224)
JNA Native: 5.2.0
MediaInfo: 17.12
p7zip: p7zip Version 16.02 (locale=C.UTF-8,Utf16=on,HugeFiles=on,64 bits,8 CPUs Intel(R) Core(TM) i7-4770R CPU @ 3.20GHz (40661),ASM,AES-NI)
unrar: UNRAR 5.50 freeware
Chromaprint: fpcalc version 1.4.3
Extended Attributes: OK
Unicode Filesystem: OK
Script Bundle: 2019-05-15 (r565)
Groovy: 2.5.6
JRE: OpenJDK Runtime Environment 11.0.4
JVM: 64-bit OpenJDK 64-Bit Server VM
CPU/MEM: 8 Core / 1 GB Max Memory / 32 MB Used Memory
OS: Linux (amd64)
HW: Linux 25436ff19aab 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
DATA: /data
Package: DOCKER
License: UNREGISTERED
Done ヾ(＠⌒ー⌒＠)ノ
when I force it to hardlink, it's successful
Code: Select all
Locking /config/amc.log
Run script [fn:amc] at [Sat Nov 16 22:21:02 UTC 2019]
Parameter: unsorted = y
Parameter: clean = y
Parameter: music = y
Parameter: artwork = y
Parameter: subtitles = en
Parameter: minLengthMS = 0
Parameter: movieFormat = {plex} - {hd}
Parameter: seriesFormat = {plex} - {hd}
Parameter: animeFormat = {plex} - {hd}
Parameter: musicFormat = {plex} - {hd}
Parameter: ut_dir = /share/downloads/complete/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO
Parameter: ut_kind = multi
Parameter: ut_title = The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO
Parameter: ut_label = N/A
Input: /share/downloads/complete/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO.mkv
Group: [Series:the mandalorian] => [The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO.mkv]
Get [English] subtitles for 1 files
CmdlineException: OpenSubtitles: Please enter your login details by calling `filebot -script fn:configure`
Rename episodes using [TheTVDB]
Auto-detected query: [the mandalorian]
Fetching episode data for [The Mandalorian]
[HARDLINK] from [/share/downloads/complete/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO.mkv] to [/share/media/TV Shows/The Mandalorian/Season 01/The Mandalorian - S01E02 - Chapter 2 - The Child - HD.mkv]
Processed 1 files
Done ヾ(＠⌒ー⌒＠)ノ
and it's very quick (5.005s)
but when I set it to duplicate, it fails to clone and then takes way longer (46.387s), which I believe is performing an actual copy
Code: Select all
Locking /config/amc.log
Run script [fn:amc] at [Sat Nov 16 22:35:36 UTC 2019]
Parameter: unsorted = y
Parameter: clean = y
Parameter: music = y
Parameter: artwork = y
Parameter: subtitles = en
Parameter: minLengthMS = 0
Parameter: movieFormat = {plex} - {hd}
Parameter: seriesFormat = {plex} - {hd}
Parameter: animeFormat = {plex} - {hd}
Parameter: musicFormat = {plex} - {hd}
Parameter: ut_dir = /share/downloads/complete/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO
Parameter: ut_kind = multi
Parameter: ut_title = The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO
Parameter: ut_label = N/A
Input: /share/downloads/complete/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO.mkv
Group: [Series:the mandalorian] => [The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO.mkv]
Get [English] subtitles for 1 files
CmdlineException: OpenSubtitles: Please enter your login details by calling `filebot -script fn:configure`
Rename episodes using [TheTVDB]
Auto-detected query: [the mandalorian]
Fetching episode data for [The Mandalorian]
[DUPLICATE] from [/share/downloads/complete/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO.mkv] to [/share/media/TV Shows/The Mandalorian/Season 01/The Mandalorian - S01E02 - Chapter 2 - The Child - HD.mkv]
cp: failed to clone '/share/media/TV Shows/The Mandalorian/Season 01/The Mandalorian - S01E02 - Chapter 2 - The Child - HD.mkv' from '/share/downloads/complete/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO.mkv': Operation not supported
Processed 1 files
Done ヾ(＠⌒ー⌒＠)ノ
I'd like to be able to maintain the fallback to copy if necessary (I'm building a container, and I want to be resilient to different volumes / block device setups) but I also want to be able to hardlink when possible.
Is there a different version I should be using that will have the updated code you described? Thanks!
Re: Failed to hardlink when --action=duplicate
Posted: 17 Nov 2019, 05:58
by rednoah
The changes mentioned in the previous post have been implemented with
r6569 and previous revisions such as
r6224 use a different implementation.
You can find the latest revision here:
viewtopic.php?f=7&t=1609
Re: Failed to hardlink when --action=duplicate
Posted: 17 Nov 2019, 07:06
by ghostserverd
Oh wow I was totally dyslexic reading that revision number. I’ll take a look at the other builds. Thanks!