Failed to hardlink when --action=duplicate

Support for Ubuntu and other Desktop Linux distributions
Post Reply
queimadus
Posts: 5
Joined: 19 May 2019, 21:35

Failed to hardlink when --action=duplicate

Post by queimadus »

I think I may have found a bug when the action is set to duplicate and the underlying OS doesn't fully support for the first approach.

After digging through the code, I saw that when the action is set to "duplicate" it will try to "clone", if that fails fallback to "duplicate" and if that fails, fallback to "copy".

The "clone" option makes use of "cp --reflink" which apparently my OS does not support (Inappropriate ioctl for device). That alone logs an error but instead of fallbacking to "duplicate" it goes straight to a "copy". (I know duplicate works because that's how I fixed my setup, and it now correctly hardlinks files)

I didn't get much into why it skipped the second fallback, I think when the "cp --reflink" fails, it still creates an empty target file which may interfere with the duplicate fallback.

PS: Running "Linux vero 3.14.29-143-osmc #1 SMP osmc-ccachefix aarch64" while using the amc script
Top
User avatar
rednoah
The Source
Posts: 23924
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:
Contact rednoah

Re: Failed to hardlink when --action=duplicate

Post by rednoah »

Here's how it works:
1. Try clone / reflink
2. Try hardlink
3. Try copy

If clone doesn't work, then it will hardlink. If hardlink doesn't work, it will copy.

:idea: Debug logging will give you more information: viewtopic.php?f=3&t=3913

queimadus wrote: ↑19 May 2019, 21:53 I think when the "cp --reflink" fails, it still creates an empty target file which may interfere with the
Please confirm that cp --reflink indeed creates an empty target file. That would be pretty stupid cp behaviour. But if that's indeed the case, then I'll remove the clone attempt altogether.
:idea: Please read the FAQ and How to Request Help.
Top
queimadus
Posts: 5
Joined: 19 May 2019, 21:35

Re: Failed to hardlink when --action=duplicate

Post by queimadus »

Seems like it does.

Code: Select all

[ osmc@vero /tmp/test ]$ ls                                                                                                                                                                                                                                          
[ osmc@vero /tmp/test ]$ echo hello > foo                                                                                                                                                                                                                            
[ osmc@vero /tmp/test ]$ cp --reflink foo bar                                                                                                                                                                                                                        
cp: failed to clone 'bar' from 'foo': Inappropriate ioctl for device
[ osmc@vero /tmp/test ]$ ls                                                                                                                                                                                                                                          
bar  foo
[ osmc@vero /tmp/test ]$ cat bar                                                                                                                                                                                                                                     
[ osmc@vero /tmp/test ]$ cat foo                                                                                                                                                                                                                                     
hello
Top
User avatar
rednoah
The Source
Posts: 23924
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:
Contact rednoah

Re: Failed to hardlink when --action=duplicate

Post by rednoah »

Noted. Will remove clone attempt from the duplicate action.
:idea: Please read the FAQ and How to Request Help.
Top
User avatar
rednoah
The Source
Posts: 23924
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:
Contact rednoah

Re: Failed to hardlink when --action=duplicate

Post by rednoah »

I've updated the code to:

Code: Select all

1. Try hardlink
2. Try clonefile (via cp -c) / reflink (cp --reflink=auto)
3. Try physical copy
:idea: To make sure we don't create empty files, 2. Try clonefile / reflink now uses cp --reflink=auto, so on Linux the cp call will implicitly take care of copying files if necessary.

:idea: @since r6569
:idea: Please read the FAQ and How to Request Help.
Top
ghostserverd
Posts: 16
Joined: 15 Nov 2019, 06:25

Re: Failed to hardlink when --action=duplicate

Post by ghostserverd »

I think I'm seeing something similar with r6224.

Code: Select all

FileBot 4.8.5 (r6224)
JNA Native: 5.2.0
MediaInfo: 17.12
p7zip: p7zip Version 16.02 (locale=C.UTF-8,Utf16=on,HugeFiles=on,64 bits,8 CPUs Intel(R) Core(TM) i7-4770R CPU @ 3.20GHz (40661),ASM,AES-NI)
unrar: UNRAR 5.50 freeware
Chromaprint: fpcalc version 1.4.3
Extended Attributes: OK
Unicode Filesystem: OK
Script Bundle: 2019-05-15 (r565)
Groovy: 2.5.6
JRE: OpenJDK Runtime Environment 11.0.4
JVM: 64-bit OpenJDK 64-Bit Server VM
CPU/MEM: 8 Core / 1 GB Max Memory / 32 MB Used Memory
OS: Linux (amd64)
HW: Linux 25436ff19aab 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
DATA: /data
Package: DOCKER
License: UNREGISTERED
Done ヾ(＠⌒ー⌒＠)ノ
when I force it to hardlink, it's successful

Code: Select all

Locking /config/amc.log
Run script [fn:amc] at [Sat Nov 16 22:21:02 UTC 2019]
Parameter: unsorted = y
Parameter: clean = y
Parameter: music = y
Parameter: artwork = y
Parameter: subtitles = en
Parameter: minLengthMS = 0
Parameter: movieFormat = {plex} - {hd}
Parameter: seriesFormat = {plex} - {hd}
Parameter: animeFormat = {plex} - {hd}
Parameter: musicFormat = {plex} - {hd}
Parameter: ut_dir = /share/downloads/complete/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO
Parameter: ut_kind = multi
Parameter: ut_title = The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO
Parameter: ut_label = N/A
Input: /share/downloads/complete/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO.mkv
Group: [Series:the mandalorian] => [The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO.mkv]
Get [English] subtitles for 1 files
CmdlineException: OpenSubtitles: Please enter your login details by calling `filebot -script fn:configure`
Rename episodes using [TheTVDB]
Auto-detected query: [the mandalorian]
Fetching episode data for [The Mandalorian]
[HARDLINK] from [/share/downloads/complete/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO.mkv] to [/share/media/TV Shows/The Mandalorian/Season 01/The Mandalorian - S01E02 - Chapter 2 - The Child - HD.mkv]
Processed 1 files
Done ヾ(＠⌒ー⌒＠)ノ
and it's very quick (5.005s)

but when I set it to duplicate, it fails to clone and then takes way longer (46.387s), which I believe is performing an actual copy

Code: Select all

Locking /config/amc.log
Run script [fn:amc] at [Sat Nov 16 22:35:36 UTC 2019]
Parameter: unsorted = y
Parameter: clean = y
Parameter: music = y
Parameter: artwork = y
Parameter: subtitles = en
Parameter: minLengthMS = 0
Parameter: movieFormat = {plex} - {hd}
Parameter: seriesFormat = {plex} - {hd}
Parameter: animeFormat = {plex} - {hd}
Parameter: musicFormat = {plex} - {hd}
Parameter: ut_dir = /share/downloads/complete/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO
Parameter: ut_kind = multi
Parameter: ut_title = The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO
Parameter: ut_label = N/A
Input: /share/downloads/complete/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO.mkv
Group: [Series:the mandalorian] => [The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO.mkv]
Get [English] subtitles for 1 files
CmdlineException: OpenSubtitles: Please enter your login details by calling `filebot -script fn:configure`
Rename episodes using [TheTVDB]
Auto-detected query: [the mandalorian]
Fetching episode data for [The Mandalorian]
[DUPLICATE] from [/share/downloads/complete/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO.mkv] to [/share/media/TV Shows/The Mandalorian/Season 01/The Mandalorian - S01E02 - Chapter 2 - The Child - HD.mkv]
cp: failed to clone '/share/media/TV Shows/The Mandalorian/Season 01/The Mandalorian - S01E02 - Chapter 2 - The Child - HD.mkv' from '/share/downloads/complete/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO/The.Mandalorian.S01E02.2019.1080p.WEBRip.X264.AC3-EVO.mkv': Operation not supported
Processed 1 files
Done ヾ(＠⌒ー⌒＠)ノ
I'd like to be able to maintain the fallback to copy if necessary (I'm building a container, and I want to be resilient to different volumes / block device setups) but I also want to be able to hardlink when possible.

Is there a different version I should be using that will have the updated code you described? Thanks!
Top
User avatar
rednoah
The Source
Posts: 23924
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:
Contact rednoah

Re: Failed to hardlink when --action=duplicate

Post by rednoah »

The changes mentioned in the previous post have been implemented with r6569 and previous revisions such as r6224 use a different implementation.

You can find the latest revision here:
viewtopic.php?f=7&t=1609
:idea: Please read the FAQ and How to Request Help.
Top
ghostserverd
Posts: 16
Joined: 15 Nov 2019, 06:25

Re: Failed to hardlink when --action=duplicate

Post by ghostserverd »

Oh wow I was totally dyslexic reading that revision number. I’ll take a look at the other builds. Thanks!
Top
Post Reply

Return to “Ubuntu & Desktop Linux”