Page 1 of 1
Keep both and index new is skipping because of exact copy
Posted: 09 Oct 2018, 02:15
by daemonofchaos
I am running the Filebot Node docker container and have File Filters > Conflict set to "Keep both and Index new file".
The command line that shows in the output window is:
Code: Select all
filebot -script 'fn:amc' /volume2/movies --output /volume2/new --action move -non-strict --order DVD --conflict index --lang en --def 'skipExtract=y' 'minLengthMS=300000' 'minFileSize=100000000' 'exec=chown -R 99:100 /volume2/*' 'seriesFormat=tv_shows/{n}/Season {s.pad(2)}/{n} - {s00e00} - {t} - {vf}.{ac}.{af}' 'movieFormat=movies/{n} ({y})/{n} ({y}) - {vf}.{ac}.{af}' 'excludeList=.excludes' --log all --log-file '/usr/local/filebot-node/data/filebot.log'
However during processing a lot of the files are being skipped due to "is an exact copy and already exists" and some are not being processed at all.
Here is an excerpt from some of the output window:
Code: Select all
Rename movies using [TheMovieDB]
Auto-detect movie from context: [/volume2/movies/Pirates of the Caribbean Dead Men Tell No Tales (2017)/Pirates of the Caribbean Dead Men Tell No Tales (2017) - 1080p.AAC.6ch.mp4]
Auto-detect movie from context: [/volume2/movies/Pirates of the Caribbean Dead Men Tell No Tales (2017)/Pirates of the Caribbean Dead Men Tell No Tales (2017) - 2160p.TrueHD.8ch.mkv]
Stripping invalid characters from new path: movies/Pirates of the Caribbean: Dead Men Tell No Tales (2017)/Pirates of the Caribbean: Dead Men Tell No Tales (2017) - 1080p.AAC.6ch
Stripping invalid characters from new path: movies/Pirates of the Caribbean: Dead Men Tell No Tales (2017)/Pirates of the Caribbean: Dead Men Tell No Tales (2017) - 2160p.TrueHD.8ch
Processed 0 files
CmdlineException: Failed to process [/volume2/movies/Pirates of the Caribbean Dead Men Tell No Tales (2017)/Pirates of the Caribbean Dead Men Tell No Tales (2017) - 1080p.AAC.6ch.mp4] because [/volume2/new/movies/Pirates of the Caribbean Dead Men Tell No Tales (2017)/Pirates of the Caribbean Dead Men Tell No Tales (2017) - 1080p.AAC.6ch.mp4] is an exact copy and already exists
However, I only have a single file -
Code: Select all
Pirates of the Caribbean Dead Men Tell No Tales (2017) - 1080p.AAC.6ch.mp4
The 2160p version of the file is not present.
Can you advise what I am doing wrong?
Re: Keep both and index new is skipping because of exact copy
Posted: 09 Oct 2018, 04:38
by rednoah
1.
You're processing the exact same file twice, which means you're you're setup is doing something strange, or unexpected, like accidentally processing the same files over and over.
--def excludeList=.excludes should indeed prevent you from processing the same file path twice.
Why are you processing the same exact file twice via different file paths? Did you download the same file twice?
Did you accidentally delete your
/volume2/new/.excludes exclude list file? Resulting in the amc script not being able to outright skip previously processed files?
2.
According to the log, you did have the same file twice before calling filebot this time.
Here:
Code: Select all
/volume2/movies/Pirates of the Caribbean Dead Men Tell No Tales (2017)/Pirates of the Caribbean Dead Men Tell No Tales (2017) - 1080p.AAC.6ch.mp4
And here:
Code: Select all
/volume2/new/movies/Pirates of the Caribbean Dead Men Tell No Tales (2017)/Pirates of the Caribbean Dead Men Tell No Tales (2017) - 1080p.AAC.6ch.mp4
3.
--conflict index will work if you only process new files that you haven't processed before, since in this case a sanity check fails before we even get to processing the new 2160p file.
Re: Keep both and index new is skipping because of exact copy
Posted: 09 Oct 2018, 16:27
by daemonofchaos
The source directory did contain duplicate files as a result of importing from backups and automated downloads. The key issue was one would be in a directory named
Code: Select all
/volume2/movies/Pirates of the Caribbean Dead Men Tell No Tales (2017)
while the other would be in
Code: Select all
/volume2/movies/Pirates of the Caribbean - Dead Men Tell No Tales (2017)
(a dash between the main title and subtitle).
As such, I was trying to use Filebot to clean this mess up. The target directory was newly created to use as a "work" directory. Once all of the files from the source directory were processed and the collection curated, then I was going to delete the source, rename "new" to "movies".
I had considered selecting "Override only if better" but was not sure if "better" was based upon audio channels, audio type (DTS vs AAC), and/or video resolution. Since I was unsure how that worked, I chose the "keep both and index new" option and planned to manually review.
Ultimately I want to end up with both a 1080p and 2160p version of the file to avoid Plex having to transcode for my clients.
Re: Keep both and index new is skipping because of exact copy
Posted: 09 Oct 2018, 17:22
by rednoah
1.
"Override only if better" will not help in this case, since the same sanity check will prevent you from processing the same 1080p files. If enabled, and if you only have one 1080p file and one 2160p file, then it'll work, but because your format includes
{vf} {ac} {af}, each file will have a unique file path, and so no file will need to be overridden. So
"Override only if better" will effectively do nothing in your particular case.
"better" is primarily defined by file size and video resolution. Currently, audio channels are not factored in. If your format were to be just
{plex} then you would end up with just the 2160p file.
2.
You can use any number of tools (maybe just simple bash scripts) to find and delete (binary / exact) duplicate files and delete all but one. Once that is done, the amc script will not complain. If it's just a handful of files, then you can just manually delete the offending source file whenever FileBot complains.