Delete Duplicate Movies

Support for Windows users
Post Reply
Spazattack4life
Posts: 12
Joined: 12 Mar 2018, 16:44

Delete Duplicate Movies

Post by Spazattack4life »

Hello,

I am looking for assistance on renaming files on a remote NAS. I have installed FB off of the windows store onto a Windows 10 Laptop. (I noticed in the Vids that I am missing the IMDB option for movies. Is this a difference in versions?) The file path is \\192.168.5.195\Movies\(Movie name here) (Year)\(Movie name here) (year) [resolution]. It doesn't give me the override option and I would like to delete any duplicate films (by keeping the one with the biggest file size). Also when it shows it in the new names area it shows \\192.168.5.195/movies/movie name (year)/Movie name (year) [Resolution}. Is this part of the issue? no matter how I change the slashes in the parser it doesn't fix it (using GUI). This isn't working no matter what I try and I could use some assistance.

Thanks in advance!
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Network File Renaming

Post by rednoah »

1.
I recommend using TheMovieDB. It's the standard database for all programs and tools. IMDb is not supported.


2.
Since you include the resolution in the file path, you will probably never have a duplicate file path. If you use a format such as \\192.168.5.195\{plex} then the same movie will result in the same file path and you'll be able to override the previous file which is a decision you have to make yourself.

If you want to search for movie duplicates, and delete lower quality files, then you can do that from the command-line:
viewtopic.php?f=4&t=5#p23171


3.
You should always use / as directory separator. On Windows, \ and / are equivalent. However, FileBot will always display / for esthetic reasons.
:idea: Please read the FAQ and How to Request Help.
Spazattack4life
Posts: 12
Joined: 12 Mar 2018, 16:44

Re: Delete Duplicate Movies

Post by Spazattack4life »

I finally got it to show me the override and continue however the override is greyed out (is it because I am using a synology disk station, I am signing into it with admin privledges on my windows files explorer before dragging and CTRL+SHIFT clicking the folders only into the original files location, for some reason files and folders errors out with long running script, and I am doing about 6 TB of movies at once). Also where do I post pictrures at so you can get more reliable data and is it possible to pull the backround log file of whats happening so that I and you can troubleshoot it?) I is it possible to call that script during when it renames so that like in powershell if($error -like "Duplicate Destination Path")(keep larger file expression goes here)? Also so I can do more research on this, what is the parser using as a "Language" if you would as I have never used it before? Thanks for the expedient reply.


P.S. I do have duplicate files as this is a shared plex repository (there is another TV Shows and Anime location that I yet have to go through, and I keep the files resolution so that I can see if we can get a higher quality of the film.)
Last edited by Spazattack4life on 12 Mar 2018, 20:21, edited 1 time in total.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Delete Duplicate Movies

Post by rednoah »

Please show me a screenshot of the error you see. Follow the link in my signature to see which image hosting website I would recommend.
:idea: Please read the FAQ and How to Request Help.
Spazattack4life
Posts: 12
Joined: 12 Mar 2018, 16:44

Re: Delete Duplicate Movies

Post by Spazattack4life »

I have it where can I post it at? I tried to drag and drop into the text box to have it attach and it doesn't do it.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Delete Duplicate Movies

Post by rednoah »

You host it somewhere and then you copy and paste the link.

Upload Screenshots => snag.gy
Upload Logs => pastebin.com

@see viewtopic.php?f=3&t=1868
:idea: Please read the FAQ and How to Request Help.
Spazattack4life
Posts: 12
Joined: 12 Mar 2018, 16:44

Re: Delete Duplicate Movies

Post by Spazattack4life »

FileBot 4.7.21 (r5319)
JNA Native: 5.2.0
MediaInfo: 0.7.99
7-Zip-JBinding: 9.20
Chromaprint: 1.4.2
Extended Attributes: OK
Unicode Filesystem: OK
Script Bundle: 2018-03-03 (r514)
Groovy: 2.4.13
JRE: Java(TM) SE Runtime Environment 9.0.4
JVM: 64-bit Java HotSpot(TM) 64-Bit Server VM
CPU/MEM: 4 Core / 3 GB Max Memory / 2 GB Used Memory
OS: Windows 10 (amd64)
Package: APPX
Spazattack4life
Posts: 12
Joined: 12 Mar 2018, 16:44

Re: Delete Duplicate Movies

Post by Spazattack4life »

Also looking at the threads you forwarded to does the AMC script delete duplicates and only keep the highest quality? because if I read the thread right then I could set it to display as what I want and then have it auto keep the highest quality. Thanks for the support.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Delete Duplicate Movies

Post by rednoah »

1.
This error means that you have multiple files that should be moved/renamed to the exact same file path, which isn't possible. You can change your format to yield a unique file path for each file, e.g. by adding {resolution} to the file path.


2.
The amc script has the --conflict auto option, which means that it'll override an existing file if a newly processed file should be moved to the exact same destination path and if it is of better media quality.

Note that a physical file path duplicate is not the same a a logical metadata duplicate. FileBot only handles the former, because it has to. The latter can be dealt with with the duplicates script.
:idea: Please read the FAQ and How to Request Help.
Spazattack4life
Posts: 12
Joined: 12 Mar 2018, 16:44

Re: Delete Duplicate Movies

Post by Spazattack4life »

So just tell me exactly how to do this either in the gui or in the command line. I not following what your saying I would like all the files to be in the format of IP\Movies\Name (year)\Name (year) [exact resolution].ext If there is a duplicate file then I want it to keep the highest quality. I just want this to work.
Spazattack4life
Posts: 12
Joined: 12 Mar 2018, 16:44

Re: Delete Duplicate Movies

Post by Spazattack4life »

Nevermind I will keep troubleshooting and getting this to work. From now on only respond if you have extra time as I am learning how to use Java Runtime 9 or whatever. I will try to compete this with the AMC script and have it pull from the move database (maybe later for shits and giggles I will try IMDB as that's what the Plex server is set to pull off of. I don't expect you to hold my hand I just F***ing hate how I can call your {Plex} thing but not filter the "CD$pi" this is where I am at now but I think I will scour the forum and try to customize the AMC script to my liking and make sure I follow the Synology NAS forum and use the Notepad++ to get it to work in addition to the windows 10 forum. I will periodically post on here where I get with it and you can chime in if you want, or provide tips and tricks to the trade. I love what you started on I just wish I could pick this up faster damn it.


Also in custom made movie formats is it possible to set it to enable conflict override so that it will just delete the file currently located at the file path location and then paste it there? its just a small usability thing that might help the noobs like me get this working a bit faster.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Delete Duplicate Movies

Post by rednoah »

1.
Set the following movie format:

Code: Select all

X:/Media/{plex.derive{' ' + [resolution]}}
That will ensure that each file will get a unique file name. You will not get "Duplicate destination path" errors this way. You will end up with multiple video files of various resolutions in each movie folder.


2.
If you are an advanced user that is familiar with the command-line, then you can use the duplicates script in a second step to delete lower resolution versions of the same movie. The GUI does not support this second step.
:idea: Please read the FAQ and How to Request Help.
Spazattack4life
Posts: 12
Joined: 12 Mar 2018, 16:44

Re: Delete Duplicate Movies

Post by Spazattack4life »

Thanks. I will look into the command line as well as the groovy/java command line.
Spazattack4life
Posts: 12
Joined: 12 Mar 2018, 16:44

Re: Delete Duplicate Movies

Post by Spazattack4life »

This is the command line input that I have saved to run the AMC script. Do you mind proof reading it or suggesting anything better? I already accidentally messed up and removed some files that I now need to re-get, sooo yeah... Anyway its below.

Code: Select all

Filebot paramfilebot -script fn:amc --output "\\192.168.5.195" --action copy -non-strict "\\192.168.5.195" --log-file "C:\Users\isaac\Desktop\amc.log" --def excludeList=C:\Users\isaac\Desktop\amc.txt --conflict auto --def subtitles=en --def clean=n --def movieFormat="\\192.168.5.195\Movies\{ny}\{fn{n.replaceAll(/[:*?"<>|?]/).replaceAll(/[`´‘’?]/, "'")} ({y})\{n.replaceAll(/[:*?"<>|?]/).replaceAll(/[`´‘’?]/, "'")} ({y}){' ['+vf}] {+lang}}" seriesFormat="\\192.168.5.195\TV Shows\{n.replaceAll(/[:*?"<>|?]/).replaceAll(/[`´‘’?]/, "'")}\Season {(episode.special ? 0 : episode.season ? s : 1).pad(2)}\ {n.replaceAll(/[:*?"<>|?]/).replaceAll(/[`´‘’?]/, "'")} S{(episode.special ? 0 : episode.season ? s : 1).pad(2)}E{episode.special ? special.pad(2) : e.pad(2)} - {t.replaceAll(/[:*?"<>|?]/).replaceAll(/[`´‘’?]/, "'")}{' ['+vf}] {+lang}" animeFormat="\\192.168.5.195\Anime\{n.replaceAll(/[:*?"<>|?]/).replaceAll(/[`´‘’?]/, "'")}\Season {(episode.special ? 0 : episode.season ? s : 1).pad(2)}\ {n.replaceAll(/[:*?"<>|?]/).replaceAll(/[`´‘’?]/, "'")} S{(episode.special ? 0 : episode.season ? s : 1).pad(2)}E{episode.special ? special.pad(2) : e.pad(2)} - {t.replaceAll(/[:*?"<>|?]/).replaceAll(/[`´‘’?]/, "'")}{' ['+vf}] {+lang}" musicFormat="\\192.168.5.195\Music\{artist}\{album}\{pi.pad(2)+' '} - {t}"
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Delete Duplicate Movies

Post by rednoah »

Absolutely not. This is not human-readable code. What does the computer say when you run this command? :lol:


:idea: I recommend using --action TEST for testing. Use --conflict skip if you don't wanna risk overriding files. You can do a few runs and see how it goes, and step by step disable these options.


:idea: Use the @file syntax for reading command-line arguments from external text files. That'll help you put your crazy format into an external text file never to be looked at again. This will also resolve issues caused by not correctly escaping command-line arguments. ;)
:idea: Please read the FAQ and How to Request Help.
Spazattack4life
Posts: 12
Joined: 12 Mar 2018, 16:44

Re: Delete Duplicate Movies

Post by Spazattack4life »

I took your advice and my results are below in addition to the output after the modification.

Code: Select all

Filebot paramfilebot -script fn:amc --output "\\192.168.5.195/Movies" --action test --conflict auto -non-strict "\\192.168.5.195\Movies" --log-file "C:\Users\isaac\Desktop\amc.log" --def excludeList=C:\Users\isaac\Desktop\amc.txt --def subtitles=en --def clean=n --def movieFormat="\\192.168.5.195\Movies\{ny}\{fn} ({y}){' ['+vf}] {+lang}}" seriesFormat="\\192.168.5.195\TV Shows\{n}\Season {(episode.special ? 0 : episode.season ? s : 1).pad(2)}\ {n} S{(episode.special ? 0 : episode.season ? s : 1).pad(2)}E{episode.special ? special.pad(2) : e.pad(2)} - {t}{' ['+vf}] {+lang}" animeFormat="\\192.168.5.195\Anime\{n}\Season {(episode.special ? 0 : episode.season ? s : 1).pad(2)}\ {n} S{(episode.special ? 0 : episode.season ? s : 1).pad(2)}E{episode.special ? special.pad(2) : e.pad(2)} - {t}{' ['+vf}] {+lang}" musicFormat="\\192.168.5.195\Music\{artist}\{album}\{pi.pad(2)+' '} - {t}"
As far as the arguments file I have not placed it in there yet but I will. Now a couple of questions. I am trying to run this remotely to a Synology NAS and so looking at the forums it seems that Linux uses \ instead of / so for the locations I start with \\ so that FileBot will go to the network location then / so it can interface with the DiskStation (also my diskstation version 6.5.1-15254 update 1 doesnt have filebot to download, do I still need to install java 9 (same version as the windows store version) onto the diskstation?)

I got it down to the error below from the Log File

Illegal usage: output folder [\\SIMBAMWR01\Movies] must be separate from input arguments [\\SIMBAMWR01\Movies]
Failure (°_°)

I want it to really search IP/Movies and IP/TV Shows and IP/Music and IP/Anime however I cant just say the IP because windows sees all those as different "Share Drives" that are hosted using the same IP. Thats also why I list out the exact naming scheme for the file names so that it can be placed in the right area, and why I am not leaving it up to the {plex} option. its also because I hate how plec uses the CD1 as a modifer and doesnt add the {Resolution} after the {y} identifier.


I tried using a wild card however I received this error, so I once again am kinda stuck in a corner.

Code: Select all

Illegal char <*> at index 21: \\SIMBAMWR01\Movies\'*'
java.nio.file.InvalidPathException: Illegal char <*> at index 21: \\SIMBAMWR01\Movies\'*'
	at net.filebot.util.FileSet.getPath(FileSet.java:90)
	at net.filebot.util.FileSet.contains(FileSet.java:82)
	at java_util_Set$contains$0.call(Unknown Source)
	at Script1.acceptFile(Script1.groovy:236)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at Script1$_run_closure47.doCall(Script1.groovy:299)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at Script1.run(Script1.groovy:299)
	at net.filebot.cli.ScriptShell.evaluate(ScriptShell.java:64)
	at net.filebot.cli.ScriptShell.runScript(ScriptShell.java:74)
	at net.filebot.cli.ArgumentProcessor.runScript(ArgumentProcessor.java:127)
	at net.filebot.cli.ArgumentProcessor.run(ArgumentProcessor.java:29)
	at net.filebot.Main.main(Main.java:112)
Also I noticed on one of the forums someone was pulling info from IMDB but earlier you reccomend not too. Is this because you haven't thouroughly tested it? Just asking. Any insight on a way forward is appreciated.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Delete Duplicate Movies

Post by rednoah »

1.
All operating systems us / as directory separator. Unix uses only / and Windows uses \ and / so you can just always use / on all operating systems.


2.
\\server\share is called a UNC path. Only Windows uses and supports UNC paths. UNC paths will not work on any operating system other than Windows, so you can't use them on Synology NAS. You'll have to first mount the SMB share and then use the local mount path.


3.
FileBot does not support IMDb so it's impossible to pull info from IMDb.


4.
FileBot for Synology NAS is currently not supported. FileBot 4.7.9 requires Java 8 and still works just fine though:
viewtopic.php?f=13&t=1802

You can support FileBot for Linux here:
https://www.patreon.com/filebot
:idea: Please read the FAQ and How to Request Help.
Spazattack4life
Posts: 12
Joined: 12 Mar 2018, 16:44

Re: Delete Duplicate Movies

Post by Spazattack4life »

So a couple of things,

First how do I mount the SMB share? I know that I can add the individual drives onto my windows by using the add network drive wizard and the UNC path, all well and good, however this does not change the fact that the input area and output area root will be the same as I will then have to place in multiple input areas and multiple output areas using the "local" path. The only way around this that I see is to have a locationcheck -Off type of argument that I could pass into it.

Second, I will change it to // for everything then.

Third, I will install the Java 8 onto the Synology box.

Fourth, I am guessing that I could run this as a regular task via windows scheduler to run once a week by creating a .bat that runs the command line version. Not that I am going to try it now, I am just thinking about the future of this.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Delete Duplicate Movies

Post by rednoah »

Synology NAS has a nice GUI to configure SMB mounts:
https://www.synology.com/en-global/know ... motevolume

Once it's mounted, the network share will appear as normal folder in your filesystem. You can mount different servers/shares to different folders.

You'll still have to have separate input and output folders, just like always, so I'm not sure why you think you need to turn off the sanity checks. Please explain in detail with examples.
:idea: Please read the FAQ and How to Request Help.
Post Reply