Something is broken

So I'm going to start from the beginning, even though it's a little too much information. I'll apologize now for the length of the post.

So, I have a couple thousand movies I've wanted to centralize for awhile now, on disks and external hard drives. I started playing with this program a few months ago and really like it, although the scripting is still over my head.

I found a free program on the net and used it to take 700 old disks I had in binders and automate the process of transferring them to a hard drive for me. So now I have these ~1100 movies sitting in a temp folder on my server. I decided on the naming format I want and decided I want to use a different program for the nfo and artwork. Before I let a rename script go automated on the folder though I wanted to test it out in small batches. I opened the program up and started dragging and dropping folders into the left rename pane, using groups of 30 movies at a time. I found that not all of the groups I have are known, so I get a lot of [DATA] instead. This is actually a little annoying for the ones that don't have groups instead of not showing up at all, but that's fine, no biggie. I've been making a list of the groups I have that haven't shown up so I can send it in after.

Question about that before I keep going. Is it better to rename the movie, then manually change [DATA] to the proper group, or email the list and rename them after the list has been updated. I think the only thing that can really be affected by this, would be the history and option of undoing the rename.

Ok, so I renamed some movies and left some for later because the groups weren't recognized and some because the folders weren't recognized. I'll go back later and manually change the folders a bit, then throw them back in and let it try again. The way the folders were named the first time, there are quite a few, probably half that filebot needs more info to process. So I'm thinking I'm glad I'm doing this manually, although it seems like it's going to take forever.

I was plugging along and matching to tmdb, then a couple times my windows explorer crashed on my vista machine and explorer restarted itself and I kept going. I'm assuming it was running out of ram, even though I have 3gb, it's a couple years old now. Then after renaming 79 folders it wouldn't rename anymore. When it was working, I'd click rename after making sure everything was good and it would take maybe a second to pop up at the top of my screen and tell me how many movies were just renamed. After I did a few rounds successfully I tried another batch and a different little window popped up saying moving 1 of 24 movies, with a bar moving along to look like it was doing something. I sat there for awhile waiting and it didn't seem to be making any progress. I cancelled that and tried it again, same thing. After doing that, the movie it was working on, first in the list showed up in the new location but wouldn't play. None of the other movies showed up. Going back to the original movie folder that movie was gone, moved already. So it wasn't in the old location and the one that was renamed in the new location won't play. I closed the program and opened again to try again. A different movie was first in the list, this time the same thing. A little window popped up saying moving, but didn't seem to be doing anything, instead of happening almost instantly like before. Again that first movie it said it was moving won't play in the new location and is gone from the old one. Both those movies are gone now. Not a HUGE deal really, but annoying.

I noticed there is now an updated version of the program, but since it was just working minutes before, I don't generally like upgrading unless I have to. I thought maybe my machine was bogging down, a memory leak somewhere or something I don't know, so I restarted hoping that would fix it. Nope. Haven't been able to rename since. It will match, but when I'm happy and press rename, just a pop up.

Suggestions?

I also thought maybe it had something to do with my server automatically spinning down hard drives if no activity for 15 minutes, but they were definitely active on later tests so I don't think that was it.

Another thing, I just tried again with my next batch to see if it is somehow fixed as quickly as it was broken. Still says moving files. I let it run for about a minute then cancelled it. Still on movie #1, where as before it would move and rename 30 movies in 1 or 2 seconds. Again, now that movie won't work anymore either. I think maybe it starts copying the movie over, very slowly for some reason, then when I cancel it it's not complete, even though it looks like it is in the new folder and somehow it adjusted something in the original instead of waiting until it was complete to remove the original. Also, not every time, but a few of these times I've tried and it tells me the destination already exists. Which would have to be from it's last attempt, but it doesn't contain a working video file.

I went to go take a look at the renaming history to see what time this problem started last night and none of the movies I renamed last night are listed. Probably because on my last attempt filebot didn't want to shut down at all so I had to end it in the task manager.


Thanks again for any help, it felt good when I was actually making progress renaming.

[rednoah]

Try running FileBot with console and see if there is any output. If it's not working for specific paths you can send me those as well.

Would be good to know FileBot version, and more importantly Java version, you can use this script to print some info:

Code: Select all

filebot -script fn:sysinfo -trust-script

I assume you're moving files on the same disk so renaming should always be almost instant. Actually from what you're telling me, with explorer crashing all the time that there is some problem with the filesystem in general and with Java accessing the filesystem. If you could somehow narrow done the problem or reproduce that would be of great help but it seems obscure. Maybe some permissions got reset when moving the file?

If you're using Java 6, just update to Java 7 and there's a good chance filesystem related problems will go away. Also keep FileBot updated, even small movie detection improvements could just save you lots of time.

Sorry, "DATA" is a group. Newer revisions only check folder/filename, had been checking the whole path before. You probably have a "data" folder. Removed "DATA" release group for now so that problem will fix itself once rg info is updated.

Also computer specs can't be an issue, filebot's memory limit is set to 256mb and it never uses more than 150mb anyway even with 1000+ movies.

So while I was waiting I tried to install the updated version, but it told me I had to uninstall the old one first through the control panel. I never know if I should do that first or try install over top. Anyways, I uninstalled the older, which was 2.62 and installed 2.64.

Then as a test I tried renaming just one movie. Again, instead of being almost instant like it was it popped up and said moving. This time I let it run and looked over at my web interface for my server while it said it was moving and sure enough my free space was being reduced by about 3MB/sec. I let it finish and it did rename and move my movie.

I don't understand why it's taking so long though, since it was working. Right now I haven't installed more drives in the server because I wanted to avoid exactly that problem of renaming across drives. These movies are staying on the same drive and shouldn't take this long. Even when I was dragging and dropping last night 30 movies into seperate folders for this purpose 29 would move instantly and 1 would transfer at 3-4MB/sec.

I don't get it. I'm using an unRAID server if anyone has any suggestions. I'll search their forums too.

Here is the new output of your suggestion, although I had to run a -clear first because it spit some errors of unclean shutdown.

Code: Select all

C:\Program Files\FileBot>filebot -script fn:sysinfo -trust-script
FileBot 2.64 (r1024)
MediaInfo: MediaInfoLib - v0.7.48
7-Zip-JBinding: OK
Java(TM) SE Runtime Environment 1.7.0_04
32-bit Java HotSpot(TM) Client VM
Windows Vista (x86)
Done ?(?????)?

I checked out one of those movies that wouldn't work anymore and it vlc gives the option to repair it, which I tried, but doesn't seem to fix it. The file will play in vlc even though it says it's broken. Again, I think somehow it got changed before the move was completed safely.

I'm checking forums for permissions, but I really don't understand why it was working and then all of a sudden instead of 2 seconds to rename 30 movies it could be an hour. Very strange.

Anyways, you were right about [DATA]. They are gone now, so then I have to ask if there is an easy way to remove that from the ones I already renamed. The problem is I like having the original filename kept and because [DATA] is now in that filename I don't know how to get rid of it.

This is the format I'm using:

Code: Select all

\\192.168.x.xxx\disk1\Video\Movies\Old Disks\{(n =~ /^[Tt]he / ? n[4..-1]+", The" : n =~ /^[Aa] / ? n[2..-1]+", A" : n).replace(":",' -').replace(";",' -')}{"  [$y]"}{"  [Rated $certification]"}{"  [Voted $rating]"}\{"[$sdhd]"}{"  [$vf]"}{"  [$source]"}{"  [$vc]"}{"  [$ac]"}{"  [$group]"}{"  $fn"}{"  [CD$pi]"}

Which on a plain jane filename spits out something like:
Bandits [2001] [Rated PG-13] [Voted 7.9]\[SD] [DivX] [MP3] [DATA] Bandits.avi

Which also because of the fn if you try to throw it back in it gets rid of the [DATA] but brings the old filename which doubles the info you get and still includes the [DATA]:
[SD] [DivX] [MP3] [SD] [DivX] [MP3] [DATA] Bandits.avi

[rednoah]

So it's all remote right? I guess somehow things get moved to your computer and then copied back again there for some reason. FileBot and maybe not even Java have any idea how Windows Vista decides the move files from A to B. Do you rename via an smb path like \\server\path\to\files? You could try mapping your data to a drive letter, that might work better.

[rednoah]

part timer wrote:

Which also because of the fn if you try to throw it back in it gets rid of the [DATA] but brings the old filename which doubles the info you get and still includes the [DATA]:
[SD] [DivX] [MP3] [SD] [DivX] [MP3] [DATA] Bandits.avi

Would removing all [...] from {fn} help?

Code: Select all

fn.replaceAll(/\[[^\]]+\]/)}

I was originally using a mapped drive letter, but tried the ip address hoping it might change something. They both seem to have the same result. Instead of just changing the folder name and being instant it copies the whole movie and then deletes the old location.

This is moving from one folder to another on the same drive.

The new code you gave does work as it removes the other items but it doesn't remove the spaces so I get about 4 spaces too many, like this:

Code: Select all

[SD] [DivX] [MP3]      Bandits.avi

Strange strange strange. I thought I had it solved. I changed the destination directory to the same directory where the 30 movies were, since I moved batches of 30 into separate folders. It worked. Instant change. Then I went to the next directory and thought I'd try change it slightly to go one directory level deeper, fail. So I changed it back to match the directory I was in this time, fail. Again it's broken, which doesn't make sense because I'm using the same format I just was.

So here is a stupid question maybe, but may also solve all my problems. Why is it that when I go to the rename section in filebot and match my files and click rename it creates a new directory and leaves an empty old directory? Why can't it simply rename the existing directory and file, since almost all of these directories contain only one file? Or maybe it can and I don't know how. I'll try change my format once again after this transfer takes forever moving these files to the same directory they are already in.

Also, I'm a little hesitant, but can I let loose a filebot script to delete all my empty directories after?

[rednoah]

part timer wrote:

The new code you gave does work as it removes the other items but it doesn't remove the spaces so I get about 4 spaces too many...

Code: Select all

fn.replaceAll(/\[[^\]]+\]/)}.trim()

[rednoah]

FileBot operates on files not folders, moves files to where you specify, creating parent folders if necessary. It would be very wierd to kinda try to 'recycle' original folders. That would be a lot of work and ugly code just for a wierd workaround for vista shares.

Delete clutter files (nfo, txt, jpg, ...) and empty folders:

Code: Select all

filebot -script fn:cleaner -trust-script /path/to/media

That trim option you gave me worked like a charm, thank you. It was something on my end. I still don't know what causes the slow transfer rates with some files and not with others. I came up with a plan to get around it though, in case anyone else runs into the same issue.

I have an external drive that is full of movies right now so I'm working on it first via usb, with instant file renames. When it's completed I'll plug it in directly to my server and transfer all the contents off and to the server. Then I'll take the other movies that are on the server now and giving me the issue and put them on the external. Hopefully these copy operations can be done while sleeping. That way I can plug the drive back in USB to my laptop, get it all sorted and back to the server to live again.

Kind of a pain this way, but as long as the big copies are done while sleeping this seems easiest to work with.

Thank you again for the help. It was my server not filebot, of course.

I love that cleaner script btw, that was awesome. I was a little hesitant at first but I'm a believer. I did modify it a bit though. I had extra clutter file types hanging around and hundreds of folders left over to deal with.

I changed the file types to:

Code: Select all

return file.hasExtension("URL", "srt", "nfo", "db", "dna", "gif", "idx", "log", "md5", "rtf", "sfv", "sub", "url", "xml", "txt", "jpg", "jpeg")

and now that I think of it should have added png, htm and html to the list too.

I love this though, since I'm dealing with the media folders, I didn't want it deleting nfo's and srt's on me, but it only selects the folder with no videos left in it. Just like the script says, only the "orphaned" files. Since all of these folders should have a video inside them, if they don't then all of those extensions can be deleted as the leftovers when my video and srt files are done being renamed. Having all of these extra extensions doesn't hurt anything either since it's only movie folders I'm processing.

Cleans it right up. Awesome, thank you.

Oh, I almost forgot, it doesn't seem to be case sensitive like the Analyze pane is. The analyze pane will show me separate NFO Nfo and nfo sometimes, this doesn't seem to differentiate fyi.


I'm sure there is a very simple way to do it, but out of laziness on my part I'll ask. What would I need to add to the cleaner to save the output to a uniquely named text file to skim through and know what was deleted?

[rednoah]

Just redirect stdout into a file.

e.g.

Code: Select all

filebot -list --q firefly > episodes.txt

@see
http://en.wikipedia.org/wiki/Redirectio ... ard_output

I was a little slow on the uptake and it took me a little bit of reading to figure out any kind of solution, but I got one.

Ran the script locally because of the file extensions I added in the post above and ran it like:

Code: Select all

filebot -script "C:\Users\Scott\Desktop\cleaner.groovy" -trust-script "N:\Movies" >> "C:\Users\Scott\Desktop\cleaner log.txt"

One > writes to the txt file and erases any previous text file and 2 >> keeps adding to the text file every time instead.

Then created a batch file for it on the desktop, so I can just double click it. It runs and does it's thing and just spits the output into a text file. I added the append option as well so each time it runs it just adds the newest results on to the end of the last run and keeps a running list.

So cool, I love it, thanks again.

Another update here, when dealing with my unRaid server. I wouldn't call it fixed yet, but it seems when using the IP address or the server name some files take a long time to be renamed or 'moved' and others don't, but it seems when I use the drive letter mapped to the server it works as it should, instantly. Maybe that will help someone else out there.

[rednoah]

Lately I've been experimenting with hooking into the native shell to do move/copy operations.

Script:

Code: Select all

rename(..., action:net.sourceforge.filebot.NativeRenameAction.MOVE)

GUI:

Code: Select all

-DuseNativeShell=true

Basically it's like Windows Explorer copying files. So maybe that'll serve as a good workaround.

You know me, that's too much code for me

By using the drive letter though it seems to (so far, knock on wood) work just fine, as it should, even though I don't see the difference. The only other thing I can think of that might have something to do with it is in the settings for my server I specified to only use the one drive for right now for my movies. There was always just the one drive so far, but maybe now that it knows for sure there can be only one that has something to do with it. I dunno maybe that or maybe the drive letter, or the combination. So far it works though.

[rednoah]

Just grab one of the latest jar, overrride the one in your install folder, and then add that line to filebot.launcher.l4j.ini:

Code: Select all

-DuseNativeShell=true

Main benefit is really that you'll get a nice move/copy progress, same as windows explorer move/copy.

rednoah wrote:

Code: Select all

fn.replaceAll(/\[[^\]]+\]/)}.trim()

What would I need to add to this code to also remove info in rounded brackets?

Code: Select all

Fast Five [2011 PG-13] ([2011]DvDrip[Eng]-FXG)

becomes

Code: Select all

Fast Five (DvDrip-FXG)

but how do I also get rid of the info in those rounded brackets?
To get this instead:

Code: Select all

Fast Five

Thanks so much, as always

[rednoah]

Code: Select all

fn.replaceAll(/\[[^\]]+\]/).replaceAll(/\([^\)]+\)/)}