CSV Issue

[DevXen]

So I'm having a problem with this:
{csv('C:/FileBot Settings/Movies/ExtraFileInfo/Test.csv').findResults{ k, v -> k.contains(n) ? "$v" : null }}

If I am checking the the csv file for Kung Fu Panda 2, and Both movies are listed in it:
Dreamworks: Kung Fu Panda [2008]
Dreamworks: Kung Fu Panda 2 [2011]

Then it fails. and returns nothing.

also if i have any other things in it.
say:
ABC Family Movie;Snow 2: Brainfreeze [2008]

it also will not return the Result.

This File has 263k of movies listed, for 'additional info' as it made sense to add it all into 1 file, instead of having 20 different files it has to check per movie.
do you maybe have any advice. to help resolve this?


Thanks,
-Dev

[rednoah]

The code is simple enough. I'm sure you'll figure it out:
https://github.com/svn2github/filebot/b ... s.java#L91

Keep in mind that csv() will give you a Map, so field 0 has to be unique, and there should only be 2 fields in total for each line.

If you need something more powerful you can always just use the standard Groovy stuff.

[DevXen]

OK, i'll look into that, thank you. I was wondering cause i did extensive testing on it, on this PC and it seemed to work fine, with every test.
then i moved it to the pc with my video files and it decided it didn't want to work. also the files, i've noticed sometimes the CSV Files will only work with UTF-8 Encoding, and other times only with ANSI Encoding. weird.

[DevXen]

Ok, I found the problem, it appears to be putting the first result, and the last result of each field 0.

Dreamworks; Movie 1
Dreamworks; Movie 2
Dreamworks; Movie 3
Dreamworks; Movie 4

It's only adding Movie 1, and movie 4 to the mapping.
hmm..

[rednoah]

As you can see by the code of csv() the format must be:

Code: Select all

key1;value
key2;value
key3;value
...

The keys must be UNIQUE, otherwise the last occurrence of a key will rule. The text file will be decoded with UTF-8 (without BOM) as is.

The result of reading this file:

Code: Select all

Dreamworks; Movie 1
Dreamworks; Movie 2
Dreamworks; Movie 3
Dreamworks; Movie 4

should be ['Dreamworks', 'Movie 4'] however you probably have an (invisible) BOM marker at the beginning of the file. Your first key is "Dreamworks" and the second key is "Dreamworks".

@see http://en.wikipedia.org/wiki/Byte_order_mark

You can easily avoid BOM issues by not using Microsoft Software as your Text Editor. I recommend Sublime.

If you use UTF-8 your editor is adding non-text invisible bytes at the beginning of the file, but it does not do that for ANSI encoded text. So if you save as ANSI you get a text file, if you save as UTF-8 you get a binary file which is mostly text after the initial non-text byte sequence. If you look at the decoded/visible text it won't make sense, but if you look at the bytes you'll see why it's not working.

[DevXen]

So looking into that, it's a great theory. but I opened the .csv in notepad++ and checked 'show all hidden characters' and it didn't show anything. then i checked the settings
and it is set to save as: UTF-8 Without BOM.

So that's likely not the issue. i did find out. that with all the other 'extra info lines i have. (roughly 7,000) it does work fine. as long as only the one Dreamworks is in it. which likely corresponds to my previous testing. I probably tested that with Pixar and as it worked, figured it would for all the animated studios I added.

that's why i was asking if this looked correct: {csv('C:/FileBot Settings/Movies/ExtraFileInfo/Test.csv').findResults{ k, v -> k.contains(n) ? "$v" : null }}
it's the only thing i can think of that would be causing the issue. but If i can't work it out. i'll just have to separate it all out into separate files. and go back to the standard csv checks for each one. it'll just get long.

also coincidentally. i did try just the DreamWorks films in their own separated file. but the standard csv default reading way. and it didn't matter what was in there, it read them all just fine.

The idea behind the extra info. is to put what company did the 3D animation, or if the movie was banned in a country, or multiple countries, or if the movie was based on a book, or a real person, or a true story, or if it's a disney channel movie. etc, etc, etc. and by having it as i have it now, the idea was it would loop through, find all the results, and apply them as applicable if there was 1 match for 3.

But Thanks for your help. I appreciate it.

[rednoah]

Well, you have the code, and it's so simple that there can't be any bugs in that piece of code itself anyway. Either the file format is wrong, or the function just works differently from what you expected.

Just copy the csv function into eclipse and debug it if you want to see what's happening step by step.

EDIT:

If you wanna read complicated structured data you can just use JSON:

Code: Select all

{'''[{"a":0, "b":1}, {"a":2, "b":3}]'''.jsonToObject() as List}

Code: Select all

{someFile.text.jsonToObject() as List}

Though I guess that might be a lot more work if you're not already generating the data yourself somehow.