Renaming files with date names (e.g. 17-april-2020) to {S00E00}

All your suggestions, requests and ideas for future development
Post Reply
Aaron
Posts: 9
Joined: 18 Apr 2020, 01:37

Renaming files with date names (e.g. 17-april-2020) to {S00E00}

Post by Aaron »

Hello all! I am a new filebot user, I purchased the license yesterday.

So for my media storage, I use a schema of /video/tv/show-title/s00e00.mp4. I don't like having messy metadata in filenames... if you think I'm wrong or naive, I am certainly open to knowing why!

Anyway, so I have a crontab that executes a script that fetches dated news programming; it's a program listed in thetvdb. Current episodes are like S45E73 for April 17, 2020, for example. So when I fetch the show from the web using a script, I'd like filebot to rename the file to the {S00E00} convention to conform to my schema mentioned above.

So my script will save the file as 17-april-2020.mp4, for example. But filebot renamed it with the wrong corresponding {S00E00}, using S45E17 instead of S45E78. I'm thinking it may have parsed the date incorrectly or something.

Does anyone know a better, more streamlined way of handling this? Or should the date of the filename be in a certain format that filebot will parse when doing a db lookup in thetvdb to get the corresponding {S00E00}?

The command used:

Code: Select all

filebot -rename 17-april-2020.mp4 --q "PBS NewsHour" --format "{S00E00}" -non-strict
User avatar
rednoah
The Source
Posts: 19127
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Renaming files with date names to {S00E00}

Post by rednoah »

:!: Since the airdate uniquely identifies your file, you don't wanna use -non-strict, because we don't want FileBot to default to the "next best, not so great" match just because nobody has entered the new data into the database yet.


The 17-april-2020 date pattern seems to be broken. I'll look into it.


The 17-April-2020 and 17-04-2020 patterns work though.



EDIT:

FileBot r7388 adds support for additional date patterns:

Code: Select all

17-april-2020
April 17, 2020
:idea: Please read the FAQ and How to Request Help.
Aaron
Posts: 9
Joined: 18 Apr 2020, 01:37

Re: Renaming files with date names (e.g. 17-april-2020) to {S00E00}

Post by Aaron »

Excellent, yes, I can certainly modify my script to output DD-MM-YYYY for a filename instead. I just used the other way because I was lazy and reused variables in the script to generate the URL to download the file from the CDN server in first place.

Actually, the ISO-8601 date format is ideal, which is YYYY-MM-DD. Then files sorted alphanumerically would be ordered chronologically. Maybe consider that pattern?

Would it make sense for filebot to ever support arguments for airdate outside of parsing it from the filename?

Thanks for the great support and an extremely handy, powerful utility.
User avatar
rednoah
The Source
Posts: 19127
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Renaming files with date names (e.g. 17-april-2020) to {S00E00}

Post by rednoah »

Aaron wrote: 18 Apr 2020, 10:11 Actually, the ISO-8601 date format is ideal, which is YYYY-MM-DD. Then files sorted alphanumerically would be ordered chronologically. Maybe consider that pattern?
YYYY-MM-DD is supported out of the box. I recommend using that one. But good thing you reported lower-case april not working. That one fell through the cracks.

Aaron wrote: 18 Apr 2020, 10:11 Would it make sense for filebot to ever support arguments for airdate outside of parsing it from the filename?
You could use --filter to select one specific episode:

Code: Select all

$ filebot -list --q "PBS NewsHour" --filter "airdate =~ /2020.04.17/" -rename xyz.mp4 --action TEST --log INFO
[TEST] from [xyz.mp4] to [PBS NewsHour - 45x78 - April 17, 2020.mp4]
:idea: Please read the FAQ and How to Request Help.
Aaron
Posts: 9
Joined: 18 Apr 2020, 01:37

Re: Renaming files with date names (e.g. 17-april-2020) to {S00E00}

Post by Aaron »

Still having some issues with file renames of dated files. But I haven't updated to FileBot 4.9.1 (r7372) yet. But none of these issues involve a lowercase month in a filename. These are without --filter.

USA date method (MM-DD-YYYY):

Code: Select all

$ touch 04-17-2020.mp4
$ filebot -rename 04-17-2020.mp4 --q "PBS NewsHour" --format "{S00E00}"
Failed to process group: {} => [/temp/04-17-2020.mp4]
Failed to identify or process any files
Failure (×_×)⌒☆
I thought perhaps the zero pad on the month (0M-DD-YYYY) might be problematic, but that didn't help.

Code: Select all

$ touch 4-17-2020.mp4
$ filebot -rename 4-17-2020.mp4 --q "PBS NewsHour" --format "{S00E00}"
Rename movies using [TheMovieDB]
Looking up movie by query [PBS NewsHour]
Failed to find a valid match: []
Failure (×_×)⌒☆
Seeing the output of it checking TheMovieDB, I thought perhaps forcing the correct db might help:

Code: Select all

$ filebot -rename 4-17-2020.mp4 --db TheTVDB --q "PBS NewsHour" --format "{S00E00}"
Rename episodes using [TheTVDB] with [Airdate Order]
Fetching episode data for [PBS NewsHour]
Failed to identify or process any files
Failure (×_×)⌒☆
EU date method (DD-MM-YYYY) failed.

Code: Select all

$ touch 17-04-2020.mp4
$ filebot -rename 17-04-2020.mp4 --q "PBS NewsHour" --format "{S00E00}"
Failed to process group: {Series=null} => [/temp/17-04-2020.mp4]
Failed to identify or process any files
Failure (×_×)⌒☆
ISO 8601 date works good (I will use this):

Code: Select all

$ touch 2020-04-17.mp4
$ filebot -rename 2020-04-17.mp4 --q "PBS NewsHour" --format "{S00E00}"
Rename episodes using [TheTVDB] with [Airdate Order]
Fetching episode data for [PBS NewsHour]
[MOVE] from [/temp/2020-04-17.mp4] to [/temp/S45E78.mp4]
Processed 1 files
Here's sysinfo for me. I am running a VM, and memory is very scant. I hope to have a more robust environment soon.

Code: Select all

$ filebot -script fn:sysinfo
FileBot 4.9.0 (r7234)
JNA Native: 6.1.0
MediaInfo: net.filebot.mediainfo.MediaInfoException: Unable to load amd64 (64-bit) native library libmediainfo.so: Unable to load library 'mediainfo':
Shared object "libmediainfo.so" not found, required by "java"
Shared object "libmediainfo.so" not found, required by "java"
Native library (freebsd-x86-64/libmediainfo.so) not found in resource path (/home/aaronkaase/bin/filebot/jar/filebot.jar)
p7zip: java.io.IOException: Cannot run program "7z": error=2, No such file or directory
FFprobe: 4.2.2
Chromaprint: java.io.IOException: Cannot run program "fpcalc": error=2, No such file or directory
Extended Attributes: OK
Unicode Filesystem: OK
Script Bundle: 2020-04-16 (r632)
Groovy: 3.0.2
JRE: OpenJDK Runtime Environment 11.0.6
JVM: 64-bit OpenJDK 64-Bit Server VM
CPU/MEM: 2 Core / 312 MB Max Memory / 23 MB Used Memory
OS: FreeBSD (amd64)
HW: FreeBSD perch 11.3-RELEASE-p7 FreeBSD 11.3-RELEASE-p7 #0: Tue Mar 17 08:32:23 UTC 2020     [email protected]:/usr/obj/usr/src/sys/GENERIC  amd64
STORAGE: ufs [/] @ 6 GB | nfs [/mnt/XigmaNAS] @ 1.4 TB
DATA: /home/aaronkaase/bin/filebot/data/aaronkaase
Package: TAR
License: FileBot License P13947000 (Valid-Until: 2021-04-24)

------------------- UPDATE AVAILABLE: FileBot 4.9.1 (r7372) --------------------

Done ヾ(@⌒ー⌒@)ノ
User avatar
rednoah
The Source
Posts: 19127
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Renaming files with date names (e.g. 17-april-2020) to {S00E00}

Post by rednoah »

You're missing --db TheTVDB, otherwise you're relying on auto-detection which doesn't have much to work with with these files.


:idea: You especially don't wanna rely on auto-detection / auto-matching on low-memory devices. Best to be explicit about everything you already know in advance. ;)


:!: Don't use MM-DD-YYYY. FileBot will work with that, but only if the numbers can't be interpreted as DD-MM-YYYY.


** as always, everything I say is based on the latest revision, and may very well not apply to older outdated revisions
:idea: Please read the FAQ and How to Request Help.
Aaron
Posts: 9
Joined: 18 Apr 2020, 01:37

Re: Renaming files with date names (e.g. 17-april-2020) to {S00E00}

Post by Aaron »

Yeah I will for sure use the ISO 8601 date format and I will also include the --db TheTVDB in the script. Anything to make it run as reliably as possible in an unattended script.

I'm spinning up a bare metal server to ditch the feeble VM that I'm running on the NAS. I'm thinking some of the errors reported by filebot's sysinfo in the VM above could perhaps be causing some erratic behavior (timeouts or whatnot).
Aaron
Posts: 9
Joined: 18 Apr 2020, 01:37

Re: Renaming files with date names (e.g. 17-april-2020) to {S00E00}

Post by Aaron »

I'm running into issues with a rename failure again. I'm not sure if the issue is FileBot, or TheTVDB though, because I manually renamed the file myself and Plex failed to associate that name with an air date. So my hunch is TheTVDB might be acting up:

Code: Select all

$ filebot -rename 2020-04-26.mp4 --db TheTVDB --q "CBS News Sunday Morning" --format "{S00E00}" -no-xattr
Rename episodes using [TheTVDB] with [Airdate Order]
Fetching episode data for [CBS News Sunday Morning]
Failed to identify or process any files
Failure (×_×)⌒☆
$ ls  
total 2613290
drwxr-xr-x   2 aaronkaase  1003     4B Apr 26 12:11 .
drwxrwxrwx  76 1003        1003    78B Apr 23 00:57 ..
-rw-r--r--   1 aaronkaase  1003   1.2G Apr 26 12:07 2020-04-26.mp4
-rw-r--r--   1 aaronkaase  1003   1.3G Apr 17 03:31 S42E30.mp4
$
Now a curious thing is that the "First Aired" field in TheTVDB was initially incorrect. It was set for 25 April when it should have been 26 April. So I edited that episode to correct the date. I suspect there's probably all sorts of caches and updates that may need to happen in the background, perhaps, before APIs will pull the change. But it's been hours now and it doesn't seem to be happening yet.

I'm running FileBot on a new server now, far more power and memory than the VM above, in case that matters.

Code: Select all

$ filebot -script fn:sysinfo
FileBot 4.9.1 (r7372)
JNA Native: 6.1.0
MediaInfo: 19.09
p7zip: p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,4 CPUs x64)
unrar: UNRAR 5.80 freeware
FFprobe: 4.2.2
Chromaprint: java.io.IOException: Cannot run program "fpcalc": error=2, No such file or directory
Extended Attributes: OK
Unicode Filesystem: OK
Script Bundle: 2020-04-20 (r634)
Groovy: 3.0.3
JRE: OpenJDK Runtime Environment 13.0.2
JVM: 64-bit OpenJDK 64-Bit Server VM
CPU/MEM: 4 Core / 2.1 GB Max Memory / 39 MB Used Memory
OS: FreeBSD (amd64)
HW: FreeBSD marlin 12.1-RELEASE-p3 FreeBSD 12.1-RELEASE-p3 GENERIC  amd64
STORAGE: ufs [/] @ 271 GB | nfs [/mnt/XigmaNAS] @ 1.4 TB
DATA: /home/aaronkaase/bin/filebot/data/aaronkaase
Package: TAR
License: FileBot License P13947000 (Valid-Until: 2021-04-24)
Done ヾ(@⌒ー⌒@)ノ
More tests:

Code: Select all

$ touch 2020-04-12.mp4
$ touch 2020-04-19.mp4
$ touch 2020-04-25.mp4
$ touch 2020-04-26.mp4
$ filebot -rename 2020-04-12.mp4 --db TheTVDB --q "CBS News Sunday Morning" --format "{S00E00}" -no-xattr
Rename episodes using [TheTVDB] with [Airdate Order]
Fetching episode data for [CBS News Sunday Morning]
[MOVE] from [/mnt/XigmaNAS/video/tv/CBS News Sunday Morning/temp/2020-04-12.mp4] to [/mnt/XigmaNAS/video/tv/CBS News Sunday Morning/temp/S42E30.mp4]
Processed 1 files
$ filebot -rename 2020-04-19.mp4 --db TheTVDB --q "CBS News Sunday Morning" --format "{S00E00}" -no-xattr
Rename episodes using [TheTVDB] with [Airdate Order]
Fetching episode data for [CBS News Sunday Morning]
[MOVE] from [/mnt/XigmaNAS/video/tv/CBS News Sunday Morning/temp/2020-04-19.mp4] to [/mnt/XigmaNAS/video/tv/CBS News Sunday Morning/temp/S42E31.mp4]
Processed 1 files
$ filebot -rename 2020-04-25.mp4 --db TheTVDB --q "CBS News Sunday Morning" --format "{S00E00}" -no-xattr
Rename episodes using [TheTVDB] with [Airdate Order]
Fetching episode data for [CBS News Sunday Morning]
Failed to identify or process any files
Failure (×_×)⌒☆
$ filebot -rename 2020-04-26.mp4 --db TheTVDB --q "CBS News Sunday Morning" --format "{S00E00}" -no-xattr
Rename episodes using [TheTVDB] with [Airdate Order]
Fetching episode data for [CBS News Sunday Morning]
Failed to identify or process any files
Failure (×_×)⌒☆
$ 
Really seems like an issue with TheTVDB. Does that happen on occasion?
User avatar
rednoah
The Source
Posts: 19127
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Renaming files with date names (e.g. 17-april-2020) to {S00E00}

Post by rednoah »

You can check your assumptions like so:

Code: Select all

filebot -list --db TheTVDB --q "CBS News Sunday Morning" --format "{airdate} {episode}"
...
2020-04-26 CBS News Sunday Morning - 42x32 - April 26, 2020
:idea: If the episode information is there, then it'll work.

:idea: If the the episode information is not there then that explains why FileBot can't find a match. The latter case is not unlikely if the data was only added or modified yesterday, and not a week in advance. (assuming there's an episode next week, it's not been added yet, so yesterdays episode probably wasn't added well in advance either, so adding next weeks episode(s) is your job now)
:idea: Please read the FAQ and How to Request Help.
Aaron
Posts: 9
Joined: 18 Apr 2020, 01:37

Re: Renaming files with date names (e.g. 17-april-2020) to {S00E00}

Post by Aaron »

rednoah wrote: 27 Apr 2020, 07:50 You can check your assumptions like so:

Code: Select all

filebot -list --db TheTVDB --q "CBS News Sunday Morning" --format "{airdate} {episode}"
...
2020-04-26 CBS News Sunday Morning - 42x32 - April 26, 2020
:idea: If the episode information is there, then it'll work.
That command is extremely helpful. So now I've discovered an inconsistency between two machines I am running FileBot with. One displays that episode, and the other does not.

This is on my Mac, it works fine:

Code: Select all

~$ filebot -list --db TheTVDB --q "CBS News Sunday Morning" --format "{airdate} {episode}" | grep 42x32
2020-04-26 CBS News Sunday Morning - 42x32 - April 26, 2020
~$
My FreeBSD box does not:

Code: Select all

$ filebot -list --db TheTVDB --q "CBS News Sunday Morning" --format "{airdate} {episode}" | grep 42x32
$ 
It does, however, list the two episodes previous to the most current one:

Code: Select all

$ filebot -list --db TheTVDB --q "CBS News Sunday Morning" --format "{airdate} {episode}" | grep 42x3
2020-04-12 CBS News Sunday Morning - 42x30 - April 12, 2020
2020-04-19 CBS News Sunday Morning - 42x31 - April 19, 2020
$
The version installed on the Mac is older than the one on the BSD box, it was the first one I installed to try out before getting the license. I just haven't gotten around to uninstalling it yet. I used brew to install it versus the tarball on the BSD box.
User avatar
rednoah
The Source
Posts: 19127
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Renaming files with date names (e.g. 17-april-2020) to {S00E00}

Post by rednoah »

FileBot will cache results client-side for a few days, so each of your machines will likely have requested the data at different times, and thus now be working with different cached data.


:idea: If you process this show once a week, then you'll always be working with the latest information, because the data from the previous week will always be stale by then. But if you're manually testing every day, then perhaps FileBot will use cached data because it's just 2-3 days old.
:idea: Please read the FAQ and How to Request Help.
Aaron
Posts: 9
Joined: 18 Apr 2020, 01:37

Re: Renaming files with date names (e.g. 17-april-2020) to {S00E00}

Post by Aaron »

Yep, that was the culprit. Okay. I'll just add -clear-cache to the scripts. Or maybe just clear them nightly or something.

Code: Select all

$ filebot -clear-cache
Clear cache
* Delete /home/aaronkaase/bin/filebot/data/aaronkaase/cache/0
$ filebot -list --db TheTVDB --q "CBS News Sunday Morning" --format "{airdate} {episode}" | grep 42x32
Initialize new disk cache: /home/aaronkaase/bin/filebot/data/aaronkaase/cache/0
2020-04-26 CBS News Sunday Morning - 42x32 - April 26, 2020
I suppose the whole point of caching is to minimize db queries, which makes sense for a community-driven site like TheTVDB.
User avatar
rednoah
The Source
Posts: 19127
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Renaming files with date names (e.g. 17-april-2020) to {S00E00}

Post by rednoah »

:!: Do not run -clear-cache unless you have reason to do so.


Doing it manually, once in a while if and when needed, no problem. Doing it preemptively for all calls, you'll get yourself banned. ;)
:idea: Please read the FAQ and How to Request Help.
kim
Power User
Posts: 1046
Joined: 15 May 2014, 16:17

Re: Renaming files with date names (e.g. 17-april-2020) to {S00E00}

Post by kim »

if you really need to "clear cache"
maybe just delete the thetvdb*.data and thetvdb*.index files
User avatar
rednoah
The Source
Posts: 19127
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Renaming files with date names (e.g. 17-april-2020) to {S00E00}

Post by rednoah »

That's the same as clearing the cache partially. Same rules apply. Doing it on a case by case manually is fine, disabling caching completely by deleting the files between each run not so much.
:idea: Please read the FAQ and How to Request Help.
Aaron
Posts: 9
Joined: 18 Apr 2020, 01:37

Re: Renaming files with date names (e.g. 17-april-2020) to {S00E00}

Post by Aaron »

Get myself banned from who? FileBot? TheTVDB? Sorry this is new to me. :oops:

I just discovered the discrepancy between the two systems I had FileBot installed was solved by clearing the cache on the discrepant system. If there's a much better solution for this, I appreciate knowing! ;)

:?: Does clearing cache impact resources for FileBot or TheTVDB? Does filebot.net maintain its own copy of these servers for its users?

By the way, I never knew what these :!: :?: :idea: emoji on phpBB forums were for before participating on this one, so I am learning volumes on this site. :D
User avatar
rednoah
The Source
Posts: 19127
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Renaming files with date names (e.g. 17-april-2020) to {S00E00}

Post by rednoah »

Both FileBot and TheTVDB have web services, that don't like to be abused. The limit is of course reasonably high so you won't run into the problem during normal usage, but if you disable caching you will almost certainly run into this problem eventually.

:idea: viewtopic.php?t=9594
:idea: Please read the FAQ and How to Request Help.
Aaron
Posts: 9
Joined: 18 Apr 2020, 01:37

Re: Renaming files with date names (e.g. 17-april-2020) to {S00E00}

Post by Aaron »

Ah, ok, no web services appreciate being abused. I had no idea FileBot maintained any sort of web services. That's very good to know and helps me understand how it works. As for TheTVDB, I think I could solve the problem of missing episodes by populating its database with future episode data... I see that done with "PBS NewsHour", but neither "CBS Sunday Morning" nor "60 Minutes".

On the subject of 60 Minutes, a news show that airs weekly here in the United States, there apparently seems to be an Australian version that FileBot seems to always pick up, even with the explicit string --q "60 Minutes".

Execution:

Code: Select all

$ filebot -list --db TheTVDB --q "60 Minutes" --format "{airdate} {episode}" | tail -5
 60 Minutes Australia - Special 33
 60 Minutes Australia - Special 34
 60 Minutes Australia - Special 35
 60 Minutes Australia - Special 36
 60 Minutes Australia - Special 37
$ filebot -list --db TheTVDB --q "60 Minutes Australia" --format "{airdate} {episode}" | tail -5
 60 Minutes Australia - Special 33
 60 Minutes Australia - Special 34
 60 Minutes Australia - Special 35
 60 Minutes Australia - Special 36
 60 Minutes Australia - Special 37
$
The same results are returned with different queries. Any way to specify an explicit lookup? I thought the omission of -non-strict at the end would enforce this.
User avatar
rednoah
The Source
Posts: 19127
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Renaming files with date names (e.g. 17-april-2020) to {S00E00}

Post by rednoah »

-non-strict means that FileBot will print all episode data for all series matches in order of relevance, instead of just the most relevant one. This is typically what you think it should be, but evidently not always. :lol:


You can use the numeric ID if you want to be more exact:

Code: Select all

$ filebot -list --db TheTVDB --q "60 Minutes" -non-strict --format "{id} {n}" | uniq
83063 60 Minutes Australia
73290 60 Minutes
266539 60 Minutes Sports
282939 60 Minutes on CNBC
$ filebot -list --db TheTVDB --q 73290 -non-strict --format "{id} {n}" | uniq
73290 60 Minutes

A more uniquely identifying alias title will do the job as well:

Code: Select all

$ filebot -list --db TheTVDB --q "60 Minutes (1968)" -non-strict --format "{id} {n} {alias}" | uniq
73290 60 Minutes [60 Minutes (1968), 60 Minutos]
83063 60 Minutes Australia [60 Minutes Australia (2008), 60 Minutes]
267271 40 Minutes [40 Minutes (1981), Forty Minutes]
:idea: Here we also see that 60 Minutes exactly is a valid alias name for 60 Minutes Australia, which explains why we get this one, and not the one from before Star Wars premiered. :lol:
:idea: Please read the FAQ and How to Request Help.
Post Reply