Filebot in python
Posted: 26 Jan 2017, 17:37
I have a python script that calls filebot to rename my files and then uses exec to call another python file. When i run the script once, filebot fails to run (does not create a log either) but when i run the same script a second time, filebot does run properly. Im running this on a headless seedbox using filebot.sh. I will explain what the script doesn and where it fails.
The script:
Steps of script:
The script will loop through each of the folders in my subfolder variable. but only the folders containing movies use FB. So the first stop is movies_cps
1 - File is downloaded to .../Transfer/movies_cps
2 - Files are unpacked/and separated into each their own subfolder
3 - They are checked for their audio and
- if they contain English audio only, they stay in movies_cps
- if they contain French audio only, they go .../Transfer/french
- if they contain DualAudio audio only, they go .../Transfer/dualmovies
4 - Script then removes all non-essential files (samples, nfo, etc..)
5 - I then run convert_files function which launches sickbeard_mp4_automator on movies_cps (or whichever folder folder_path is equal to)
6 - SickMp4Auto output's the file to the final_path variable which would equal to /media/sdu1/home/saitoh183/private/Transfer/manage/movies/(cps,french,dualmovies)
7 - Next I check if there is content in the final_path and if there is, Run Filebot on final_path.
Now I added logging to try and capture the error but my logging comes up empty. I added an info line to see when filebot is called and when it ends and in the log, those lines are both there but nothing in between. I also enabled logs in the filebot command and set it to ALL but nothing was generated either.
So then if I take one of the files that are currently in final_path and move them back to their original location. Then I run the script again, my function convert_files will not convert the file cuz it has already been converted and it will move it directly to final_path, then the script will move on to Filebot, and filebot will do it things and generate logs and the files will be renamed.
So I don't know when on the first pass filebot is failing and doesn't generate any sort of logs but maybe you would have an idea or maybe a way for me to see what is going on with FB on the first attempt.
The script:
Code: Select all
#!/usr/bin/env python
import os
import re
import subprocess
from os.path import getsize
import shutil
import fnmatch
import logging
from logging.handlers import RotatingFileHandler
import argparse
import time
import unpack_transfer
#Set time zone
os.environ['TZ'] = 'EST+05EDT,M4.1.0,M10.5.0'
time.tzset()
#Parser get external vairables
parser = argparse.ArgumentParser()
parser.add_argument('-v','--verbose', help="Enable debug",action="store_true")
args = parser.parse_args()
log_filename = "/media/sdu1/home/saitoh183/private/logs/MediaMgmt_remote.log"
logger = logging.getLogger('Media_Processing')
if args.verbose:
logger.setLevel(logging.DEBUG)
dbug = '-v'
else:
logger.setLevel(logging.INFO)
dbug = ""
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
fh = RotatingFileHandler(log_filename, maxBytes=100000, backupCount=5)
fh.setFormatter(formatter)
logger.addHandler(fh)
console = logging.StreamHandler()
console.setLevel(logging.DEBUG)
console.setFormatter(formatter)
logger.addHandler(console)
def get_child2(path):
return [item for item in os.listdir(path) if item not in ('.stignore', '.stfolder') and getsize(os.path.join(path,item)) > 104857600]
def get_child(path):
return [item for item in os.listdir(path) if item not in ('.stignore', '.stfolder')]
def splitup(fdir,mediafolder,dir_):
logger.debug('FDIR: {}.'.format(fdir))
count = len(get_child2(fdir))
logger.debug('LENGH: {}.'.format(count))
newfdir = os.path.join(mediafolder, dir_)
if count >= 1:
logger.info('Splitting up Folders in {}.'.format(fdir))
try:
shutil.move(fdir, mediafolder)
return newfdir
except Exception:
pass
logger.info('Splitting up files in {}.'.format(newfdir))
for root, dirs, files2 in os.walk(newfdir):
for file_ in files2:
dst = os.path.join(mediafolder,file_)
current_file = os.path.join(root, file_)
logger.debug('Processing {}.'.format(current_file))
filesize = getsize(current_file)
if filesize > 104857600:
logger.debug('Destination1: {}.'.format(dst))
logger.debug('Destination2: {}.'.format(mediafolder))
if not os.path.exists(dst):
os.makedirs(dst)
logger.info('Moving {} to {}.'.format(current_file,dst))
shutil.move(current_file, dst)
def splitupmovies(mediafolder):
logger.info('Splitting up...Checking folder {}.'.format(mediafolder))
for path, subdirs, files in os.walk(mediafolder):
logger.debug('root folder: {}.'.format(path))
logger.debug('directories: {}.'.format(subdirs))
logger.debug('files: {}.'.format(files))
for dir_ in subdirs:
fdir = os.path.join(path,dir_)
newdir = splitup(fdir,mediafolder,dir_)
logger.debug('NEWDIR: {}'.format(newdir))
if newdir != None:
newcount = len(get_child2(newdir))
logger.debug('NEWCOUNT: {}'.format(newcount))
if newcount > 1:
splitup(newdir,mediafolder,dir_)
def clean_files(folder):
for path, subdirs, files in os.walk(folder):
logger.info('Removing small files from {}.'.format(path))
for name in files:
current_file = os.path.join(path, name)
# Check if file is more than 100mb
try:
filesize = getsize(current_file)
except os.error as e:
logger.error('Was unable to get filesize of {} ({}).'.format(name,e))
# File doesn't exist or is inaccessible
pass
# Remove all files that are 100mb and less from every subfolder
if name not in ('.stignore', '.stfolder') and filesize <= 104857600 or fnmatch.fnmatch(name,'name.r*'):
logger.debug('Removing file {}.'.format(name))
os.remove(current_file)
def convert_files(filepath):
convert_mp4 = subprocess.Popen([
"/media/sdu1/home/saitoh183/private/script/Convert/manual.py",
'-a', '-i', filepath,
'-pr','-m', final_path
],stdout=subprocess.PIPE).communicate()[0]
logger.debug('{}'.format(convert_mp4))
logger.info('End of conversion of files in {}'.format(filepath))
# Remove all empty directories:
for root, dirs, files in os.walk(filepath, topdown=False):
logger.info('Checking for empty directories in {}.'.format(root))
logger.debug('root folder: {}.'.format(root))
logger.debug('directories: {}.'.format(dirs))
logger.debug('files: {}.'.format(files))
for dir_ in dirs:
try:
logger.info('Removing directory {}.'.format(dir_))
os.rmdir(os.path.join(root, dir_))
except OSError as e:
logger.warning( '{}'.format(e))
def check_audio(f_path):
for path, subdirs, files in os.walk(f_path):
logger.info('Verifiying for Dual Audio Movies in {}.'.format(path))
logger.debug('path folder: {}.'.format(path))
logger.debug('directories: {}.'.format(subdirs))
logger.debug('files: {}.'.format(files))
for name in files:
if any(name.endswith(ext) for ext in ('mkv', 'avi', 'mp4')):
current_file = os.path.join(path, name)
logger.debug('PROCESSING: {}'.format(current_file))
audio = subprocess.check_output([
"mediainfo",
'--Inform=General;%AudioCount%', current_file
])
logger.debug('File {} has {} audio stream(s).'.format(name,audio))
language = subprocess.check_output([
"mediainfo",
'--Inform=Audio;%Language%', current_file
])
logger.debug('File {} has the following language(s): {}'.format(name,language))
if int(audio) > 1 and not fnmatch.fnmatch(language,'fr*fr'): # Dual audio
if os.path.join(folder_path, name) == current_file:
# File exist at the root
logger.info('Moving file {} to dualmovies.'.format(name))
dst = os.path.join(subfolders[5], name)
os.rename(current_file, dst)
else:
# File is in subfolder, move subfolder
logger.info('Moving directory {} to dualmovies.'.format(path))
shutil.move(path, subfolders[5])
elif "Francais" in language or "fr" in language: # French movie
if os.path.join(folder_path, name) == current_file:
# File exist at the root
logger.info('Moving file {} to french.'.format(name))
dst = os.path.join(subfolders[4], name)
os.rename(current_file, dst)
else:
# File is in subfolder, move subfolder
logger.info('Moving directory {} to french.'.format(path))
shutil.move(path, subfolders[4])
elif language == "":
logger.info('Audio language was not found for {}. It will stay in {}.'.format(name,folder_path))
else:
continue
def get_command(command, extra):
cmd = list(command)
cmd.extend(extra)
return cmd
filebot_cmd = [
"/media/sdu1/home/saitoh183/filebot/filebot.sh", #ie 'C:\Program Files\Filebot\filebot.exe'
# I recommend replacing 'fn:amc' with the path to a locally saved copy of
# amc.groovy. 'fn:amc' downloads the script every time you run filebot.exe
# from filebot.net. If filebot.net is down, your file processing won't be
# able to run. You can download the script from here:
# https://github.com/filebot/scripts/blob/devel/amc.groovy
'-script', 'fn:amc',
# Save log files to same directory where this script file is located
# '--log-file', logfiles,
'--action', 'move',
'--def', 'clean=y',
'-no-xattr',
#'--log=Fine'
]
# Declare variables
source = "/media/sdu1/home/saitoh183/private/Transfer"
importing = os.path.join(source, "importing")
subfolders = [
os.path.join(source, "movies_cps"),
os.path.join(source, "sonarr_tv"),
os.path.join(source, "sickrage_tv"),
os.path.join(source, "ufc"),
os.path.join(source, "french"),
os.path.join(source, "dualmovies")
]
importfolders = [
os.path.join(importing,"clone"),
os.path.join(source, "manage", "movies," "cps"),
os.path.join(source, "manage", "movies", "ufc"),
os.path.join(source, "manage", "movies", "dualmovies"),
os.path.join(source, "manage", "movies", "french")
]
for folder_path in subfolders:
logger.info('Processing path {}.'.format(folder_path))
childcount = len(get_child(folder_path))
logger.debug('Number of items in {}: {}.'.format(folder_path,childcount))
if not childcount:
logger.debug('There is nothing to process in {}.'.format(folder_path))
logger.info('-----------------------------------------------------------------')
continue
# Convert media files and assign final import path
if "sonarr_tv" in folder_path:
unpack_transfer.transfer(folder_path.rsplit('/',1)[1])
final_path = importfolders[0]
elif "sickrage_tv" in folder_path:
unpack_transfer.transfer(folder_path.rsplit('/',1)[1])
final_path = importfolders[0
elif "movies_cps" in folder_path:
unpack_transfer.transfer(folder_path.rsplit('/',1)[1])
final_path = importfolders[1]
splitupmovies(folder_path)
check_audio(folder_path)
new_filebot = get_command(filebot_cmd, [
'--def', 'movieFormat=/media/sdu1/home/saitoh183/private/Transfer/manage/movies/cps/{plex[1..-1]}',
'--def', 'ut_label=Movies',
'--def', 'exec= \"\"python /media/sdu1/home/saitoh183/private/script/rclone_move.py -fp \"{file}\"\"\"'
])
elif "ufc" in folder_path:
final_path = importfolders[2]
new_filebot = get_command(filebot_cmd, [
'--def', 'movieFormat=/media/sdu1/home/saitoh183/private/Transfer/manage/movies/ufc/{plex[1..-1]}',
'--def', 'ut_label=Movies',
'--def', 'exec= \"\"python /media/sdu1/home/saitoh183/private/script/rclone_move.py -fp \"{file}\"\"\"'
])
elif "dualmovies" in folder_path:
final_path = importfolders[3]
new_filebot = get_command(filebot_cmd, [
'--def', 'movieFormat=/media/sdu1/home/saitoh183/private/Transfer/manage/movies/dualmovies/{plex[1..-1]}',
'--def', 'ut_label=Movies',
'--def', 'exec= \"\"python /media/sdu1/home/saitoh183/private/script/rclone_move.py -fp \"{file}\" -fr \"{localize.french.name} ({y})\" -en \"{ny}\"\"\"'
])
elif "french" in folder_path:
final_path = importfolders[4]
new_filebot = get_command(filebot_cmd, [
'--def', 'movieFormat=/media/sdu1/home/saitoh183/private/Transfer/manage/movies/french/{plex[1..-1]}',
'--def', 'ut_label=Movies',
'--lang', 'fr',
'--def', 'exec= \"\"python /media/sdu1/home/saitoh183/private/script/rclone_move.py -fp \"{file}\"\"\"'
])
clean_files(folder_path)
logger.info('Starting converstion in {}'.format(folder_path))
convert_files(folder_path)
if any(re.findall(r'ufc|french|dualmovies|cps', final_path, re.IGNORECASE)):
childcount = len(get_child(final_path))
if childcount:
new_filebot.extend(['-non-strict', final_path])
logger.info('Filebot Processing Started...')
FB = subprocess.Popen(new_filebot,stdout=subprocess.PIPE).communicate()[0]
logger.info('Filebot Processing Ended...')
logger.debug('{}'.format(FB))
logger.info('-----------------------------------------------------------------')
The script will loop through each of the folders in my subfolder variable. but only the folders containing movies use FB. So the first stop is movies_cps
1 - File is downloaded to .../Transfer/movies_cps
2 - Files are unpacked/and separated into each their own subfolder
3 - They are checked for their audio and
- if they contain English audio only, they stay in movies_cps
- if they contain French audio only, they go .../Transfer/french
- if they contain DualAudio audio only, they go .../Transfer/dualmovies
4 - Script then removes all non-essential files (samples, nfo, etc..)
5 - I then run convert_files function which launches sickbeard_mp4_automator on movies_cps (or whichever folder folder_path is equal to)
6 - SickMp4Auto output's the file to the final_path variable which would equal to /media/sdu1/home/saitoh183/private/Transfer/manage/movies/(cps,french,dualmovies)
7 - Next I check if there is content in the final_path and if there is, Run Filebot on final_path.
Now I added logging to try and capture the error but my logging comes up empty. I added an info line to see when filebot is called and when it ends and in the log, those lines are both there but nothing in between. I also enabled logs in the filebot command and set it to ALL but nothing was generated either.
So then if I take one of the files that are currently in final_path and move them back to their original location. Then I run the script again, my function convert_files will not convert the file cuz it has already been converted and it will move it directly to final_path, then the script will move on to Filebot, and filebot will do it things and generate logs and the files will be renamed.
So I don't know when on the first pass filebot is failing and doesn't generate any sort of logs but maybe you would have an idea or maybe a way for me to see what is going on with FB on the first attempt.
Logging in with SSH and running scripts and running scripts via cron is completely different. It works via SSH because your .profile or .bashrc fix your environment, whereas these user/shell specific startup scripts are not called for cron jobs.
The file ufc.208.ppv.720p.hdtv.x264-plutonium.mp4 just happens to not be working because it's a shitty file. I'm sure you have tested with more than 1 single shitty file before you conclude that "filebot is not working" right? Some file just don't work.