What does System Information say? How much memory do you have? How much memory would you expect FileBot to use when running?
FileBot primarily works with file paths. The actual file size of each video file generally does not matter. We would expect that processing a large video file does not use significantly more or less memory than processing a small video file.