[Scan-Script] TidyIt - Media Library House Cleaning

Share your scripts or request scripts with specific features.
Forum rules
Please keep the forum clean - one topic per script. Questions not related to a specific script should be posted in Support forum.
l2g
Posts: 228
Joined: 27 Jun 2014, 22:13
Contact:

[Scan-Script] TidyIt - Media Library House Cleaning

Post by l2g » 18 May 2015, 20:53

TidyIt Scheduler Script for NZBGet
TidyIt just performs basic house cleaning on your library by eliminating garbage files and meta data that have accumulated over time.

For example:
If you delete a video file through a third party application such as KODI (XBMC) or Plex, you'll notice that it leaves the old directories in place. It also leave lingering meta information behind such as poster info, subtitles, etc. This script is intended to just scan your library and eliminate any lingering content that is no longer being referenced or used.

TidyIt It operates in 3 modes:
  • Preview: a Log Only mode which only prints what it would have done. By default, this is the mode the script always operates in unless you explicitly specify otherwise. This is a great mode for just surveying your media library from afar (through log files) without doing anything to it!
  • Delete: The Delete mode actually does the removal of the data flagged to be handled. You might want to choose this option once you're satisfied with it's cleaning algorithm (seen from the Preview mode's output). Alternatively you can check out the Move mode (identified next) which is a little less of an extreme approach (in comparison to this option).
  • Move: Similar to the Delete mode (explained above) except content is moved to a directory of your choice instead of being removed. This gives you the ability to at least survey what the script considered to be no good and make a final decision for yourself.
TidyIt utilizes the pynzbget I wrote which simplifies NZBGet script development.

TidyIt NZBGet Script Details
Author: Chris Caron <lead2gold@gmail.com>
Current Version: 0.8.0
Release Date: Jul 19th, 2019
License: GPLv3
Source: GitHub / Direct Download Link

b]Updates[/b]:
  • Jul 19th, 2019 Update (v0.8.0):
    • Small bugfixes dating back to 2015 :oops:
    • Python 3 Support Added
    • Carrying supported packages dropped and replaced with a requirements.txt file.
  • Sep 25th, 2015 (v0.4.0):
    • Original Build
Installation
The following needs to be ran in order to pull the packages this script depends on into your environment:

Code: Select all

# The requirements.txt can be found included in this package
pip install -r requirements.txt
Note: This added step can appear annoying, but it only needs to be done once. It also allows this package to work with their latest versions (properly patched by those who maintain them).

Since this script operates as a Scheduling Plugin, you'll need to go into the Scheduler settings of NZBGet and add this script there in order for it to be ran. Schedule it to run whenever you want and remember that it runs in a Preview mode out of the box.

Features:
  • Can scan more then one video library and recursively scans them all for videos and their containing folders.
  • It supports safe-entries to over-ride the entire scripts functionality. If a directory or file is found that matches what you've defined as a safe-entry, then it will effectively cause the script to automatically ignore it (including subdirectories within) and all of it's contents.
  • It can detect and remove zero byte files, empty directories since they're just wasting space anyway.
  • You can optionally identify content you want to always remove if detected such as '*-sample.mkv'. By default there is nothing defined in this section.
  • Eliminates unused/orphan subtitles and other meta information that have been downloaded (and reside in the same directory as other video files). It's smart enough to ignore metadata that is already associated with a video file.
  • The script ignores any directories or files that is less then 1 hour old. This is intentional (but configurable) setting used to prevent cases where another processes is managing the same library.
    * A CLI (Command Line Interface) allows you to the TidyIt.py manually if you like from the command line now. Run it with --help (-h) to see some options if you like. A simple command might be:
    • * TidyIt.py /a/path/to/your/library /another/path/to/another/library which simply scans the libraries and only prints what it would like to do. It will never actually preform any action (so don't worry about screwing anything up here!). This is equivalent to NZBGet's Disabled mode TidyIt offers.
      * TidyIt.py -c /a/path/to/your/library /another/path/to/another/library is much more harsh. The --clean (-c) switch invokes the Delete mode. Anything marked for handling is deleted. Make sure you're comfortable with your settings before choosing this option.
      * TidyIt.py -p /path/to/temporary/trash /a/path/to/your/library /another/path/to/another/library is a little more laid back. The --move-path (-p) switch invokes the Move mode. Anything marked for handling is moved into the path you specify instead. If the path doesn't exist, then it'll be created.
Caution: When using the Delete mode, I must stress that once content has been removed, it can't be restored unless you have a backup elsewhere. I will not be held liable for any irrecoverable data caused by this script. This caution is only necessary for those who stray away from the default settings. Otherwise, there is enough bulletproofing in place that will prevent any catastrophic from happening.

CLI Help Menu

Code: Select all

Usage: TidyIt.py [options] [scandir1 [scandir2 [...]]]

Options:
  -h, --help            show this help message and exit
  -n ENCODING, --encoding=ENCODING
                        The system encoding to use (utf-8, ISO-8859-1, etc).
                        The default value is 'UTF-8'.
  -s ENTRIES, --safe-entries=ENTRIES
                        If a safe-entry file/dir is located within a path
                        scanned then the path is ignored. Use safe-entry files
                        (or dirs) to intentionally ignore directories of your
                        choice that reside in your video library. You can
                        specify more then one safe-entry by separating them
                        with a comma (,). The default value(s) are
                        '.tidysafe'.
  -t ENTRIES, --always-trash=ENTRIES
                        Identify any file extensions you wish to always trash
                        if matched. By default this is not set.You can specify
                        more then one trash entry by separating each of them
                        with a comma (,).
  -M ENTRIES, --meta-content=ENTRIES
                        Identify any files and/or directories that should be
                        treated as meta content. Meta content is only handled
                        if it's the last thing within a media directory.You
                        can specify more then one meta entry by separating
                        each of them with a comma (,).  By Default the
                        following are already defined: 'Thumbs.db', '@eaDir',
                        '.DS_Store', '.AppleDouble', '__MACOSX'.
  -m SIZE_IN_MB, --video-minsize=SIZE_IN_MB
                        Specify the minimum size a video must be before it's
                        treated as part of your collection. This value is used
                        to diffentiate between video file and samples files.
                        This value is interpreted in MB (Megabytes) and
                        defaults to 150 MB.
  -a AGE_IN_SEC, --min-age=AGE_IN_SEC
                        Specify the minimum age a directory and/or file must
                        be before considering it for processing. This value is
                        interpreted in seconds and defaults to 3600 sec(s).
  -c, --clean           Unless this switch is specified, this script only runs
                        in a log only mode (a dry-run) allowing you to see the
                        actions the script would have otherwise performed.
                        This switch can be combined with the --move-path (-p)
                        switch to move handled instead.
  -p PATH, --move-path=PATH
                        Identifiy the path to place content into instead of
                        removing it.  By specifying a --move-path, the --clean
                        (-c) switch is implied however handled content is
                        moved instead of being removed.
  -L FILE, --logfile=FILE
                        Send output to the specified logfile instead of
                        stdout.
  -D, --debug           Debug Mode
Last edited by l2g on 19 Jul 2019, 23:13, edited 17 times in total.

kloaknet
Posts: 337
Joined: 23 Jul 2014, 08:52

Re: [Scan-Script] TidyIt - Media Library House Cleaning

Post by kloaknet » 19 May 2015, 14:27

you're keeping yourself busy :-), but i noticed that the GitHub page link doesnt work atm ( https://github.com/caronc/nzbget-tidy ). Got no Plex yet, so nothing else to report :oops:

l2g
Posts: 228
Joined: 27 Jun 2014, 22:13
Contact:

Re: [Scan-Script] TidyIt - Media Library House Cleaning

Post by l2g » 19 May 2015, 17:13

Doh, you're right! I corrected the link ; it should have read: https://github.com/caronc/nzbget-tidyit instead.

Thanks Kloaknet :)

CBers
Posts: 140
Joined: 24 Aug 2014, 11:35

Re: [Scan-Script] TidyIt - Media Library House Cleaning

Post by CBers » 20 May 2015, 09:20

@l2g - it might be worth posting this on the Emby (MediaBrowser) forums.

There's a large user-base over there that might be interested in your offering.

l2g
Posts: 228
Joined: 27 Jun 2014, 22:13
Contact:

Re: [Scan-Script] TidyIt - Media Library House Cleaning

Post by l2g » 24 May 2015, 16:24

Official release working: Direct link to v0.0.2.

I rewrote a massive amount of it to preform some additional smarts into the handling of lingering meta information within the meta directories themselves. I still ask that anyone using this script run it in it's (Disabled) mode first just to see what it intends to do. If you agree with it's decision making, then it's safe to turn it over to the Enabled mode. I added a lot of logging into this release too; If you turn on Debuging mode (or -D (--debug) in the CLI), it will actually tell you why it wants to remove the file/dir.. Reasons such as:
  • A zero byte file
  • A file within a directory that has no associated videos at all
  • A file is meta data with no associated video to work with it.
Alternatively with debug mode on, you'll get reasoning as to why something isn't being removed (that you might have expected to the script to handle) such as:
  • A safe file was found (causing it to ignore the directory being scanned)
  • A file was found that was not identified as an 'extra', this is kind of like a safe file. The script won't remove extensions you didn't tell it about (otherwise it may be making a dangerous mistake by removing it).
  • The file could have been a meta file and it's associated movie it matches against was found (therefore causing the file not to be erased).
--
@CBers: I'm not familiar with that app, but upon visiting the site, I see that your a moderator of it. You're more then welcome to share the script with your community if you think it would be useful to them. At this time i don't' really want to maintain 2 forums every time I do an update :). Thanks for sharing your site all the same!

l2g
Posts: 228
Joined: 27 Jun 2014, 22:13
Contact:

Re: [Scan-Script] TidyIt - Media Library House Cleaning

Post by l2g » 31 May 2015, 01:59

Release v0.0.3 (Direct Download Link) in place.

The release details can be found at the head of this thread (quick link).

moddigi1
Posts: 39
Joined: 14 Apr 2013, 11:44

Re: [Scan-Script] TidyIt - Media Library House Cleaning

Post by moddigi1 » 26 Jul 2015, 21:39

Love this script, thanks for your efforts :)

kloaknet
Posts: 337
Joined: 23 Jul 2014, 08:52

Re: [Scan-Script] TidyIt - Media Library House Cleaning

Post by kloaknet » 11 Aug 2015, 17:45

Hi again!

got some questions, running it currently in test mode. The script gave me these comments:

Code: Select all

Potential Execution (os meta data): ...\Thumbs.db
and

Code: Select all

Unhandled entry found: ..\file.rar
what do those mean?

I guess those are informative only (verbose on btw)

and only these comments are related to actual removing?

Code: Select all

PREVIEW ONLY: Remove DIRECTORY: folder dir

Might it be an idea to move the deleted stuff to the recycle bin as option? In that way you even has an extra level of security when running the script, but arent sure yet if you configured it right?

l2g
Posts: 228
Joined: 27 Jun 2014, 22:13
Contact:

Re: [Scan-Script] TidyIt - Media Library House Cleaning

Post by l2g » 11 Aug 2015, 19:55

kloaknet wrote:

Code: Select all

Potential Execution (os meta data): ...\Thumbs.db
Basically a file was found categorized as meta data. It's flagged to be deleted, but only in situations where no video (media) files were found that correspond to the same directory it was found in.
kloaknet wrote:

Code: Select all

Unhandled entry found: ..\file.rar
In these cases, it found a file in a directory that has not been flagged as meta data (for potential removal) or important data (such as .mkv, etc). It also didn't fit in a 3rd category that handles ignorable filetypes (kind of like safe markers). Unhandled files basically cause the tidyit script to avoid scanning further (in this directory tree) and will move to the next.
kloaknet wrote: I guess those are informative only (verbose on btw)
Exactly; it's more for debugging. You know for situations where you might disagree or not understand why the script wants to eliminate a dir/file/subtitle, etc. You can get justification as to where it got the thought process from with the verbose/debug mode on. :)
kloaknet wrote: and only these comments are related to actual removing?

Code: Select all

PREVIEW ONLY: Remove DIRECTORY: folder dir
That's right, if you're running from the command line without the -c (or Disabled Mode from NZBGet), then it only prints what it would have done. It won't do anything damaging.
kloaknet wrote: Might it be an idea to move the deleted stuff to the recycle bin as option? In that way you even has an extra level of security when running the script, but arent sure yet if you configured it right?
This is a really awesome idea! I love it!
What i might do instead though is just allow you to optionally specify a directory you want to move to-be-deleted content to (because not all OS's have a recycle bin). But it's still an great idea!

Edit: I pushed a few changes to the master branch that will make the logging a little more easier to read when on verbose mode.

kloaknet
Posts: 337
Joined: 23 Jul 2014, 08:52

Re: [Scan-Script] TidyIt - Media Library House Cleaning

Post by kloaknet » 12 Aug 2015, 16:39

l2g wrote:
kloaknet wrote:

Code: Select all

Unhandled entry found: ..\file.rar
In these cases, it found a file in a directory that has not been flagged as meta data (for potential removal) or important data (such as .mkv, etc). It also didn't fit in a 3rd category that handles ignorable filetypes (kind of like safe markers). Unhandled files basically cause the tidyit script to avoid scanning further (in this directory tree) and will move to the next.
the above example is related to a movie with a .rar file with a .sub file included. Its not used because subliminal ;) is way easier. But from the above I understand that such a folder won't be removed, even when the movie is deleted? Its a difficult issue to cope with I guess, because how do you know its not part of a bigger archive? Maybe check the file contents?

Ill just hit subscribe to this topic too, so Ill test the option for file deletion. Maybe you can add an option to specify the directory where to send the crap files (preferably with windows option to just dump them into the bin, because I am lazy :oops: )

Post Reply

Who is online

Users browsing this forum: No registered users and 37 guests