[Schd-Script] DirWatch - Multiple NZB-File Drop Support

Share your scripts or request scripts with specific features.
Forum rules
Please keep the forum clean - one topic per script. Questions not related to a specific script should be posted in Support forum.
l2g
Posts: 228
Joined: 27 Jun 2014, 22:13
Contact:

[Schd-Script] DirWatch - Multiple NZB-File Drop Support

Post by l2g » 01 Nov 2016, 01:34

Introduction
An NZBGet Scheduler script that allows you to identify additional locations you intend to place NZB-Files in outside of what is already configured. Basically NZBGet limits you to identifying one directory it should scan for NZB-Files in.

This script allows you to do the following:
  • Push NZB-Files (including compressed ones) residing in multiple directories and on multiple systems to a central NZBGet Server.
  • Optionally assign fixed categories to any NZB-Files found within a specific directory.
  • Full CLI support, so you can run all this from multiple servers in a cron to achieve the same results.
  • It can also just be a very basic file moving tool reading from a series of local directories and moving it to one central one.
The script was created in response to this topic which led to this github issue basically requesting a WatchDir feature. The idea behind the request was so that the user could have NZBGet process NZB-Files from a DropBox directory in addition to the NzbDir it already scans. Well... this is the (NZBGet) scripted solution.

This script utilizes the pynzbget I wrote which simplifies NZBGet script development.

DirWatch NZBGet Script Details
Author: Chris Caron <lead2gold@gmail.com>
Current Version: 0.5.0
Release Date: Jul 19th, 2019
License: GPLv3
Source: GitHub / Direct Download Link

Updates:
  • Jul 19th, 2019 Update (v0.5.0):
    • Directory Watch files created to manage the handling of nzbfiles that might have already been handled.
    • Small bugfixes dating back to 2017 :oops:
    • Python 3 Support Added
    • Carrying supported packages dropped and replaced with a requirements.txt file.
  • Feb 19th, 2017 Update (v0.2.0):
    • Supports category -> directory assignments. Now you can scan directories such as /Movies for NZB-Files and assign each NZB-File the 'movie' category (as a example). There is no limit to the number of directories you scan or the number of categories you wish to use.
    • Supports remote NZB-File loading from across the network. You can now run this script on any server/pc in your network and report NZB-Files directly to your central NZBGet server.
  • Nov 22nd, 2016 Update (v0.1.0):
    Now runs concurrently next to NZBGet and can poll your files every 30 seconds or more. Configuration will be incredibly easy when NZBGet v18 rolls out because it will support auto-launching things. But for now it still works great; you just need to set a schedule to have it launch.

    Once NZBGet spawns an instance from the scheduler; it will remain in memory and run on it's own cycle. Every 30 seconds, every 60 seconds; it'll run for whatever you tell it to. It will shut down when NZBGet shuts down and also when it gets a Reload signal.

    If another (NZBGet) scheduled time rolls by and tries to launch another instance, that instance will gracefully close itself off.
  • Nov 9th, 2016 Update (v0.0.3):
    Made an official release out of some additional testing and minor bug fixes.
  • Nov 1st, 2016 Update (v0.0.2):
    Minor release adding:
    • Support for Zip files; the Zip files themselves are moved if they only contain NZB-Files within them.
Installation
The following needs to be ran in order to pull the packages this script depends on into your environment:

Code: Select all

# The requirements.txt can be found included in this package
pip install -r requirements.txt
Note: This added step can appear annoying, but it only needs to be done once. It also allows this package to work with their latest versions (properly patched by those who maintain them).

How do I make it Work?
From the NZBGet settings, you can configure all of the directories you want to scan. You can also set a Poll time (no smaller then 30 seconds). This is the time the script will automatically check your specified directories for new NZB-Files that it should hand off to NZBGet.

To get the script running, you need to create a Schedule; you can do this from inside NZBGet. From the Settings -> Scheduler section, you can choose to [Add Task].

You'll need to specify the following:
  • TaskX.Time:
    • Setting for NZBGet Versions < v18: *:00,*:05,*:10,*:15,*:20,*:25,*:30,*:35,*:40,*:45,*:50,*:55
      This basically does a check every 5 minutes; This means if you restart NZBGet or reload it's configuration for whatever reason; you'll just have to wait up to 5 minutes before the script launches and takes over (using it's configured poll time).
    • Setting for NZBGet Versions >= v18: *
      A new feature allowing NZBGet to automatically start up any Scheduled service if you simply define a single asterix (*) as it's scheduled start time. So your configuration for time entries just need to be.
  • TaskX.Weekdays: 1-7
  • TaskX.Command: Script
  • TaskX.Param: DirWatch.py
After you got that part set up, you only need to tell the script of the directories (and optionally category assignments) you want to perform. You can do this from the DIRWATCH configuration option that should appear on the left hand side.
In the WatchPaths location, you can specify as many directories as you like (use a comma and space to delimit more the one entry).

You can additionally (optionally) associate a category with any directory you add by simply adding a ?category=name where name identifies the category itself. For example, you might set up a WatchPaths path such as:

Code: Select all

/path/to/NZBFiles/Movies?category=movie, /path/to/NZBFiles/Shows?category=tv, /path/to/NZBFiles/General
The above would:
  • associate the category 'movie' with all (NZB-File) content found in /path/to/NZBFile/Movies.
  • associate the category 'tv' with all (NZB-File) content found in /path/to/NZBFile/Shows.
  • just process NZB-Files found in /path/to/NZBFile/Shows as they are (no additional assignments).
The word ?category= can get lengthy when attached to lots of directories. So if you want, ?c= and ?cat= will all accomplish the same thing.

Command Line
Here is what the tool offers so far (from the command line):

Code: Select all

Usage: DirWatch.py [options] [SrcDir1 [SrcDir2 [...]]]

Options:
  -h, --help            show this help message and exit
  -t DIR, --target-dir=DIR
                        The directory you want to move found NZB-Files from
                        the identified source directories to. This option is
                        required if not using the --remote (-r) switch.
  -a AGE_IN_SEC, --min-age=AGE_IN_SEC
                        Specify the minimum age a NZB-File must be before
                        considering it for processing. This value is
                        interpreted in seconds and defaults to 30 sec(s). This
                        is just a safety switch to prevent us from creating a
                        racing condition where an NZB-File is still being
                        written to disk at the same time as we're trying to
                        process it.
  -s SIZE_IN_KB, --max-archive-size=SIZE_IN_KB
                        Specify the maximum size a detected compressed file
                        can be before ignoring it. If the found compressed
                        file is within this specified value, it's contents
                        will be scanned to see if it (only) contains NZB-
                        Files. These types of files would qualify to be moved
                        as well. Set this value to Zero (0) to not process
                        compressed files. The value is interpreted in
                        Kilobytes and has a default value of 150if not
                        otherwise specified.
  -p, --preview         This is like a test switch; the actions the script
                        would have otherwise performed are instead just
                        printed to the screen.
  -L FILE, --logfile=FILE
                        Send output to the specified logfile instead of
                        stdout.
  -u API_URL, --api-url=API_URL
                        Specify the URL of the NZB-Get API server such as:
                        nzbget://user:pass@control.nzbget.host (to access
                        insecure port 6789),
                        nzbgets://user:pass@control.nzbget.host (to access
                        secure port 6791),
                        nzbget://user:pass@control.nzbget.host:port (to
                        specify your own insecure port), and
                        nzbgets://user:pass@control.nzbget.host:port (to
                        specify your own secure port).  By default
                        nzbget://127.0.0.1 is used.
  -r, --remote-push     Perform a remote push to NZBGet. This allows you to
                        scan directories for NZB-Files on different machines
                        and still remotely push them to your central NZBGet
                        server.
  -D, --debug           Debug Mode
Most of my testing is done using the Command Line Interface I attached to it.

Code: Select all

# Scan your library for NZB-Files and print to the sreen what
# you plan on doing (observe the -p switch):
python DirWatch.py -p -t /path/to/NZBGet/NzbDir 
    /path/to/location/with/NZB-Files

# Happy with the plan of action? Just drop the -p switch and the
# matched NZB-Files will be moved:
python DirWatch.py -t /path/to/NZBGet/NzbDir 
    /path/to/location/with/NZB-Files
You can specify as many paths on the command line as you like too if you want:

Code: Select all

# Scan your libraries for NZB-Files and move them
# into: /path/to/NZBGet/NzbDir
python DirWatch.py -t /path/to/NZBGet/NzbDir \
    /home/joe/Downloads \
    /home/jason/Downloads \
    /home/trevor/Downloads \
    /home/joe/Dropbox/NZBFiles \
    /home/jason/Dropbox/NZBFiles \
    /home/trevor/Dropbox/NZBFiles \

# Don't forget to use the power of shells wildcards to accomplish the same feat:
python DirWatch.py -t /path/to/NZBGet/NzbDir \
    /home/*/Downloads /home/*/Dropbox/NZBFiles
But you can also take what is documented here and put it in a crontab too:

Code: Select all

# $> crontab -e
# Scan a directory every 2 minutes
*/2 * * * * /path/to/DirWatch.py -t /path/to/NZBGet/NzbDir ~/DropBox/NZB-Files
You can also post your content directly to NZBGet:

Code: Select all

# $> crontab -e
# Scan a directory every 2 minutes and post the content directly to nzbget
*/2 * * * * /path/to/DirWatch.py -r -u nzbget://nzbget.host.name ~/DropBox/NZB-Files
Don't forget category assignments:

Code: Select all

# $> crontab -e
# Scan a directory every 2 minutes and post the content directly to nzbget
# Also assigns everything found in the NZB-Files the category 'movie' prior
# to loading it
*/2 * * * * /path/to/DirWatch.py -r -u nzbget://nzbget.host.name ~/DropBox/NZB-Files?c=movie
Last edited by l2g on 19 Jul 2019, 22:53, edited 12 times in total.

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: [Schd-Script] DirWatch - Multiple NZB-File Drop Support

Post by hugbug » 01 Nov 2016, 11:02

Great script, thanks for your work!

One thing: on windows it's not uncommon to have spaces in directory names. For example MS OneDrive client (for cloud storage service) by default creates directory with spaces ("OneDrive - Username").

I would simply not use space as path separator but you may find another solution like surrounding paths with quotation marks, although that would require a more sophisticated parsing.

l2g
Posts: 228
Joined: 27 Jun 2014, 22:13
Contact:

Re: [Schd-Script] DirWatch - Multiple NZB-File Drop Support

Post by l2g » 01 Nov 2016, 12:12

hugbug wrote:Great script, thanks for your work!
Thanks man! :)
hugbug wrote:One thing: on windows it's not uncommon to have spaces in directory names. For example MS OneDrive client (for cloud storage service) by default creates directory with spaces ("OneDrive - Username").

I would simply not use space as path separator but you may find another solution like surrounding paths with quotation marks, although that would require a more sophisticated parsing.
I corrected the wording as you're correct; the good news is the script should already handle this. It can detect 2 separate paths with spaces if it sees "C:\Path C:\Path2" but putting a comma enforces it. I get what you're saying with the spaces though, C:\Program Files\My Dir is totally treated as a single path. Anyway, long story short; paths with spaces should work. Thanks for your feedback!

l2g
Posts: 228
Joined: 27 Jun 2014, 22:13
Contact:

Re: [Schd-Script] DirWatch - Multiple NZB-File Drop Support

Post by l2g » 10 Nov 2016, 02:59

Fully tested and working now; incremented the version to v0.0.3 and removed all pre-release notices.

Documentation updated as well.

Direct download link to v0.0.3.

Hopefully it will prove useful to someone!

l2g
Posts: 228
Joined: 27 Jun 2014, 22:13
Contact:

Re: [Schd-Script] DirWatch - Multiple NZB-File Drop Support

Post by l2g » 23 Nov 2016, 04:24

Sorry for the spam people; this is the last bump for this script as I believe it's completed now.

Updated minor to v0.1.0.

Direct download link to v0.1.0.

zombiehoffa
Posts: 8
Joined: 22 Jan 2017, 16:09

Re: [Schd-Script] DirWatch - Multiple NZB-File Drop Support

Post by zombiehoffa » 22 Jan 2017, 16:14

Hi,

Would it be possible for you to update this script so that it supports sub directories in the nzbwatch directory? bare bones it would be super awesome to be able to specify a sub directory to put all additional files this script adds into. Super mega awesome bonus feature would be to be able to specify a per additional watch directory sub directory.

i.e. bare bones awesome feature addition would be to put all nzbs this script adds in a sub folder i.e. all found nzbs into /officialnzbgetwatchdir/zombiehoffa
super mega awesome bonus feature:
watchdir1 -> /officialwatchdir/zombiehoffa
watchdir2 -> /officialwatchdir/Ronald
etc.

Alternatively, support for categories on import would be awesome, since that's basically what I'm trying to make happen by asking you to allow sub directories.

l2g
Posts: 228
Joined: 27 Jun 2014, 22:13
Contact:

Re: [Schd-Script] DirWatch - Multiple NZB-File Drop Support

Post by l2g » 23 Jan 2017, 02:51

zombiehoffa wrote: ...to put all nzbs this script adds in a sub folder i.e. all found nzbs into /officialnzbgetwatchdir/zombiehoffa
super mega awesome bonus feature:
watchdir1 -> /officialwatchdir/zombiehoffa
watchdir2 -> /officialwatchdir/Ronald
etc.
Hi; thanks for your feedback. With respect to your question, do your sub-directories change that often? I only ask, because your problem can be solved with the scripts current state... just add the directories /officialwatchdir/zombiehoffa, /officialwatchdir/Ronald to the WatchPaths. It will now scan both those directories; got more? No problem, just add them (separate the paths with a comma).
zombiehoffa wrote: Alternatively, support for categories on import would be awesome, since that's basically what I'm trying to make happen by asking you to allow sub directories.
Categories detection is already built into NZBGet; they are detected from the NZB-Files themselves (including the ones this script finds). Check out Categories in the settings.

For example, adding a category 'movies' and set the aliases to movies* is usually a pretty good start. Then add one called 'tv' and set the aliases to hdtv, tv*, s??e??. Now you're already handling to types of categories already. You can adapt from that and add as many categories as you want. It's the aliases section that does all the work for you here.

zombiehoffa
Posts: 8
Joined: 22 Jan 2017, 16:09

Re: [Schd-Script] DirWatch - Multiple NZB-File Drop Support

Post by zombiehoffa » 24 Jan 2017, 01:50

l2g wrote:
zombiehoffa wrote: ...to put all nzbs this script adds in a sub folder i.e. all found nzbs into /officialnzbgetwatchdir/zombiehoffa
super mega awesome bonus feature:
watchdir1 -> /officialwatchdir/zombiehoffa
watchdir2 -> /officialwatchdir/Ronald
etc.
Hi; thanks for your feedback. With respect to your question, do your sub-directories change that often? I only ask, because your problem can be solved with the scripts current state... just add the directories /officialwatchdir/zombiehoffa, /officialwatchdir/Ronald to the WatchPaths. It will now scan both those directories; got more? No problem, just add them (separate the paths with a comma).
zombiehoffa wrote: Alternatively, support for categories on import would be awesome, since that's basically what I'm trying to make happen by asking you to allow sub directories.
Categories detection is already built into NZBGet; they are detected from the NZB-Files themselves (including the ones this script finds). Check out Categories in the settings.

For example, adding a category 'movies' and set the aliases to movies* is usually a pretty good start. Then add one called 'tv' and set the aliases to hdtv, tv*, s??e??. Now you're already handling to types of categories already. You can adapt from that and add as many categories as you want. It's the aliases section that does all the work for you here.

Hi,

that's not quite it. If want to take nzbs from outside the officialwatchdir and instead of putting them in the officialwatchdir, I want to put them in officialwatchdir/someotherdir specified per external directory. nzbget automatically interprets subfolders in the officialwatchdir as a category and adds nzbs in subdirs of the official watchdir to that category. I have custom categories so the nzb category in the nzb isn't useful to me.

l2g
Posts: 228
Joined: 27 Jun 2014, 22:13
Contact:

Re: [Schd-Script] DirWatch - Multiple NZB-File Drop Support

Post by l2g » 25 Jan 2017, 01:09

zombiehoffa wrote: that's not quite it. If want to take nzbs from outside the officialwatchdir and instead of putting them in the officialwatchdir, I want to put them in officialwatchdir/someotherdir specified per external directory. nzbget automatically interprets subfolders in the officialwatchdir as a category and adds nzbs in subdirs of the official watchdir to that category. I have custom categories so the nzb category in the nzb isn't useful to me.
I think I'm following what you're asking; I spoke to HugBug about your request and he gave me some great advice (as he always does). Since I already have a WatchDir that works pretty well (comma separated directories), here is what I propose:

How about I add a CategoryWatchDir?

But here's the catch; I'm not going to look for sub-directories within the directories you specify here. It will be your job to specify all of the directories. So... consider the following (all directories identified by their category):
  • /my/path/to/my/nzbs/customcategory01
  • /my/path/to/my/nzbs/customcategory02
  • /my/path/to/my/nzbs/customcategory03
You'll be required to specify the CategoryWatchDir (comma separated) path as: /my/path/to/my/nzbs/customcategory01, /my/path/to/my/nzbs/customcategory02, /my/path/to/my/nzbs/customcategory03

I will take the last directory name in each of these paths (if they exist) and treat them as a category name before handling any NZB-File i find within it. So with respect to the above example, every nzbfile (or zip of nzbfile) in /my/path/to/my/nzbs/customcategory01 will be treated as being a part of the category 'customcategory01'. Would that work for you?
Last edited by l2g on 25 Jan 2017, 01:53, edited 1 time in total.

zombiehoffa
Posts: 8
Joined: 22 Jan 2017, 16:09

Re: [Schd-Script] DirWatch - Multiple NZB-File Drop Support

Post by zombiehoffa » 25 Jan 2017, 01:37

That would be fabulous as long as I can specify an unlimited number of dirs per category (i.e. as long as I am not limited to one dir, one category)

Post Reply

Who is online

Users browsing this forum: No registered users and 34 guests