[SQS-Script] Completion-Propagation/DMCA/Retention check

Share your scripts or request scripts with specific features.
Forum rules
Please keep the forum clean - one topic per script. Questions not related to a specific script should be posted in Support forum.
hugbug
Developer & Admin
Posts: 7541
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: [SQS-Script] Completion - Post completion check script

Post by hugbug » 28 Jan 2015, 21:26

Does the DUPE check happen before or after the NZB_ADDED?
The dupe check is performed after the nzb is processed by scan-scripts before adding it to queue. Dupes are moved to history without executing of queue-scripts.
I thought of a workaround for the above
I'll add a new env var for scripts, something like "NZBNA_QUEUEDFILE" which will point directly to the nzb-file (.nzb.X.queued). For older NZBGet version you will need the workaround.

kloaknet
Posts: 335
Joined: 23 Jul 2014, 08:52

Re: [SQS-Script] Completion - Post completion check script

Post by kloaknet » 29 Jan 2015, 15:04

I'll add a new env var for scripts, something like "NZBNA_QUEUEDFILE" which will point directly to the nzb-file (.nzb.X.queued). For older NZBGet version you will need the workaround.
Cool, then Ill write the NZBID and NZBNA_QUEUEDFILE to a log file, so that I can retrieve the correct file name when the file check etc is called from the scan, scheduler or queue script. (because the just added file is not always directly checked on completion, only the oldest highest prio file within X hours in the queue will be checked and eventually resumed first, the others remain paused), Thanks!

kloaknet
Posts: 335
Joined: 23 Jul 2014, 08:52

Re: [SQS-Script] Completion - Post completion check script

Post by kloaknet » 31 Jan 2015, 11:56

Note that I edited this post within an hour after posting.

@hugbug
I'll add a new env var for scripts, something like "NZBNA_QUEUEDFILE" which will point directly to the nzb-file (.nzb.X.queued). For older NZBGet version you will need the workaround.
Today I were trying to figure out / implement the future option, and noticed that you wanted to add it as a queue (NZBNA) call instead of a scan (NZBNP) call. When it would be available only when a file is added in the queue, there would not be an option to store/get the full nzb extension for DUPEs, because DUPEs won't get called via a queue call. (the script needs to check the DUPEs before they are moved back into the queue: it could be that the nzb in the queue is propagating, as well might are the DUPEs in the history, no need to rotate the file from and to the history all the time during bad propagation for like hours).

I also encountered the following issue with my scripts and thoughts: When the scan script call happens there is no NZBID yet, and the files are also not queued, so an option as NZBNP_QUEUEDFILE would not be possible.
This makes a thought of me not possible, I cannot log the items the script pauses via the scan call, because I cannot ID them (although the NZBNP_NZBNAME is given, there could be similar ones, with the same NZBNP_NZBNAME).

A little shorter:
- How to know each QUEUEDFILE full extension for each NZBID?
- maybe easiest for you hugbug to add the extension in 'history' and 'listgroups' option. Too keep data to minimum, limit it to full extension (.nzb.X.queued) only as separate item ('NZBExtension') or maybe add the .X.queued to 'NZBFilename', as that is the actual file name on the disk.
- Is there a way to know which nzb (NZBID) the script paused in the scan call?

sorry for all these questions.

hugbug
Developer & Admin
Posts: 7541
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: [SQS-Script] Completion - Post completion check script

Post by hugbug » 31 Jan 2015, 21:31

It's possible to store arbitrary data per nzb using pp-parameters. To store a string:

Code: Select all

print('[NZB] NZBPR_myvar=my value')
To read the data later (possibly in other script):

Code: Select all

myvar = os.environ.get('NZBPR_myvar', '')
Using this feature the scan script can store the timestamp and the size of nzb-file in the nzb pp-parameters. Later this info can be used to better identify queued-files. Then you don't need NZBNP_QUEUEDFILE and the script will work with all NZBGet versions.
kloaknet wrote:This makes a thought of me not possible, I cannot log the items the script pauses via the scan call, because I cannot ID them (although the NZBNP_NZBNAME is given, there could be similar ones, with the same NZBNP_NZBNAME).
Pp-parameters should help here too.
kloaknet wrote:- Is there a way to know which nzb (NZBID) the script paused in the scan call?
Here too.
kloaknet wrote:- How to know each QUEUEDFILE full extension for each NZBID?
- maybe easiest for you hugbug to add the extension in 'history' and 'listgroups' option. Too keep data to minimum, limit it to full extension (.nzb.X.queued) only as separate item ('NZBExtension') or maybe add the .X.queued to 'NZBFilename', as that is the actual file name on the disk.
As described above. Alternatively your scan-script could make a copy of nzb-file under unique name (in a specific directory) and store the name of the file as pp-parameter.

General hint: When working with pp-parameters use a unique string as prefix for all parameter names to avoid conflicts with other scripts.

kloaknet
Posts: 335
Joined: 23 Jul 2014, 08:52

Re: [SQS-Script] Completion - Post completion check script

Post by kloaknet » 01 Feb 2015, 12:42

thanks for the tips. I will try to figure it out with the NZBPR trick.

Searching the forums and the change log gives me the impression that the NZBPR_ function for recalling data does not work for scheduler scripts. Also tried if it maybe did (v14.0) by using:

Code: Select all

os.environ['NZBPR_addtime, NZBID]
but no luck.

I think I will manage by using the (NZBPR_)addtime in the output from the 'listgroups' and 'history' instead.

Just noted that also besides the .nzb.X.queued is not shown, the _2 is also taken from the NZB_names (for 2 different NZB with identical name added at the same time). Only place where it is available is in the NZBNP_FILENAME. (in the 'listgroups' and 'history', the NZBfilename is also without the _2), so will have to store that one too, because the _2 is kept in the actual name, and the nzb names will be '.nzb.queued' and '_2.nzb.queued'


btw, got the SSL option working, will post that version when the above is working too.

hugbug
Developer & Admin
Posts: 7541
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: [SQS-Script] Completion - Post completion check script

Post by hugbug » 01 Feb 2015, 13:02

Pp-parameters are per-nzb. Of course they are not passed to scheduler-scripts, since they are called without nzb context.
Yes, you can extract pp-parameters from "listgroups" or "history", when the pp-parameters are needed in the scheduler-script. In the queue-script it's easier to use "NZBPR_" (and saves extra RPC-calls).

To set pp-parameters from scheduler-script use RPC-method
editqueue with Command set to "GroupSetParameter" or "HistorySetParameter".
kloaknet wrote:ust noted that also besides the .nzb.X.queued is not shown, the _2 is also taken from the NZB_names (for 2 different NZB with identical name added at the same time). Only place where it is available is in the NZBNP_FILENAME.
Don't use NZBName, that's a user friendly display name, and it can also be changed by user.

kloaknet
Posts: 335
Joined: 23 Jul 2014, 08:52

Re: [SQS-Script] Completion - Post completion check script

Post by kloaknet » 08 Feb 2015, 09:23

Updated the script to 0.1.1 See first post http://nzbget.net/forum/viewtopic.php?f=8&t=1736 for the download and new info.


It now includes SSL/TLS support. It does not check the certificate, and I blocked the use of SSLv2 and SSLv3, in line with the python help: https://docs.python.org/2/library/ssl.h ... OCOL_SSLv2

Code: Select all

context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
context.verify_mode = ssl.CERT_NONE
context.options |= ssl.OP_NO_SSLv2
context.options |= ssl.OP_NO_SSLv3
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
 sockets[i] = context.wrap_socket(s)

Predicting the actual queued file based on testing how NZBget responds to new items with identical file names, added the var NZBPR_CnpNZBFileName. Had some issues extracting the data from the 'listgroups' (or 'history') via JSON etc later on, from the the NZB 'Parameters' but converted the list in 'Parameters' to a string and manually extracted the queued file name from it, maybe there is a easier way than this?:

Code: Select all

 # extracting filename from job the hard way
sj = str(job['Parameters'])
loc = sj.rfind("u'CnpNZBFileName', u'Value': u'")
nzb_filename = sj[loc+31:-3]

kloaknet
Posts: 335
Joined: 23 Jul 2014, 08:52

Re: [SQS-Script] Completion - Propagation/DMCA/Retention che

Post by kloaknet » 22 Feb 2015, 18:15

Updated the script to 0.1.2 See first post http://nzbget.net/forum/viewtopic.php?f=8&t=1736 for the download and new info.

The script is now also able to check DUPEs when the item in the queue is not yet older than X hours and incomplete. Mainly useful for RSS feed users, because as far as I know automation programs like Sonarr only push 1 item at a time (DUPEs are not likely to occur).

sumofatguy
Posts: 10
Joined: 14 Mar 2015, 20:27

Re: [SQS-Script] Completion - Propagation/DMCA/Retention che

Post by sumofatguy » 24 Mar 2015, 22:39

I've played around with this script a bit, and I'm just curious if it only works with one (main) server, or if it has the ability to check your backup block servers if a file is missing on the main. Thanks!

kloaknet
Posts: 335
Joined: 23 Jul 2014, 08:52

Re: [SQS-Script] Completion - Propagation/DMCA/Retention che

Post by kloaknet » 25 Mar 2015, 15:33

That option is not implemented in the current code, (I don't have backup accounts :oops: ), but it is must be possible that I create an option to check the failing articles on your backup server.

The quick and dirty method would be to just check all articles also for you backup server if more than X% fails <- rather easy to implement.
But I think it would be better to specifically check for the failed articles only at your back up provider, but implementing this will take some more time, and expect the feature in a month. (Atm I don't have much free time).

Will it be enough to just add the option for 1 back-up provider, or is it necessary to add more? Please note that when you want to use a back-up provider, the amount of articles to check should be higher, otherwise you could easily start downloading, while not enough articles are available.

Currently I have worked out the option to force a failure (instead of marking BAD), and force a 100% check on items without pars. I will first release that version somewhere next week.

Post Reply

Who is online

Users browsing this forum: No registered users and 3 guests