[SQS-Script] Completion-Propagation/DMCA/Retention check

Share your scripts or request scripts with specific features.
Forum rules
Please keep the forum clean - one topic per script. Questions not related to a specific script should be posted in Support forum.
Post Reply
kloaknet
Posts: 308
Joined: 23 Jul 2014, 08:52

[SQS-Script] Completion-Propagation/DMCA/Retention check

Post by kloaknet » 15 Jan 2015, 18:14

Latest version:
Completion 1.1.0
Completion110.zip
completion 1.1.0.
(18.21 KiB) Downloaded 84 times
Notes:
- To run the script you currently need NZBGet version 15.0 or higher (18+ advised), and python 2.7.9+ (does not work for python 3). For versions of NZBGet 18+ configuring the script is way more simple.
- To get a good insight of why the script does certain things you might not expect, please turn on 'Verbose' logging on the script settings page. This most likely will show you all you need in the NZBGet logs.

General:
NZBGet ‘extension script’ that checks if the data in the NZB file is sufficiently complete at your usenet provider(s), before starting the download. If incomplete it would wait for a certain period and check the completion of the NZB file again. This check is done by requesting the header status, and is in normal cases done within seconds (like 1 - 5 sec. for a 1 GB file). This method is significantly faster than when NZBGet would report a failure, after actual downloading a (part of) the files that end up incomplete. The script is typically useful for issues related to:
- very recent posts,
- failed downloads, which after a while are just ok (propagation issues),
- incomplete posts,
- taken down posts (DMCA, etc.),
- old posts,
- long par repair times,
- downloading (parts of) NZB files beyond repair,
- unnecessary use of expensive block / slow fill accounts.

The above would generally result in an (error) messages like ‘missing articles’, ‘unable to repair’ or ‘additional par files required’, ‘not enough par-blocs’, etc. The script avoids these messages.

How to install, NZBGet 18+:
- put the script in your script folder of NZBGet (ScriptDir under PATHS in Settings),
- go to Settings -> EXTENSION SCRIPTS, hit ‘Choose’ at the ‘Extensions’ option, and thick Completion.py,
- if u use category mapping, also go to Settings -> CATEGORIES, and select the script for CategoryX.Extension too.
- optionally configure the settings under Settings -> COMPLETION, although the default settings would normally suffice.
- optionally remove all propagation delays in NZBGet, your indexer, and/or Sonarr and the likes, as this script handles it all.

How to install, pre NZBGet 18:
- put the script in your script folder of NZBget (ScriptDir under PATHS in Settings),
- optionally configure the settings under Settings -> COMPLETION, although the default settings would normally suffice.
- enable the script as:
- SCAN SCRIPT, via Settings -> EXTENSION SCRIPTS
- QUEUE SCRIPT, via Settings -> EXTENSION SCRIPTS
- SCHEDULER SCRIPT, via Settings -> SCHEDULER, enable it for every X minutes (*:00,*:15, *:30,*:45), everyday (1-7), and tick ‘Script’. Checking each 15 minutes should be sufficient.
- if u use category mapping, also go to Settings -> CATEGORIES, and select the script for CategoryX.Extension too.
- optionally remove all propagation delays in NZBGet, your indexer, and/or Sonarr and the likes, as this script handles it all.

Make sure your PATHS settings are correct, only use \ for windows, / for unix. Don't use trailing slashes either.

What does it do:
- scans all incoming NZBs, turns them PAUSED (optional for typical categories, e.g. for your RSS feeds categories or TV shows).
- uses all in NZBGet specified usenet providers, (optional for selected servers as main servers and fill servers).
- when the scheduler is triggered, or when an NZB is added, downloaded, deleted or marked, a check is started.
- on each, check, the oldest, highest priority item in the queue will be checked, if OK: Resume NZB, if not OK: check for possible DUPEs in history or check next paused NZB in queue.
- if no queued / downloading NZB (with equal or lower priority than by the script paused NZBs) is in the queue, and NZBGet is not paused, Y% of the header statuses of by the script paused NZB will be checked (to see if the actual files are available at the usenet provider(s))
- if not OK after X hours, the file will be moved to history, and marked as BAD / FAILED, so new NZBs (e.g. RSS) will be downloaded, or programs like Sonarr will be informed, so they can send an alternative NZB file.
- when the check fails, it is possible to let the script check the history for possible working duplicate NZB files. This creates the option to download a working NZB that is not actually in the queue, but listed as DUPE in the history. (via CheckDupes option, DUPEs are generally a result of using RSS feeds).
- the script will download the highest priority, oldest item in the queue first. Priority goes above age. (optionally NZBs within X hours can be 'prioritized' over files older than X hours).
- check the settings page of the script to get more details on all the script options.
- The NZBget log file, with the Verbose option turned on, will look like this:

Code: Select all

Sun May 14 07:25:25 2017	INFO	nzbget 19.0-testing-r1991 service-mode
Sun May 14 07:25:25 2017	INFO	Scheduler: unpausing download
Sun May 14 07:25:25 2017	INFO	Executing scheduler-script Completion.py
Sun May 14 07:25:26 2017	INFO	Completion: [V] scheduler_call()
Sun May 14 07:25:26 2017	INFO	Completion: [V] Empty queue
Sun May 14 07:26:25 2017	INFO	Queue Some.NZB-FiLE @ inde.xer
Sun May 14 07:26:26 2017	INFO	Executing scan-script Completion.py for Some.NZB-FiLE.nzb
Sun May 14 07:26:27 2017	INFO	Completion: [V] scan_call()
Sun May 14 07:26:27 2017	INFO	Completion: [V] Expected queued file name: "Some.NZB-FiLE.nzb.queued"
Sun May 14 07:26:27 2017	INFO	Completion: [V] Pausing: "Some.NZB-FiLE.nzb"
Sun May 14 07:26:27 2017	INFO	Adding collection Some.NZB-FiLE.nzb to queue
Sun May 14 07:26:27 2017	INFO	Collection Some.NZB-FiLE added to queue
Sun May 14 07:26:27 2017	INFO	Executing queue-script Completion.py for Some.NZB-FiLE
Sun May 14 07:26:28 2017	INFO	Completion: [V] queue_call()
Sun May 14 07:26:28 2017	INFO	Completion: [V] lock_file()
Sun May 14 07:26:28 2017	INFO	Completion: [V] server_time= 1494739588
Sun May 14 07:26:28 2017	INFO	Completion: [V] New completion.lock file created.
Sun May 14 07:26:28 2017	INFO	Completion: [V] nzbget_paused()
Sun May 14 07:26:28 2017	INFO	Completion: [V] Waiting for NZBGet to end downloading
Sun May 14 07:26:28 2017	INFO	Completion: [V] Downloading for NZBGet paused
Sun May 14 07:26:28 2017	INFO	Completion: [V] Paused UNSORTED NZBs in queue that will be processed:
Sun May 14 07:26:28 2017	INFO	Completion: [V] * Some.NZB-FiLE.nzb.queued, Age: 10.1 hours, Priority: 0
Sun May 14 07:26:28 2017	INFO	Completion: [V] Ignoring sorting priority of items older than AgeLimit of 4 hours
Sun May 14 07:26:28 2017	INFO	Completion: [V] Paused and SORTED NZBs in queue that will be processed:
Sun May 14 07:26:28 2017	INFO	Completion: [V] * Some.NZB-FiLE.nzb.queued, Age: 10.1 hours, Priority: 0
Sun May 14 07:26:28 2017	INFO	Completion: [V] get_nzb_status(nzb=[213, 'Some.NZB-FiLE.nzb.queued', 1494703365, 886, 'NONE', 0])
Sun May 14 07:26:28 2017	INFO	Completion: Checking: "Some.NZB-FiLE.nzb.queued"
Sun May 14 07:26:28 2017	INFO	Completion: [V] get_nzb_data(fname=C:\ProgramData\NZBGet\nzb\Some.NZB-FiLE.nzb.queued)
Sun May 14 07:26:28 2017	INFO	Completion: [V] Amount of to be checked articles increased to about 100 articles.
Sun May 14 07:26:28 2017	INFO	Completion: [V] NZB contains 414 articles, 363 rar articles, 51 par2 articles.
Sun May 14 07:26:28 2017	INFO	Completion: [V] 121 rar articles will be checked.
Sun May 14 07:26:28 2017	INFO	Completion: Maximum failed articles limit for NZB: 11.4%
Sun May 14 07:26:28 2017	INFO	Completion: [V] get_server_settings(nzb_age=1494703365)
Sun May 14 07:26:28 2017	INFO	Completion: [V] All news servers after filtering on Active, Servers, FillServers + AgeLimit and Retention,  BEFORE filtering on NZBGet ServerX.Group:
Sun May 14 07:26:28 2017	INFO	Completion: [V] * news.nzbget.net:563, SSL: True, connections: 6
Sun May 14 07:26:28 2017	INFO	Completion: [V] * free.nzbget.net:119, SSL: False, connections: 3
Sun May 14 07:26:28 2017	INFO	Completion: [V] * ssl.getnzb.com:443, SSL: True, connections: 1
Sun May 14 07:26:28 2017	INFO	Completion: [V] All active news servers AFTER filtering and sorting on NZBGet ServerX.Group:
Sun May 14 07:26:28 2017	INFO	Completion: [V] * news.nzbget.net:563, SSL: True, connections: 6
Sun May 14 07:26:28 2017	INFO	Completion: [V] * free.nzbget.net:119, SSL: False, connections: 3
Sun May 14 07:26:28 2017	INFO	Completion: [V] * ssl.getnzb.com:443, SSL: True, connections: 1
Sun May 14 07:26:28 2017	INFO	Completion: Using server: news.nzbget.net
Sun May 14 07:26:28 2017	INFO	Completion: [V] Creating sockets for server: news.nzbget.net
Sun May 14 07:26:28 2017	INFO	Completion: [V] Using IPv4 for news.nzbget.net
Sun May 14 07:26:28 2017	INFO	Completion: [V] Socket 0 created.
Sun May 14 07:26:28 2017	INFO	Completion: [V] Socket 1 created.
Sun May 14 07:26:28 2017	INFO	Completion: [V] Socket 2 created.
Sun May 14 07:26:28 2017	INFO	Completion: [V] Socket 3 created.
Sun May 14 07:26:29 2017	INFO	Completion: [V] Socket 4 created.
Sun May 14 07:26:29 2017	INFO	Completion: [V] Socket 5 created.
Sun May 14 07:26:29 2017	INFO	Completion: Requested [1/121] articles, 0 failed.
Sun May 14 07:26:29 2017	INFO	Completion: Requested [13/121] articles, 0 failed.
Sun May 14 07:26:29 2017	INFO	Completion: Requested [30/121] articles, 0 failed.
Sun May 14 07:26:29 2017	INFO	Completion: Requested [60/121] articles, 0 failed.
Sun May 14 07:26:29 2017	INFO	Completion: Requested [90/121] articles, 0 failed.
Sun May 14 07:26:29 2017	INFO	Completion: Requested [121/121] articles, 0 failed.
Sun May 14 07:26:29 2017	INFO	Completion: All requested article replies received, 0 failed.
Sun May 14 07:26:29 2017	INFO	Completion: [V] Socket 1 closed.
Sun May 14 07:26:29 2017	INFO	Completion: [V] Socket 2 closed.
Sun May 14 07:26:29 2017	INFO	Completion: [V] Socket 3 closed.
Sun May 14 07:26:29 2017	INFO	Completion: [V] Socket 4 closed.
Sun May 14 07:26:29 2017	INFO	Completion: [V] Socket 5 closed.
Sun May 14 07:26:29 2017	INFO	Completion: [V] Socket 0 closed.
Sun May 14 07:26:29 2017	INFO	Completion: Failed ratio for server: news.nzbget.net: 0.0%. Server check completed in 1.81 sec.
Sun May 14 07:26:29 2017	INFO	Completion: [V] Total failed ratio: 0.0%
Sun May 14 07:26:29 2017	INFO	Completion: Resuming: "Some.NZB-FiLE.nzb.queued"
Sun May 14 07:26:30 2017	INFO	Completion: Overall check completed in 1.91 sec.
Sun May 14 07:26:30 2017	INFO	Completion: [V] nzbget_resume()
Sun May 14 07:26:30 2017	INFO	Completion: [V] Downloading for NZBGet resumed
Sun May 14 07:26:30 2017	INFO	Completion: [V] del_lock_file()
Sun May 14 07:26:30 2017	INFO	Completion: [V] completion.lock file deleted
Sun May 14 07:26:30 2017	INFO	Successfully downloaded Some.NZB-FiLE\Some.NZB-FiLE.nfo
Sun May 14 07:26:30 2017	INFO	Successfully downloaded Some.NZB-FiLE\Some.NZB-FiLE-sample.par2
Sun May 14 07:26:33 2017	INFO	Successfully downloaded Some.NZB-FiLE\Some.NZB-FiLE.r12
...
...
Sun May 14 07:30:00 2017	INFO	Executing scheduler-script Completion.py
Sun May 14 07:30:00 2017	INFO	Completion: [V] scheduler_call()
Sun May 14 07:30:00 2017	INFO	Completion: [V] Empty queue
Sun May 14 07:45:00 2017	INFO	Executing scheduler-script Completion.py
Sun May 14 07:45:00 2017	INFO	Completion: [V] scheduler_call()
Sun May 14 07:45:00 2017	INFO	Completion: [V] Empty queue
Release history:
Completion 0.1.0. :
Changes:
- First public release
- Requires NZBget 14.0+

Completion 0.1.1. :
Changes:
- Will now work for NZBget 13.0
- Includes SSL/TLS support, but requires python 2.7.9
- Stores the .lock file in temp_dir/completion/
- Only items paused by the (scan) script will be checked on completion and resumed
- 'Fixed' an issue based on that not NZBget doesn't provide the complete/correct file name when queued, resulting in checking the wrong file when NZB files with the same name were in the queue/history.

Completion 0.1.2.:
Changes:
- Fixed a bug for rare occasions that the 'expected file name' could end with nzb..queued instead of nzb.queued, resulting in not be able to locate the correct nzb.
- NZBGet History can now be checked for not hidden DUPES when the CheckDupes option is set to 'Yes' or 'SameScore', this checks for possible DUPEs that could replace the not sufficient complete item in the queue, not older than X hours.

Completion 0.1.3.:
Changes:
- When archives without or just 1 par file are found, all the archives will be checked when option FullCheckNoPars is enabled.
- The option ForceFailure can now be used to inform programs like Sonarr that a download failed, when the DUPE / BAD marking is not supported.
- Fixed an issue where an exact match on failure ratio would result in a BAD nzb, resulting in marking nzb files with a max failure ratio of 0.0% as BAD.
-Various minor fixes in output messages.

Completion 0.2.2.:
Changes:
- Extract news-server login data directly from the NZBGet settings
- Use only enabled news-servers
- Filter all but 1 news-server based on same ServerX.group rating
- Removed option to disable pause NZBget during check, obsolete because check will only start when no active items in the queue.
- Check article availability on each active filtered news servers.
- Added a delay of 200 ms / num connections per server when no reply is received, avoiding continuous looping when waiting for reply
- Fixed an endless loop when a server wasn't configured correctly. (user error or server down)
- Fixed a path naming issue for windows users, related to the double slash, script returned nzb to queue without checking completion.
- Fixed an issue were the last nntp server replies ()equal to the number of connections per provider) weren't taken into account in the actual amount of failed articles, causing small files to be marked ok, while they weren't
- FailureLimitCorrection removed as option, as there is now support for unlimited amount of backup servers

Completion 0.2.3.:
Changes:
- When other scripts also added a parameter to a NZB, the script couldn't extract the file name of the nzb to be checked, causing a move to the queue without checking. Typically happened for Sonarr.
Completion023.zip (Downloaded 464 times)

Completion 0.2.4.:
Changes:
- Correct some default settings back to its intended general usage settings instead of debug settings.
- Some code clean up.
- Added option to filter on servers the script should use.
- Added option to avoid downloading nzb when X percent of 1st news server fails.
Completion024.zip (Downloaded 212 times)

Completion 0.2.6.:
Changes:
- Fixed issue for servers without login requirements, minor tweaks to NNTP message handling.
- Added fill server option, fill servers will only be used in last check.
- Bug ending script after looping due to provider issue, but keeping NZBget paused.
- Fixed minor output issue, noted by JackD.
- Fixed issue in duplicate handling with VERBOSE off, noted by JackD.
- Added check duration time for each used server.
Completion026.zip (Downloaded 167 times)

Completion 0.2.7.:[/b]
Changes:
- Fixed an issue with MaxFailure=0 limit for duplicate items, marking bad/forcing failure while the post was OK
Completion027.zip (Downloaded 59 times)

Completion 0.2.8.:
Changes:
- Fixed an issue where NZB file names that contain a '-sign would return an error in where the NZB could not be found during checks... NZB file naming should stick with using only the . and - sign :@.
- Passwords now don't end up in the logs for VERBOSE logging. For EXTREME logging they will still appear.
- Fixed an issue when NZBGet was downloading a file not paused by script prior to check would result in an error in nzbget_paused
- Better error handling on news-server connection issues. Failing news-server will be skipped during check, instead of stopping, resulting in checking the file over and over.
- Check duration per server is now shown correctly, was not shown if not all requested replies were received.
- Clearer messaging.
- When script crashes, and completion.lock file remains longer than 60 min in tmp folder, the old .lock file will be overwritten, assuming the script is not running for more than 60 minutes for 1 nzb. This may avoid completely paused queues when a rare error occurs that won't be there the next check.
- Fixed an issue in when no .par files would be included, and using more than 1 news-server, the script would continue checking the articles on the 2nd server, even after no missing articles where detected, resulting in a crash of the script. Issue as reported by JackD.
Completion028.zip (Downloaded 111 times)

Completion 0.2.10.:
Changes:
- Fixed an issue related to use of strange characters in headers, resulting in a crash, reported by joshuader6
- Presumably fixed issue reported by fovert, for a misplaced if statement resulting in the parameter success not being defined before use.
- Fixed an issue in when the last news server had connection errors on all connections, the failed_ratio would be set to 0 marking the file OK, while it shouldn't. This would only happen in very rare cases. As noticed by dagoob.
- Fixed an issue in where on rare occasions the script would crash due to wrong loop exits due to wrongly assumed upper bound on rar_msg_ids array. As reported by user Crowley, and presumably earlier on by JackD.
- Corrected several script status messages that might not appeared for other servers than the main server.
- In module get_dupe_nzb_status the wrong variable was used, failed_ratio instead of rar_msg_ids, causing script to crash on found DUPEs in history as alternatives for failing items in the queue.
Completion0210.zip (Downloaded 119 times)

Completion 1.0.0.:
Changes:
- Added a suggested feature by user geogolem to check priority of paused and unpaused items, and sent paused items with higher priority than the active download also back to queue, and let NZBGet handle the download priority.
- Fixed an issue where the script was not triggered when performing queue operations (like delete) when the related file was not paused by the script.
- Fixed an issue that caused the socket error message "The operation did not complete (read) (_ssl.c:1752)" on sockets that already had received the last reply from the news-provider, speeding up the script.
- Socket creation now is done for only the server(s) that will actually be used, instead of creating the sockets for all listed servers prior to the check. This speeds up the script, especially when a lot of newsservers listed in the settings.
- Fixed an issue when more sockets than articles were made, causing some warning messages, and might trigger the number of connections exceeded warning messages in NZBGet.
- Added option to check NZBGet server retention value, so it may skip the check for a typical server with lower specified retention than post age. Requires NZBGet 15.0+
- Updated the extraction of article data from the NZB file, speeding up the data extraction process by ~80%. Change of if statements into else ifs, find() replaced by in, HTML parsing at the end, not for each line. Especially noticable on files with many articles.
- Some serious code clean up.
- Added an option to limit the maximum amount of articles that will be checked overruling the % option.
- Fixed an issue for when an NZB file does not contain any rars, and the script did crash. NZB will now be resumed, and NZBGet will most likely mark it as FAILED. As reported by user konubywy.
- Added an option that sends a QUIT message to newsserver after all messages received, and closes the used sockets. In this way the connections might be freed for the actual downloading, solving an issue where NZBGet would return a message that too many connections are being used for a typical server.
- Added an option that waits till NZBGet expectedly closed the connections to the news servers, avoiding that the script would trigger a message that too many connections are being used for a typical server, resulting in a failed check on that news server.
- The ForceFailure option now actually searches for the smallest par and other file, to limit the data use and time of forcing the failure.
- Rebuild of the error catching on slow replies.
- Script now is able to use the 'Unified extension scripts' features that will be introduced in NZBGet 18, and simplifies the scripts use.
- Script in NZBGet 18 won't be called for each downloaded file inside the NZB, only noticable in VERBOSE mode.
- Fixed an issue when only 1 article would be checked, the script crashed as no group was defined. Reported by dagoob.
- Added an option to increase the minimal amount of articles that will be checked overruling the % option.
- Added IPv6 support thanks to user dagoob.
Completion100.zip
completion 1.0.0.
(18.71 KiB) Downloaded 904 times

Completion 1.1.0.:
Changes:
- Fixed an issue in where a filename might contained a } sign, resulting in the no such nzb file message. Issue reported by user barenaked.
- Added check for correct python version.
- Removed Prioritize option
- Added option AgeSortLimit
- Added the option to run a scheduler check manually, using the button option introduced in NZBget v19, works only in NZBget 19+
- Added check on correct use of path separators, if incorrect a warning message will be shown.
Completion110.zip
completion 1.1.0.
(18.21 KiB) Downloaded 84 times
Last edited by kloaknet on 01 Sep 2017, 15:31, edited 44 times in total.

kloaknet
Posts: 308
Joined: 23 Jul 2014, 08:52

Re: [SQPP-Script] Completion - Post completion check script

Post by kloaknet » 15 Jan 2015, 18:43

Got a bunch of questions/comments for hugbug about the script, hope you can be of help ;) :

1. I am not sure if it will work for older versions than v14.0. Based on going through the release notes down to V11, I think the script might work partly for V11 ( but without enabling it as queue script. It should work completely for V13, but not 100% sure. I added some version check in the script, so atm it can only run in V14 or later, couldn't find a robust way to get it check for V13+.

2. For the file to work together with FailureLink.py, I need to set the history status to a typical failure status. This is however not possible. Marking the file BAD won't trigger FailureLink.py atm. An option could be that the nzb will be destroyed (like removing all but one file of a group), and returned to the queue. But that would make the NZB unusable for when the NZB is destroyed by mistake.

3. I noticed that NZBget determines the critical health based on all PAR2 files in the queue, and it happens that a critical failure % of like 20% is shown. But this includes the % for samples and subs, while the actual main file only has a max fail limit of like 10%. Haven't had any issues with it tbh.

4. The possible changing of Categories (via SETTINGS -> CATEGORIES) happens after the Scan scripts have been run. Is this intended?

5. Would implementing the option to use of SSL be easy or ...?

6. Will implementing a PAR2 check in the script be easy, so that the amount of actual available repair blocks is determined by the script. ( amount of PAR articles doesn't seem to equal the amount of repair blocks ).

hugbug
Developer & Admin
Posts: 6585
Joined: 09 Sep 2008, 11:58
Location: Germany (NRW)

Re: [SQPP-Script] Completion - Post completion check script

Post by hugbug » 16 Jan 2015, 08:52

kloaknet wrote:1. couldn't find a robust way to get it check for V13+.
Version check is usually simple - check if a certain option introduced in the version in question has exist. For example according to ChangeLog option "NzbProcess" was renamed to "ScanScript". Then you can check if the option ScanScript was passed to the script:

Code: Select all

if os.environ.get("NZBOP_SCANSCRIPT") == None:
    error
Or test the version directly - Extension_scripts#Testing_NZBGet_version.
2. For the file to work together with FailureLink.py, I need to set the history status to a typical failure status.
FailureLink.py is just a script. Make necessary changes there required to work together with your script. Scripts can transfer info using pp-vars. For example to set myvar from your script (bash example):

Code: Select all

echo "[NZB] NZBPR_myvar=my value";
Then you can add an extra check to FailureLink.py:

Code: Select all

if os.environ.get('NZBPR_myvar') == 'my value':
3. I noticed that NZBget determines the critical health based on all PAR2 files in the queue, and it happens that a critical failure % of like 20% is shown. But this includes the % for samples and subs, while the actual main file only has a max fail limit of like 10%. Haven't had any issues with it tbh.
Samples and subs are small. It's not possible for them to have such a big impact (10% vs. 20%).
4. The possible changing of Categories (via SETTINGS -> CATEGORIES) happens after the Scan scripts have been run. Is this intended?
Not sure what you mean.
5. Would implementing the option to use of SSL be easy or ...?
You mean in python? In python is everything simple :)
6. Will implementing a PAR2 check in the script be easy, so that the amount of actual available repair blocks is determined by the script. ( amount of PAR articles doesn't seem to equal the amount of repair blocks ).
Par-check is a complex thing. The calculation of the correct amount of par-blocks before download of files is not possible. An nzb-file doesn't contain enough info to determine how articles map to par2-blocks.

kloaknet
Posts: 308
Joined: 23 Jul 2014, 08:52

Re: [SQS-Script] Completion - Post completion check script

Post by kloaknet » 18 Jan 2015, 15:53

1. Thanks for the 'NZBOP_SCANSCRIPT' for the version check for v13. Because I expect that the script also would work for v11 (only not for the Queue part), but I am not sure. Did the options 'NZBNP_NZBNAME' and 'NZBSP_TASKID' already existed in v11? I cannot extract that info from the change logs.


2. Thanks for the tip, atm I don't have the intention to create / include the FailureLink script in the queue. On second thoughts, the script is mainly intended for use together with RSS feeds, and therefore DUPE items will be returned from the History when marking a file BAD.


4. Via SETTINGS -> CATEGORIES it is possible to change the category of an incoming item, like TV - HD to TV by using CategoryX.Aliases. This happens after the Scan script is called. So when you use the script to pause incoming NZBs with the TV category, they are not detected by the script, because they still have the TV - HD category. So is it intended that the Category changes happen after the scan script? (its not an issue, but then I should mention it in the script, to avoid confusion why it doesnt seems to be working as expected).


5. :) I just started with Python, google helps a lot, but I don't know how to handle certificates etc, and have no expierence with that. This gave me an idea: http://www.heikkitoivonen.net/blog/2008 ... python-26/, but I dont know when a certificate is safe or not etc.


6. In the far future I could might use NZBGet to grab all pars (pause all rar files), only if close to critical health, and see how many repair blocks there are available. Maybe there is some data I can extract from NZBget, when option ParCheck is on :roll:, or is the ParCheck purely a post processing thing?


Currently working on implementing to Check possible dupes in the history, when an item with the same dupekey as in the download queue fails the check:

EDIT: ADDED:
7. The script extracts the NZB name via the rpc-api 'listgroups' or 'history' command. However, I couldn't find an indication in the json string how DUPEs with identical NZB names are named (.nzb.X.queued). How to know which NZB has no X, and which NZB will have 2, 3, etc? Can it be done by using 'HistoryTime' or 'NZBID', so is the lowest NZBID the one with .nzb.queued, the next one .nzb.2.queued, etc? Although that won't work when someone fully deletes .nzb.2.queued from the history, when there is also a .nzb.3.queued ;) . NZBget must somewhere know which .nzb.X.queued is in the queue / history, is there maybe another option I could call to get that info?

hugbug
Developer & Admin
Posts: 6585
Joined: 09 Sep 2008, 11:58
Location: Germany (NRW)

Re: [SQS-Script] Completion - Post completion check script

Post by hugbug » 26 Jan 2015, 21:04

kloaknet wrote:1. Thanks for the 'NZBOP_SCANSCRIPT' for the version check for v13. Because I expect that the script also would work for v11 (only not for the Queue part), but I am not sure. Did the options 'NZBNP_NZBNAME' and 'NZBSP_TASKID' already existed in v11? I cannot extract that info from the change logs.
NZBNP_NZBNAME exists in v11. NZBSP_TASKID is for scheduler scripts, which are available since v13.
I don't see reasons why you would bother about supporting older versions.
4. Via SETTINGS -> CATEGORIES it is possible to change the category of an incoming item, like TV - HD to TV by using CategoryX.Aliases. This happens after the Scan script is called.
Are you sure about this? The code suggests the category is changed before calling scan-scripts.
but I don't know how to handle certificates etc
Me neither. I don't check theme at all.
In the far future I could might use NZBGet to grab all pars (pause all rar files), only if close to critical health, and see how many repair blocks there are available. Maybe there is some data I can extract from NZBget, when option ParCheck is on :roll:, or is the ParCheck purely a post processing thing?
Yes, it's purely a post processing thing. Trying to par-check during download is too complex and isn't worth the efforts.
Currently working on implementing to Check possible dupes in the history, when an item with the same dupekey as in the download queue fails the check:
May be you should perform the completion check from the Queue-Script called when the download starts. In that case you could pause the download during download. If the check fails you can mark the download as BAD (like FakeDetector does) and the dupe handling will google automatically.
The script extracts the NZB name via the rpc-api 'listgroups' or 'history' command. However, I couldn't find an indication in the json string how DUPEs with identical NZB names are named (.nzb.X.queued). How to know which NZB has no X, and which NZB will have 2, 3, etc? Can it be done by using 'HistoryTime' or 'NZBID', so is the lowest NZBID the one with .nzb.queued, the next one .nzb.2.queued, etc? Although that won't work when someone fully deletes .nzb.2.queued from the history, when there is also a .nzb.3.queued ;) . NZBget must somewhere know which .nzb.X.queued is in the queue / history, is there maybe another option I could call to get that info?
There is an internal property QueuedFilename but it's not exposed via API. The names are usually long and adding this rarely useful info to the response doesn't seem right. I could pass the var to scripts (queue-script and pp-script) however, that doesn't eat any resources. Can you explain why you need this info?

barabbas
Posts: 24
Joined: 04 Dec 2014, 17:35

Re: [SQS-Script] Completion - Post completion check script

Post by barabbas » 27 Jan 2015, 13:18

Out of interest, I never heard that python's XML-RPC is slow with large amounts of data. Did you find out by yourself or do you have a source on that?

hugbug
Developer & Admin
Posts: 6585
Joined: 09 Sep 2008, 11:58
Location: Germany (NRW)

Re: [SQS-Script] Completion - Post completion check script

Post by hugbug » 27 Jan 2015, 13:55

barabbas wrote:Out of interest, I never heard that python's XML-RPC is slow with large amounts of data. Did you find out by yourself or do you have a source on that?
What part of this topic you refer to?

barabbas
Posts: 24
Joined: 04 Dec 2014, 17:35

Re: [SQS-Script] Completion - Post completion check script

Post by barabbas » 27 Jan 2015, 16:02

Sorry, the script source code says:
Connect to NZBGet and call an RPC-API-method without using of python's XML-RPC. XML-RPC is easy to use but it is slow for large amounts of data.

hugbug
Developer & Admin
Posts: 6585
Joined: 09 Sep 2008, 11:58
Location: Germany (NRW)

Re: [SQS-Script] Completion - Post completion check script

Post by hugbug » 27 Jan 2015, 16:12

That's a comment from FakeDetector-script.
For explanation see the second post in topic [Queue-Script] FakeDetector - early detection of fake videos (the link should scroll to the second post but may not scroll in all browsers).

kloaknet
Posts: 308
Joined: 23 Jul 2014, 08:52

Re: [SQS-Script] Completion - Post completion check script

Post by kloaknet » 28 Jan 2015, 17:22

@barabbas, the basic way how the script works was implemented by an other guy, he must have copied it from the script hugbug mentioned. I continued with it, so maybe some snippets are from other scripts, that I didnt know about, and just kept the comments in place :oops: .

@hugbug, thanks for helping me again.
1. I don't see reasons why you would bother about supporting older versions.
ill stick with v13 then. If others would like to have it run on older version, let me know
4. Via SETTINGS -> CATEGORIES it is possible to change the category of an incoming item, like TV - HD to TV by using CategoryX.Aliases. This happens after the Scan script is called.
Are you sure about this? The code suggests the category is changed before calling scan-scripts.
You must be correct, I enabled the category filter in my code again, and now it works :oops:. Sorry wasting your time.

I just tested the 6 old versions of the script, no problems now. Did spend a day or 2 figuring out why I had the issue, about 10 weeks ago, I couldn't figure out why some stuff wasn't paused, while I expected it to be, because the category seemed ok, extensive debugging (category printing to the logs etc) showed that the category change happened after the scan call somehow. (did send the nzbs directly from the indexer) It works ok atm, so lets ignore it! :?

5. So i just can create an SSL connection without checking certificates of the news providers, sounds strange when you try to make a secure and safe connection :P. Ill just try something!

6. Even better, lets just skip the par checks etc.
May be you should perform the completion check from the Queue-Script called when the download starts. In that case you could pause the download during download. If the check fails you can mark the download as BAD (like FakeDetector does) and the dupe handling will google automatically.
The script is called when an NZBNA_EVENT NZB_ADDED occurs. I currently assume the item that is 'added' (or returned from history) is paused. Only paused items are checked. (Does the DUPE check happen before or after the NZB_ADDED?, then dupes will be paused also when returned due to BAD marking). Items that fail after X hours are marked BAD by the script, to let the DUPE return. But when there are already dupes within the X hours, that are OK, i like to switch them out with the one thats incomplete in the queue. This brings me to the next question and your reply:
7. There is an internal property QueuedFilename but it's not exposed via API. The names are usually long and adding this rarely useful info to the response doesn't seem right. I could pass the var to scripts (queue-script and pp-script) however, that doesn't eat any resources. Can you explain why you need this info?
To check an NZB on completion, the script reads the actual NZB for all the articles, files and servers (to cover cross posting of the items in the one NZB over different a.b.groups). Therefore I need to know which NZB i need to check. It happens several times that an RSS feed returns 2 NZBs with exactly the same name (but different size, obfustication, etc, so its not deleted directly by NZBget as a duplicate, but kept as a DUPE). Because I got 2 of the same files names in the history / listgroups, and the part after .nzb. is not shown, I don't know which file (NZBID) I check (atm its just hardcoded to nzb.queued in my code :shock:). So basically I only need the part .nzb.X.queued, not the whole file name + dir.

I thought of a workaround for the above, maybe you can tell me if it will work (even with DUPE check of in NZBget maybe?), so you don't need to tweak NZBget only for me, and the script will work on v13+:
- list all NZBs with the same NzbName / DupeKey, connected to their NZBID.
- list all NZBs with the nzb file name in the directory, check if they have a matching NzbName with NZBget, get the file creation time (in miliseconds)
- from the above, connect the oldest NZBID with the oldest creation time, hoping the oldest file is actually the one with the oldest NZBID?
(Its just a bit more of coding, and then I always need to call listgroups, history and list the files in the nzb dir)

Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests