Completion 1.1.0 Notes:
- To run the script you currently need NZBGet version 15.0 or higher (18+ advised), and python 2.7.9+ (does not work for python 3, but maybe this version does). For versions of NZBGet 18+ configuring the script is way more simple.
- To get a good insight of why the script does certain things you might not expect, please turn on 'Verbose' logging on the script settings page. This most likely will show you all you need in the NZBGet logs.
General:
NZBGet ‘extension script’ that checks if the data in the NZB file is sufficiently complete at your usenet provider(s), before starting the download. If incomplete it would wait for a certain period and check the completion of the NZB file again. This check is done by requesting the header status, and is in normal cases done within seconds (like 1 - 5 sec. for a 1 GB file). This method is significantly faster than when NZBGet would report a failure, after actual downloading a (part of) the files that end up incomplete. The script is typically useful for issues related to:
- very recent posts,
- failed downloads, which after a while are just ok (propagation issues),
- incomplete posts,
- taken down posts (DMCA, etc.),
- old posts,
- long par repair times,
- downloading (parts of) NZB files beyond repair,
- unnecessary use of expensive block / slow fill accounts.
The above would generally result in an (error) messages like ‘missing articles’, ‘unable to repair’ or ‘additional par files required’, ‘not enough par-blocs’, etc. The script avoids these messages.
How to install, NZBGet 18+:
- put the script in your script folder of NZBGet (ScriptDir under PATHS in Settings),
- go to Settings -> EXTENSION SCRIPTS, hit ‘Choose’ at the ‘Extensions’ option, and thick Completion.py,
- if u use category mapping, also go to Settings -> CATEGORIES, and select the script for CategoryX.Extension too.
- optionally configure the settings under Settings -> COMPLETION, although the default settings would normally suffice.
- optionally remove all propagation delays in NZBGet, your indexer, and/or Sonarr and the likes, as this script handles it all.
How to install, pre NZBGet 18:
- put the script in your script folder of NZBget (ScriptDir under PATHS in Settings),
- optionally configure the settings under Settings -> COMPLETION, although the default settings would normally suffice.
- enable the script as:
- SCAN SCRIPT, via Settings -> EXTENSION SCRIPTS
- QUEUE SCRIPT, via Settings -> EXTENSION SCRIPTS
- SCHEDULER SCRIPT, via Settings -> SCHEDULER, enable it for every X minutes (*:00,*:15, *:30,*:45), everyday (1-7), and tick ‘Script’. Checking each 15 minutes should be sufficient.
- if u use category mapping, also go to Settings -> CATEGORIES, and select the script for CategoryX.Extension too.
- optionally remove all propagation delays in NZBGet, your indexer, and/or Sonarr and the likes, as this script handles it all.
Make sure your PATHS settings are correct, only use \ for windows, / for unix. Don't use trailing slashes either.
What does it do:
- scans all incoming NZBs, turns them PAUSED (optional for typical categories, e.g. for your RSS feeds categories or TV shows).
- uses all in NZBGet specified usenet providers, (optional for selected servers as main servers and fill servers).
- when the scheduler is triggered, or when an NZB is added, downloaded, deleted or marked, a check is started.
- on each, check, the oldest, highest priority item in the queue will be checked, if OK: Resume NZB, if not OK: check for possible DUPEs in history or check next paused NZB in queue.
- if no queued / downloading NZB (with equal or lower priority than by the script paused NZBs) is in the queue, and NZBGet is not paused, Y% of the header statuses of by the script paused NZB will be checked (to see if the actual files are available at the usenet provider(s))
- if not OK after X hours, the file will be moved to history, and marked as BAD / FAILED, so new NZBs (e.g. RSS) will be downloaded, or programs like Sonarr will be informed, so they can send an alternative NZB file.
- when the check fails, it is possible to let the script check the history for possible working duplicate NZB files. This creates the option to download a working NZB that is not actually in the queue, but listed as DUPE in the history. (via CheckDupes option, DUPEs are generally a result of using RSS feeds).
- the script will download the highest priority, oldest item in the queue first. Priority goes above age. (optionally NZBs within X hours can be 'prioritized' over files older than X hours).
- check the settings page of the script to get more details on all the script options.
- The NZBget log file, with the Verbose option turned on, will look like this:
Code: Select all
Sun May 14 07:25:25 2017 INFO nzbget 19.0-testing-r1991 service-mode
Sun May 14 07:25:25 2017 INFO Scheduler: unpausing download
Sun May 14 07:25:25 2017 INFO Executing scheduler-script Completion.py
Sun May 14 07:25:26 2017 INFO Completion: [V] scheduler_call()
Sun May 14 07:25:26 2017 INFO Completion: [V] Empty queue
Sun May 14 07:26:25 2017 INFO Queue Some.NZB-FiLE @ inde.xer
Sun May 14 07:26:26 2017 INFO Executing scan-script Completion.py for Some.NZB-FiLE.nzb
Sun May 14 07:26:27 2017 INFO Completion: [V] scan_call()
Sun May 14 07:26:27 2017 INFO Completion: [V] Expected queued file name: "Some.NZB-FiLE.nzb.queued"
Sun May 14 07:26:27 2017 INFO Completion: [V] Pausing: "Some.NZB-FiLE.nzb"
Sun May 14 07:26:27 2017 INFO Adding collection Some.NZB-FiLE.nzb to queue
Sun May 14 07:26:27 2017 INFO Collection Some.NZB-FiLE added to queue
Sun May 14 07:26:27 2017 INFO Executing queue-script Completion.py for Some.NZB-FiLE
Sun May 14 07:26:28 2017 INFO Completion: [V] queue_call()
Sun May 14 07:26:28 2017 INFO Completion: [V] lock_file()
Sun May 14 07:26:28 2017 INFO Completion: [V] server_time= 1494739588
Sun May 14 07:26:28 2017 INFO Completion: [V] New completion.lock file created.
Sun May 14 07:26:28 2017 INFO Completion: [V] nzbget_paused()
Sun May 14 07:26:28 2017 INFO Completion: [V] Waiting for NZBGet to end downloading
Sun May 14 07:26:28 2017 INFO Completion: [V] Downloading for NZBGet paused
Sun May 14 07:26:28 2017 INFO Completion: [V] Paused UNSORTED NZBs in queue that will be processed:
Sun May 14 07:26:28 2017 INFO Completion: [V] * Some.NZB-FiLE.nzb.queued, Age: 10.1 hours, Priority: 0
Sun May 14 07:26:28 2017 INFO Completion: [V] Ignoring sorting priority of items older than AgeLimit of 4 hours
Sun May 14 07:26:28 2017 INFO Completion: [V] Paused and SORTED NZBs in queue that will be processed:
Sun May 14 07:26:28 2017 INFO Completion: [V] * Some.NZB-FiLE.nzb.queued, Age: 10.1 hours, Priority: 0
Sun May 14 07:26:28 2017 INFO Completion: [V] get_nzb_status(nzb=[213, 'Some.NZB-FiLE.nzb.queued', 1494703365, 886, 'NONE', 0])
Sun May 14 07:26:28 2017 INFO Completion: Checking: "Some.NZB-FiLE.nzb.queued"
Sun May 14 07:26:28 2017 INFO Completion: [V] get_nzb_data(fname=C:\ProgramData\NZBGet\nzb\Some.NZB-FiLE.nzb.queued)
Sun May 14 07:26:28 2017 INFO Completion: [V] Amount of to be checked articles increased to about 100 articles.
Sun May 14 07:26:28 2017 INFO Completion: [V] NZB contains 414 articles, 363 rar articles, 51 par2 articles.
Sun May 14 07:26:28 2017 INFO Completion: [V] 121 rar articles will be checked.
Sun May 14 07:26:28 2017 INFO Completion: Maximum failed articles limit for NZB: 11.4%
Sun May 14 07:26:28 2017 INFO Completion: [V] get_server_settings(nzb_age=1494703365)
Sun May 14 07:26:28 2017 INFO Completion: [V] All news servers after filtering on Active, Servers, FillServers + AgeLimit and Retention, BEFORE filtering on NZBGet ServerX.Group:
Sun May 14 07:26:28 2017 INFO Completion: [V] * news.nzbget.net:563, SSL: True, connections: 6
Sun May 14 07:26:28 2017 INFO Completion: [V] * free.nzbget.net:119, SSL: False, connections: 3
Sun May 14 07:26:28 2017 INFO Completion: [V] * ssl.getnzb.com:443, SSL: True, connections: 1
Sun May 14 07:26:28 2017 INFO Completion: [V] All active news servers AFTER filtering and sorting on NZBGet ServerX.Group:
Sun May 14 07:26:28 2017 INFO Completion: [V] * news.nzbget.net:563, SSL: True, connections: 6
Sun May 14 07:26:28 2017 INFO Completion: [V] * free.nzbget.net:119, SSL: False, connections: 3
Sun May 14 07:26:28 2017 INFO Completion: [V] * ssl.getnzb.com:443, SSL: True, connections: 1
Sun May 14 07:26:28 2017 INFO Completion: Using server: news.nzbget.net
Sun May 14 07:26:28 2017 INFO Completion: [V] Creating sockets for server: news.nzbget.net
Sun May 14 07:26:28 2017 INFO Completion: [V] Using IPv4 for news.nzbget.net
Sun May 14 07:26:28 2017 INFO Completion: [V] Socket 0 created.
Sun May 14 07:26:28 2017 INFO Completion: [V] Socket 1 created.
Sun May 14 07:26:28 2017 INFO Completion: [V] Socket 2 created.
Sun May 14 07:26:28 2017 INFO Completion: [V] Socket 3 created.
Sun May 14 07:26:29 2017 INFO Completion: [V] Socket 4 created.
Sun May 14 07:26:29 2017 INFO Completion: [V] Socket 5 created.
Sun May 14 07:26:29 2017 INFO Completion: Requested [1/121] articles, 0 failed.
Sun May 14 07:26:29 2017 INFO Completion: Requested [13/121] articles, 0 failed.
Sun May 14 07:26:29 2017 INFO Completion: Requested [30/121] articles, 0 failed.
Sun May 14 07:26:29 2017 INFO Completion: Requested [60/121] articles, 0 failed.
Sun May 14 07:26:29 2017 INFO Completion: Requested [90/121] articles, 0 failed.
Sun May 14 07:26:29 2017 INFO Completion: Requested [121/121] articles, 0 failed.
Sun May 14 07:26:29 2017 INFO Completion: All requested article replies received, 0 failed.
Sun May 14 07:26:29 2017 INFO Completion: [V] Socket 1 closed.
Sun May 14 07:26:29 2017 INFO Completion: [V] Socket 2 closed.
Sun May 14 07:26:29 2017 INFO Completion: [V] Socket 3 closed.
Sun May 14 07:26:29 2017 INFO Completion: [V] Socket 4 closed.
Sun May 14 07:26:29 2017 INFO Completion: [V] Socket 5 closed.
Sun May 14 07:26:29 2017 INFO Completion: [V] Socket 0 closed.
Sun May 14 07:26:29 2017 INFO Completion: Failed ratio for server: news.nzbget.net: 0.0%. Server check completed in 1.81 sec.
Sun May 14 07:26:29 2017 INFO Completion: [V] Total failed ratio: 0.0%
Sun May 14 07:26:29 2017 INFO Completion: Resuming: "Some.NZB-FiLE.nzb.queued"
Sun May 14 07:26:30 2017 INFO Completion: Overall check completed in 1.91 sec.
Sun May 14 07:26:30 2017 INFO Completion: [V] nzbget_resume()
Sun May 14 07:26:30 2017 INFO Completion: [V] Downloading for NZBGet resumed
Sun May 14 07:26:30 2017 INFO Completion: [V] del_lock_file()
Sun May 14 07:26:30 2017 INFO Completion: [V] completion.lock file deleted
Sun May 14 07:26:30 2017 INFO Successfully downloaded Some.NZB-FiLE\Some.NZB-FiLE.nfo
Sun May 14 07:26:30 2017 INFO Successfully downloaded Some.NZB-FiLE\Some.NZB-FiLE-sample.par2
Sun May 14 07:26:33 2017 INFO Successfully downloaded Some.NZB-FiLE\Some.NZB-FiLE.r12
...
...
Sun May 14 07:30:00 2017 INFO Executing scheduler-script Completion.py
Sun May 14 07:30:00 2017 INFO Completion: [V] scheduler_call()
Sun May 14 07:30:00 2017 INFO Completion: [V] Empty queue
Sun May 14 07:45:00 2017 INFO Executing scheduler-script Completion.py
Sun May 14 07:45:00 2017 INFO Completion: [V] scheduler_call()
Sun May 14 07:45:00 2017 INFO Completion: [V] Empty queue
Completion 0.1.0. :
Changes:
- First public release
- Requires NZBget 14.0+
Completion 0.1.1. :
Changes:
- Will now work for NZBget 13.0
- Includes SSL/TLS support, but requires python 2.7.9
- Stores the .lock file in temp_dir/completion/
- Only items paused by the (scan) script will be checked on completion and resumed
- 'Fixed' an issue based on that not NZBget doesn't provide the complete/correct file name when queued, resulting in checking the wrong file when NZB files with the same name were in the queue/history.
Completion 0.1.2.:
Changes:
- Fixed a bug for rare occasions that the 'expected file name' could end with nzb..queued instead of nzb.queued, resulting in not be able to locate the correct nzb.
- NZBGet History can now be checked for not hidden DUPES when the CheckDupes option is set to 'Yes' or 'SameScore', this checks for possible DUPEs that could replace the not sufficient complete item in the queue, not older than X hours.
Completion 0.1.3.:
Changes:
- When archives without or just 1 par file are found, all the archives will be checked when option FullCheckNoPars is enabled.
- The option ForceFailure can now be used to inform programs like Sonarr that a download failed, when the DUPE / BAD marking is not supported.
- Fixed an issue where an exact match on failure ratio would result in a BAD nzb, resulting in marking nzb files with a max failure ratio of 0.0% as BAD.
-Various minor fixes in output messages.
Completion 0.2.2.:
Changes:
- Extract news-server login data directly from the NZBGet settings
- Use only enabled news-servers
- Filter all but 1 news-server based on same ServerX.group rating
- Removed option to disable pause NZBget during check, obsolete because check will only start when no active items in the queue.
- Check article availability on each active filtered news servers.
- Added a delay of 200 ms / num connections per server when no reply is received, avoiding continuous looping when waiting for reply
- Fixed an endless loop when a server wasn't configured correctly. (user error or server down)
- Fixed a path naming issue for windows users, related to the double slash, script returned nzb to queue without checking completion.
- Fixed an issue were the last nntp server replies ()equal to the number of connections per provider) weren't taken into account in the actual amount of failed articles, causing small files to be marked ok, while they weren't
- FailureLimitCorrection removed as option, as there is now support for unlimited amount of backup servers
Completion 0.2.3.:
Changes:
- When other scripts also added a parameter to a NZB, the script couldn't extract the file name of the nzb to be checked, causing a move to the queue without checking. Typically happened for Sonarr.
Completion023.zip (Downloaded 464 times)
Completion 0.2.4.:
Changes:
- Correct some default settings back to its intended general usage settings instead of debug settings.
- Some code clean up.
- Added option to filter on servers the script should use.
- Added option to avoid downloading nzb when X percent of 1st news server fails.
Completion024.zip (Downloaded 212 times)
Completion 0.2.6.:
Changes:
- Fixed issue for servers without login requirements, minor tweaks to NNTP message handling.
- Added fill server option, fill servers will only be used in last check.
- Bug ending script after looping due to provider issue, but keeping NZBget paused.
- Fixed minor output issue, noted by JackD.
- Fixed issue in duplicate handling with VERBOSE off, noted by JackD.
- Added check duration time for each used server.
Completion026.zip (Downloaded 167 times)
Completion 0.2.7.:[/b]
Changes:
- Fixed an issue with MaxFailure=0 limit for duplicate items, marking bad/forcing failure while the post was OK
Completion027.zip (Downloaded 59 times)
Completion 0.2.8.:
Changes:
- Fixed an issue where NZB file names that contain a '-sign would return an error in where the NZB could not be found during checks... NZB file naming should stick with using only the . and - sign :@.
- Passwords now don't end up in the logs for VERBOSE logging. For EXTREME logging they will still appear.
- Fixed an issue when NZBGet was downloading a file not paused by script prior to check would result in an error in nzbget_paused
- Better error handling on news-server connection issues. Failing news-server will be skipped during check, instead of stopping, resulting in checking the file over and over.
- Check duration per server is now shown correctly, was not shown if not all requested replies were received.
- Clearer messaging.
- When script crashes, and completion.lock file remains longer than 60 min in tmp folder, the old .lock file will be overwritten, assuming the script is not running for more than 60 minutes for 1 nzb. This may avoid completely paused queues when a rare error occurs that won't be there the next check.
- Fixed an issue in when no .par files would be included, and using more than 1 news-server, the script would continue checking the articles on the 2nd server, even after no missing articles where detected, resulting in a crash of the script. Issue as reported by JackD.
Completion028.zip (Downloaded 111 times)
Completion 0.2.10.:
Changes:
- Fixed an issue related to use of strange characters in headers, resulting in a crash, reported by joshuader6
- Presumably fixed issue reported by fovert, for a misplaced if statement resulting in the parameter success not being defined before use.
- Fixed an issue in when the last news server had connection errors on all connections, the failed_ratio would be set to 0 marking the file OK, while it shouldn't. This would only happen in very rare cases. As noticed by dagoob.
- Fixed an issue in where on rare occasions the script would crash due to wrong loop exits due to wrongly assumed upper bound on rar_msg_ids array. As reported by user Crowley, and presumably earlier on by JackD.
- Corrected several script status messages that might not appeared for other servers than the main server.
- In module get_dupe_nzb_status the wrong variable was used, failed_ratio instead of rar_msg_ids, causing script to crash on found DUPEs in history as alternatives for failing items in the queue.
Completion0210.zip (Downloaded 119 times)
Completion 1.0.0.:
Changes:
- Added a suggested feature by user geogolem to check priority of paused and unpaused items, and sent paused items with higher priority than the active download also back to queue, and let NZBGet handle the download priority.
- Fixed an issue where the script was not triggered when performing queue operations (like delete) when the related file was not paused by the script.
- Fixed an issue that caused the socket error message "The operation did not complete (read) (_ssl.c:1752)" on sockets that already had received the last reply from the news-provider, speeding up the script.
- Socket creation now is done for only the server(s) that will actually be used, instead of creating the sockets for all listed servers prior to the check. This speeds up the script, especially when a lot of newsservers listed in the settings.
- Fixed an issue when more sockets than articles were made, causing some warning messages, and might trigger the number of connections exceeded warning messages in NZBGet.
- Added option to check NZBGet server retention value, so it may skip the check for a typical server with lower specified retention than post age. Requires NZBGet 15.0+
- Updated the extraction of article data from the NZB file, speeding up the data extraction process by ~80%. Change of if statements into else ifs, find() replaced by in, HTML parsing at the end, not for each line. Especially noticable on files with many articles.
- Some serious code clean up.
- Added an option to limit the maximum amount of articles that will be checked overruling the % option.
- Fixed an issue for when an NZB file does not contain any rars, and the script did crash. NZB will now be resumed, and NZBGet will most likely mark it as FAILED. As reported by user konubywy.
- Added an option that sends a QUIT message to newsserver after all messages received, and closes the used sockets. In this way the connections might be freed for the actual downloading, solving an issue where NZBGet would return a message that too many connections are being used for a typical server.
- Added an option that waits till NZBGet expectedly closed the connections to the news servers, avoiding that the script would trigger a message that too many connections are being used for a typical server, resulting in a failed check on that news server.
- The ForceFailure option now actually searches for the smallest par and other file, to limit the data use and time of forcing the failure.
- Rebuild of the error catching on slow replies.
- Script now is able to use the 'Unified extension scripts' features that will be introduced in NZBGet 18, and simplifies the scripts use.
- Script in NZBGet 18 won't be called for each downloaded file inside the NZB, only noticable in VERBOSE mode.
- Fixed an issue when only 1 article would be checked, the script crashed as no group was defined. Reported by dagoob.
- Added an option to increase the minimal amount of articles that will be checked overruling the % option.
- Added IPv6 support thanks to user dagoob.
Completion 1.1.0.:
Changes:
- Fixed an issue in where a filename might contained a } sign, resulting in the no such nzb file message. Issue reported by user barenaked.
- Added check for correct python version.
- Removed Prioritize option
- Added option AgeSortLimit
- Added the option to run a scheduler check manually, using the button option introduced in NZBget v19, works only in NZBget 19+
- Added check on correct use of path separators, if incorrect a warning message will be shown.