[SQS-Script] Completion-Propagation/DMCA/Retention check

Share your scripts or request scripts with specific features.
Forum rules
Please keep the forum clean - one topic per script. Questions not related to a specific script should be posted in Support forum.
Post Reply
binreader
Posts: 20
Joined: 07 Oct 2014, 03:00

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by binreader » 23 Oct 2015, 06:28

reloading nzbget dont help, neither did toggle prioritization off. script basically pauses nzbget and chills. i wait for someone else to continue from here.

kloaknet
Posts: 335
Joined: 23 Jul 2014, 08:52

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by kloaknet » 28 Oct 2015, 16:09

binreader wrote:reloading nzbget dont help, neither did toggle prioritization off. script basically pauses nzbget and chills. i wait for someone else to continue from here.
I checked your issue with the 023 script version on my Linux machine, and didn't have an issue with the script. The only thing I can come up with now is that you changed your dir settings in NZBget, via Settings, Paths.

For windows machines, use the \ in the NzbDir field, like: ${MainDir}\nzb

For linux machines, use the / in the NzbDir field, like: ${MainDir}/nzb

Note that when you notice that the issue is related to the (back) slash, please download the script again, because the proposed changes earlier wouldnt make it work if the issue is related to the above.


Otherwise Ill have to provide you with additional coding to retrieve the paths that your NZBget version is using etc.

war59312
Posts: 45
Joined: 12 Feb 2013, 04:52
Location: U.S.A
Contact:

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by war59312 » 28 Nov 2015, 20:36

Hi,

Nice script, Sadly it is not working for me.

Only ScanScript installed.

Set as first QueueScript. Followed by FakeDetector and PasswordDetector. Set for SCHEDULER every 1 hour.

Running nzbget 16.3 on Ubuntu 14.04 LTS.

Start of Detail log:

Code: Select all

Sat Nov 28 15:19:37 2015	INFO	nzbget 16.3 daemon-mode
Sat Nov 28 15:19:49 2015	INFO	SickBeard connected to drop off FileName.nzb any moment now.
Sat Nov 28 15:19:49 2015	INFO	Queue FileName @ api.nzbgeek.info
Sat Nov 28 15:19:50 2015	DETAIL	Downloading FileName @ api.nzbgeek.info
Sat Nov 28 15:19:51 2015	DETAIL	Download FileName @ api.nzbgeek.info completed
Sat Nov 28 15:19:51 2015	INFO	Executing scan-script Completion023.py for FileName.nzb
Sat Nov 28 15:19:51 2015	INFO	Completion023: [V] Expected queued file name: "FileName.nzb.queued"
Sat Nov 28 15:19:51 2015	INFO	Completion023: [V] Pausing: "FileName.nzb"
Sat Nov 28 15:19:51 2015	INFO	Adding collection FileName.nzb to queue
Sat Nov 28 15:19:51 2015	INFO	Collection FileName added to queue
Sat Nov 28 15:19:51 2015	INFO	Executing queue-script Completion023.py for FileName
Sat Nov 28 15:19:52 2015	INFO	Completion023: [V] lock_file()
Sat Nov 28 15:19:52 2015	INFO	Completion023: [V] Old completion.lock file overwritten
Sat Nov 28 15:19:52 2015	INFO	Completion023: [E] get_prio_nzb(jobs=
Sat Nov 28 15:19:52 2015	INFO	Completion023: [E] {u'NZBName': u'FileName', u'Category': u'tv', u'DeleteStatus': u'NONE', u'Log': [], u'ExParStatus': u'NONE', u'Parameters': [{u'Name': u'*Unpack:', u'Value': u'yes'}, {u'Name': u'nzbToMedia/DeleteSamples.py:', u'Value': u'yes'}, {u'Name': u'nzbToMedia/nzbToSickBeard.py:', u'Value': u'yes'}, {u'Name': u'EMail.py:', u'Value': u'yes'}, {u'Name': u'NotifyXBMC.py:', u'Value': u'yes'}, {u'Name': u'CnpNZBFileName', u'Value': u'FileName.nzb.queued'}], u'MoveStatus': u'NONE', u'RemainingSizeLo': 1126253271, u'UnpackStatus': u'NONE', u'MaxPriority': 100, u'MinPostTime': 1448678125, u'PostInfoText': u'NONE', u'RemainingSizeHi': 0, u'ParTimeSec': 0, u'Health': 1000, u'PostTotalTimeSec': 0, u'LastID': 23, u'PausedSizeMB': 1074, u'FileCount': 73, u'UnpackTimeSec': 0, u'Status': u'PAUSED', u'PostStageProgress': 4893483, u'PostStageTimeSec': 0, u'FileSizeMB': 1074, u'Deleted': False, u'ExtraParBlocks': 0, u'Succ
Sat Nov 28 15:19:52 2015	INFO	Completion023: )
Sat Nov 28 15:19:52 2015	INFO	Completion023: [V] Found paused item in queue: "FileName.nzb.queued"
Sat Nov 28 15:19:52 2015	INFO	Completion023: [V] nzbget_paused()
Sat Nov 28 15:19:52 2015	INFO	Completion023: [V] Waiting for NZBGet to end downloading
Sat Nov 28 15:19:52 2015	INFO	Completion023: [V] Downloading for NZBGet paused
Sat Nov 28 15:19:52 2015	INFO	Completion023: [V] Ignoring priority of items older than AgeLimit of 4 hours
Sat Nov 28 15:19:52 2015	INFO	Completion023: [V] get_nzb_status(nzb=[23, 'FileName.nzb.queued', 1448678141, 895, u'SickBeard-248861-4.1', 400])
Sat Nov 28 15:19:52 2015	INFO	Completion023: Checking: "FileName.nzb.queued"
Sat Nov 28 15:19:52 2015	INFO	Completion023: [V] get_nzb_data(fname=/DownloadDirectory/FileName.nzb.queued)
Sat Nov 28 15:19:52 2015	INFO	Completion023: [V] NZB contains 1461 articles, 1320 rar articles, 141 par2 articles
Sat Nov 28 15:19:52 2015	INFO	Completion023: [V] 330 articles will be checked
Sat Nov 28 15:19:52 2015	INFO	Completion023: [V] Maximum failed articles limit: 10.5%
Sat Nov 28 15:19:52 2015	INFO	Completion023: [E] check_failure_status(rar_msg_ids=[[u'FileName [02/73] - "FileName.part01.rar" yEnc (1/21)', 0, ['alt.binaries.bloaf'], '565912ed$0$43583$b1db1813$ca8b28d9@NewsProvider', -1], [u'FileName [02/73] - "FileName.part01.rar" yEnc (1/21)', 0, ['alt.binaries.bloaf'], '565912ed$0$44005$b1db1813$ca8b28d9@NewsProvider', -1], [u'FileName [02/73] - "FileName.part01.rar" yEnc (1/21)', 0, ['alt.binaries.bloaf'], '565912ed$0$43626$b1db1813$ca8b28d9@NewsProvider', -1], [u'FileName [02/73] - "FileName.part01.rar" yEnc (1/21)', 0, ['alt.binaries.bloaf'], '565912ed$0$43644$b1db1813$ca8b28d9@NewsProvider', -1], [u'FileName [02/73] - "FileName.part01.rar" yEnc (1/21)', 0, ['alt.binaries.bloaf'], '565912ed$1$43773$b1db1813$ca8b28d9@news.astra
Sat Nov 28 15:19:52 2015	INFO	Completion023: gEAL8rcr0KAz0KGOE02bd5I4VMf [11/73] - "FileName.part10.rar" yEnc (1/21)', 0, ['alt.binaries.bloaf'], '565912ef$2$43607$b1db1813$ca8b28d9@NewsProvider', -1], [u'FileName [12/73] - "FileName.part11.rar" yEnc (1/21)', 0, ['alt.binaries.bloaf'], '565912ef$2$43655$b1db1813$ca8b28d9@NewsProvider', -1], [u'FileName [12/73] - "FileName.part11.rar" yEnc (1/21)', 0, ['alt.binaries.bloaf'], '565912ef$3$43577$b1db1813$ca8b28d9@NewsProvider', -1], [u'FileName [12/73] - "FileName.part11.rar" yEnc (1/21)', 0, ['alt.binaries.bloaf'], '565912ef$3$43773$b1db1813$ca8b28d9@NewsProvider', -1], [u'FileName [12/73] - "FileName.part11.rar" yEnc (1/21)', 0, ['alt.binaries.bloaf'], '565912ef$3$43775$b1db1813$ca8b28d9@NewsProvider', -1], [u'iElMIzgEAL8rcr0KAz0KGOE02bd5I
Sat Nov 28 15:19:52 2015	INFO	Completion023: AL8rcr0KAz0KGOE02bd5I4VMf.part20.rar" yEnc (1/21)', 0, ['alt.binaries.bloaf'], '565912f1$3$43629$b1db1813$ca8b28d9@NewsProvider', -1], [u'FileName [22/73] - "FileName.part21.rar" yEnc (1/21)', 0, ['alt.binaries.bloaf'], '565912f2$0$43626$b1db1813$ca8b28d9@NewsProvider', -1], [u'FileName [22/73] - "FileName.part21.rar" yEnc (1/21)', 0, ['alt.binaries.bloaf'], '565912f2$0$43773$b1db1813$ca8b28d9@NewsProvider', -1], [u'FileName [22/73] - "FileName.part21.rar" yEnc (1/21)', 0, ['alt.binaries.bloaf'], '565912f2$1$43655$b1db1813$ca8b28d9@NewsProvider', -1], [u'FileName [22/73] - "FileName.part21.rar" yEnc (1/21)', 0, ['alt.binaries.bloaf'], '565912f2$0$43672$b1db1813$ca8b28d9@NewsProvider', -1], [u'FileName [22/73] - "iElMIzgEAL8rcr0KAz0KGOE02bd5I4V
Sat Nov 28 15:19:52 2015	INFO	Completion023: 1)', 0, ['alt.binaries.bloaf'], '565912f4$1$43577$b1db1813$ca8b28d9@NewsProvider', -1], [u'FileName [31/73] - "FileName.part30.rar" yEnc (1/21)', 0, ['alt.binaries.bloaf'], '565912f4$1$43671$b1db1813$ca8b28d9@NewsProvider', -1], [u'FileName [32/73] - "FileName.part31.rar" yEnc (1/21)', 0, ['alt.binaries.bloaf'], '565912f4$1$43627$b1db1813$ca8b28d9@NewsProvider', -1], [u'FileName [32/73] - "FileName.part31.rar" yEnc (1/21)', 0, ['alt.binaries.bloaf'], '565912f4$1$43754$b1db1813$ca8b28d9@NewsProvider', -1], [u'FileName [32/73] - "FileName.part31.rar" yEnc (1/21)', 0, ['alt.binaries.bloaf'], '565912f4$2$43583$b1db1813$ca8b28d9@NewsProvider', -1], [u'FileName [32/73] - "FileName.part31.rar" yEnc (1/21)', 0, ['alt.binaries.
Sat Nov 28 15:19:52 2015	INFO	Completion023: 71$b1db1813$ca8b28d9@NewsProvider', -1], [u'FileName [41/73] - "FileName.part40.rar" yEnc (1/21)', 0, ['alt.binaries.bloaf'], '565912f6$2$43583$b1db1813$ca8b28d9@NewsProvider', -1], [u'FileName [42/73] - "FileName.part41.rar" yEnc (1/21)', 0, ['alt.binaries.bloaf'], '565912f6$2$43670$b1db1813$ca8b28d9@NewsProvider', -1], [u'FileName [42/73] - "FileName.part41.rar" yEnc (1/21)', 0, ['alt.binaries.bloaf'], '565912f6$2$43677$b1db1813$ca8b28d9@NewsProvider', -1], [u'FileName [42/73] - "FileName.part41.rar" yEnc (1/21)', 0, ['alt.binaries.bloaf'], '565912f6$3$43671$b1db1813$ca8b28d9@NewsProvider', -1], [u'FileName [42/73] - "FileName.part41.rar" yEnc (1/21)', 0, ['alt.binaries.bloaf'], '565912f6$2$43792$b1db1813$ca8b28d9@ne
Sat Nov 28 15:19:52 2015	INFO	Completion023: u'FileName [51/73] - "FileName.part50.rar" yEnc (1/21)', 0, ['alt.binaries.bloaf'], '565912f9$0$43708$b1db1813$ca8b28d9@NewsProvider', -1], [u'FileName [51/73] - "FileName.part50.rar" yEnc (1/21)', 0, ['alt.binaries.bloaf'], '565912f9$0$43582$b1db1813$ca8b28d9@NewsProvider', -1], [u'FileName [52/73] - "FileName.part51.rar" yEnc (1/21)', 0, ['alt.binaries.bloaf'], '565912f9$0$43672$b1db1813$ca8b28d9@NewsProvider', -1], [u'FileName [52/73] - "FileName.part51.rar" yEnc (1/21)', 0, ['alt.binaries.bloaf'], '565912f9$0$43632$b1db1813$ca8b28d9@NewsProvider', -1], [u'FileName [52/73] - "FileName.part51.rar" yEnc (1/21)', 0, ['alt.binaries.bloaf'], '565912f9$0$43670$b1db1813$ca8b28d9@NewsProvider', -1], [u'iElMIzgEAL8rcr0KAz0KG
Sat Nov 28 15:19:52 2015	INFO	Completion023: FileName.part60.rar" yEnc (1/21)', 0, ['alt.binaries.bloaf'], '565912fb$1$43677$b1db1813$ca8b28d9@NewsProvider', -1], [u'FileName [61/73] - "FileName.part60.rar" yEnc (1/21)', 0, ['alt.binaries.bloaf'], '565912fb$1$44005$b1db1813$ca8b28d9@NewsProvider', -1], [u'FileName [62/73] - "FileName.part61.rar" yEnc (1/21)', 0, ['alt.binaries.bloaf'], '565912fb$1$43709$b1db1813$ca8b28d9@NewsProvider', -1], [u'FileName [62/73] - "FileName.part61.rar" yEnc (1/21)', 0, ['alt.binaries.bloaf'], '565912fb$1$43672$b1db1813$ca8b28d9@NewsProvider', -1], [u'FileName [62/73] - "FileName.part61.rar" yEnc (1/21)', 0, ['alt.binaries.bloaf'], '565912fb$2$43648$b1db1813$ca8b28d9@NewsProvider', -1], [u'FileName [62/73] - "iElMIzgEAL8rcr0KAz0KGOE
Sat Nov 28 15:19:52 2015	INFO	Completion023: [V] all active news servers BEFORE filtering on NZBGet ServerX.Group: [['0', '0', 'Server', '563', 'UserName', 'Password', True, '25']]
Sat Nov 28 15:19:52 2015	INFO	Completion023: [V] all active news servers AFTER filtering on NZBGet ServerX.Group: [['0', '0', 'Server', '563', 'UserName', 'Password', True, '25']]
Sat Nov 28 15:19:52 2015	INFO	Completion023: [V] Using SSL connection for server: Server
Sat Nov 28 15:19:52 2015	INFO	Completion023: Exception LINE: 865: 'module' object has no attribute 'SSLContext'
Sat Nov 28 15:19:52 2015	INFO	Completion023: Using server: Server
Is this line the problem?:

Code: Select all

Sat Nov 28 15:19:52 2015   INFO   Completion023: Exception LINE: 865: 'module' object has no attribute 'SSLContext'
Server is using a trusted self signed cert for nzbget.

File gets paused and this is logged (Verbose and Extreme enabled) 60 times a second:

Code: Select all

Completion023: [E] Slow reply from server, waiting 8 ms to avoid looping
Waited 10 minutes and just same error logged thousands of times. Nothing else happens.

Update: Solved problem, but this warning is still being logged a lot.

Thanks,

Will

Update:

Whoops. I just noticed first post says it requires "python 2.7.9".

Sadly, Ubuntu 14.04 TLS is still on Python 2.7.6. Really should add a check into script to verify python version.

SSLContext was introduced in Python 2.7.9. Perhaps have an old way to support old versions. After all, it is Ubuntu LTS version. Thanks.

On the hunt for trusted Ubuntu 14.04 Python 2.7.10 (latest build) PPA.

Update 2:

OK, deadsnakes' PPA appears to be trusted.

Fixed Python Via:

Code: Select all

sudo add-apt-repository ppa:fkrull/deadsnakes-python2.7
sudo apt-get update
sudo apt-get install python2.7
Wow, launchpad.net is slow today: Fetched 27.1 MB in 3min 2s (149 kB/s)

Anyways, problem solved. Now this script is working as expected, of course.

But, what's up still with slow reply warning?

kloaknet
Posts: 335
Joined: 23 Jul 2014, 08:52

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by kloaknet » 29 Nov 2015, 09:15

Thanks for helping yourself solving the issues :D. The 2.7.9 requirement is mentioned in the first topic, and in the script, tbh, I never bothered about adding a check for python versions. Python 3 prolly also dont work. Ill add it to the todo list for the script for further updates.

about the slow server reply warning, thats because the script loops till it gets a reply from the server. If there wouldn't be a pause, and your machine is very fast, it just uses 100% cpu of a core, while not needed:

do
if msg received
exit loop
loop

So I added a small break to avoid useless (very fast) looping. A way to solve it for you is to disable the Extreme and Verbose logging, as all works fine, and you won't be bothered will all the messages.

war59312
Posts: 45
Joined: 12 Feb 2013, 04:52
Location: U.S.A
Contact:

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by war59312 » 29 Nov 2015, 18:22

OK, no worries.

Thanks m8.

kloaknet
Posts: 335
Joined: 23 Jul 2014, 08:52

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by kloaknet » 06 May 2016, 07:31

Updated the script, now you can specify which servers to use for the check on propagation / retention. Also there is a limit option added that gives you the option to force a pause/stop on the check when X% of the articles fail at your main server, to avoid using fill accounts too much. Script update can be downloaded here: http://forum.nzbget.net/viewtopic.php?f=8&t=1736

kloaknet
Posts: 335
Joined: 23 Jul 2014, 08:52

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by kloaknet » 23 May 2016, 09:50

there is a minor issue with the script, Only when deleting an item that is being checked by the script from history, it somehow let crash nzbget without any (debug) notice :oops: This seems only be the case for Nzbget v17 for now. So basically it won't affect nzbget when you don't do manual full history deletes while the spinning nzbget is orange. For automation there is no issue.

JackD
Posts: 16
Joined: 15 Jul 2016, 18:17

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by JackD » 15 Jul 2016, 18:41

Reporting issue and a fix ...

The symptom is that under a certain condition, this module can wait forever in an authentication loop.

Background ... my ISP provides network news where the username and password are required to be blank given that NNTP data is transported over a direct connection to the customer cable network.

The conventional NNTP authentication process flow is in use but instead of sending back "281" Authentication accepted, the NNTP reply here is 501. Thankfully, this is clearly visible by reading the logs when "EXTREME" log mode is enabled, therefore, other users will be able to confirm this response for themselves.

The effect is that this module pauses a download as usual but waits forever for a response that will never arrive -- therefore, the download will never unpause.

A solution: in the module "PP-script Version: 0.2.4", at line 631 reads ...

Code: Select all

        elif server_reply in ('281'):
            # 281 Authentication accepted
Changing the above to ...

Code: Select all

        elif server_reply in ('281', '501'):
            # 281 Authentication accepted
fixes the issue and the module then performs as designed.

Enjoy.

JackD
SickGear
---------------------------

kloaknet
Posts: 335
Joined: 23 Jul 2014, 08:52

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by kloaknet » 16 Jul 2016, 06:56

thanks for reporting the issue.

the 501 msg is acutally an error message in the NNTP commands, so the 501 is not a confirmation you are 'logged in'.

Could you provide the messages and codes prior to your fix, so what does the log report before you get the 501 msg in your logs?

JackD
Posts: 16
Joined: 15 Jul 2016, 18:17

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by JackD » 16 Jul 2016, 11:28

This is a placeholder padding post.

My response continues on the next page because I had to split the
post into two parts due to phpBB forum post length restrictions.
Last edited by JackD on 16 Jul 2016, 11:34, edited 2 times in total.

Post Reply

Who is online

Users browsing this forum: No registered users and 4 guests