[SQS-Script] Completion-Propagation/DMCA/Retention check

Share your scripts or request scripts with specific features.
Forum rules
Please keep the forum clean - one topic per script. Questions not related to a specific script should be posted in Support forum.
kloaknet
Posts: 337
Joined: 23 Jul 2014, 08:52

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by kloaknet » 19 Dec 2016, 15:15

Crowley wrote:No errors, but the check seems to be slower than before.
Edit:
Or it might be my main news server, the speed have gone randomly up and down the last few days.
Thanks for your reply, the past days I have been mainly working on increasing the speed of the script, so it has become faster ;). Your version already contains a speed improvement compared to the other because it only creates the necessary sockets, instead of all for each news server.

It will soon include 80% faster analysis of the nzb, ignoring a check for servers outside the retention limit as specified in NZBget, a max number of articles that will be checked (usefull for files of like 20 GB +).

Note that each socket creation has a time-out of 2 sec, so when 1 of your connections fail, it add an additional 2 sec per failed connection. Checking about 5% (what i use) takes for me like 2.5 sec in total for a file of 1.5 Gb, without any issues on the news-server end. There might be a news server specific optimum for the specified number of connections available in the NZBget settings, as the script uses those to build the sockets for the script. More sockets take more time to create, more sockets cost less time to check the headers :roll: :lol:, so it is a bit of trade. Note that my general experience with NZBget and downloading, is that more sockets don't always speed up the download, also there is an optimum.

Crowley
Posts: 37
Joined: 22 Nov 2016, 09:43

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by Crowley » 19 Dec 2016, 17:23

Must have been a server issue, I'm using 25% and checking a 30 Gb file took over 5 minutes yesterday but only 70 seconds today. All in all, it's working fine.

konubywy
Posts: 16
Joined: 05 Nov 2015, 15:07

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by konubywy » 19 Dec 2016, 18:22

Noticed nzbget was paused this morning and saw this in the log:

Code: Select all

warning	Mon Dec 19 2016 08:12:45	Completion: Script seems to be running for more than 30 minutes and has most likely crashed. Check your logs and report the log and errors at http://forum.nzbget.net/viewtopic.php?f=8&t=1736
Went investigating and saw this in my logs:

Code: Select all

Mon Dec 19 00:42:12 2016	     INFO    Completion: Using server: news.frugalusenet.com
Mon Dec 19 00:42:12 2016        INFO    Completion: Traceback (most recent call last):
Mon Dec 19 00:42:12 2016        INFO    Completion:   File "/opt/nzbget/scripts/Completion.py", line 1560, in <module>
Mon Dec 19 00:42:12 2016        INFO    Completion:     main()
Mon Dec 19 00:42:12 2016        INFO    Completion:   File "/opt/nzbget/scripts/Completion.py", line 1540, in main
Mon Dec 19 00:42:12 2016        INFO    Completion:     if 'NZBNA_NZBNAME' in os.environ: queue_call()
Mon Dec 19 00:42:12 2016        INFO    Completion:   File "/opt/nzbget/scripts/Completion.py", line 1466, in queue_call
Mon Dec 19 00:42:12 2016        INFO    Completion:     get_prio_nzb(paused_jobs)
Mon Dec 19 00:42:12 2016        INFO    Completion:   File "/opt/nzbget/scripts/Completion.py", line 1411, in get_prio_nzb
Mon Dec 19 00:42:12 2016        INFO    Completion:     if get_nzb_status(nzb):
Mon Dec 19 00:42:12 2016        INFO    Completion:   File "/opt/nzbget/scripts/Completion.py", line 422, in get_nzb_status
Mon Dec 19 00:42:12 2016        INFO    Completion:     failed_ratio = check_failure_status(rar_msg_ids, failed_limit, nzb[2])
Mon Dec 19 00:42:12 2016        INFO    Completion:   File "/opt/nzbget/scripts/Completion.py", line 1166, in check_failure_status
Mon Dec 19 00:42:12 2016        INFO    Completion:     check_send_server_reply(sockets[i], reply, group, id,
Mon Dec 19 00:42:12 2016        INFO    Completion: UnboundLocalError: local variable 'group' referenced before assignment
This is with v0.2.10. Let me know if you need anything else.

kloaknet
Posts: 337
Joined: 23 Jul 2014, 08:52

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by kloaknet » 19 Dec 2016, 19:59

konubywy wrote:Noticed nzbget was paused this morning and saw this in the log:

Code: Select all

warning	Mon Dec 19 2016 08:12:45	Completion: Script seems to be running for more than 30 minutes and has most likely crashed. Check your logs and report the log and errors at http://forum.nzbget.net/viewtopic.php?f=8&t=1736
Went investigating and saw this in my logs:

Code: Select all

Mon Dec 19 00:42:12 2016	     INFO    Completion: Using server: news.frugalusenet.com
Mon Dec 19 00:42:12 2016        INFO    Completion: Traceback (most recent call last):
Mon Dec 19 00:42:12 2016        INFO    Completion:   File "/opt/nzbget/scripts/Completion.py", line 1560, in <module>
Mon Dec 19 00:42:12 2016        INFO    Completion:     main()
Mon Dec 19 00:42:12 2016        INFO    Completion:   File "/opt/nzbget/scripts/Completion.py", line 1540, in main
Mon Dec 19 00:42:12 2016        INFO    Completion:     if 'NZBNA_NZBNAME' in os.environ: queue_call()
Mon Dec 19 00:42:12 2016        INFO    Completion:   File "/opt/nzbget/scripts/Completion.py", line 1466, in queue_call
Mon Dec 19 00:42:12 2016        INFO    Completion:     get_prio_nzb(paused_jobs)
Mon Dec 19 00:42:12 2016        INFO    Completion:   File "/opt/nzbget/scripts/Completion.py", line 1411, in get_prio_nzb
Mon Dec 19 00:42:12 2016        INFO    Completion:     if get_nzb_status(nzb):
Mon Dec 19 00:42:12 2016        INFO    Completion:   File "/opt/nzbget/scripts/Completion.py", line 422, in get_nzb_status
Mon Dec 19 00:42:12 2016        INFO    Completion:     failed_ratio = check_failure_status(rar_msg_ids, failed_limit, nzb[2])
Mon Dec 19 00:42:12 2016        INFO    Completion:   File "/opt/nzbget/scripts/Completion.py", line 1166, in check_failure_status
Mon Dec 19 00:42:12 2016        INFO    Completion:     check_send_server_reply(sockets[i], reply, group, id,
Mon Dec 19 00:42:12 2016        INFO    Completion: UnboundLocalError: local variable 'group' referenced before assignment
This is with v0.2.10. Let me know if you need anything else.
thanks for your reply. Looking at the code it seems rather odd that 'group' had no value defined. It could have been that the NZB file was not containing any groups, or that somewhere and loop is smaller than it should :?: . But when possible provide me with all the completion messages for that NZB prior to the error, and the NZB file. You can sent the data to me using the PM. The NZB you can open with notepad, and paste the content in https://justpaste.it and share the link via PM.

kloaknet
Posts: 337
Joined: 23 Jul 2014, 08:52

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by kloaknet » 20 Dec 2016, 15:50

Guys, as reported by konubywy, the script fails when 0 rar articles are in the NZB. This would mean the NZB file is most likely an extra posted NZB file for repairing an already downloaded NZB.

What is the most logical thing to do with such a file?

- I would assume it is a wrong NZB and mark it BAD or let it fail, so it doesn't waste download time / blocks. I will include some warning message on the NZB file, so the user may understand it got moved to the history as FAILED/BAD.

- or do you guys think that the file should be moved into the download queue (unpaused) so it may will be downloaded by NZBget?

dagoob
Posts: 21
Joined: 15 Jan 2016, 23:27

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by dagoob » 20 Dec 2016, 16:30

But what if the content simply isn't rar'ed? Say it's tiny like an ebook? I'd say if they user added the NZB they wanted to download it: The script should check it, and proceed based on those results.

Incidentally, the 030 testing version is so much faster for me that in my testing a 100% check only takes about as long as a 10% check used to take. :shock: It's so fast now, that it's lead to a few "502 Too many connections" on my first server if it was downloading right before a check. (I added an extra 2 second pause after the 'cool-down loop' which seems to have mostly solved this if you don't want to handle the error more elegantly).

Crowley
Posts: 37
Joined: 22 Nov 2016, 09:43

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by Crowley » 20 Dec 2016, 16:56

For some reason it still isn't any faster for me. Might be due to my low power N3150 CPU. But it works, a recent 1Gb file with 25% check took 19 seconds. Not too bad.

Edit:

Did more testing, always with the same NZB between script versions. 15GB file with 0.30 and the previous version "with a lot of spamming". Previous version manages to check in 60 seconds, new one takes 65. With a 1.7GB file, new version checks in 35 seconds and the older in 3.8 seconds(!). The result was weird, so I deleted and tested again, and now it took 8 seconds with the new version. Lastly I tested a 45 GB file, New one: 99 seconds, old one 82 seconds.

For some reason, the older version is consistently faster for me. Most of the time only by a little amount though.

dagoob
Posts: 21
Joined: 15 Jan 2016, 23:27

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by dagoob » 20 Dec 2016, 17:55

How many servers you have and how many of them are needed to complete the check is a factor.

I have quite a lot of different block accounts some of which are slow. The old version would make all of those connections then maybe pass the check with my main server; the newer version only makes the connections as it needs them so in my case the extra time for creating all those connections is reduced and often removed.

If you've only got, say, two servers and the script needs to connect to them both to pass the check, then it won't be any faster.

kloaknet
Posts: 337
Joined: 23 Jul 2014, 08:52

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by kloaknet » 20 Dec 2016, 17:58

thanks for your useful feedback guys!
dagoob wrote:But what if the content simply isn't rar'ed? Say it's tiny like an ebook? I'd say if they user added the NZB they wanted to download it: The script should check it, and proceed based on those results.

Incidentally, the 030 testing version is so much faster for me that in my testing a 100% check only takes about as long as a 10% check used to take. :shock: It's so fast now, that it's lead to a few "502 Too many connections" on my first server if it was downloading right before a check. (I added an extra 2 second pause after the 'cool-down loop' which seems to have mostly solved this if you don't want to handle the error more elegantly).
I just noticed NZBget handles a file with only par files in it well, so I'll just resume those files now. This would also solve your ebook example. Note that it relies on rar and par files to check for completion, other file types won't be supported. Larger posts should always be rared so it should not affect speed.

The too many connections error in NNTP messages does not exists, some providers use 481, others 482 (those have actually a different meaning), and yours 502 :-), You may noticed the message on that, as it is not covered by the script. So its hard to tackle that one based on replies from the news server.

I noticed several serious 482 issues too on my provider during busy hours, but at less traffic hours, there is no error at all. Besides that, for me there is like 5 sec between the last reply received, and NZBget starting with the download, and it still returns the 482... I have it listed to look for a solution, but :roll: . I solved it to not use all my 30 but just 4 connections, as it does not affect download speed, and reduces load, checks faster on small files. The script closes the sockets also after all messages are received, so its on the news server end maybe that doesn't end the connection. But maybe an additional 1 - 2 sec delay on resuming the NZB might be an option to avoid the connection issues for NZBget, also a similar pause after pausing NZBget might can be implemented, but it will make the script overal ~4 sec slower :-). I will test if I can send the QUIT command to the news server, that might help too (https://tools.ietf.org/html/rfc3977).
Crowley wrote:For some reason it still isn't any faster for me. Might be due to my low power N3150 CPU. But it works, a recent 1Gb file with 25% check took 19 seconds. Not too bad.
May send me some logs (with option verbose on) using a PM, then I can see where the limited speed of the script happens, maybe its the reading, the socket creation, the communication with the news-server, etc.

Crowley
Posts: 37
Joined: 22 Nov 2016, 09:43

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by Crowley » 20 Dec 2016, 18:03

dagoob wrote:How many servers you have and how many of them are needed to complete the check is a factor.

I have quite a lot of different block accounts some of which are slow. The old version would make all of those connections then maybe pass the check with my main server; the newer version only makes the connections as it needs them so in my case the extra time for creating all those connections is reduced and often removed.

If you've only got, say, two servers and the script needs to connect to them both to pass the check, then it won't be any faster.
I have one main server and one block. I've set it to use the main server for the checks. I use 20 connections, it's enough to max out my connection, any more wont help.
kloaknet wrote:
May send me some logs (with option verbose on) using a PM, then I can see where the limited speed of the script happens, maybe its the reading, the socket creation, the communication with the news-server, etc.
Sure, I'll edit this post with the log when done. (Or just send them via pm, like suggested)

Post Reply

Who is online

Users browsing this forum: No registered users and 27 guests