[SQS-Script] Completion-Propagation/DMCA/Retention check

Share your scripts or request scripts with specific features.
Forum rules
Please keep the forum clean - one topic per script. Questions not related to a specific script should be posted in Support forum.
dagoob
Posts: 21
Joined: 15 Jan 2016, 23:27

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by dagoob » 20 Dec 2016, 18:19

Crowley wrote:I've set it to use the main server for the checks.
If it's only using the one server then the script is opening the same number of connections in both version of the script, so you won't see the kind of speed change I've observed.

@kloaknet To be clear my "To many connections" have only occurred during checking (immediately after pausing a download). I've not seen it in NZBGet's connections after downloading resumes.

Crowley
Posts: 30
Joined: 22 Nov 2016, 09:43

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by Crowley » 20 Dec 2016, 18:26

dagoob wrote:
Crowley wrote:I've set it to use the main server for the checks.
If it's only using the one server then the script is opening the same number of connections in both version of the script, so you won't see the kind of speed change I've observed.
I get that, but for some reason it is slower. The larger the file, the more the difference is. But it's nothing to worry about, the times are not that dramatic.

Anyway, a log from a recent test sent via PM.

kloaknet
Posts: 336
Joined: 23 Jul 2014, 08:52

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by kloaknet » 21 Dec 2016, 18:14

Find attached some updated test version. However it won't support single line NZB files atm, as I broke that trying to speed up the script :D .

New is the max article limit, and using the QUIT message to tell newsserver the completion script connections are finished, solving the issue that NZBget may fail due to max connection limit to download after the script has ran. It doesnt solve hitting the connection number limit when the script starts. I have posted a question about that for hugbug himself.

Please run the script with VERBOSE on when using this version, this to be able to track some possible errors :shock:.
Last edited by kloaknet on 21 Jan 2017, 11:44, edited 1 time in total.

konubywy
Posts: 16
Joined: 05 Nov 2015, 15:07

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by konubywy » 21 Dec 2016, 19:10

Been running test version 030 for the past day or two and, subjectively speaking, it seems to run A LOT faster than v0.2.10. As for version 031, what exactly do you mean 'it won't support single line NZB files atm'?

konubywy
Posts: 16
Joined: 05 Nov 2015, 15:07

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by konubywy » 21 Dec 2016, 19:21

Also, is there a way to have this script run on nzb files that are added from a RSS feed that was added within nzbget itself (i.e not from sonarr, couchpotato, or manually added) via the RSS Feeds settings page?

kloaknet
Posts: 336
Joined: 23 Jul 2014, 08:52

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by kloaknet » 22 Dec 2016, 07:22

In very rare cases an NZB file might be a file with only a single line of data, with all XML data on a single line. The read that data, the script first needs to split all data before it can read it. That part is not working, but no worries, its very rare.

The script was originally build for RSS feeds, thats why there is also the DUPE functionality in the script. (checking possible ok dupes when the item in the queue fails within the AgeLimit). If you don't specify any Categories for the script, it should just pick up each incoming NZB file and pause it, even if it comes from a feed. If you put your RSS feed NZBs in a typical category (i.e. FeedX.Category), you could also limit it to only that category via the script settings.

Crowley
Posts: 30
Joined: 22 Nov 2016, 09:43

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by Crowley » 22 Dec 2016, 17:27

The New version is faster, but it also tests a lot less articles. Hopefully it doesn't cause issues with propagation, I raised the amount to check since I did get some failures with the default 10%.

kloaknet
Posts: 336
Joined: 23 Jul 2014, 08:52

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by kloaknet » 23 Dec 2016, 08:53

The current test version should not overwrite your settings of your previous version. It should only have the new option that limits the max amount articles to 1000 (default value), or you could increase that to some big number.

When an NZB is DMCA-ed (mainly all articles gone), propagating (all articles missing after a certain point) or out of retention (all articles gone), a low number / % would be sufficient. I am not really familiar with NZBs missing a lot of random articles, or what % is than the best amount to check. Most usenet providers promote with like 99% or more completeness on their servers, so I guess the amount of failures when checking like 10% (each Xth article) should in general be enough. Maybe some others can also drop their experience with the script.

The script does not check if the PAR files are available, so if say max failure ratio is 10%, and 5% of the articles fails, and more than 50% of the PAR blocks are missing, the download would still fail, even though the script checked ok. Maybe this was the case for your failed downloads?
Note that checking if the PAR files are OK is not simply checking the amount of articles. If I understand correctly how PAR files work, a single missing article in each PAR file would make a repair action almost certainly fail, while just maybe less than 5% of the PAR articles are missing. This makes checking of PAR files a bit more complicated than the RAR check currently in the script.

@Crowley, you have a PM ;)

Crowley
Posts: 30
Joined: 22 Nov 2016, 09:43

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by Crowley » 23 Dec 2016, 09:15

Indeed, I missed the new option. I seem to remember having PAR errors, so that could very well be the reason for failures. I guess I'll try the default value again and monitor how it goes.

Sorry about the PM, didn't notice, I will do that test when I have the time.

kloaknet
Posts: 336
Joined: 23 Jul 2014, 08:52

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by kloaknet » 24 Dec 2016, 14:52

Attached version 0.3.2. This version should be stable, as all known issues are solved, and put a lot of tests on it. Feel free to test it a bit too, and if no issues arrise, I will move this version to the first page. Compared to 0.3.1. the hitting max connection issues should be resolved.



If you like this script don't forget to support the guy that made it all possible developping NZBGet, thus by hitting the donate button on this page: http://nzbget.net :twisted:, it is that time of the year isn't it?
Last edited by kloaknet on 21 Jan 2017, 11:44, edited 1 time in total.

Post Reply

Who is online

Users browsing this forum: No registered users and 3 guests