[SQS-Script] Completion-Propagation/DMCA/Retention check

Share your scripts or request scripts with specific features.
Forum rules
Please keep the forum clean - one topic per script. Questions not related to a specific script should be posted in Support forum.
garyleecn
Posts: 25
Joined: 04 Sep 2015, 20:35

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by garyleecn » 23 May 2017, 19:38

kloaknet wrote:
garyleecn wrote:just add this to my setup (nzbget 17.1 on Synology),
and I realize a problem.

when there's a queue in my NZBGet,
it will download the first in the queue (of course), then this task will stuck at "PP-QUEUED" status, until the 2nd task finishes downloading. at which point NZBGet will start post-process (repair, unpack, run PPscripts) on the first task, and start downloading 3rd task. but now the 2nd task will again stuck at "PP-QUEUED" until the 3rd finishes downloading, so on and so forth.

my setup was running smoothly for more than a year, and the only thing I did is adding completion.py as "scan script" and "queue script", the setting in completion.py is untouched as default, and scriptPauseQueus is set to NO.


so any idea on how should I fix this?
thanks :)
Thanks for reporting this feature, I would expect that it is related to the setting scriptPauseQueue setting, but you thought about that already ;) . But there are 2 other options that might be related: UnpackPauseQueue and ParPauseQueue, and i think those also need to be set to No.

I haven't notice the issue (4 cores, 3 for unpack, all above 3 settings on NO), but I think it happens because:
- when a complete download is finished,
- no other items are downloading,
- the script gets the first position in the PP-queue list, (par check, unpack and scripts),
- the script goes full throttle for the next NZB to check as first item in the PP-queue,
- unpauses an nzb
- and NZBget thinks, lets download instead of doing some PP-ing like unpack.

But I am not 100% sure, please provide some logs without the above 2 options on your values, some logs with the 2 options above on No.
How many cores does your NAS have?

If that doesnt help, we may need hugbug, as I can't see an option to prioritize unpacking or use an NZBNA_EVENT that triggers after the PP-scripts are done. I start my script on these options:

Code: Select all

    event = os.environ['NZBNA_EVENT']
    if (event == 'NZB_ADDED' or event == 'NZB_DOWNLOADED' or event == 'NZB_DELETED' or event == 'NZB_MARKED'):
The issue probably is related to the last section on this page: https://github.com/nzbget/nzbget/wiki/Queue-scripts



Important side note, if you don't let the script also run as a scheduler script'(NZBget versions below 18), it could worse the'availability due to possible take downs. If the nzb is not there yet while it is propagating, the script checks it, and notices it is not ~90% there, and leaves the nzb paused. If now no other nzbs are added for say 8 hours, the nzb would NOT be checked again in this time window, and it could easily be taken down in the 8 hour time window. The scheduler part is of real importance to have the script working for propagation issues!

oddly enough, it kinda fixed itself...
what I did is, remove completion script from queue/scan scripts (btw, this is the only script I use, besides another PP script for some certain categories), reload the NZBGet, then add them back, reload NZBGet again, now it is working just fine...

btw, is it possible to have it ignore the age, just move from top to bottom of the queue?

kloaknet
Posts: 337
Joined: 23 Jul 2014, 08:52

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by kloaknet » 25 May 2017, 08:59

garyleecn wrote:
oddly enough, it kinda fixed itself...
what I did is, remove completion script from queue/scan scripts (btw, this is the only script I use, besides another PP script for some certain categories), reload the NZBGet, then add them back, reload NZBGet again, now it is working just fine...

btw, is it possible to have it ignore the age, just move from top to bottom of the queue?

For very small files I have seen happen something similar, but can't put my finger behind it. It is related to how the PP actions are queued/prioritized in NZBget. If you walk into the issue again, please drop some logs, if needed via PM.

I am not happy with adding the option to respect NZBget queue order, and have seen it asked before. I hope below explains it:
- Option AgeLimit is there to prioritize items with a high risk of being DMCA-ed, therefore this value should be max up to 24 h
- Option AgeLimit is there to keep retrying items that are propagating due to a bad provider, propagation delay could easily be 4 hours (Astraweb :evil:) , so a good minimum would be 4 hours.
- Files that are (some days) outside the AgeLimit, have a very low risk of being taken down if they are still there, so no need to rush to grab them prior to files within AgeLimit.
- Older files within AgeLimit will have more chance on being DMCA-ed, and less chance on having propagating issues.
- Most stuff you download would most likely be automated, so the NZBs are grabbed when they arrive, and have a very limited file age, and within The AgeLimit.
- Automated stuff arrives in your queue when you are at work, sleep, whatever, so prioritizing on availability (age) is more important than queue order.
- If you add manually items, and you want them to be downloaded first, increase their priority in NZBget, you are already doing stuff manually. If you don't want to do it manually, lower the priority of automatically added items (via Sonarr, RSS feeds, ...)
- Why download and old item (manually added) first, when young items are in the queue with a high risk of being taken down.
- Furthermore, for series / episodes posted by the same poster, they are 99% of the time in the episode order, so the oldest, first ep, will be grabbed first.


In short, not respecting age, just like people not adding the scheduler settings, would affect the effectiveness of the script. To me it feels like people not understanding the principle/goal of the script. You will have the same effect by simply using the fixed propagation delay in NZBget. Just respecting priority without age can be added, but I need a good reason from you guys to add it, and support testing it. May also take a good look at the code, read the comments in it explaining the choices I made.

garyleecn
Posts: 25
Joined: 04 Sep 2015, 20:35

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by garyleecn » 25 May 2017, 21:47

kloaknet wrote:
garyleecn wrote:
oddly enough, it kinda fixed itself...
what I did is, remove completion script from queue/scan scripts (btw, this is the only script I use, besides another PP script for some certain categories), reload the NZBGet, then add them back, reload NZBGet again, now it is working just fine...

btw, is it possible to have it ignore the age, just move from top to bottom of the queue?

For very small files I have seen happen something similar, but can't put my finger behind it. It is related to how the PP actions are queued/prioritized in NZBget. If you walk into the issue again, please drop some logs, if needed via PM.

I am not happy with adding the option to respect NZBget queue order, and have seen it asked before. I hope below explains it:
- Option AgeLimit is there to prioritize items with a high risk of being DMCA-ed, therefore this value should be max up to 24 h
- Option AgeLimit is there to keep retrying items that are propagating due to a bad provider, propagation delay could easily be 4 hours (Astraweb :evil:) , so a good minimum would be 4 hours.
- Files that are (some days) outside the AgeLimit, have a very low risk of being taken down if they are still there, so no need to rush to grab them prior to files within AgeLimit.
- Older files within AgeLimit will have more chance on being DMCA-ed, and less chance on having propagating issues.
- Most stuff you download would most likely be automated, so the NZBs are grabbed when they arrive, and have a very limited file age, and within The AgeLimit.
- Automated stuff arrives in your queue when you are at work, sleep, whatever, so prioritizing on availability (age) is more important than queue order.
- If you add manually items, and you want them to be downloaded first, increase their priority in NZBget, you are already doing stuff manually. If you don't want to do it manually, lower the priority of automatically added items (via Sonarr, RSS feeds, ...)
- Why download and old item (manually added) first, when young items are in the queue with a high risk of being taken down.
- Furthermore, for series / episodes posted by the same poster, they are 99% of the time in the episode order, so the oldest, first ep, will be grabbed first.


In short, not respecting age, just like people not adding the scheduler settings, would affect the effectiveness of the script. To me it feels like people not understanding the principle/goal of the script. You will have the same effect by simply using the fixed propagation delay in NZBget. Just respecting priority without age can be added, but I need a good reason from you guys to add it, and support testing it. May also take a good look at the code, read the comments in it explaining the choices I made.
I totally understand your stand point, minimize the risk of DMCA.
however, one draw back of this is, when you have a large queue, say pages of tasks. it happens when you add an old show to your playlist, and it will need to download everything to date all together. and then since the script completely disregards the queue order, we will have no control (well, we can set priorities, but talk about that later) over which one to download first. normally I'll want 1st season to be downloaded first so I can start watch or something, but maybe the 5th season has the oldest age and will be downloaded first, or maybe s07e05 will be the oldest, I don't know.
also, because I don't know which task is being downloaded, it can be buried in pages of tasks, I can't monitor it. which means if it stalls, there's no way I can no unless I manually look through pages.
and I know I can set priorities and sort tasks by age, but when a task fails, sonarr/radarr or similar will re-add another task, which leads back to square 1...

kloaknet
Posts: 337
Joined: 23 Jul 2014, 08:52

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by kloaknet » 26 May 2017, 07:32

garyleecn wrote: I totally understand your stand point, minimize the risk of DMCA.
however, one draw back of this is, when you have a large queue, say pages of tasks. it happens when you add an old show to your playlist, and it will need to download everything to date all together. and then since the script completely disregards the queue order, we will have no control (well, we can set priorities, but talk about that later) over which one to download first. normally I'll want 1st season to be downloaded first so I can start watch or something, but maybe the 5th season has the oldest age and will be downloaded first, or maybe s07e05 will be the oldest, I don't know.
also, because I don't know which task is being downloaded, it can be buried in pages of tasks, I can't monitor it. which means if it stalls, there's no way I can no unless I manually look through pages.
and I know I can set priorities and sort tasks by age, but when a task fails, sonarr/radarr or similar will re-add another task, which leads back to square 1...
Some possible options I came up with based on your input:
- When your download queue is longer than 10 items, an override should be in place to ignore age? (Causes possible failure of recent items due to DMCA)
- Or 2 inputs for the AgeLimit, 1 for the lower bound covering propagation (~2 - ~8 hours), 1 for the upper bound for DMCA (~4 - ~24 hours), and then all beyond this window, respect queue order and priority, remove/change the PrioritizeOld option? (would mean new stuff will still get priority, but old stuff will follow queue).

My guess is that when 1 item fails in Sonarr, a new item will be added at the bottom of the queue, so there it won't help ;) that one is up to markus101 from Sonarr to fix then :D.

barenaked
Posts: 21
Joined: 20 May 2017, 13:27

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by barenaked » 02 Jun 2017, 15:29

Hey kloaknet, have you ever encountered that the nzb filename check fails for passworded nzbs used by pfftn script?

The files are named like this: Title{{password}}

Code: Select all

INFO	Fri Jun 02 2017 14:32:20	Completion: Overall check completed in 0.09 sec.
WARNING	Fri Jun 02 2017 14:32:20	Completion: The NZB file Title{{password does not seem to exist, resuming NZB.
ERROR	Fri Jun 02 2017 14:32:20	Completion: No such nzb file.
INFO	Fri Jun 02 2017 14:32:20	Completion: [V] get_nzb_data(fname=/volume1/downloads/nzbget/nzb/Title{{password)
INFO	Fri Jun 02 2017 14:32:20	Completion: Checking: "Title{{password"
INFO	Fri Jun 02 2017 14:32:20	Completion: [V] get_nzb_status(nzb=[26, 'Title{{password', 1430410026, 949, 'NONE', 0])
INFO	Fri Jun 02 2017 14:32:20	Completion: [V] * Title{{password, Age: 18332.4 hours, Priority: 0
INFO	Fri Jun 02 2017 14:32:20	Completion: [V] Paused and SORTED NZBs in queue that will be processed:
INFO	Fri Jun 02 2017 14:32:20	Completion: [V] Ignoring sorting priority of items older than AgeLimit of 4 hours
INFO	Fri Jun 02 2017 14:32:20	Completion: [V] * Title{{password, Age: 18332.4 hours, Priority: 0
INFO	Fri Jun 02 2017 14:32:20	Completion: [V] Paused UNSORTED NZBs in queue that will be processed:
INFO	Fri Jun 02 2017 14:32:20	Completion: [V] Downloading for NZBGet paused
Looks like it's either not understanding the two "}}" at the end of the filename or its still looking for the part "{{password}}" which has been removed by pfftn after applying the password in nzbget.

Pfftn is running before completion does, should I change the order of the scripts or is it something that needs to be fixed?

kloaknet
Posts: 337
Joined: 23 Jul 2014, 08:52

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by kloaknet » 03 Jun 2017, 07:02

barenaked wrote:
02 Jun 2017, 15:29
Hey kloaknet, have you ever encountered that the nzb filename check fails for passworded nzbs used by pfftn script?

Looks like it's either not understanding the two "}}" at the end of the filename or its still looking for the part "{{password}}" which has been removed by pfftn after applying the password in nzbget.

Pfftn is running before completion does, should I change the order of the scripts or is it something that needs to be fixed?
When an nzb is paused (directly after adding, using the scan script option), it also stores the name of the nzbfile into nzbget as a variable. In this way the script knows which files belong to a typical nzb, not all file info is otherwise available in the NZBget queue data (extensions nzb.queued, nzb.2.queued, _2.nzb.queued). So the script relies on this when it is asked to do a scan. But from what I understand the pfftn script does do a rename, and then cannot find the nzb anymore.

I would advice to set the completion script to be executed before the pfftn script, so the file name is still the same as when it was added. The password is not needed for the completion script.

But I am not sure if the above solves the issue, as the }} clearly seems to be missing when trying to retrieve the nzb anyway. Could you provide the part just before your log data too, so the completion message '[V] scan_call()" and '[V] Expected queued file name: xxxxxx ?

kloaknet
Posts: 337
Joined: 23 Jul 2014, 08:52

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by kloaknet » 03 Jun 2017, 07:37

update on the above:

if you change line 348, that has this code:

Code: Select all

    loc = s.find('}')
with this:

Code: Select all

    loc = s.rfind('}')
It most likely would solve your issue. Will be added in a future version too.

kloaknet
Posts: 337
Joined: 23 Jul 2014, 08:52

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by kloaknet » 03 Jun 2017, 09:36

kloaknet wrote:
26 May 2017, 07:32
Some possible options I came up with based on your input:
- When your download queue is longer than 10 items, an override should be in place to ignore age? (Causes possible failure of recent items due to DMCA)
- Or 2 inputs for the AgeLimit, 1 for the lower bound covering propagation (~2 - ~8 hours), 1 for the upper bound for DMCA (~4 - ~24 hours), and then all beyond this window, respect queue order and priority, remove/change the PrioritizeOld option? (would mean new stuff will still get priority, but old stuff will follow queue).
Any comment on the option you guys prefer, or is the option not needed anymore?

barenaked
Posts: 21
Joined: 20 May 2017, 13:27

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by barenaked » 03 Jun 2017, 11:49

Thank you so much for the tip. When I'm home I'll try your suggestions and report back.

I'll also look into the log if the part you ask for is still there.

Here is the rest of the log

Code: Select all

INFO	Fri Jun 02 2017 14:32:20	Completion: [V] get_nzb_data(fname=/nzbget/nzb/Title{{password)
INFO	Fri Jun 02 2017 14:32:20	Completion: Checking: "Title{{password"
INFO	Fri Jun 02 2017 14:32:20	Completion: [V] get_nzb_status(nzb=[26, 'Title{{password', 1430410026, 949, 'NONE', 0])
INFO	Fri Jun 02 2017 14:32:20	Completion: [V] * Title{{password, Age: 18332.4 hours, Priority: 0
INFO	Fri Jun 02 2017 14:32:20	Completion: [V] Paused and SORTED NZBs in queue that will be processed
INFO	Fri Jun 02 2017 14:32:20	Completion: [V] Ignoring sorting priority of items older than AgeLimit of 4 hours
INFO	Fri Jun 02 2017 14:32:20	Completion: [V] * Title{{password, Age: 18332.4 hours, Priority: 0
INFO	Fri Jun 02 2017 14:32:20	Completion: [V] Paused UNSORTED NZBs in queue that will be processed:
INFO	Fri Jun 02 2017 14:32:20	Completion: [V] Downloading for NZBGet paused
INFO	Fri Jun 02 2017 14:32:20	Completion: [V] Waiting for NZBGet to end downloading
INFO	Fri Jun 02 2017 14:32:20	Completion: [V] nzbget_paused()
INFO	Fri Jun 02 2017 14:32:20	Completion: [V] New completion.lock file created.
INFO	Fri Jun 02 2017 14:32:20	Completion: [V] server_time= 1496406740
INFO	Fri Jun 02 2017 14:32:20	Completion: [V] lock_file()
INFO	Fri Jun 02 2017 14:32:20	Completion: [V] queue_call()
INFO	Fri Jun 02 2017 14:32:20	Executing queue-script Completion.py for Title
OK so one of the two did the trick, I'll change the pp script order next time to see what the code change did. But so far no error, maybe a hint to run your script before others that change the nzb in the script tutorial could help.

kloaknet
Posts: 337
Joined: 23 Jul 2014, 08:52

Re: [SQS-Script] Completion-Propagation/DMCA/Retention check

Post by kloaknet » 05 Jun 2017, 10:27

My best guess is that NZBget does not change the actual file name, but only the name within NZBget. That it works now, as u mentioned above, seems to support that. So no reordering of the scripts needed.

Post Reply

Who is online

Users browsing this forum: No registered users and 29 guests