Page 3 of 4

Re: Only download NZB if required - RSS feeds?

Posted: 31 Jul 2018, 12:44
by hugbug
get39678 wrote:
31 Jul 2018, 12:23
The reason of mentioning the Paused state was should any item added to the Queue in the Paused state have the URL downloaded? It should still be duplicate checked/scored and moved to History if required but it is Paused so URL is not needed until it Resumed?
The paused state applies to nzbs only. URLs cannot be paused. This is because pause applies to files within nzb and when all files are paused the nzb item is shown as paused. There is a special flag for URL telling it to pause all files after nzb is fetched. We can change current behaviour so that the add-files-as-paused flag of URL would work as pause for URL instead. That would make it possible to pause/resume URLs. On the other side an URL added via webui with paused checkbox will be added as paused URL. It will not be possible to fetch URL and pause added NZB as it works now.

The change is probably not necessary if we implement "do not start URL fetching until all RSS items are processed".

Re: Only download NZB if required - RSS feeds?

Posted: 31 Jul 2018, 14:08
by get39678
Thanks for the explanation.

I am only making (uninformed) suggestions so please do whatever you want and think is best.

Thinking further you are correct if you implement "do not start URL fetching until all RSS items are processed" it will not really matter. The only possible thing would be if all the matches for the same dupekey are paused will this mean a nzb download as one would end up in paused the queue (and dupes in history) - such a rare case though probably not worth considering.

This is probably better than making it possible to pause URLs especially if it removes other functionality that other users may be using.

Re: Only download NZB if required - RSS feeds?

Posted: 07 Aug 2018, 09:50
by get39678
stack trace
As mentioned above I do not know exactly what causes the problem (most of the time it works fine) but it seems to occur infrequently when a scheduled RSS retrieve is being processed and the items are being processed.

Code: Select all

ERROR	Segmentation fault, tracing...
ERROR	Obtained 11 stack frames
ERROR	/home/user1/nzbget/bin/nzbget(_Z14PrintBacktracev+0x3a) [0x47ba7a]
ERROR	/lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x7fbf452324b0]
ERROR	/home/user1/nzbget/bin/nzbget(_ZN15DupeCoordinator8NzbFoundEP13DownloadQueueP7NzbInfo+0x8bf) [0x4c40cf]
ERROR	/home/user1/nzbget/bin/nzbget(_ZN7Subject6NotifyEPv+0x52) [0x5077b2]
ERROR	/home/user1/nzbget/bin/nzbget(_ZN14UrlCoordinator13AddUrlToQueueESt10unique_ptrI7NzbInfoSt14default_deleteIS1_EEb+0x92) [0x4eafb2]
ERROR	/home/user1/nzbget/bin/nzbget(_ZN15FeedCoordinator13FeedCompletedEP14FeedDownloader+0x458) [0x452fc8]
ERROR	/home/user1/nzbget/bin/nzbget(_ZN7Subject6NotifyEPv+0x52) [0x5077b2]
ERROR	/home/user1/nzbget/bin/nzbget(_ZN13WebDownloader3RunEv+0x1f2) [0x444382]
ERROR	/home/user1/nzbget/bin/nzbget(_ZN6Thread14thread_handlerEPv+0x105) [0x50a095]
ERROR	/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7fbf46c6d6ba]
ERROR	/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fbf453043dd]
"do not start URL fetching until all RSS items are processed"
Apart from this occasional crash and the issue where URLs fetching is started before all the RSS items are processed this feature seems to be working great. Obviously update the thread and I will try any develop version changes.

Thanks

Re: Only download NZB if required - RSS feeds?

Posted: 25 Aug 2018, 16:09
by hugbug
I've made a fix and believe the crash should not occur anymore.

Re: Only download NZB if required - RSS feeds?

Posted: 27 Aug 2018, 09:19
by get39678
I will give it a try and report back if any problems. If it was dependant on the number of items it probably explains why it was so intermittent.

Re: Only download NZB if required - RSS feeds?

Posted: 07 Sep 2018, 22:50
by get39678
hugbug wrote:
25 Aug 2018, 16:09
I've made a fix and believe the crash should not occur anymore.
No crashes since using this version. Can't be 100% sure it is fixed due to it being intermittent but it looks like it is. Thanks
hugbug wrote:
31 Jul 2018, 12:44
The change is probably not necessary if we implement "do not start URL fetching until all RSS items are processed".
I realise your time is limited but if you do get the chance to consider this feature please update the thread. I can see no definite pattern to this I just think the number of unneeded nzbs that are downloaded is just dependant on the speed of the rss provider and perhaps how many simultaneous downloads you have set so setting it not to start URL fetching until all the RSS items are processed should fix it.

Re: Only download NZB if required - RSS feeds?

Posted: 26 Apr 2019, 18:00
by get39678
get39678 wrote:
07 Sep 2018, 22:50
hugbug wrote:
31 Jul 2018, 12:44
The change is probably not necessary if we implement "do not start URL fetching until all RSS items are processed".
I realise your time is limited but if you do get the chance to consider this feature please update the thread. I can see no definite pattern to this I just think the number of unneeded nzbs that are downloaded is just dependant on the speed of the rss provider and perhaps how many simultaneous downloads you have set so setting it not to start URL fetching until all the RSS items are processed should fix it.
hugbug just wondering if any kind of workaround for the "dont start URL fetching until all RSS items are processed" problem has popped in to mind?

It is so much better since this feature was implemented but it "leaking" these few API searches has a big impact when there is a limit on the number of API searches per day.

Re: Only download NZB if required - RSS feeds?

Posted: 01 May 2019, 14:55
by hugbug
I've created issue #630 Start URL fetching only after all RSS items are processed on tracker to not forget about this.

Re: Only download NZB if required - RSS feeds?

Posted: 12 May 2019, 12:11
by kjKmcQN3
Is there a way to revert back to the way NZBGet downloaded NZBs from RSS feeds prior to v21? The reason I would like to do so is that I now have to manually clear duplicate NZBs inadvertently bookmarked on sites in order to stop them from cluttering up the bookmarks section of the site (DOG is a good example.) Earlier, such bookmarks went straight to NZBGet's History and could easily be used/cleared as necessitated, which I found to be a much easier approach.

I tried to see if I could turn off the new behavior by playing around with the setting in NZBGet but couldn't find anything. Thanks for your help and continued efforts on this excellent project.

Re: Only download NZB if required - RSS feeds?

Posted: 12 May 2019, 17:02
by hugbug
@kjKmcQN3:
There is no old/new behaviour switch. There were so many changes in implementing this feature, I'm not sure if I can easily add an old-switch but I'll look into this.