Only download NZB if required - RSS feeds?

Discuss newly added features or request new features.
hugbug
Developer & Admin
Posts: 7418
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: Only download NZB if required - RSS feeds?

Post by hugbug » 31 Jul 2018, 12:44

get39678 wrote:
31 Jul 2018, 12:23
The reason of mentioning the Paused state was should any item added to the Queue in the Paused state have the URL downloaded? It should still be duplicate checked/scored and moved to History if required but it is Paused so URL is not needed until it Resumed?
The paused state applies to nzbs only. URLs cannot be paused. This is because pause applies to files within nzb and when all files are paused the nzb item is shown as paused. There is a special flag for URL telling it to pause all files after nzb is fetched. We can change current behaviour so that the add-files-as-paused flag of URL would work as pause for URL instead. That would make it possible to pause/resume URLs. On the other side an URL added via webui with paused checkbox will be added as paused URL. It will not be possible to fetch URL and pause added NZB as it works now.

The change is probably not necessary if we implement "do not start URL fetching until all RSS items are processed".

get39678
Posts: 219
Joined: 09 Jun 2014, 10:49

Re: Only download NZB if required - RSS feeds?

Post by get39678 » 31 Jul 2018, 14:08

Thanks for the explanation.

I am only making (uninformed) suggestions so please do whatever you want and think is best.

Thinking further you are correct if you implement "do not start URL fetching until all RSS items are processed" it will not really matter. The only possible thing would be if all the matches for the same dupekey are paused will this mean a nzb download as one would end up in paused the queue (and dupes in history) - such a rare case though probably not worth considering.

This is probably better than making it possible to pause URLs especially if it removes other functionality that other users may be using.

get39678
Posts: 219
Joined: 09 Jun 2014, 10:49

Re: Only download NZB if required - RSS feeds?

Post by get39678 » 07 Aug 2018, 09:50

stack trace
As mentioned above I do not know exactly what causes the problem (most of the time it works fine) but it seems to occur infrequently when a scheduled RSS retrieve is being processed and the items are being processed.

Code: Select all

ERROR	Segmentation fault, tracing...
ERROR	Obtained 11 stack frames
ERROR	/home/user1/nzbget/bin/nzbget(_Z14PrintBacktracev+0x3a) [0x47ba7a]
ERROR	/lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x7fbf452324b0]
ERROR	/home/user1/nzbget/bin/nzbget(_ZN15DupeCoordinator8NzbFoundEP13DownloadQueueP7NzbInfo+0x8bf) [0x4c40cf]
ERROR	/home/user1/nzbget/bin/nzbget(_ZN7Subject6NotifyEPv+0x52) [0x5077b2]
ERROR	/home/user1/nzbget/bin/nzbget(_ZN14UrlCoordinator13AddUrlToQueueESt10unique_ptrI7NzbInfoSt14default_deleteIS1_EEb+0x92) [0x4eafb2]
ERROR	/home/user1/nzbget/bin/nzbget(_ZN15FeedCoordinator13FeedCompletedEP14FeedDownloader+0x458) [0x452fc8]
ERROR	/home/user1/nzbget/bin/nzbget(_ZN7Subject6NotifyEPv+0x52) [0x5077b2]
ERROR	/home/user1/nzbget/bin/nzbget(_ZN13WebDownloader3RunEv+0x1f2) [0x444382]
ERROR	/home/user1/nzbget/bin/nzbget(_ZN6Thread14thread_handlerEPv+0x105) [0x50a095]
ERROR	/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7fbf46c6d6ba]
ERROR	/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fbf453043dd]
"do not start URL fetching until all RSS items are processed"
Apart from this occasional crash and the issue where URLs fetching is started before all the RSS items are processed this feature seems to be working great. Obviously update the thread and I will try any develop version changes.

Thanks

hugbug
Developer & Admin
Posts: 7418
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: Only download NZB if required - RSS feeds?

Post by hugbug » 25 Aug 2018, 16:09

I've made a fix and believe the crash should not occur anymore.

get39678
Posts: 219
Joined: 09 Jun 2014, 10:49

Re: Only download NZB if required - RSS feeds?

Post by get39678 » 27 Aug 2018, 09:19

I will give it a try and report back if any problems. If it was dependant on the number of items it probably explains why it was so intermittent.

get39678
Posts: 219
Joined: 09 Jun 2014, 10:49

Re: Only download NZB if required - RSS feeds?

Post by get39678 » 07 Sep 2018, 22:50

hugbug wrote:
25 Aug 2018, 16:09
I've made a fix and believe the crash should not occur anymore.
No crashes since using this version. Can't be 100% sure it is fixed due to it being intermittent but it looks like it is. Thanks
hugbug wrote:
31 Jul 2018, 12:44
The change is probably not necessary if we implement "do not start URL fetching until all RSS items are processed".
I realise your time is limited but if you do get the chance to consider this feature please update the thread. I can see no definite pattern to this I just think the number of unneeded nzbs that are downloaded is just dependant on the speed of the rss provider and perhaps how many simultaneous downloads you have set so setting it not to start URL fetching until all the RSS items are processed should fix it.

get39678
Posts: 219
Joined: 09 Jun 2014, 10:49

Re: Only download NZB if required - RSS feeds?

Post by get39678 » 26 Apr 2019, 18:00

get39678 wrote:
07 Sep 2018, 22:50
hugbug wrote:
31 Jul 2018, 12:44
The change is probably not necessary if we implement "do not start URL fetching until all RSS items are processed".
I realise your time is limited but if you do get the chance to consider this feature please update the thread. I can see no definite pattern to this I just think the number of unneeded nzbs that are downloaded is just dependant on the speed of the rss provider and perhaps how many simultaneous downloads you have set so setting it not to start URL fetching until all the RSS items are processed should fix it.
hugbug just wondering if any kind of workaround for the "dont start URL fetching until all RSS items are processed" problem has popped in to mind?

It is so much better since this feature was implemented but it "leaking" these few API searches has a big impact when there is a limit on the number of API searches per day.

hugbug
Developer & Admin
Posts: 7418
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: Only download NZB if required - RSS feeds?

Post by hugbug » 01 May 2019, 14:55

I've created issue #630 Start URL fetching only after all RSS items are processed on tracker to not forget about this.

kjKmcQN3
Posts: 75
Joined: 29 Aug 2015, 15:09

Re: Only download NZB if required - RSS feeds?

Post by kjKmcQN3 » 12 May 2019, 12:11

Is there a way to revert back to the way NZBGet downloaded NZBs from RSS feeds prior to v21? The reason I would like to do so is that I now have to manually clear duplicate NZBs inadvertently bookmarked on sites in order to stop them from cluttering up the bookmarks section of the site (DOG is a good example.) Earlier, such bookmarks went straight to NZBGet's History and could easily be used/cleared as necessitated, which I found to be a much easier approach.

I tried to see if I could turn off the new behavior by playing around with the setting in NZBGet but couldn't find anything. Thanks for your help and continued efforts on this excellent project.

hugbug
Developer & Admin
Posts: 7418
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: Only download NZB if required - RSS feeds?

Post by hugbug » 12 May 2019, 17:02

@kjKmcQN3:
There is no old/new behaviour switch. There were so many changes in implementing this feature, I'm not sure if I can easily add an old-switch but I'll look into this.

Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests