[New Feature] Simultaneous NZB downloading

Discuss newly added features or request new features.
patrioticaction
Posts: 3
Joined: 17 Feb 2016, 05:35

[New Feature] Simultaneous NZB downloading

Post by patrioticaction » 17 Feb 2016, 06:15

Usenet providers move old binaries (>1000 days) to "cold" storage tiers, which are typically high-capacity, rotational disks. The nature of rotational disk is that its capacity scales exponentially in relation to the time to store and retrieve files. This behavior causes old binaries to download very slowly relative to the available bandwidth (<5MB/sec download for >15MB/sec connection). This problem will continue to be exacerbated for perhaps the next 5-10 years due to the confluence of four factors: increasing binary sizes, increasing binary retention, increasing internet bandwidth, and HDD price per GB being lower than SSD.

While we cannot do much to influence how Usenet providers store their binaries, the remainder of unused bandwidth could be put towards the completion of other binaries on other, similarly slow storage tiers as consolation. NZBGet has the potential to lead the market in this respect by adding support for the concurrent download of multiple NZBs. The alternative solution (multiple instances of NZBGet) is cumbersome, feels wrong at a philosophical level and seemingly proves out the need for such a feature.

Please consider this request. If there is significant support, I am willing to contribute the code to make it happen and submit a pull request on GitHub.
Last edited by patrioticaction on 17 Feb 2016, 09:43, edited 1 time in total.

hugbug
Developer & Admin
Posts: 7539
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: [New Feature] Simultaneous NZB downloading

Post by hugbug » 17 Feb 2016, 06:31

What are the criterias to decide how to split available connections between nzbs?
If you take away connections from the top nzb that will make it download even slower and compromise the whole idea of queue management (order of nzbs and their priorities).

patrioticaction
Posts: 3
Joined: 17 Feb 2016, 05:35

Re: [New Feature] Simultaneous NZB downloading

Post by patrioticaction » 17 Feb 2016, 09:20

If I understand you correctly, the main concern here is that you intend for the top priority NZB to get all of the available bandwidth so that it finishes as soon as possible. I agree that this should take precedence over any other NZB. However, if there is a surplus of bandwidth, then we should allow some of the connections to be allocated for the downloading of lower priority NZBs to increase the total download rate without diminishing higher priority NZBs download rates.

User configuration
Considering the steep diminishing returns of connections per NZB, the user could set a minimum number of connections per NZB (e.g. 4 or 5 connections) as well as the maximum number of simultaneous NZB downloads allowed.

Increasing from one to N
Whenever there is an assumed surplus of bandwidth (i.e. the difference between some rolling average of the current total download rate and the speed limit is >5%) the following logic will apply. Since the real surplus of bandwidth is an unknown quantity, we will first need to make a prediction about the download rate of the highest priority NZB(s) and the total download rate after allocating a connection to a lower priority NZB (e.g. download rate of the highest priority NZB(s) is >99% of previous measurement and total download rate is >101% of previous measurement). Then, we test the prediction by reallocating one connection (after the article finishes downloading) from the highest priority NZB(s) to the NZB with the next highest priority, waiting 10 seconds, and checking to see if the outcome is within our predicted thresholds for one of two cases:
1) If the outcome is inside the predicted thresholds, then we continue the process of reallocating additional connections to the NZB with the next highest priority until there is no significant benefit to the total download rate -- at which point we will allocate connections to an even lower priority NZB (unless it would violate the "minimum number of connections per NZB" or "maximum number of simultaneous NZB downloads" rules specified by the user).
2) If the outcome is outside of the predicted thresholds, then we give the connection back (discarding any incomplete article download) to the higher priority NZB and wait for an exponentially increasing time to test the prediction again (starting at say 10 seconds and ending at 1800 seconds, resetting when the highest priority NZB finishes).

Bandwidth Squeeze
If at any time the total download rate decreases in a significant way (e.g. total download rate is <99% of previous measurement) then we will remove a connection (discarding any incomplete article download) from the lowest priority NZB and allocate it to the highest priority NZB. Then, we will check to see if allocating this connection to the highest priority NZB has increased that NZB's download rate (e.g. highest NZB download rate is >101% of previous measurement). If reallocating this connection has increased the download rate of the highest priority NZB in a significant way after 10 seconds, then we will continue reallocating connections. This check will run frequently (every 5 seconds).

Miscellaneous
1) Each connection will be considered unique so as to provide a way to distinguish connections which have recently been reallocated due to the "bandwidth squeeze" algorithm so that the "increasing from one to n" algorithm will not reallocate a second connection.
2) When the highest priority NZB finishes downloading, all of its connections will be allocated to the next highest priority NZB.

Thoughts?

hugbug
Developer & Admin
Posts: 7539
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: [New Feature] Simultaneous NZB downloading

Post by hugbug » 17 Feb 2016, 18:52

Pretty complex mechanics here.

The feature will improve bandwidth usage only in certain situations, which is: a queue containing both slow and high-speed items. Moreover these nzbs must be intermixed; otherwise if the slow and high-speed nzbs are grouped together trying of the next nzb will not help.

From this standpoint - how real are such situations? You either download very recent nzbs (through automation) of you need to fill the backlog. Recent nzbs is no issue here. If you backfill a show you would have a list of episodes, which most likely be all of the same speed. Therefore the remaining use case for the feature is backfilling of movies, where each nzb can be of different age.
patrioticaction wrote:if there is a surplus of bandwidth, then we should allow some of the connections to be allocated for the downloading of lower priority NZBs
Removing connections from top nzb will reduce the speed of this nzb, resulting in even longer download time. At the end the time required to download the whole queue will not improve.

patrioticaction
Posts: 3
Joined: 17 Feb 2016, 05:35

Re: [New Feature] Simultaneous NZB downloading

Post by patrioticaction » 18 Feb 2016, 02:57

hugbug wrote:Pretty complex mechanics here.
This is perhaps the most important point in this thread. It is not clear how many edge cases will be surfaced by adding this functionality and could become a real nightmare to debug. Is the technical debt worth having the functionality (assuming it works 100% of the time)?
hugbug wrote:The feature will improve bandwidth usage only in certain situations, which is: a queue containing both slow and high-speed items. Moreover these nzbs must be intermixed; otherwise if the slow and high-speed nzbs are grouped together trying of the next nzb will not help.

From this standpoint - how real are such situations? You either download very recent nzbs (through automation) of you need to fill the backlog. Recent nzbs is no issue here. If you backfill a show you would have a list of episodes, which most likely be all of the same speed. Therefore the remaining use case for the feature is backfilling of movies, where each nzb can be of different age.
The compelling use case for such a feature, which you have identified, is when you are backfilling older content. More specifically, this feature is only useful when the top of your queue contains multiple, older NZBs. The algorithm, when faced with a situation where the top of your queue is a new NZBs, will bias towards giving all available connections to the highest priority NZB.
hugbug wrote:Removing connections from top nzb will reduce the speed of this nzb, resulting in even longer download time. At the end the time required to download the whole queue will not improve.
This is assuming that the top NZB is recent (<60 days). Also, the top NZB could have 5, 10, 20 or even 50 connections allocated to it. After 5-10 connections are allocated the increase in speed for each additional connection is nearly imperceptible.

The algorithm would, in the worst case, cause 10 seconds of diminished speed for the top NZB, assuming that NZB was utilizing all available bandwidth (and then again every n seconds after the completion of one article, with n increasing exponentially to the upper-bound of 30 minutes). If the top NZB speed was close to the speed limit, then no action would be taken and therefore the top NZB speed would be unchanged.

Thanks for thinking through this with me, I can see why this wasn't something originally included in nzbget or sabnzbd. It's really hard to get it right for a small win.

hugbug
Developer & Admin
Posts: 7539
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: [New Feature] Simultaneous NZB downloading

Post by hugbug » 18 Feb 2016, 08:33

There might be another way to improve speed for old nzbs. This depends however of the reasons causing slow speed, which I'd like to clarify.

For my Highwind account the speed seems to be the same regardless of nzb age. I can always saturate my 100 Mbps connection.

So what is the issue with hard disks? Is it the latency or the raw read speed is also slow? If the problem is the latency only it should be possible to mitigate the issue by increasing the amount of connections. An example: let's say for recent nzbs you can saturate your bandwidth with 10 connections. To download one article you need (for example) one second. Assume a latency for old articles is half a second. In that case the download speed will be significantly decreased - 25-50% slower than normal. But if you increase the number of connections to 15 or even more that should bring the speed back to normal. Many connections will wait for response but you will always have 10 connections downloading at full speed.

Once again, this will work only if the raw transfer speed is the same once the article is fetched from the storage. I can't test it but if you have a provider where you can reproduce the issue you could make tests with different number of connections and post your results.

Typically I suggest to use as few connections as possible because the higher number of threads increase the thread management overhead and the overall system responsiveness suffers. Therefore it's better to use 10 connections if that's enough to saturate the line, even if the provider allows 50 connections. Nonetheless we are talking about tests here. The results may help to develop a solution for older nzbs which will not necessary mean a high number of connections for high-speed nzbs.

wolfpack
Posts: 9
Joined: 04 Aug 2013, 14:14

Re: [New Feature] Simultaneous NZB downloading

Post by wolfpack » 21 Feb 2016, 02:29

I love the level of technical details involved in this discussion ...

I do have to say though ... I have not seen that much difference when downloading old posts (old as in 4+ years old).

I'm usually a fan of KISS, the simpler the better ... and although this is a complex change, I can't help but wonder what the completed execution would look like :)

neilt0
Posts: 261
Joined: 10 Sep 2008, 19:11

Re: [New Feature] Simultaneous NZB downloading

Post by neilt0 » 22 Feb 2016, 05:54

When you have a fast connection the differences in download speed are noticeable.

When my line speed was 20mbps, download speeds never changed, no matter how old the NZB.

With a 150mbps connection, new stuff comes in at 18.5MB/sec, older stuff as low as 4MB/sec.

NSPs: SuperNews and Astraweb.

hugbug
Developer & Admin
Posts: 7539
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: [New Feature] Simultaneous NZB downloading

Post by hugbug » 22 Feb 2016, 11:13

@everyone who has speed issues with older nzbs:
  • Is this reproducible with the same nzb?
  • What is your normal connection number configured?
  • Please make tests with different number of connections (1,2,4,8, etc. up to max number allowed by provider) and post speed results.

neilt0
Posts: 261
Joined: 10 Sep 2008, 19:11

Re: [New Feature] Simultaneous NZB downloading

Post by neilt0 » 24 Feb 2016, 05:19

Yes, it's reproducible and maxing out connections won't fix it (already at max connections - 30).

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest