Slow download speed with large queue

Get help, report and discuss bugs.
mittens
Posts: 6
Joined: 30 Aug 2017, 00:11

Slow download speed with large queue

Post by mittens » 30 Aug 2017, 02:02

Hey everyone, I have been using nzbget for a couple months now and am running the latest version.

It has worked perfectly well with a queue of <500 items - getting between 80-100MB/s, using two providers.

However, when I have a huge queue of ~6,000 nzbs (37TB in total) the speed drops down to around 5MB/s, and never goes above 10MB/s.


Server specs are Hetzner EX41-SSD:
2*500GB SSD in Raid0
i7-6700 Quad-Core (10k rated bench)
32GB RAM


I have plenty RAM left over (over 12GB when other programs are running) and CPU is maybe 20% in total, plenty disk space etc.



These are some of the settings i'm running:

Article Cache: 500
FlushQueue No
DirectWrite Yes
WriteBuffer 1024

PostStrat Rocket

ParCheck Auto
ParRepair Yes
ParScan Extended
ParQuick Yes
ParBuffer 2000
ParRename Yes

DirectUnpack Yes
UnpackPauseQueue No
UnpackCleanupDisk Yes


I tried adding these too:
https://forum.nzbget.net/viewtopic.php? ... 6e5#p18946


The file sizes i'm downloading are anywhere from 200MB-25GB, the same speed applies no matter.

Has anyone else encountered this - is there a way to fix it or slowly add to the queue instead of having ~6,000 queued? :)


Thanks for reading. :)

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: Slow download speed with large queue

Post by hugbug » 30 Aug 2017, 04:31

mittens wrote:
30 Aug 2017, 02:02
is there a way to fix it or slowly add to the queue instead of having ~6,000 queued?
6000 items sound a bit extreme.
How about manually deleting some of items (put them in history) and later manually adding them back (from history)?

Can you send me your queue (content of QueueDir) for a test?

mittens
Posts: 6
Joined: 30 Aug 2017, 00:11

Re: Slow download speed with large queue

Post by mittens » 30 Aug 2017, 12:01

hugbug wrote:
30 Aug 2017, 04:31
mittens wrote:
30 Aug 2017, 02:02
is there a way to fix it or slowly add to the queue instead of having ~6,000 queued?
6000 items sound a bit extreme.
How about manually deleting some of items (put them in history) and later manually adding them back (from history)?

Can you send me your queue (content of QueueDir) for a test?
Aha moving to history seems to fix the problem. I sent you a PM with my queue.

Although i'm unsure how I can go about only re-adding for example, 200 without having to manually click each of them. If I click the top box it will select every single nzb. :D

Image

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: Slow download speed with large queue

Post by hugbug » 30 Aug 2017, 13:53

Use "shift+click".

mittens
Posts: 6
Joined: 30 Aug 2017, 00:11

Re: Slow download speed with large queue

Post by mittens » 30 Aug 2017, 16:48

hugbug wrote:
30 Aug 2017, 13:53
Use "shift+click".
Shift click on the Status selects them all. :o

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: Slow download speed with large queue

Post by hugbug » 30 Aug 2017, 18:35

Click on checkbox of the first item, then shift+click on checkbox of the last item.

mittens
Posts: 6
Joined: 30 Aug 2017, 00:11

Re: Slow download speed with large queue

Post by mittens » 30 Aug 2017, 18:44

hugbug wrote:
30 Aug 2017, 18:35
Click on checkbox of the first item, then shift+click on checkbox of the last item.
Ohh very nice! Thank you very much. :)


Do you see the slow speed from a large queue as just "a thing that happens" or is there a way of preventing it?

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: Slow download speed with large queue

Post by hugbug » 30 Aug 2017, 18:58

When downloader looks for the next article to download it goes through all files in queue to find the one with the highest priority. The more files are queued the more work needs to be done, the work which isn't downloading but just the managing of queue. There are certainly ways to optimize the code to reduce that amount of work but that requires reorganizing of queue structure - not a trivial thing to do.

I'll see what I can do but that's clearly not a typical use case and I'm not sure how much time I should invest into this.

mittens
Posts: 6
Joined: 30 Aug 2017, 00:11

Re: Slow download speed with large queue

Post by mittens » 30 Aug 2017, 19:12

hugbug wrote:
30 Aug 2017, 18:58
When downloader looks for the next article to download it goes through all files in queue to find the one with the highest priority. The more files are queued the more work needs to be done, the work which isn't downloading but just the managing of queue. There are certainly ways to optimize the code to reduce that amount of work but that requires reorganizing of queue structure - not a trivial thing to do.

I'll see what I can do but that's clearly not a typical use case and I'm not sure how much time I should invest into this.
Aha I see, I can't imagine it would be a problem for many people or very often so I understand the low prio.

Thank you again for your help! ;)

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: Slow download speed with large queue

Post by hugbug » 02 Sep 2017, 17:48

I've made a couple of improvements documented in https://github.com/nzbget/nzbget/issues/438.

Do you want to try this?
If you compile yourself feel free to compile from develop-branch. If you use installer from NZBGet download page - let me know your OS and CPU type and I'll send you an installer for current develop.

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 36 guests