Page 1 of 1

Download complete NZB, then postprocess?

Posted: 09 Oct 2019, 05:50
by Boullie
Hey all!

Since I started playing around with virtualization (currently running 3 physical servers with about 4-5 Virtual Servers on each of them), I am having an issue with the (Virtual) machine (assigned 4GB Ram, 4-core CPU) running Windows Server 2016 Standard) I use for NZBGet (version 21.0, also tried the testing build but the issue is the same). I'm not sure if the issue I'll be describing also happened prior to the virtualization with the bare metal installation (which used NZBGet 19.0, also on Windows Server 2016 Standard), but still...
The bottleneck in this case isn't the hardware, as RAM usage hardly goes above 2GB and CPU-usage is within the 25-50% usage range on average.

So here goes...
I notice terrible download speeds (<2 megabyte per second, on a 200mbit line) when NZBGet is downloading certain NZB's. This seems to be the case ONLY when the NZB consists of several .partXXXX.rar files.
When monitoring the queue, I see it hopping from "Downloading" to "Queued" every other second or so.
It appears as if NZBGet is trying to extract the (part.)rar-file as soon as it's downloaded.

I do not believe this to be a bug in NZBGet, but it working as expected due to some setting somewhere.
Unfortunately, the only setting I could find that might have any influence on this is the "directunpack" feature. I disabled this and restarted NZBGet, yet the issue remains the same.

What I do believe that could solve the issue (where older formats of archiving were used) is that NZBGet downloads the ENTIRE NZB (thus all files + par-files) BEFORE attempting ANY repair/extract actions.

Does anyone of you guys have a clue on how to achieve this?

Thanks in advance
Boullie

Worth mentioning: I have already tried disabling the FlushQueue option (as explained here), but this didn't solve the issue.

Re: Download complete NZB, then postprocess?

Posted: 12 Oct 2019, 12:06
by hugbug
For the speed it shouldn't matter which files (rar or not) are downloaded. Sometimes downloading of older files may be way lower because news servers may store them on slower medium.

If you have a reproducible issue with certain nzb (reproducible means you can download it multiple times with the same issue), it's best to test this nzb then on a machine with nzbget installed directly without virtualisation. Virtualisation and virus scanners may have a great impact on speed.

Another good test is to try another downloader (for example SABnzbd) with the same nzb-file in the same machine/VM.

Re: Download complete NZB, then postprocess?

Posted: 12 Oct 2019, 12:15
by Boullie
Tried downloading through grabit & sabnzbd, both download at full speed... :-/

Re: Download complete NZB, then postprocess?

Posted: 12 Oct 2019, 12:19
by hugbug
Are you still interested in using nzbget? If you do - we can investigate this further but this requires many tests.

Re: Download complete NZB, then postprocess?

Posted: 12 Oct 2019, 12:30
by Boullie
Very!

I use several applications who have built-in features to flawlessly interract with NZB-get. With SabNZBd & Grabit I need to perform all actions manually.

Re: Download complete NZB, then postprocess?

Posted: 12 Oct 2019, 12:48
by hugbug
We need to determine:
  • Does the speed issue occur with certain nzbs?
  • Does the issue have something to do with virtualisation?

First of all reset nzbget to its default settings. For this backup you current config file and then use command "Reset to factory defaults" in nzbget tray icon menu. Then open settings in web-interface and use "restore settings" from SYSTEM section but select only "NEWS SERVERS". You can also enter news server settings manually instead of import.

Then for the tests disable options "FlushQueue", "DirectUnpack", "DirectRename" and "ContinuePartial" (tip: you can use search box at the top right of the page to search for settings).

Now try to reproduce the speed issue. If you found some nzb which shows the issue, try it multiple times. A perfect test-nzb would show similar speed each time.

Now, after you can reproduce the speed issue in your VM it would be great if you could install nzbget into a hardware machine and test it with the same settings and same nzb there. This shouldn't be necessary the host machine used for virtualisation but it should be preferably in the same network.

During tests pleas monitor system usage, in particular CPU usage (how high is CPU usage of nzbget process? are there any other processes with high CPU usage, for example an antivirus?) and disk write usage (the same question about nzbget process or other processes). Tip: after changing the option the nzb should be downloaded form the beginning.

Another test: activate option "SkipWrite" and try downloading again. This option disables writing of downloaded files and to some extend minimise effect of disk write speed (empty output files are still created though). Any change on speed with SkipWrite "on" and "off"?

Re: Download complete NZB, then postprocess?

Posted: 13 Oct 2019, 08:32
by Boullie
Update:
Seems it's all NZB's, not just a few... I think I just didn't notice this in the past.

Tried the suggested settings and there is a slight improvement. Now reaching speeds of around 5MB/s.

This still isn't close to the speeds I get with the other downloaders (with the exact same configuration), unfortunately.

What I did notice is that when I change all working directories to local directories on the VM, download speed is drastically improved (around 15MB/s).

Even though this could be a suitable solution, I would need to make sure each download is post processed before the next one is started, to avoid issues with disk space...

Re: Download complete NZB, then postprocess?

Posted: 13 Oct 2019, 10:36
by hugbug
Set all directories except DestDir to local drive. The DestDir can be on a remote drive (NAS). The downloaded files will be unpacked into DestDir directly.