[New Feature] Article memory cache

Discuss newly added features or request new features.
hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: [New Feature] Article memory cache

Post by hugbug » 30 Jul 2014, 19:48

If direct write is disabled there is no flushing until all articles for a file are downloaded. If the cache is full the other articles (which do not fit into the cache) will be saved into temporary files just like they would if the cache were disabled.

The premature flushing in DirectWrite=off is possible but it is not always optimal. Users may use separate drives for temp-directory and destination directory. In that case two writings into destination directory (once zeros and then actual data) is bad. Another option would be needed to control the behavior.

At the moment the DirectWrite+ArticleCache looks like the best fit for most system.

Macs (as a notable popular system not supporting sparse files) are probably better served with DirectWrite=off+ArticleCache=1GB. Macs have a lot of memory anyway. I don't know other systems not supporting sparse files, which I should care about.

mannibis
Posts: 60
Joined: 29 Jul 2014, 15:10

Re: [New Feature] Article memory cache

Post by mannibis » 31 Jul 2014, 22:43

Upgraded to latest testing version.
I have NZBGet on a Mac-Mini (2.5 Ghz i5 w/ 8 GB Ram) downloading directly to my NAS that's mounted via AFP.

I have DirectWrite ON, Article Cache 500 MB, and WriteBuffer as 1024.

Dude, my speed went up 3-4 MB/s....I don't know if I hit the optimal sweet spot or not, but I tried 2,000 MB for Article Cache and my speeds were around 28 MB/s. Having it at 0 slowed me down to around 19 MB/s for some reason. But after setting it to 500 MB, I'm getting 38 MB/s consistently, whereas with V13 Stable I could only hit 35 MB/s consistently.

Nice work man! I noticed you mentioned that Mac's should have DirectWrite Off, but is that only for Mac's downloading to their own hard drive (HFS), right? Right now I have it on, but it's writing directly to my NAS (EXT4).

Either way, I'm going to leave it as is since I'm pretty much saturating my connection (300/20). Just leaving this here for anyone who is trying to figure out what to set ArticleCache to.

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: [New Feature] Article memory cache

Post by hugbug » 31 Jul 2014, 23:03

There shouldn't be such thing as sweet spot of article cache size :)

In the "status and statistics"-dialog there is a new line "Article Cache" showing the amount of memory currently used. Please take a look there when trying different cache sizes. For downloads consisting of rar-files say 200 MB the used memory should not be much higher than 200 MB (may be a little bit when it is flushing one file and downloading another at the same moment). Meaning the ArticleCache set to 500 MB or 2000 MB should give the same results.

The inability of Macs to create spares files refers indeed to HFS only. The kernel itself does support sparse files on other file systems. I don't know for sure if it works via NFS or AFP though. I hope it does.

This is how you can test it: find an nzb-file with very large rar-files (preferably 500 MB, try bluray images). Start download, open download volume graph. At the moments it switches from one file to another it creates an output file on disk. Sparse files are created very fast, but the system not supporting sparse files needs several seconds to write 500 MB of zeros. If that happens you should see a drop lasting few seconds on the volume graph. When monitoring you can open the "status and statistics"-dialog in another browser window to watch the used Article Cache simultaneously - when it comes near 500 MB you know it will start flushing one file and creating a new output file.

mannibis
Posts: 60
Joined: 29 Jul 2014, 15:10

Re: [New Feature] Article memory cache

Post by mannibis » 31 Jul 2014, 23:37

I didn't notice any spikes down to 0 lasting seconds every 500 MB, but I did notice spikes that lasted half a second every once in a while (not corresponding to Article Cache size). Either way, I tried setting it to 1 GB and was getting the same speeds. As to whether I should leave DirectWrite on or off, I don't know. I've always had it on and haven't experienced any slow speed or unpacking (upacks of 10 gb usually take 5 min.)

Post Reply

Who is online

Users browsing this forum: No registered users and 18 guests