Using tmpfs for the temp folder to speed up downloading

Discuss newly added features or request new features.
hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: Using tmpfs for the temp folder to speed up downloading

Post by hugbug » 06 Jul 2014, 10:13

Thanks for testing.
I was interested in effect brought by WriteBufferSize. Can you please make another test with "DirectWrite on, RamDisk off, WriteBufferSize 0"?

TRaSH
Posts: 54
Joined: 23 Mar 2014, 10:11

Re: Using tmpfs for the temp folder to speed up downloading

Post by TRaSH » 06 Jul 2014, 12:22

DirectWrite on, RamDisk off, WriteBufferSize 0

Start Download : 14:01:31
Finish Download : 14:14:53
Total Download : 00:13:22

Start UnRar : 14:14:55
Finish UnRar : 14:20:47
Total UnRar : 00:05:52

Total Job : 00:19:14


i also edited my other post by adding this info there,
perhaps i will give it next weekend in the morning a new try with all the settings,
beeing that the download tool longer this time perhaps because of the time of the day

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: Using tmpfs for the temp folder to speed up downloading

Post by hugbug » 21 Jul 2014, 10:40

Let's see how [New Feature] Article memory cache compares to RAM-disk setup.

The cache should be more effective in memory usage (when used in DirectWrite mode), is compatible with option ContinuePartial and is of course much easier to activate than configure a RAM-disk.

Please use the new thread for further discussion (unless your posts are not related to article cache).

TRaSH
Posts: 54
Joined: 23 Mar 2014, 10:11

Re: Using tmpfs for the temp folder to speed up downloading

Post by TRaSH » 21 Jul 2014, 17:07

if there's a testing version for windows i could give it a try

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: Using tmpfs for the temp folder to speed up downloading

Post by hugbug » 21 Jul 2014, 17:52

Forum member salami has compiled a windows binary and posted a link in the other topic.

neilt0
Posts: 261
Joined: 10 Sep 2008, 19:11

Re: Using tmpfs for the temp folder to speed up downloading

Post by neilt0 » 12 Sep 2014, 17:34

hugbug wrote:Thanks for sharing.

Since you already have a working test environment ;) how about testing DirectWrite with EXT3 and EXT4?
Could you temporary attach another drive (small or big, doesn't matter) and use it for InterDir? Would be very interesting to see how EXT3 and EXT4 compare to each other and to ReiserFS.
At some point soon, unRAID will have the ability to use ReiserFS, ext4, BTRFS or XFS (all of those apart from ext4 are in the current Beta now).
I plan on reformatting my cache drive to something other than ReiserFS. That's the drive that gets written to the most -- it will hold the interdir. Unpacking will go to the array, which will remain as ReiserFS for a while.

What do you think would be the best file system to help with faster writes in nzbget? Or is it all bit moot now with the article cache? I still see speed dropouts in the bandwidth graph in nzbget when it writes out (say) a 500MB .rar.

Cheers,

Neil.

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: Using tmpfs for the temp folder to speed up downloading

Post by hugbug » 12 Sep 2014, 18:58

I don't know much of these FS.

Do you have DirectWrite active now? Please post graphs.

neilt0
Posts: 261
Joined: 10 Sep 2008, 19:11

Re: Using tmpfs for the temp folder to speed up downloading

Post by neilt0 » 12 Sep 2014, 19:23

hugbug wrote:I don't know much of these FS.

Do you have DirectWrite active now? Please post graphs.
No, because I'm on ReiserFS!

Currently, my cache drive is 2TB and is pretty full. I'm emptying a 200GB drive in another server and plan on testing different FS using that. It's slower than the 2TB drive, but if it's empty, I think it might keep up with the download speeds.

neilt0
Posts: 261
Joined: 10 Sep 2008, 19:11

Re: Using tmpfs for the temp folder to speed up downloading

Post by neilt0 » 13 Sep 2014, 12:27

I did a little reading up and people say BTRFS is a bit faster than XFS with multiple simultaneous file operations. It's not as mature as XFS, but the cache drive is only for temporary files, so I don't really care. I'll migrate my array drives to XFS, but the "new" cache drive was formatted as BTRFS.

I replaced my 2TB 5900rpm 3.5" drive that was formatted as ReiserFS with a 2.5" 7200rpm drive formatted as BTRFS. I know the 2.5" is a slower drive, as it used to be too slow to use as a cache drive. The unpacking was very slow, in particular.

To my surprise, download speeds are better. I've gone from about 18MB/sec to 18.5MB/sec, peaking at 19MB/sec. I haven't done any formal tests, but unpacking seems to be the same or slightly faster, which was a big surprise.

I turned on DirectWrite and there was a big slowdown -- no formal testing yet, but I don't think I'll be using it with BTRFS.

Cheers,

Neil.

Post Reply

Who is online

Users browsing this forum: No registered users and 17 guests