How to enable multi thread unpack?

Get help, report and discuss bugs.
Post Reply
defcon_three
Posts: 5
Joined: 14 Jan 2018, 11:20
Location: Germany

How to enable multi thread unpack?

Post by defcon_three » 02 Jul 2021, 19:54

Hi,

I have a Celeron J1900 4core synology box, where Nzbget runs on Docker.

Now I got a upgrade to 500Mbit/s fiber connection. Having "Direct Unpack" enabled I have a problem that unpacking is much slower then downloading speed. On downloading 3 RARs only 1 RAR get unpacked simultaneously.

According to TOP: Downloading takes ~70% of 1 CPU core, unpacking ~100% of 1 CPU core.

Is there any possibility to remove the unpacking bottleneck?
How is it possible to enable multi thread unpacking (using unrar 5.90)?

hugbug
Developer & Admin
Posts: 7625
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: How to enable multi thread unpack?

Post by hugbug » 02 Jul 2021, 21:18

Usually video files are not compressed and rar is used just as container. That means the issue isn't uncompressing but reading and writing of data. Therefore a faster disk system should help.

For optimal performance the directories InterDir and DestDir should be on different physical disks. Even better if you can use an SSD for InterDir. Also make sure to use article cache in nzbget, it decreases disk access significantly.

As for multithreaded unpacking in unrar - nzbget does nothing to prevent unrar from doing this. If unrar supports such feature at all.

You may also Test if disabling DirectUnpack is a better option for you. Without DirectUnpack and with UnpackPauseQueue=yes the unpack will be performed separately and can use all CPU and disk resources. On a very fast connection the total performance may increase.

defcon_three
Posts: 5
Joined: 14 Jan 2018, 11:20
Location: Germany

Re: How to enable multi thread unpack?

Post by defcon_three » 03 Jul 2021, 12:52

Thank you hugbug.
For all temporary directories I am using a SSD. I just tested a manual unrar on the RAR filed without nzbget directly on the SSD. I got the same result: CPU bound to 100% of 1 core, 25 MB/s unrar Performance.

Do you know any unrar version with multithread unpacking?

Edit: Found a 7z multithread version inside my NZBGet docker. Unfortunately not supporting RAR. According to 7z benchmark my CPU has 85MB/s on 4 cores and 21 MB/s on 1 core unpacking speed. Still looking for a multithread version of unrar.

hugbug
Developer & Admin
Posts: 7625
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: How to enable multi thread unpack?

Post by hugbug » 03 Jul 2021, 15:34

Try to find unrar compiled specifically for your device. Maybe there is one already on the system partition?

defcon_three
Posts: 5
Joined: 14 Jan 2018, 11:20
Location: Germany

Re: How to enable multi thread unpack?

Post by defcon_three » 05 Jul 2021, 20:25

I tried many different versions of unrar and unrar-unfree as well as 7z-full.
No improvement.

Do you ever have seen a unrar/7z version which can unpack password protected RARs in multiple threads?

hugbug
Developer & Admin
Posts: 7625
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: How to enable multi thread unpack?

Post by hugbug » 05 Jul 2021, 23:38

Rar is a proprietary format. If unrar from rarlab can't do this - nothing can. Test unrar.exe on Windows if it can use more than one thread with your test archive.

Unrar for Linux is also provided by rarlab but in a source code form. It is possible that if compiled with wrong settings it works suboptimal. That's why you should test the Windows exe downloaded from rarlab site.

Post Reply

Who is online

Users browsing this forum: No registered users and 6 guests