Multithreaded par repair?

Discuss newly added features or request new features.
swtor
Posts: 11
Joined: 22 Jun 2014, 23:02

Multithreaded par repair?

Post by swtor » 22 Jun 2014, 23:04

Hello

I noticed that my nas is only using one core to par repait, resulting in rather long delays.
Is it possible to have par2 use both cores? Par should be able to do this, if i am not mistaken?
Is there an option in nzbget?

Yil
Posts: 49
Joined: 26 May 2014, 02:44

Re: Multithreaded par repair?

Post by Yil » 23 Jun 2014, 04:04

I've looked at the par repair library and in the future I should have a number of enhancements, but multi-threaded repair won't really speed things up that much. The limiting factor is disk I/O in almost all cases (unless perhaps you have a fast ssd). Luckily I think there are some ways to reduce disk use, but I haven't tried them yet.

swtor
Posts: 11
Joined: 22 Jun 2014, 23:02

Re: Multithreaded par repair?

Post by swtor » 23 Jun 2014, 14:14

Nice to hear, thank you for the feedback.
On my synology nas, one core is at 100%, and nzbget process takes 100% cpu time, which suggests additional cpu power could help
Since most nas have not so fast cpus i think, at least in my case, the cpu is the bottleneck.
Repairing takes 3-4 times longer than downloading.
Isn't there a simple cmd line param with par2 to make it use all cores, sry if thats incredibly naive ;)

Edit: after some googling i found a sabnzbd multicore par2 link which directed me to a par2 for linux with multicore support
Is it possible to integrate that somehow?
http://www.chuchusoft.com/par2_tbb/download.html
Last edited by swtor on 23 Jun 2014, 14:24, edited 1 time in total.

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: Multithreaded par repair?

Post by hugbug » 23 Jun 2014, 14:24

Standard par2-tool par2cmdline uses only one CPU. There is a special version par2+tbb made by other developer which can use multiple cores. It works on x86 CPUs though.

NZBGet uses libpar2 which is based on standard par2cmdline and can't use multiple cores. There is a wish to make a multicore version of libpar2 using code from par2+tbb but this is not a trivial task. And even if it succeeds it will work on x86 only.

swtor
Posts: 11
Joined: 22 Jun 2014, 23:02

Re: Multithreaded par repair?

Post by swtor » 23 Jun 2014, 14:25

I c, thank you!

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: Multithreaded par repair?

Post by hugbug » 23 Jun 2014, 14:31

How long the repair takes for you?
You can calculates the time from timestamps in the log file (or in the postprocesslog.txt created by Logger.py).
It's important to distinguish between verify time and repair time.
There is an idea how to reduce the verify time. It'd be nice to see how much impact it will have on the overall verify/repair time.

Can you please post few examples (download size, verify time, repair time)?

swtor
Posts: 11
Joined: 22 Jun 2014, 23:02

Re: Multithreaded par repair?

Post by swtor » 23 Jun 2014, 14:54

Ok so i have 2 examples here:

7.4GB
Verify took:
Sun Jun 22 06:08:15 2014 INFO Verifying xxx 1st file
Sun Jun 22 06:39:45 2014 INFO Need more 453 par-block(s) ... last file

no repair on this one, not enough blocks

5GB
verify:

Mon Jun 23 01:08:36 2014 INFO Collection xxx with health 99.1% needs par-check
Mon Jun 23 01:21:37 2014 INFO Need more 50 par-block(s) for xxx/141547

this took 13 minutes

repair:
Mon Jun 23 01:22:51 2014 INFO Repairing xxx/141547
Mon Jun 23 02:49:36 2014 INFO Verifying repaired file xxx

repair took 27 minutes.

verify and repair together 39 minutes.

While downloading took ~6 minutes, on the same nas.

these are the only 2 samples i have right now.

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: Multithreaded par repair?

Post by hugbug » 23 Jun 2014, 15:00

Thanks for the numbers.

Does your NAS have x86 CPU?

swtor
Posts: 11
Joined: 22 Jun 2014, 23:02

Re: Multithreaded par repair?

Post by swtor » 23 Jun 2014, 15:07

Unfortunately not.
Its a dualcore ppc
cpu : e500v2
clock : 1066.666000MHz

Synology 2013+ i believe.

neilt0
Posts: 261
Joined: 10 Sep 2008, 19:11

Re: Multithreaded par repair?

Post by neilt0 » 23 Jun 2014, 17:20

Yil wrote:I've looked at the par repair library and in the future I should have a number of enhancements, but multi-threaded repair won't really speed things up that much. The limiting factor is disk I/O in almost all cases (unless perhaps you have a fast ssd). Luckily I think there are some ways to reduce disk use, but I haven't tried them yet.
That is incorrect. It will be partly dependent on CPU and disk speed, but for many (most?) configurations, multicore will drastically reduce repair times.

Post Reply

Who is online

Users browsing this forum: No registered users and 24 guests