[New Feature] Multithreading par-repair

Discuss newly added features or request new features.
hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

[New Feature] Multithreading par-repair

Post by hugbug » 02 Sep 2014, 23:08

The most wanted feature is now available for all NZBGet users:
  • doesn't depend on other libraries and works everywhere, on all platforms and all CPUs (with multiple cores), no special compiling steps are required;
  • new option ParThreads to set the number of threads for repairing;
  • new option ParBuffer to define the memory limit to use during par-repair;
Few test results
All tests made for the following nzb:
  • Total size: 3.8 GB, 78 files 50MB each
  • Block size: 2 304 000 bytes
  • Block count: 1664
  • Missing files: 2 (100 MB total, 44 blocks total)
For 100 MB of data which needs to be restored, the optimal buffer size is 100 MB (can be set to a higher value but only 100 MB will be used).

Results:
  • Windows PC, Core2 Duo, 2.6 GHz, 4 cores:
    • 1 thread, 128MB buffer, Repair time: 0:06:32
    • 2 threads, 128MB buffer, Repair time: 0:03:51
    • 3 threads, 128MB buffer, Repair time: 0:02:31
    • 4 threads, 128MB buffer, Repair time: 0:01:50
    • 6 threads, 128MB buffer, Repair time: 0:01:43
  • Macbook Core i5, 2.4 GHz, 2 cores + hyper threading (4 logical cores):
    • 1 thread, 16MB buffer, Repair time: 0:04:55
    • 1 thread, 128MB buffer, Repair time: 0:02:25
    • 4 threads, 128MB buffer, Repair time: 0:01:20
    • 6 threads, 128MB buffer, Repair time: 0:01:18
    • For comparison: par2cmdline+tbb (multicore), 128MB buffer, Repair time: 0:01:16
  • Vu+ duo SAT receiver, MIPS CPU with 2 cores 400 MHz:
    • 1 thread, 16MB buffer, Repair time: 1:10:28
    • 2 threads, 16MB buffer, Repair time: 0:45:39
    • 2 threads, 32MB buffer, Repair time: 0:42:58
Measured was only repair time. Verification time isn't included. With quick par-verification NZBGet has a big advantage over par2cmdline.

Implemented in v14-testing-r1116.

neilt0
Posts: 261
Joined: 10 Sep 2008, 19:11

Re: [New Feature] Multithreading par-repair

Post by neilt0 » 03 Sep 2014, 09:19

Fantastic!

mannibis
Posts: 60
Joined: 29 Jul 2014, 15:10

Re: [New Feature] Multithreading par-repair

Post by mannibis » 03 Sep 2014, 12:38

Great work hugbug!

neilt0
Posts: 261
Joined: 10 Sep 2008, 19:11

Re: [New Feature] Multithreading par-repair

Post by neilt0 » 03 Sep 2014, 12:59

I broke down and upgraded unRAID to 6.0 and set up a Docker so I could easily get the latest builds and the multicore repair. It's working great!

Thanks,

Neil.

mannibis
Posts: 60
Joined: 29 Jul 2014, 15:10

Re: [New Feature] Multithreading par-repair

Post by mannibis » 03 Sep 2014, 15:07

Any timeline on a release for new testing version for OSX?

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: [New Feature] Multithreading par-repair

Post by hugbug » 03 Sep 2014, 15:10

Before I make a testing release I'd like the few most active forum members to test it. If you want, I'll send you the binary for testing.

vergessen
Posts: 23
Joined: 19 Feb 2014, 15:36

Re: [New Feature] Multithreading par-repair

Post by vergessen » 03 Sep 2014, 15:39

This is working great on my machine. debian with i7. I have tested and broken a couple of files on purpose and I have run into no issues.

mannibis
Posts: 60
Joined: 29 Jul 2014, 15:10

Re: [New Feature] Multithreading par-repair

Post by mannibis » 03 Sep 2014, 15:56

I'd love to test it out for OSX.

Thanks hugbug
Last edited by mannibis on 22 Feb 2015, 22:30, edited 2 times in total.

salami
Posts: 68
Joined: 14 Apr 2014, 11:09
Location: Switzerland

Re: [New Feature] Multithreading par-repair

Post by salami » 03 Sep 2014, 16:01

Tried a couple of broken jobs on Windows, working perfectly so far! Am I correct that the total of ParBuffer and ArticleCache has to be below ~1900MB in 32bit mode?

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: [New Feature] Multithreading par-repair

Post by hugbug » 03 Sep 2014, 19:06

That's correct about 1900MB. I guess I should add a validation check.

Post Reply

Who is online

Users browsing this forum: No registered users and 22 guests