Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Anyone know of a comparison of similar tools. I've tinkered with this one: https://github.com/klauspost/reedsolomon/

I believe it's compatible with the Java implementation used by Backblaze and others.



One of the authors of Tahoe-LAFS and the original creator of zfec, Zooko, co-authored a paper on the performance of such codes.

http://nisl.wayne.edu/Papers/Tech/code-pf-fast09.pdf

Luigi Rizzo is the original author of the fec C library.

I co-maintain zfec (and the Debian package of it) along with Zooko. A new release of zfec is planned to be out soonish.

edit: removing the <> delimiters around the URL. Sorry!


HN sadly does not support <> URL delimiters, so your link ends up 404. Here is it without delimiters:

http://nisl.wayne.edu/Papers/Tech/code-pf-fast09.pdf


Thanks. Updated the link.


I don't know of any cross-library comparison benchmark. Also, if a library does ship with an internal benchmark, it usually uses a different number of data/parity shards and different shard size, compared to other library benchmarks, which makes comparison difficult.

I wrote an MIT-licensed ReedSolomon module for Node [1], with native C++ multi-core throughput. The internal benchmark (node benchmark.js) provides latency and throughput measurements for data=10, parity=4 for a variety of shard sizes, and the number of data/parity shards can be adjusted for comparison.

It's not yet using optimized assembler or SIMD like Klaus Post's implementation, but it is compatible and also based on Backblaze's implementation, as well as including a pure Javascript implementation for browser use.

[1] https://github.com/jorangreef/reed-solomon


I ported Klaus' library to Haskell, and redid the whole SIMD implementation (see below for the extras this brings): https://github.com/NicolasT/reedsolomon/

In my library, the SSSE3 code also has AVX and AVX2 equivalents, providing higher throughput. There are also implementations with specific optimized instructions for ARM NEON, ARM64, PPC64/AltiVec and Power8. All of this in C (with intrinsics), not ASM, so making use of your C compiler's instruction scheduling, loop unrolling,... See https://github.com/NicolasT/reedsolomon/blob/master/cbits/re...


The one you linked to is excellent. It has hand crafted SSE3 assembler for enormous speed. It is a bit difficult to compare the benchmarks from the one in the original posting to Klaus's as they aren't done at the same (N,K) parameters though.


Papers usually compare against jerasure: http://jerasure.org/jerasure/jerasure




Consider applying for YC's Summer 2026 batch! Applications are open till May 4

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: