Entropy Coders

[Yann]

Entropy coders reduce a set of symbols according to their appearance probabilities.
They are typically used as stage 2 of many compression programs, including zip, rar or 7zip.

1) Range0 v0.7
New : small encoding & decoding speed improvements in extreme situations (badly or highly compressible data)
This is a semi-static Range Coder implementation. Data is divided into blocks of 128KB. Each compressed block starts with a frequency count header.
Compression speed : 145-188 MB/s
Decoding speed : 77-148 MB/s

2) Huff0 v0.9
New : Improved multi-threading code (benchmark mode only)
This is a semi-static huffman coder implementation. Data is divided into blocks of 16KB. Each compressed block starts with a huffman tree header.
Compression speed : 190-285 MB/s
Decoding speed : 184-280 MB/s

3) HuffX v0.1
New : Initial release, merge huff0 and range0, multi-threading available in benchmark mode
This is a mixed huffman and range coder, trying to provide the best of both worlds. Data is divided into blocks of 16KB.
Compression speed : 179-284 MB/s
Decoding speed : 141-271 MB/s


Detailed performance assessment :
Benchmark platform : Core 2 Duo E8400 (3GHz), Window Seven 32-bits



Comparing with other entropy coders :

Shindlet, from Piotr Tarsa, an optimized order-0 dynamic range coder written in assembler.
Compression speed : 18 MB/s
Decoding speed : 16.5 MB/s
enwik8 ratio : 1.590

o0c from Eugene Shelwien, a reference dynamic range coder
Compression speed : 5 MB/s
Decoding speed : 4.7 MB/s
enwik8 ratio : 1.585

o0rc from Eugene Shelwien; an adaptative dynamic range coder, with best-in-class compression ratio & excellent speed
Compression speed : 16 MB/s
Decoding speed : 15.2 MB/s
enwik8 ratio : 1.633 (best ratio)

[Yann]

Range0

Range0 v0.7
small encoding & decoding speed improvements in extreme situations (badly or highly compressible data)

Range0 v0.6
Encoding & decoding speed improvements in extreme situations (badly or highly compressible data)

Range0 v0.5
Improved speed (compression 20%, decoding 10%) at the expense of compression rate (-0.1%)

Range0 v0.4
Slightly improved decoding speed

Range0 v0.3
Slightly improved compression rate; Corrected behavior with zero-filled sequences

Range0 v0.2
Slightly improved compression rate, compression animation report (cosmetic)

Range0 v0.1
Initial release


Huff0

Huff0 v0.9
Improved multi-threading code (benchmark mode only)

Huff0 v0.8
Multi-threading code (benchmark mode only)

Huff0 v0.7
Speed improvement in corner-case situations (badly or highly compressible data)

Huff0 v0.6
Significant speed improvement in extreme situations (badly or highly compressible data)

Huff0 v0.5
Small speed improvement

Huff0 v0.4
Improved compression rate

Huff0 v0.3
More flexible I/O, more precise benchmark, slightly improved compression rate

Huff0 v0.2
improved decoding speed

Huff0 v0.1
initial release


HuffX

HuffX v0.1
Initial release, merge huff0 and range0, multi-threading available in benchmark mode