LZ4 - Very Fast compressor => x2.07 @ 295 MB/s

[Yann]

Note : LZ4 homepage has changed, please update your bookmark. The new homepage is here : http://fastcompression.blogspot.com/p/lz4.html

LZ4 is a particularly fast compressor using well-known LZ77 (Lempel-Ziv) algorithm.

Originally based on LZP2, it provides much better compression ratio on text files and reaches impressive decompression speed, beyond 500MB/s for text and 1GB/s (!) for binaries (test system : Core 2 Duo E8400). These speeds are scalable with multi-threading modes.
According to several compression benchmark websites, LZ4 is the fastest compressor and decoder.

LZ4 now features several compression modes, with different trade-off. The higher compression ratios use the MMC (Morphing Match Chain) search algorithm (explained here).

All compression modes generate *.lz4 compressed files.


Download :
v1.2b : LZ4 (win32)
- Corrected : MMC on specific patterns
- Corrected : Pipe mode (stdin)
- Corrected : zero-byte file
- benchmark mode can process multiple input files and provide overall summary


Benchmark evaluation :
File sample totalizing about 4GB.
In-memory benchmark results (command -b), excluding I/O considerations (such as hard drive).
Benchmark platform : Core 2 Duo E8400 (3GHz), Window Seven 32-bits

Compression level	Compression Ratio	Speed	Decoding
Single Thread
c0	2.075	295 MB/s	1.02 GB/s
c1	2.343	35.4 MB/s	1.09 GB/s
c2	2.403	17.6 MB/s	1.15 GB/s
With 2 Threads
c0	2.075	575 MB/s	1.81 GB/s
c1	2.343	69.0 MB/s	1.84 GB/s
c2	2.403	34.8 MB/s	1.91 GB/s

See benchmark results below

[Yann]

"General" version :
v1.2b : LZ4
- Corrected : MMC on specific patterns
- Corrected : Pipe mode (stdin)
- Corrected : zero-byte file
- benchmark mode can process multiple input files and provide overall summary
v1.2 : LZ4
- c2 compression level speed increase
- decompression speed increase
- pipe mode (stdin & stdout) and "null" redirection
- benchmark can process multiple input files
v1.1 : LZ4
- c1 mode : multi threading available
- Stats animation while compressing
- Silent mode (-s)
v1.0 : LZ4
- Integrate all compression modes

"Ultra Fast" versions :
v0.9 : LZ4
- Multi-threading available for file compression
v0.8 : LZ4
- Significant speed boost on binary files
- Improved multi-threading code (benchmark mode only)
v0.7 : LZ4
- Multi-threading compression & decompression (benchmark mode only)
v0.6 : LZ4
- Improved decoding speed
v0.5 : LZ4
- small compression speed improvement
v0.4 : LZ4
- Improved decoding speed
v0.3 : LZ4
- small compression speed and ratio improvement
v0.2 : LZ4
- initial release

"Optimal Compression" versions :
v0.1 : LZ4OC "Optimal Compression"
- initial release; Simple Hash Chain with max search limit 256 (note : superseded by faster HC versions using MMC)

"High Compression" versions :
v0.9 : LZ4HC "High Compression"
- De-Overlap : Improved decoding speed
v0.8 : LZ4HC "High Compression"
- Multi-Level promotions : substantial compression speed improvement
v0.7 : LZ4HC "High Compression"
- Fusion (full segment & MMC integration) : small compression speed improvement
v0.6 : LZ4HC "High Compression"
- Fusion0 : Improved compression speed
- Improved decoding speed
- Corrected progress animation with large files (>4GB)
v0.5 : LZ4HC "High Compression"
- Segments generalisation : Improved compression speed; minor compression rate improvement
v0.4 : LZ4HC "High Compression"
- Segments 0 : Large compression speed improvement on binary files; minor compression rate improvement
v0.3 : LZ4HC "High Compression"
- Changed : improved compression speed, using Morphing Match Chain (MMC) search algorithm
v0.2 : LZ4HC "High Compression"
- Changed : improved compression ratio
v0.1 : LZ4HC "High Compression"
initial release, using simple hash-chain search methodology

[Yann]

1) LZ4 on Squeeze Chart, by Stephan Busch : fastest compressor

2) LZ4 on Compression Ratings, by Sami Runsas : fastest compressor

3) LZ4 on Large Text Compression Benchmark, by Matt Mahoney : fastest decoder, and pareto frontier for compression speed/ratio

4) Fast compression algorithms comparison : by Richard Sim

5) Graphical comparison of fast compressors (outdated)

[Yann]

Morphing Match Chain explanation :

Homepage = http://fastcompression.blogspot.com/p/mmc-morphing-match-chain.html

Discussion : encode.ru post

[stfox]

J'ai pas accès à ma messagerie, alors je te laisse un petit commentaire :
j'ai remarqué qu'en décompression LZ4 utilise au minimum 2 threads, même si on lui spécifie -t1,
étonnant ? Tu peux le voir dans le gestionnaire des tâches en affichant la colonne thread, mais
aussi par le fait qu'un dual core tourne à 100% alors qu'avec un seul thread il ne devrait tourner
qu'à 50%

A++
Bruno

[Yann]

Alors, je suppose que tu parles du mode I/O (normal) ?

Dans ce cas, j'ai peut-être une explication :
En plus du thread de compression ou de decompression, il y a un thread I/O.

Il y a toujours un thread I/O. Il s'ajoute au nombre de threads spécifié par -t#.

C'est surtout par simplicité que j'ai choisi ce design.
La liste de tâches est alors "encadrée" par le thread principal qui s'occupe de l'I/O.

Comme dans la pub : on peut faire plus lent, mais c'est plus compliqué...

[Sponsored content]