LZ4 compression

2 respuestas [Último envío]
nadebula.1984
Desconectado/a
se unió: 05/01/2018

I wanted to make a backup of several large software packages officially released in bzip2 or xz format. The extraction of official packages took a minute or more. Compression could take even longer.

I typed the compression command (tar + lz4) in the terminal. When I lifted my finger from the Enter key, the compression completed. 2 or 3 seconds at most. The resulting .tar.lz4 file is only ~10% larger than the original .tar.bz2 or .tar.xz package.

Given the fact that I need to backup TBs of experimental data in my lab, I feel LZ4 suitable for my application. (Zstd is not fast enough.)

jxself
Desconectado/a
se unió: 09/13/2010

Yes, lz4 is very fast.

https://en.wikipedia.org/wiki/LZ4_(compression_algorithm)

nadebula.1984
Desconectado/a
se unió: 05/01/2018

According to the official benchmarks, the decompression speed of latest LZ4 approaches 5 GB/s (based on high-performance CPU), even faster than a PCIe x4 NVMe SSD.

On my most powerful mobile workstation (Xeon processor, 64 GiB RAM, two NVMe SSDs plus one SATA SSD), it does instantly decompress almost anything.