dtp/vendor/github.com/klauspost/compress/zstd

zstd

Zstandard is a real-time compression algorithm, providing high compression ratios. It offers a very wide range of compression / speed trade-off, while being backed by a very fast decoder. A high performance compression algorithm is implemented. For now focused on speed.

This package provides compression to and decompression of Zstandard content.

This package is pure Go and without use of "unsafe".

The zstd package is provided as open source software using a Go standard license.

Currently the package is heavily optimized for 64 bit processors and will be significantly slower on 32 bit processors.

Installation

Install using go get -u github.com/klauspost/compress. The package is located in github.com/klauspost/compress/zstd.

Compressor

Status:

STABLE - there may always be subtle bugs, a wide variety of content has been tested and the library is actively used by several projects. This library is being fuzz-tested for all updates.

There may still be specific combinations of data types/size/settings that could lead to edge cases, so as always, testing is recommended.

For now, a high speed (fastest) and medium-fast (default) compressor has been implemented.

• The "Fastest" compression ratio is roughly equivalent to zstd level 1.
• The "Default" compression ratio is roughly equivalent to zstd level 3 (default).
• The "Better" compression ratio is roughly equivalent to zstd level 7.
• The "Best" compression ratio is roughly equivalent to zstd level 11.

In terms of speed, it is typically 2x as fast as the stdlib deflate/gzip in its fastest mode. The compression ratio compared to stdlib is around level 3, but usually 3x as fast.

Usage

An Encoder can be used for either compressing a stream via the io.WriteCloser interface supported by the Encoder or as multiple independent tasks via the EncodeAll function. Smaller encodes are encouraged to use the EncodeAll function. Use NewWriter to create a new instance that can be used for both.

To create a writer with default options, do like this:

// Compress input to output.
func Compress(in io.Reader, out io.Writer) error {
    enc, err := zstd.NewWriter(out)
    if err != nil {
        return err
    }
    _, err = io.Copy(enc, in)
    if err != nil {
        enc.Close()
        return err
    }
    return enc.Close()
}

Now you can encode by writing data to enc. The output will be finished writing when Close() is called. Even if your encode fails, you should still call Close() to release any resources that may be held up.

The above is fine for big encodes. However, whenever possible try to reuse the writer.

To reuse the encoder, you can use the Reset(io.Writer) function to change to another output. This will allow the encoder to reuse all resources and avoid wasteful allocations.

Currently stream encoding has 'light' concurrency, meaning up to 2 goroutines can be working on part of a stream. This is independent of the WithEncoderConcurrency(n), but that is likely to change in the future. So if you want to limit concurrency for future updates, specify the concurrency you would like.

You can specify your desired compression level using WithEncoderLevel() option. Currently only pre-defined compression settings can be specified.

Future Compatibility Guarantees

This will be an evolving project. When using this package it is important to note that both the compression efficiency and speed may change.

The goal will be to keep the default efficiency at the default zstd (level 3). However the encoding should never be assumed to remain the same, and you should not use hashes of compressed output for similarity checks.

The Encoder can be assumed to produce the same output from the exact same code version. However, the may be modes in the future that break this, although they will not be enabled without an explicit option.

This encoder is not designed to (and will probably never) output the exact same bitstream as the reference encoder.

Also note, that the cgo decompressor currently does not report all errors on invalid input, omits error checks, ignores checksums and seems to ignore concatenated streams, even though it is part of the spec.

Blocks

For compressing small blocks, the returned encoder has a function called EncodeAll(src, dst []byte) []byte.

EncodeAll will encode all input in src and append it to dst. This function can be called concurrently, but each call will only run on a single goroutine.

Encoded blocks can be concatenated and the result will be the combined input stream. Data compressed with EncodeAll can be decoded with the Decoder, using either a stream or DecodeAll.

Especially when encoding blocks you should take special care to reuse the encoder. This will effectively make it run without allocations after a warmup period. To make it run completely without allocations, supply a destination buffer with space for all content.

import "github.com/klauspost/compress/zstd"

// Create a writer that caches compressors.
// For this operation type we supply a nil Reader.
var encoder, _ = zstd.NewWriter(nil)

// Compress a buffer. 
// If you have a destination buffer, the allocation in the call can also be eliminated.
func Compress(src []byte) []byte {
    return encoder.EncodeAll(src, make([]byte, 0, len(src)))
} 

You can control the maximum number of concurrent encodes using the WithEncoderConcurrency(n) option when creating the writer.

Using the Encoder for both a stream and individual blocks concurrently is safe.

Performance

I have collected some speed examples to compare speed and compression against other compressors.

• file is the input file.
• out is the compressor used. zskp is this package. zstd is the Datadog cgo library. gzstd/gzkp is gzip standard and this library.
• level is the compression level used. For zskp level 1 is "fastest", level 2 is "default"; 3 is "better", 4 is "best".
• insize/outsize is the input/output size.
• millis is the number of milliseconds used for compression.
• mb/s is megabytes (2^20 bytes) per second.

Silesia Corpus:
http://sun.aei.polsl.pl/~sdeor/corpus/silesia.zip

This package:
file    out     level   insize      outsize     millis  mb/s
silesia.tar zskp    1   211947520   73101992    643     313.87
silesia.tar zskp    2   211947520   67504318    969     208.38
silesia.tar zskp    3   211947520   64595893    2007    100.68
silesia.tar zskp    4   211947520   60995370    8825    22.90

cgo zstd:
silesia.tar zstd    1   211947520   73605392    543     371.56
silesia.tar zstd    3   211947520   66793289    864     233.68
silesia.tar zstd    6   211947520   62916450    1913    105.66
silesia.tar zstd    9   211947520   60212393    5063    39.92

gzip, stdlib/this package:
silesia.tar gzstd   1   211947520   80007735    1654    122.21
silesia.tar gzkp    1   211947520   80136201    1152    175.45

GOB stream of binary data. Highly compressible.
https://files.klauspost.com/compress/gob-stream.7z

file        out     level   insize  outsize     millis  mb/s
gob-stream  zskp    1   1911399616  235022249   3088    590.30
gob-stream  zskp    2   1911399616  205669791   3786    481.34
gob-stream  zskp    3   1911399616  175034659   9636    189.17
gob-stream  zskp    4   1911399616  165609838   50369   36.19

gob-stream  zstd    1   1911399616  249810424   2637    691.26
gob-stream  zstd    3   1911399616  208192146   3490    522.31
gob-stream  zstd    6   1911399616  193632038   6687    272.56
gob-stream  zstd    9   1911399616  177620386   16175   112.70

gob-stream  gzstd   1   1911399616  357382641   10251   177.82
gob-stream  gzkp    1   1911399616  359753026   5438    335.20

The test data for the Large Text Compression Benchmark is the first
10^9 bytes of the English Wikipedia dump on Mar. 3, 2006.
http://mattmahoney.net/dc/textdata.html

file    out level   insize      outsize     millis  mb/s
enwik9  zskp    1   1000000000  343848582   3609    264.18
enwik9  zskp    2   1000000000  317276632   5746    165.97
enwik9  zskp    3   1000000000  292243069   12162   78.41
enwik9  zskp    4   1000000000  262183768   82837   11.51

enwik9  zstd    1   1000000000  358072021   3110    306.65
enwik9  zstd    3   1000000000  313734672   4784    199.35
enwik9  zstd    6   1000000000  295138875   10290   92.68
enwik9  zstd    9   1000000000  278348700   28549   33.40

enwik9  gzstd   1   1000000000  382578136   9604    99.30
enwik9  gzkp    1   1000000000  383825945   6544    145.73

Highly compressible JSON file.
https://files.klauspost.com/compress/github-june-2days-2019.json.zst

file                        out level   insize      outsize     millis  mb/s
github-june-2days-2019.json zskp    1   6273951764  699045015   10620   563.40
github-june-2days-2019.json zskp    2   6273951764  617881763   11687   511.96
github-june-2days-2019.json zskp    3   6273951764  524340691   34043   175.75
github-june-2days-2019.json zskp    4   6273951764  470320075   170190  35.16

github-june-2days-2019.json zstd    1   6273951764  766284037   8450    708.00
github-june-2days-2019.json zstd    3   6273951764  661889476   10927   547.57
github-june-2days-2019.json zstd    6   6273951764  642756859   22996   260.18
github-june-2days-2019.json zstd    9   6273951764  601974523   52413   114.16

github-june-2days-2019.json gzstd   1   6273951764  1164400847  29948   199.79
github-june-2days-2019.json gzkp    1   6273951764  1125417694  21788   274.61

VM Image, Linux mint with a few installed applications:
https://files.klauspost.com/compress/rawstudio-mint14.7z

file                    out level   insize      outsize     millis  mb/s
rawstudio-mint14.tar    zskp    1   8558382592  3667489370  20210   403.84
rawstudio-mint14.tar    zskp    2   8558382592  3364592300  31873   256.07
rawstudio-mint14.tar    zskp    3   8558382592  3158085214  77675   105.08
rawstudio-mint14.tar    zskp    4   8558382592  2965110639  857750  9.52

rawstudio-mint14.tar    zstd    1   8558382592  3609250104  17136   476.27
rawstudio-mint14.tar    zstd    3   8558382592  3341679997  29262   278.92
rawstudio-mint14.tar    zstd    6   8558382592  3235846406  77904   104.77
rawstudio-mint14.tar    zstd    9   8558382592  3160778861  140946  57.91

rawstudio-mint14.tar    gzstd   1   8558382592  3926257486  57722   141.40
rawstudio-mint14.tar    gzkp    1   8558382592  3962605659  45113   180.92

CSV data:
https://files.klauspost.com/compress/nyc-taxi-data-10M.csv.zst

file                    out level   insize      outsize     millis  mb/s
nyc-taxi-data-10M.csv   zskp    1   3325605752  641339945   8925    355.35
nyc-taxi-data-10M.csv   zskp    2   3325605752  591748091   11268   281.44
nyc-taxi-data-10M.csv   zskp    3   3325605752  530289687   25239   125.66
nyc-taxi-data-10M.csv   zskp    4   3325605752  476268884   135958  23.33

nyc-taxi-data-10M.csv   zstd    1   3325605752  687399637   8233    385.18
nyc-taxi-data-10M.csv   zstd    3   3325605752  598514411   10065   315.07
nyc-taxi-data-10M.csv   zstd    6   3325605752  570522953   20038   158.27
nyc-taxi-data-10M.csv   zstd    9   3325605752  517554797   64565   49.12

nyc-taxi-data-10M.csv   gzstd   1   3325605752  928656485   23876   132.83
nyc-taxi-data-10M.csv   gzkp    1   3325605752  922257165   16780   189.00

Decompressor

Staus: STABLE - there may still be subtle bugs, but a wide variety of content has been tested.

This library is being continuously fuzz-tested, kindly supplied by fuzzit.dev. The main purpose of the fuzz testing is to ensure that it is not possible to crash the decoder, or run it past its limits with ANY input provided.

Usage

The package has been designed for two main usages, big streams of data and smaller in-memory buffers. There are two main usages of the package for these. Both of them are accessed by creating a Decoder.

For streaming use a simple setup could look like this:

import "github.com/klauspost/compress/zstd"

func Decompress(in io.Reader, out io.Writer) error {
    d, err := zstd.NewReader(in)
    if err != nil {
        return err
    }
    defer d.Close()
    
    // Copy content...
    _, err = io.Copy(out, d)
    return err
}

It is important to use the "Close" function when you no longer need the Reader to stop running goroutines. See "Allocation-less operation" below.

For decoding buffers, it could look something like this:

import "github.com/klauspost/compress/zstd"

// Create a reader that caches decompressors.
// For this operation type we supply a nil Reader.
var decoder, _ = zstd.NewReader(nil)

// Decompress a buffer. We don't supply a destination buffer,
// so it will be allocated by the decoder.
func Decompress(src []byte) ([]byte, error) {
    return decoder.DecodeAll(src, nil)
} 

Both of these cases should provide the functionality needed. The decoder can be used for concurrent decompression of multiple buffers. It will only allow a certain number of concurrent operations to run. To tweak that yourself use the WithDecoderConcurrency(n) option when creating the decoder.

Dictionaries

Data compressed with dictionaries can be decompressed.

Dictionaries are added individually to Decoders. Dictionaries are generated by the zstd --train command and contains an initial state for the decoder. To add a dictionary use the WithDecoderDicts(dicts ...[]byte) option with the dictionary data. Several dictionaries can be added at once.

The dictionary will be used automatically for the data that specifies them. A re-used Decoder will still contain the dictionaries registered.

When registering multiple dictionaries with the same ID, the last one will be used.

It is possible to use dictionaries when compressing data.

To enable a dictionary use WithEncoderDict(dict []byte). Here only one dictionary will be used and it will likely be used even if it doesn't improve compression.

The used dictionary must be used to decompress the content.

For any real gains, the dictionary should be built with similar data. If an unsuitable dictionary is used the output may be slightly larger than using no dictionary. Use the zstd commandline tool to build a dictionary from sample data. For information see zstd dictionary information.

For now there is a fixed startup performance penalty for compressing content with dictionaries. This will likely be improved over time. Just be aware to test performance when implementing.

Allocation-less operation

The decoder has been designed to operate without allocations after a warmup.

This means that you should store the decoder for best performance. To re-use a stream decoder, use the Reset(r io.Reader) error to switch to another stream. A decoder can safely be re-used even if the previous stream failed.

To release the resources, you must call the Close() function on a decoder. After this it can no longer be reused, but all running goroutines will be stopped. So you must use this if you will no longer need the Reader.

For decompressing smaller buffers a single decoder can be used. When decoding buffers, you can supply a destination slice with length 0 and your expected capacity. In this case no unneeded allocations should be made.

Concurrency

The buffer decoder does everything on the same goroutine and does nothing concurrently. It can however decode several buffers concurrently. Use WithDecoderConcurrency(n) to limit that.

The stream decoder operates on

• One goroutine reads input and splits the input to several block decoders.
• A number of decoders will decode blocks.
• A goroutine coordinates these blocks and sends history from one to the next.

So effectively this also means the decoder will "read ahead" and prepare data to always be available for output.

Since "blocks" are quite dependent on the output of the previous block stream decoding will only have limited concurrency.

In practice this means that concurrency is often limited to utilizing about 2 cores effectively.

Benchmarks

These are some examples of performance compared to datadog cgo library.

The first two are streaming decodes and the last are smaller inputs.

BenchmarkDecoderSilesia-8                          3     385000067 ns/op     550.51 MB/s        5498 B/op          8 allocs/op
BenchmarkDecoderSilesiaCgo-8                       6     197666567 ns/op    1072.25 MB/s      270672 B/op          8 allocs/op

BenchmarkDecoderEnwik9-8                           1    2027001600 ns/op     493.34 MB/s       10496 B/op         18 allocs/op
BenchmarkDecoderEnwik9Cgo-8                        2     979499200 ns/op    1020.93 MB/s      270672 B/op          8 allocs/op

Concurrent performance:

BenchmarkDecoder_DecodeAllParallel/kppkn.gtb.zst-16                28915         42469 ns/op    4340.07 MB/s         114 B/op          0 allocs/op
BenchmarkDecoder_DecodeAllParallel/geo.protodata.zst-16           116505          9965 ns/op    11900.16 MB/s         16 B/op          0 allocs/op
BenchmarkDecoder_DecodeAllParallel/plrabn12.txt.zst-16              8952        134272 ns/op    3588.70 MB/s         915 B/op          0 allocs/op
BenchmarkDecoder_DecodeAllParallel/lcet10.txt.zst-16               11820        102538 ns/op    4161.90 MB/s         594 B/op          0 allocs/op
BenchmarkDecoder_DecodeAllParallel/asyoulik.txt.zst-16             34782         34184 ns/op    3661.88 MB/s          60 B/op          0 allocs/op
BenchmarkDecoder_DecodeAllParallel/alice29.txt.zst-16              27712         43447 ns/op    3500.58 MB/s          99 B/op          0 allocs/op
BenchmarkDecoder_DecodeAllParallel/html_x_4.zst-16                 62826         18750 ns/op    21845.10 MB/s        104 B/op          0 allocs/op
BenchmarkDecoder_DecodeAllParallel/paper-100k.pdf.zst-16          631545          1794 ns/op    57078.74 MB/s          2 B/op          0 allocs/op
BenchmarkDecoder_DecodeAllParallel/fireworks.jpeg.zst-16         1690140           712 ns/op    172938.13 MB/s         1 B/op          0 allocs/op
BenchmarkDecoder_DecodeAllParallel/urls.10K.zst-16                 10432        113593 ns/op    6180.73 MB/s        1143 B/op          0 allocs/op
BenchmarkDecoder_DecodeAllParallel/html.zst-16                    113206         10671 ns/op    9596.27 MB/s          15 B/op          0 allocs/op
BenchmarkDecoder_DecodeAllParallel/comp-data.bin.zst-16          1530615           779 ns/op    5229.49 MB/s           0 B/op          0 allocs/op

BenchmarkDecoder_DecodeAllParallelCgo/kppkn.gtb.zst-16             65217         16192 ns/op    11383.34 MB/s         46 B/op          0 allocs/op
BenchmarkDecoder_DecodeAllParallelCgo/geo.protodata.zst-16        292671          4039 ns/op    29363.19 MB/s          6 B/op          0 allocs/op
BenchmarkDecoder_DecodeAllParallelCgo/plrabn12.txt.zst-16          26314         46021 ns/op    10470.43 MB/s        293 B/op          0 allocs/op
BenchmarkDecoder_DecodeAllParallelCgo/lcet10.txt.zst-16            33897         34900 ns/op    12227.96 MB/s        205 B/op          0 allocs/op
BenchmarkDecoder_DecodeAllParallelCgo/asyoulik.txt.zst-16         104348         11433 ns/op    10949.01 MB/s         20 B/op          0 allocs/op
BenchmarkDecoder_DecodeAllParallelCgo/alice29.txt.zst-16           75949         15510 ns/op    9805.60 MB/s          32 B/op          0 allocs/op
BenchmarkDecoder_DecodeAllParallelCgo/html_x_4.zst-16             173910          6756 ns/op    60624.29 MB/s         37 B/op          0 allocs/op
BenchmarkDecoder_DecodeAllParallelCgo/paper-100k.pdf.zst-16       923076          1339 ns/op    76474.87 MB/s          1 B/op          0 allocs/op
BenchmarkDecoder_DecodeAllParallelCgo/fireworks.jpeg.zst-16       922920          1351 ns/op    91102.57 MB/s          2 B/op          0 allocs/op
BenchmarkDecoder_DecodeAllParallelCgo/urls.10K.zst-16              27649         43618 ns/op    16096.19 MB/s        407 B/op          0 allocs/op
BenchmarkDecoder_DecodeAllParallelCgo/html.zst-16                 279073          4160 ns/op    24614.18 MB/s          6 B/op          0 allocs/op
BenchmarkDecoder_DecodeAllParallelCgo/comp-data.bin.zst-16        749938          1579 ns/op    2581.71 MB/s           0 B/op          0 allocs/op

This reflects the performance around May 2020, but this may be out of date.

Zstd inside ZIP files

It is possible to use zstandard to compress individual files inside zip archives. While this isn't widely supported it can be useful for internal files.

To support the compression and decompression of these files you must register a compressor and decompressor.

It is highly recommended registering the (de)compressors on individual zip Reader/Writer and NOT use the global registration functions. The main reason for this is that 2 registrations from different packages will result in a panic.

It is a good idea to only have a single compressor and decompressor, since they can be used for multiple zip files concurrently, and using a single instance will allow reusing some resources.

See this example for how to compress and decompress files inside zip archives.

Contributions

Contributions are always welcome. For new features/fixes, remember to add tests and for performance enhancements include benchmarks.

For general feedback and experience reports, feel free to open an issue or write me on Twitter.

This package includes the excellent github.com/cespare/xxhash package Copyright (c) 2016 Caleb Spare.