用于研究和生产的熵编码器

constriction 库提供了一组可组合的熵编码算法，重点关注正确性、多功能性、易用性、压缩性能和计算效率。constriction 的目标有三

1. 通过提供一组可组合的原始算法（例如，你可以轻松地将范围编码器切换为ANS编码器，而无需找到一个新的库或更改你表示精确可逆熵模型的方式）来促进对新型无损和有损压缩方法的研究；
2. 通过为Python（用于在研究代码上快速原型设计）和Rust（将成功的原型转换为独立二进制文件、库或WebAssembly模块）提供类似API和二进制兼容的熵编码器来简化从研究代码到部署软件的过渡；
3. 通过在一个统一的框架内提供各种熵编码原始算法来作为教学资源。查看我们从数据压缩大学课程中获得的额外教学材料（包含一些使用constriction（附带解决方案）的问题集）。

更多信息： 项目网站

实时演示： 这里有一个Web应用程序，它最初是一个Python机器学习研究项目，后来通过使用WebAssembly模块中的constriction将其转换为Web应用程序。

快速入门

将以下内容添加到您的Cargo.toml

[dependencies]
constriction = "0.3.5"
probability = "0.20.3" # Not strictly required but used in many examples.

编码示例

在此示例中，我们将使用量化的高斯分布作为熵模型来编码一些符号。每个符号将由具有不同均值和标准差的高斯分布来模拟（这样示例就不会太简单）。我们将使用probability crate来处理高斯分布，因此请确保您在Cargo.toml中具有以下依赖项

probability = "0.17"

现在，我们来对一些符号进行编码（即压缩）。在这里，我们将使用非对称数字系统（ANS）编码器，因为它速度快、压缩性能好。我们将在下面讨论如何用范围编码器或类似霍夫曼编码的符号编码来替换ANS编码器。

use constriction::stream::{stack::DefaultAnsCoder, model::DefaultLeakyQuantizer};
use probability::distribution::Gaussian;

fn encode_sample_data() -> Vec<u32> {
    // Create an empty ANS Coder with default word and state size:
    let mut coder = DefaultAnsCoder::new();

    // Some made up data and entropy models for demonstration purpose:
    let symbols = [23i32, -15, 78, 43, -69];
    let means = [35.2, -1.7, 30.1, 71.2, -75.1];
    let stds = [10.1, 25.3, 23.8, 35.4, 3.9];

    // Create an adapter that integrates 1-d probability density functions over bins
    // `[n - 0.5, n + 0.5)` for all integers `n` from `-100` to `100` using fixed point
    // arithmetic with default precision, guaranteeing a nonzero probability for each bin:
    let quantizer = DefaultLeakyQuantizer::new(-100..=100);

    // Encode the data (in reverse order, since ANS Coding operates as a stack):
    coder.encode_symbols_reverse(
        symbols.iter().zip(&means).zip(&stds).map(
            |((&sym, &mean), &std)| (sym, quantizer.quantize(Gaussian::new(mean, std)))
    )).unwrap();

    // Retrieve the compressed representation (filling it up to full words with zero bits).
    coder.into_compressed().unwrap()
}

assert_eq!(encode_sample_data(), [0x421C_7EC3, 0x000B_8ED1]);

解码示例

现在，让我们从压缩表示中重建样本数据。

use constriction::stream::{stack::DefaultAnsCoder, model::DefaultLeakyQuantizer, Decode};
use probability::distribution::Gaussian;

fn decode_sample_data(compressed: Vec<u32>) -> Vec<i32> {
    // Create an ANS Coder with default word and state size from the compressed data:
    // (ANS uses the same type for encoding and decoding, which makes the method very flexible
    // and allows interleaving small encoding and decoding chunks, e.g., for bits-back coding.)
    let mut coder = DefaultAnsCoder::from_compressed(compressed).unwrap();

    // Same entropy models and quantizer we used for encoding:
    let means = [35.2, -1.7, 30.1, 71.2, -75.1];
    let stds = [10.1, 25.3, 23.8, 35.4, 3.9];
    let quantizer = DefaultLeakyQuantizer::new(-100..=100);

    // Decode the data:
    coder.decode_symbols(
        means.iter().zip(&stds).map(
            |(&mean, &std)| quantizer.quantize(Gaussian::new(mean, std))
    )).collect::<Result<Vec<_>, _>>().unwrap()
}

assert_eq!(decode_sample_data(vec![0x421C_7EC3, 0x000B_8ED1]), [23, -15, 78, 43, -69]);

练习

尝试上述示例，验证解码是否能够恢复原始数据。然后看看constriction如何使替换ANS编码器为范围编码器变得多么简单，通过以下替换

在编码器中，

• 将constriction::stream::stack::DefaultAnsCoder替换为constriction::stream::queue::DefaultRangeEncoder；并且
• 将coder.encode_symbols_reverse替换为coder.encode_symbols（由于范围编码器作为队列操作，即先进先出，你不再需要按反向顺序编码符号）。你还需要在文件顶部添加一行use constriction::stream::Encode;，以便将特性行为encode_symbols引入作用域。

use constriction::stream::{
    model::DefaultLeakyQuantizer,
    queue::{DefaultRangeEncoder, DefaultRangeDecoder},
    Encode, Decode,
};
use probability::distribution::Gaussian;

fn encode_sample_data() -> Vec<u32> {
    // Create an empty Range Encoder with default word and state size:
    let mut encoder = DefaultRangeEncoder::new();

    // Same made up data, entropy models, and quantizer as in the ANS Coding example above:
    let symbols = [23i32, -15, 78, 43, -69];
    let means = [35.2, -1.7, 30.1, 71.2, -75.1];
    let stds = [10.1, 25.3, 23.8, 35.4, 3.9];
    let quantizer = DefaultLeakyQuantizer::new(-100..=100);

    // Encode the data (this time in normal order, since Range Coding is a queue):
    encoder.encode_symbols(
        symbols.iter().zip(&means).zip(&stds).map(
            |((&sym, &mean), &std)| (sym, quantizer.quantize(Gaussian::new(mean, std)))
    )).unwrap();

    // Retrieve the (sealed up) compressed representation.
    encoder.into_compressed().unwrap()
}

fn decode_sample_data(compressed: Vec<u32>) -> Vec<i32> {
    // Create a Range Decoder with default word and state size from the compressed data:
    let mut decoder = DefaultRangeDecoder::from_compressed(compressed).unwrap();

    // Same entropy models and quantizer we used for encoding:
    let means = [35.2, -1.7, 30.1, 71.2, -75.1];
    let stds = [10.1, 25.3, 23.8, 35.4, 3.9];
    let quantizer = DefaultLeakyQuantizer::new(-100..=100);

    // Decode the data:
    decoder.decode_symbols(
        means.iter().zip(&stds).map(
            |(&mean, &std)| quantizer.quantize(Gaussian::new(mean, std))
    )).collect::<Result<Vec<_>, _>>().unwrap()
}

let compressed = encode_sample_data();

// We'll get a different compressed representation than in the ANS Coding
// example because we're using a different entropy coding algorithm ...
assert_eq!(compressed, [0x1C31EFEB, 0x87B430DA]);

// ... but as long as we decode with the matching algorithm we can still reconstruct the data:
assert_eq!(decode_sample_data(compressed), [23, -15, 78, 43, -69]);