Struct QuantizedBytes

pub struct QuantizedBytes {
    pub bytes: Bytes,
    pub scheme: QuantizationScheme,
    pub num_elements: usize,
}

Quantized data bytes representation.

Notes

1. The quantized values are packed into 32-bit unsigned integers. For example, int8 quantized values pack 4 grouped values into a single u32. When unpacking these values, we make sure to retrieve only the meaningful values (and ignore the alignment padding).
2. Quantization parameters are appended to the tensor data. As such, the last bytes always correspond to the scale parameter. If the quantization scheme includes an offset (zero-point) parameter, it is next to last.

Fields

bytes: Bytes

The quantized values and quantization parameters represented as bytes.

scheme: QuantizationScheme

The quantization scheme.

num_elements: usize

The number of quantized elements.

Implementations

impl QuantizedBytes

pub fn new<E>(value: Vec<E>, strategy: QuantizationStrategy) -> QuantizedBytes

where E: Element,

Creates a new quantized bytes representation.

pub fn into_vec_i8(self) -> (Vec<i8>, QParams<f32, i8>)

Returns the int8 quantized values with the quantization parameters.

pub fn dequantize(self) -> (Vec<f32>, QParams<f32, i8>)

Dequantizes the data according to its quantization scheme.