Struct SqliteDataset 

pub struct SqliteDataset<I> { /* private fields */ }

This struct represents a dataset where all items are stored in an SQLite database. Each instance of this struct corresponds to a specific table within the SQLite database, and allows for interaction with the data stored in the table in a structured and typed manner.

The database is read and written with Turso, which implements the engine in Rust rather than linking libsqlite3, so the files stay interchangeable with any other SQLite tooling. Note that this does not make the build free of C: turso pulls simsimd for its SIMD kernels, which compiles C unconditionally.

The SQLite database must contain a table with the same name as the split field. This table should have a primary key column named row_id, which is used to index the rows in the table. The row_id should start at 1, while the corresponding dataset index should start at 0, i.e., row_id = index + 1.

Table columns can be represented in two ways:

1. The table can have a column for each field in the I struct. In this case, the column names in the table should match the field names of the I struct. The field names can be a subset of column names and can be in any order.

For the supported field types, refer to SQLite data types. Each column is handed to serde as the value its storage class implies: INTEGER as i64, REAL as f64, TEXT as String, and BLOB as a byte sequence. Booleans are read back from integers, and a nullable column maps to an Option field. Reading a NULL into a bare f32 or f64 field yields NaN.

A Vec<u8> field derives as a sequence, so a large BLOB column is walked one byte at a time. Tagging the field #[serde(with = "serde_bytes")] moves the blob across instead, which is substantially cheaper for columns of any size.

2. The fields in the I struct can be serialized into a single column item in the table. In this case, the table should have a single column named item of type BLOB. This is useful when the I struct contains complex fields that cannot be mapped to a SQLite type, such as nested structs, vectors, etc. The serialization is done using MessagePack.

Note: The code automatically figures out which of the above two cases is applicable, and uses the appropriate method to read the data from the table.

Implementations

impl<I> SqliteDataset<I>

pub fn from_db_file<P>( db_file: P, split: &str, ) -> Result<SqliteDataset<I>, SqliteDatasetError>

where P: AsRef<Path>,

Initializes a SqliteDataset from a SQLite database file and a split name.

pub fn db_file(&self) -> PathBuf

Get the database file name.

pub fn split(&self) -> &str

Get the split name.

Trait Implementations

impl<I> Dataset<I, SqliteDatasetError> for SqliteDataset<I>

where I: Clone + Send + Sync + DeserializeOwned,

fn get(&self, index: usize) -> Result<I, SqliteDatasetError>

Get an item from the dataset.

Panics

Panics if index >= len().

fn get_many(&self, indexes: Vec<usize>) -> Result<Vec<I>, SqliteDatasetError>

Get multiple items from the dataset in a single query.

Panics

Panics if indexes[i] >= len().

fn len(&self) -> usize

Return the number of rows in the dataset.

fn is_empty(&self) -> bool

Checks if the dataset is empty.

fn iter(&self) -> DatasetIterator<'_, I, E>

where Self: Sized,

Returns an iterator over the dataset.

impl<I> Debug for SqliteDataset<I>

where I: Debug,

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.