Struct SqliteDatasetWriter 

pub struct SqliteDatasetWriter<I> { /* private fields */ }

This SqliteDatasetWriter struct is a SQLite database writer dedicated to storing datasets. It retains the current writer’s state and its database connection.

Being thread-safe, this writer can be concurrently used across multiple threads.

Typical applications include:

• Generation of a new dataset
• Storage of preprocessed data or metadata
• Enlargement of a dataset’s item count post preprocessing

Implementations

impl<I> SqliteDatasetWriter<I>

where I: Clone + Send + Sync + Serialize + DeserializeOwned,

pub fn new<P>( db_file: P, overwrite: bool, ) -> Result<SqliteDatasetWriter<I>, SqliteDatasetError>

where P: AsRef<Path>,

Creates a new instance of SqliteDatasetWriter.

Arguments

• db_file - A reference to the Path that represents the database file path.
• overwrite - A boolean indicating if the existing database file should be overwritten.

Returns

• A Result which is Ok if the writer could be created, Err otherwise.

pub fn write(&self, split: &str, item: &I) -> Result<usize, SqliteDatasetError>

Serializes and writes an item to the database. The item is written to the table for the specified split. If the table does not exist, it is created. If the table exists, the item is appended to the table. The serialization is done using the MessagePack

Arguments

• split - A string slice that defines the data split for writing (e.g., “train”, “test”).
• item - A reference to the item to be written to the database.

Returns

• A Result containing the index of the inserted row if successful, an error otherwise.

The index is provisional until Self::set_completed returns Ok. It belongs to a row in a transaction that has not committed yet, and a batch that is later rolled back gives its rowids back to the engine. A run whose set_completed fails publishes nothing, so indexes recorded during it must be discarded along with it.

pub fn set_completed(&mut self) -> Result<(), SqliteDatasetError>

Marks the dataset as completed and persists the temporary database file.

Trait Implementations

impl<I> Debug for SqliteDatasetWriter<I>

where I: Debug,

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl<I> Drop for SqliteDatasetWriter<I>

fn drop(&mut self)

Removes the write-ahead log beside the temporary database.

gix_tempfile owns {name}.db.tmp, but it knows nothing about the -wal sidecar turso writes next to it. That sidecar holds the tail of the dataset, so a writer abandoned before set_completed would otherwise leave the bulk of what it wrote on disk with nothing left to reference it.

This covers an ordinary drop only. gix_tempfile::signal::setup still takes the .db.tmp away on SIGINT or SIGTERM, but Drop does not run there, so a signalled process leaves the sidecar behind; init clears it on the next run.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.