Struct MhaCache
pub struct MhaCache<B>where
B: Backend,{ /* private fields */ }Expand description
Cache for the Multi Head Attention layer.
To be used during inference when decoding tokens.
Implementations§
§impl<B> MhaCache<B>where
B: Backend,
impl<B> MhaCache<B>where
B: Backend,
pub fn autoregressive() -> MhaCache<B>
pub fn autoregressive() -> MhaCache<B>
Initialize a cache for autoregressive inference.
pub fn autoregressive_cross_attention() -> MhaCache<B>
pub fn autoregressive_cross_attention() -> MhaCache<B>
Initialize a cache for autoregressive inference, but with a fixed memory used for keys and values (cross-attention).
Auto Trait Implementations§
impl<B> Freeze for MhaCache<B>where
<B as Backend>::FloatTensorPrimitive: Freeze,
<B as Backend>::QuantizedTensorPrimitive: Freeze,
impl<B> RefUnwindSafe for MhaCache<B>where
<B as Backend>::FloatTensorPrimitive: RefUnwindSafe,
<B as Backend>::QuantizedTensorPrimitive: RefUnwindSafe,
impl<B> Send for MhaCache<B>
impl<B> Sync for MhaCache<B>
impl<B> Unpin for MhaCache<B>
impl<B> UnwindSafe for MhaCache<B>where
<B as Backend>::FloatTensorPrimitive: UnwindSafe,
<B as Backend>::QuantizedTensorPrimitive: UnwindSafe,
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more