Package com.codedstream.otterstream.inference.cache

Class ModelCache<K,​V>

  • Type Parameters:
    K - the type of cache keys (typically String hashes)
    V - the type of cached values (typically InferenceResult)
    public class ModelCache<K,​V>
extends Object
    Thread-safe LRU cache for storing ML model predictions and inference results.

    This cache uses Caffeine for high-performance caching with automatic eviction based on size and time constraints. Ideal for caching inference results in streaming applications to reduce latency and computational overhead.

    Features:

    • Thread-safe operations for concurrent Flink streams
    • Automatic eviction based on LRU policy
    • Time-based expiration for stale data
    • Configurable maximum size

    Usage Example:

    
 // Create cache for 10,000 entries, expiring after 30 minutes
 ModelCache<String, InferenceResult> cache =
     new ModelCache<>(10000, 30);

 // Store prediction
 cache.put("input-hash-123", predictionResult);

 // Retrieve cached prediction
 InferenceResult cached = cache.get("input-hash-123");
 if (cached != null) {
     // Use cached result
 }
    Since:
    1.0.0
    Author:
    Nestor Martourez, Sr Software and Data Streaming Engineer @ CodedStreams

    • Constructor Detail

      • ModelCache

        public ModelCache​(long maximumSize,
                  long expireAfterWriteMinutes)
        Creates a new model cache with specified size and expiration policy.
        Parameters:
        maximumSize - maximum number of entries to store
        expireAfterWriteMinutes - time in minutes before entries expire

    • Method Detail

      • get

        public V get​(K key)
        Retrieves a cached value if present.
        Parameters:
        key - the cache key
        Returns:
        the cached value, or null if not found or expired

      • put

        public void put​(K key,
                V value)
        Stores a value in the cache.
        Parameters:
        key - the cache key
        value - the value to cache

      • invalidate

        public void invalidate​(K key)
        Removes a specific entry from the cache.
        Parameters:
        key - the key to invalidate

      • invalidateAll

        public void invalidateAll()
        Clears all entries from the cache.

      • size

        public long size()
        Returns the approximate current size of the cache.
        Returns:
        estimated number of entries in cache