Class LocalInferenceEngine<T>
- java.lang.Object
-
- com.codedstream.otterstream.inference.engine.LocalInferenceEngine<T>
-
- Type Parameters:
T- the type of the loaded model
- All Implemented Interfaces:
InferenceEngine<T>
- Direct Known Subclasses:
OnnxInferenceEngine,PmmlInferenceEngine,TensorFlowGraphDefEngine,TensorFlowInferenceEngine,TensorFlowSavedModelEngine,TorchScriptInferenceEngine,XGBoostInferenceEngine
public abstract class LocalInferenceEngine<T> extends Object implements InferenceEngine<T>
Abstract base class for local inference engines that load models from files.Provides common functionality for engines that run models locally on the Flink TaskManager, as opposed to remote inference via API calls.
For Engine Implementers:
Extend this class and implement:
infer(Map)- single inference logicinferBatch(Map[])- batch inference logicgetCapabilities()- engine capabilitiesInferenceEngine.getMetadata()- model metadata
You can either:
- Provide a
ModelLoadervia constructor, OR - Override
loadModelDirectly(ModelConfig)to handle loading yourself
Example Implementation:
{@code public class MyEngine extends LocalInferenceEngine{ - Since:
- 1.0.0
- Author:
- Nestor Martourez, Sr Software and Data Streaming Engineer @ CodedStreams
- See Also:
InferenceEngine
-
-
Nested Class Summary
-
Nested classes/interfaces inherited from interface com.codedstream.otterstream.inference.engine.InferenceEngine
InferenceEngine.EngineCapabilities
-
-
Field Summary
Fields Modifier and Type Field Description protected booleaninitializedprotected TloadedModelprotected ModelConfigmodelConfigprotected ModelLoader<T>modelLoader
-
Constructor Summary
Constructors Constructor Description LocalInferenceEngine()
-
Method Summary
All Methods Instance Methods Abstract Methods Concrete Methods Modifier and Type Method Description voidclose()Closes the inference engine and releases all resources.abstract InferenceEngine.EngineCapabilitiesgetCapabilities()Gets the capabilities of this inference engine.ModelConfiggetModelConfig()Gets the configuration used to initialize this engine.abstract InferenceResultinfer(Map<String,Object> inputs)Performs inference on a single input.abstract InferenceResultinferBatch(Map<String,Object>[] batchInputs)Performs batch inference on multiple inputs.voidinitialize(ModelConfig config)Initializes the inference engine with the given configuration.booleanisReady()Checks if the engine is ready for inference operations.protected voidloadModelDirectly(ModelConfig config)Override this method for engines that handle their own model loading.-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface com.codedstream.otterstream.inference.engine.InferenceEngine
getMetadata
-
-
-
-
Field Detail
-
modelConfig
protected ModelConfig modelConfig
-
loadedModel
protected T loadedModel
-
modelLoader
protected ModelLoader<T> modelLoader
-
initialized
protected boolean initialized
-
-
Method Detail
-
initialize
public void initialize(ModelConfig config) throws InferenceException
Description copied from interface:InferenceEngineInitializes the inference engine with the given configuration.Loads the model and prepares the engine for inference operations.
- Specified by:
initializein interfaceInferenceEngine<T>- Parameters:
config- model configuration- Throws:
InferenceException- if initialization fails
-
loadModelDirectly
protected void loadModelDirectly(ModelConfig config) throws InferenceException
Override this method for engines that handle their own model loading.Called during
initialize(ModelConfig)if no ModelLoader is provided.- Parameters:
config- model configuration containing path and options- Throws:
InferenceException- if model loading fails
-
infer
public abstract InferenceResult infer(Map<String,Object> inputs) throws InferenceException
Description copied from interface:InferenceEnginePerforms inference on a single input.- Specified by:
inferin interfaceInferenceEngine<T>- Parameters:
inputs- map of input name to input value- Returns:
- inference result containing predictions
- Throws:
InferenceException- if inference fails
-
inferBatch
public abstract InferenceResult inferBatch(Map<String,Object>[] batchInputs) throws InferenceException
Description copied from interface:InferenceEnginePerforms batch inference on multiple inputs.Batch inference is typically more efficient than multiple single inferences.
- Specified by:
inferBatchin interfaceInferenceEngine<T>- Parameters:
batchInputs- array of input maps- Returns:
- inference result containing batch predictions
- Throws:
InferenceException- if inference fails
-
isReady
public boolean isReady()
Description copied from interface:InferenceEngineChecks if the engine is ready for inference operations.- Specified by:
isReadyin interfaceInferenceEngine<T>- Returns:
- true if engine is initialized and ready
-
getModelConfig
public ModelConfig getModelConfig()
Description copied from interface:InferenceEngineGets the configuration used to initialize this engine.- Specified by:
getModelConfigin interfaceInferenceEngine<T>- Returns:
- model configuration
-
close
public void close() throws InferenceExceptionDescription copied from interface:InferenceEngineCloses the inference engine and releases all resources.After calling this method, the engine should not be used again.
- Specified by:
closein interfaceInferenceEngine<T>- Throws:
InferenceException- if cleanup fails
-
getCapabilities
public abstract InferenceEngine.EngineCapabilities getCapabilities()
Description copied from interface:InferenceEngineGets the capabilities of this inference engine.- Specified by:
getCapabilitiesin interfaceInferenceEngine<T>- Returns:
- engine capabilities (batching, GPU support, etc.)
-
-