Package com.codedstream.otterstream.inference.engine

Class LocalInferenceEngine<T>

java.lang.Object

com.codedstream.otterstream.inference.engine.LocalInferenceEngine<T>

Type Parameters:

T - the type of the loaded model

All Implemented Interfaces:

InferenceEngine<T>

Direct Known Subclasses:

OnnxInferenceEngine, PmmlInferenceEngine, TensorFlowGraphDefEngine, TensorFlowInferenceEngine, TensorFlowSavedModelEngine, TorchScriptInferenceEngine, XGBoostInferenceEngine

public abstract class LocalInferenceEngine<T>
extends Object
implements InferenceEngine<T>

Abstract base class for local inference engines that load models from files.

Provides common functionality for engines that run models locally on the Flink TaskManager, as opposed to remote inference via API calls.

For Engine Implementers:

Extend this class and implement:

• infer(Map) - single inference logic
• inferBatch(Map[]) - batch inference logic
• getCapabilities() - engine capabilities
• InferenceEngine.getMetadata() - model metadata

You can either:

• Provide a ModelLoader via constructor, OR
• Override loadModelDirectly(ModelConfig) to handle loading yourself

Example Implementation:

{@code
 public class MyEngine extends LocalInferenceEngine {

Since:

1.0.0

Author:

Nestor Martourez, Sr Software and Data Streaming Engineer @ CodedStreams

See Also:

InferenceEngine

Nested Class Summary

Nested classes/interfaces inherited from interface com.codedstream.otterstream.inference.engine.InferenceEngine

InferenceEngine.EngineCapabilities

Field Summary

Fields

Modifier and Type	Field	Description
protected boolean	initialized
protected T	loadedModel
protected ModelConfig	modelConfig
protected ModelLoader<T>	modelLoader

Constructor Summary

Constructors

Constructor	Description
LocalInferenceEngine()

Method Summary

Modifier and Type	Method	Description
void	close()	Closes the inference engine and releases all resources.
abstract InferenceEngine.EngineCapabilities	getCapabilities()	Gets the capabilities of this inference engine.
ModelConfig	getModelConfig()	Gets the configuration used to initialize this engine.
abstract InferenceResult	infer(Map<String,Object> inputs)	Performs inference on a single input.
abstract InferenceResult	inferBatch(Map<String,Object>[] batchInputs)	Performs batch inference on multiple inputs.
void	initialize(ModelConfig config)	Initializes the inference engine with the given configuration.
boolean	isReady()	Checks if the engine is ready for inference operations.
protected void	loadModelDirectly(ModelConfig config)	Override this method for engines that handle their own model loading.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.codedstream.otterstream.inference.engine.InferenceEngine

getMetadata

Field Detail

modelConfig

protected ModelConfig modelConfig

loadedModel

protected T loadedModel

modelLoader

protected ModelLoader<T> modelLoader

initialized

protected boolean initialized

Constructor Detail

LocalInferenceEngine

public LocalInferenceEngine()

Method Detail

initialize

public void initialize(ModelConfig config)
                throws InferenceException

Description copied from interface: InferenceEngine

Initializes the inference engine with the given configuration.

Loads the model and prepares the engine for inference operations.

Specified by:

initialize in interface InferenceEngine<T>

Parameters:

config - model configuration

Throws:

InferenceException - if initialization fails

loadModelDirectly

protected void loadModelDirectly(ModelConfig config)
                          throws InferenceException

Override this method for engines that handle their own model loading.

Called during initialize(ModelConfig) if no ModelLoader is provided.

Parameters:

config - model configuration containing path and options

Throws:

InferenceException - if model loading fails

infer

public abstract InferenceResult infer(Map<String,Object> inputs)
                               throws InferenceException

Description copied from interface: InferenceEngine

Performs inference on a single input.

Specified by:

infer in interface InferenceEngine<T>

Parameters:

inputs - map of input name to input value

Returns:

inference result containing predictions

Throws:

InferenceException - if inference fails

inferBatch

public abstract InferenceResult inferBatch(Map<String,Object>[] batchInputs)
                                    throws InferenceException

Description copied from interface: InferenceEngine

Performs batch inference on multiple inputs.

Batch inference is typically more efficient than multiple single inferences.

Specified by:

inferBatch in interface InferenceEngine<T>

Parameters:

batchInputs - array of input maps

Returns:

inference result containing batch predictions

Throws:

InferenceException - if inference fails

isReady

public boolean isReady()

Description copied from interface: InferenceEngine

Checks if the engine is ready for inference operations.

Specified by:

isReady in interface InferenceEngine<T>

Returns:

true if engine is initialized and ready

getModelConfig

public ModelConfig getModelConfig()

Description copied from interface: InferenceEngine

Gets the configuration used to initialize this engine.

Specified by:

getModelConfig in interface InferenceEngine<T>

Returns:

model configuration

close

public void close()
           throws InferenceException

Description copied from interface: InferenceEngine

Closes the inference engine and releases all resources.

After calling this method, the engine should not be used again.

Specified by:

close in interface InferenceEngine<T>

Throws:

InferenceException - if cleanup fails

getCapabilities

public abstract InferenceEngine.EngineCapabilities getCapabilities()

Description copied from interface: InferenceEngine

Gets the capabilities of this inference engine.

Specified by:

getCapabilities in interface InferenceEngine<T>

Returns:

engine capabilities (batching, GPU support, etc.)