Module Overview

Dependency Graph

All engine modules depend on ml-inference-core. The SQL module depends on ml-inference-core and optionally on whichever engine modules are on the runtime classpath. Flink is always provided scope — it must never be bundled in the shaded JAR.

otter-streams (parent pom)
|
+-- ml-inference-core          <-- InferenceEngine, InferenceResult, ModelCache, metrics
|
+-- otter-stream-sql           <-- depends on: ml-inference-core
|
+-- otter-stream-onnx          <-- depends on: ml-inference-core, onnxruntime
+-- otter-stream-tensorflow    <-- depends on: ml-inference-core, tensorflow-core-platform
+-- otter-streams-xgboost      <-- depends on: ml-inference-core, xgboost4j
+-- otter-stream-pmml          <-- depends on: ml-inference-core, jpmml-evaluator
+-- otter-stream-pytorch       <-- depends on: ml-inference-core, djl-pytorch-engine
+-- otter-stream-remote        <-- depends on: ml-inference-core, okhttp, aws-sdk
|
+-- otter-stream-examples      (not published)

ml-inference-core

The foundation of the library. Every other module declares a compile dependency on this artifact. It contains no ML framework code — only the contracts, configuration objects, caching layer, and Flink async function wiring that all engines share.

Key classes

Runtime layer (com.codedstream.otterstream.runtime) — new in 0.1.0

OtterRuntime is an optional facade that sits above the classes in the table above. It does not replace them — every existing class continues to work exactly as before and does not require the runtime to function. What it adds is a Provider SPI (so engine implementations are discovered rather than hardcoded) and a Lifecycle Manager (so a model version can be validated, warmed, and atomically swapped in without ever exposing a partially-initialized engine to callers).

OtterRuntime runtime = OtterRuntime.builder()
    .metrics(new MetricsCollector())
    .build(); // auto-discovers otter-stream-* providers on the classpath via ServiceLoader

ManagedModel model = runtime.deploy(ModelConfig.builder()
    .modelId("fraud-detector")
    .modelPath("file:///models/fraud_detection.onnx")
    .format(ModelFormat.ONNX)
    .modelVersion("1.0")
    .build());

InferenceResult result = runtime.infer("fraud-detector", inputs);

// later — hot-swap to v2; no code change needed at the call site above.
// the old engine is drained (in-flight requests finish) before it's closed.
runtime.deploy(v2Config);

runtime.close();

Provider coverage: ONNX, TensorFlow, PyTorch, XGBoost, PMML, remote HTTP, and SageMaker all ship an InferenceProvider and are auto-discovered. Vertex AI and gRPC remote clients are not yet covered by the SPI — their constructors require an InferenceConfig at construction time rather than following the no-arg-constructor-then-initialize(ModelConfig) pattern the rest of the engines use, so they can't be created generically yet. They remain fully usable directly (outside OtterRuntime); aligning them with the rest of the SPI is tracked as a follow-up.

Dynamic model loading & rollback

OtterRuntime.watch(modelId, interval) polls the configured ModelRegistry on a per-model schedule and deploys new versions automatically as they're registered — no code needs to call deploy() by hand. Every swap, whether triggered manually or by the watcher, drains the outgoing engine (waits for its in-flight requests to finish, up to a configurable timeout) before closing it, so a hot swap under load never severs an in-progress call. rollback() re-deploys the last known-good version if a new one turns out to be bad.

runtime.watch("fraud-detector", Duration.ofSeconds(30));
// ... new versions registered in the ModelRegistry are now picked up automatically ...
runtime.unwatch("fraud-detector");

// something's wrong with the latest deploy — go back to the previous version
runtime.rollback("fraud-detector");

Shadow & canary deployments

Canary: deploy a candidate version alongside the current primary at a chosen traffic percentage; each call to infer() is routed to the canary or the primary per that split. promoteCanary() makes it the new primary (and drains/retires the old one); rollbackCanary() discards it, leaving the primary untouched.

Shadow: deploy a version that receives a sampled, asynchronous copy of traffic purely for comparison — it never affects what the caller receives, and a ShadowListener is notified with each (primary, shadow) result pair.

// canary: 10% of traffic to the candidate
runtime.deployCanary(v2Config, 10);
// looks good after watching metrics/logs for a while...
runtime.promoteCanary("fraud-detector");
// ...or if it doesn't:
// runtime.rollbackCanary("fraud-detector");

// shadow: mirror 5% of traffic to a candidate, compare offline, no user-facing impact
runtime.addShadowListener((modelId, shadowVersion, primary, shadow, error) -> {
    if (error == null && !primary.equals(shadow)) {
        log.warn("Shadow mismatch for {} v{}", modelId, shadowVersion);
    }
});
runtime.deployShadow("fraud-detector", candidateConfig, 0.05);
runtime.stopShadow("fraud-detector");

Maven

<dependency>
  <groupId>com.codedstreams</groupId>
  <artifactId>ml-inference-core</artifactId>
  <version>0.0.4</version>
</dependency>

Feature store providers

Implementations of the FeatureProvider SPI (runtime.spi.FeatureProvider). Unlike InferenceProviders, feature providers always need connection details (a host, a JDBC URL, a base URL) to be useful, so they're constructed directly by your application rather than auto-discovered — the same reasoning that keeps ModelCache and MetricsCollector explicit constructor calls instead of SPI-discovered singletons. OtterRuntime does not yet wire feature lookups into the inference call path itself; these providers are ready to call directly from your own pre-processing step.

RedisFeatureProvider redis = new RedisFeatureProvider("localhost", 6379, "features:user:");
Map<String, Object> values = redis.fetch("42", List.of("age", "country", "ltv_score"));

JdbcFeatureProvider jdbc = new JdbcFeatureProvider(
        "jdbc:postgresql://localhost:5432/features", "reader", "secret",
        "user_features", "user_id");
Map<String, Object> row = jdbc.fetch("42", List.of("age", "country"));

FeastFeatureProvider feast = new FeastFeatureProvider("http://localhost:6566", "driver_id");
Map<String, Object> feats = feast.fetch("1001",
        List.of("driver_hourly_stats:conv_rate", "driver_hourly_stats:acc_rate"));

otter-stream-rules, otter-stream-rules-drools, otter-stream-kafka

Post-inference decision engine and result publishing. Full documentation, including architecture diagrams and the expression language reference, lives on its own page: Rule Engine →

otter-stream-sql

Provides everything needed to use Otter Streams from Flink SQL without writing Java operators. The module registers a scalar UDF, a lookup table function, and a full dynamic table connector (ml-inference) that the Flink planner discovers via Java SPI.

Key classes

Maven

<dependency>
  <groupId>com.codedstreams</groupId>
  <artifactId>otter-stream-sql</artifactId>
  <version>0.0.4</version>
</dependency>

otter-stream-onnx

Wraps Microsoft's ONNX Runtime Java API. Supports CPU, CUDA, and TensorRT execution providers with automatic provider fallback. Input tensors are constructed from Map<String, Object> with automatic float32 / int64 conversion.

Supported execution providers

Maven

<dependency>
  <groupId>com.codedstreams</groupId>
  <artifactId>otter-stream-onnx</artifactId>
  <version>0.0.4</version>
</dependency>

otter-stream-tensorflow

Loads TensorFlow SavedModel directories and performs inference via the official TensorFlow Java API. Signature names and tensor names are discovered automatically from the model's saved_model.pb.

Maven

<dependency>
  <groupId>com.codedstreams</groupId>
  <artifactId>otter-stream-tensorflow</artifactId>
  <version>0.0.4</version>
</dependency>

otter-streams-xgboost

Loads XGBoost models using XGBoost4J. Converts feature maps to a DMatrix and invokes Booster.predict(). Thread-safe via per-call DMatrix construction. Supports binary classification (binary:logistic), multi-class, and regression objectives.

Supported file formats

Maven

<dependency>
  <groupId>com.codedstreams</groupId>
  <artifactId>otter-streams-xgboost</artifactId>
  <version>0.0.4</version>
</dependency>

otter-stream-pmml

Evaluates PMML 4.x documents using the JPMML-Evaluator library. Supports logistic regression, decision trees, random forests, gradient boosted trees, neural networks, and naïve Bayes. Preprocessing transformations defined inside the PMML document are applied automatically.

Maven

<dependency>
  <groupId>com.codedstreams</groupId>
  <artifactId>otter-stream-pmml</artifactId>
  <version>0.0.4</version>
</dependency>

otter-stream-remote

Routes inference calls to external endpoints: REST APIs, AWS SageMaker, GCP Vertex AI, and Azure ML. Uses OkHttp for HTTP/1.1 and gRPC for binary protocol communication. Includes configurable retry, timeout, and circuit-breaker policies. Useful when the model is too large to embed in the Flink JAR or when a dedicated model server is preferred.

Maven

<dependency>
  <groupId>com.codedstreams</groupId>
  <artifactId>otter-stream-remote</artifactId>
  <version>0.0.4</version>
</dependency>