Fermentation ML: Advisory-First Process Control at the Plant Floor

We built the Fermentation Realtime ML System as an on-premises industrial ML platform for fermentation process optimization. It runs entirely on a single server inside the facility — no cloud dependency, no internet requirement for any real-time function.

The core capability is 60-minute trajectory forecasting for fermentation tanks. The system ingests live sensor data (temperature, Brix, CO2, pH, dissolved oxygen) via an MQTT broker, computes rolling-window features at 10-second cadence, and runs two model types: XGBoost soft sensors for estimating lab-quality state variables from cheap sensor signals, and LSTM+attention networks for sequence forecasting.

The output is not an autonomous command. It is a ranked list of setpoint recommendations — each scored, each carrying a human-readable rationale, each requiring operator approval before anything writes to the plant control layer.

Deployment is a Docker Compose stack: MQTT broker, Python inference service (FastAPI), React dashboard, and SQLite database, all on one server or VM.

---

Why advisory-first rather than autonomous control?

Most industrial ML implementations sit at one of two extremes: a dashboard curiosity that operators ignore, or a fully autonomous system that operators fear. We designed a deliberate middle layer.

Advisory-first addresses operator trust directly. The ML layer suggests; the operator decides. Hard safety constraints (maximum temperature, maximum ramp rate, maximum delta per action) are enforced in code before any recommendation is published. "Hold current setpoint" is always present on the ranked list. There is always a conservative fallback.

The delivery model reinforces this through six milestones (M0-M5): from data collection only (M0) through advisory recommendations (M2-M3) to shadow mode at M4 — where the complete pipeline runs against live plant data with zero write risk, accumulating tracked recommendation quality for weeks before any controlled mode is considered. M5 enables controlled writes, still with operator approval.

The energy penalty term in the scoring function encodes sustainability awareness directly into the recommendation ranking: large temperature deltas are penalized against fermentation outcome score, making the conservative choice also the economical one.

---

How the LSTM+attention architecture works

# Sequence: 60 time steps (10 minutes at 10s cadence)
# Model: 2-layer LSTM (64 hidden units), dropout 0.2
# Attention layer before output projection

# Feature windows declared once, shared by training and inference:
# configs/default.yaml -> rolling windows: [5min, 10min, 30min, 60min]
# FeatureBuilder instantiated in both training pipeline and inference runtime
# Model bundle validates features.json schema at load time before serving

Train/serve feature consistency is enforced architecturally. The same FeatureBuilder class runs in both the offline training pipeline and the online inference runtime. Each trained model is packaged as a self-contained versioned bundle: manifest.json, features.json, metrics.json, metadata.json (including git commit hash and data snapshot hash), and inference_config.json. Any historical model state is fully reproducible from the bundle alone.

---

Where it applies

The system is a reference implementation designed to be forked for different fermentation facilities or adapted for other process industries. The advisory-first pattern, the shadow mode validation gate, and the versioned model bundle format are all portable patterns independent of the fermentation domain. Any regulated environment where autonomous control is not acceptable without a proven track record — food production, chemical processing, pharmaceuticals — can adopt this approach and progress through the same milestone-based delivery model.