Feature Stores
Training-serving skew — where features computed at training time differ subtly from those computed at serving time — is one of the most insidious production ML bugs. Feature stores solve this by centralising feature computation, ensuring the exact same logic runs in both training pipelines and real-time serving, and enabling feature reuse across teams and models.
The Problem: Training-Serving Skew
Consider a fraud model that uses "number of transactions in the last hour" as a feature. During training, this is computed from historical batch data. In production, it's computed in real time from a streaming database. Any difference in how "last hour" is computed — timezone handling, null treatment, windowing boundaries — silently degrades model performance.
Training pipeline: Python script reads from S3 → computes features with Pandas. Serving pipeline: Java microservice reads from Redis → computes features differently. Skew is invisible until production accuracy drops.
One feature definition (Python). Training retrieves historical point-in-time features from the offline store. Serving retrieves the same feature from the online store. Same logic, guaranteed consistency.
Feature Store Architecture
Offline Store
Historical feature values for training. Backed by data warehouses (BigQuery, Snowflake, S3 + Parquet). Supports point-in-time correct joins — retrieves the feature value as it existed at the time of each training example (prevents data leakage).
Online Store
Low-latency feature retrieval for real-time inference. Backed by Redis, DynamoDB, or Bigtable. Features are pre-computed and cached. P99 latency <5ms for most use cases.
Feast: Open-Source Feature Store
Feast is the most widely used open-source feature store. Here's a complete example:
# 1. Define feature views (feature_repo/features.py)
from feast import Entity, FeatureView, Field, FileSource
from feast.types import Float32, Int64
from datetime import timedelta
customer = Entity(name="customer_id", description="Customer identifier")
customer_stats = FeatureView(
name="customer_transaction_stats",
entities=[customer],
ttl=timedelta(days=7),
schema=[
Field(name="tx_count_1h", dtype=Int64),
Field(name="tx_amount_avg", dtype=Float32),
Field(name="days_since_join", dtype=Int64),
],
source=FileSource(path="data/customer_stats.parquet",
timestamp_field="event_timestamp"),
)
# 2. Apply registry
# feast apply
# 3. Training: retrieve historical features
from feast import FeatureStore
import pandas as pd
store = FeatureStore(repo_path=".")
training_df = store.get_historical_features(
entity_df=pd.DataFrame({
"customer_id": [1, 2, 3],
"event_timestamp": pd.to_datetime(["2024-01-01", "2024-01-01", "2024-01-01"])
}),
features=["customer_transaction_stats:tx_count_1h",
"customer_transaction_stats:tx_amount_avg"],
).to_df()
# 4. Serving: retrieve online features (after materialisation)
# feast materialize-incremental $(date -u +"%Y-%m-%dT%H:%M:%S")
online_features = store.get_online_features(
features=["customer_transaction_stats:tx_count_1h"],
entity_rows=[{"customer_id": 42}],
).to_dict()Feature Store Options Compared
Point-in-Time Correct Joins
The most critical feature store capability. During training, naive joins use the latest feature value — but this leaks future information. If you're predicting whether a customer will churn in March, you shouldn't use their April transaction count as a feature.
A model trained with future feature values will look great in offline evaluation but fail completely in production. Point-in-time joins ensure each training row only uses feature values available before the prediction timestamp. Always use this for time-series and event-based ML problems.
When Do You Need a Feature Store?
You Need One When...
- Multiple models use the same features
- You've had training-serving skew bugs
- Feature computation takes >10 minutes
- >3 data scientists sharing features
- Real-time feature latency matters (<10ms)
- Compliance requires feature auditability
Probably Don't Need One If...
- Only 1–2 models in production
- Features are simple (no windowing/aggregation)
- Batch predictions only (no real-time serving)
- Small team with tight code review
- Early-stage project, still iterating
- Infrastructure cost is a concern
Before a full feature store, a shared feature computation library (a Python package that both training and serving import) eliminates most skew. This is 20% of the complexity for 80% of the benefit. Adopt Feast when feature sharing across teams becomes the bottleneck.
Frequently Asked Questions
How does a feature store handle real-time (streaming) features?
Streaming features require a stream processor (Kafka + Flink/Spark Streaming) to continuously compute aggregations and write to the online store. Tecton and Hopsworks have native streaming support. With Feast, you handle stream processing separately (e.g., with Flink) and write results to Feast's online store. Real-time features add significant infrastructure complexity — batch features recomputed hourly cover most use cases.
What is feature materialisation?
Materialisation is the process of computing and writing feature values into the online store so they can be retrieved at serving time with low latency. You run feast materialize (batch) or feast materialize-incremental on a schedule (e.g., every hour) to keep the online store fresh. Without materialisation, online feature retrieval would require recomputing from raw data on every request — too slow.
Can I use a feature store with LLMs?
Yes, for structured features used alongside LLMs. In a RAG system, you might store pre-computed document embeddings in a feature store / vector database, or use a feature store to provide user context features (subscription tier, language preference, interaction history) that get included in the LLM prompt. The feature store handles the structured data side; the vector database handles semantic retrieval.
Frequently Asked Questions
What is a feature store in machine learning?
A feature store is a central place to define, store, and serve the input features your machine learning models use. It runs the same feature logic for both training and live predictions, so a model sees identical data in development and production.
What is training-serving skew?
Training-serving skew is when a feature is computed slightly differently during training than during live serving — for example, different timezone or null handling. The model trained on one version then receives subtly different inputs in production, which quietly hurts accuracy.
How does a feature store prevent training-serving skew?
It defines each feature once and reuses that exact definition everywhere. Training pipelines and the real-time serving layer both call the same feature logic from the store, so there is no chance for the two to drift apart.
What is the difference between Feast and Tecton?
Feast is a popular open-source feature store you run and manage yourself. Tecton is a fully managed commercial platform built by the original Feast creators, adding hosted infrastructure, real-time pipelines, and enterprise features on top of the same core idea.