Boxkite

What do you do?

Boxkite simplifies model monitoring by capturing feature and inference distributions used in model training and comparing them against real time production distributions via Prometheus and Grafana

How much does it cost?

Free, open source

Feature List

0.5 seconds to process 1 million data points (training)

Sub millisecond p99 latency (serving)

Supports sampling for large data sets

Aggregates histograms from multiple server replicas (using PromQL)

Separate counters for discrete and continuous variables (ie. categorical and numeric features)

Initializes serving histogram bins from training data set (based on Freedman-Diaconis rule)

Handles unseen data, nan, None, inf, and negative values

One metric for each counter type (no confusion over which metric to choose)

Default configuration supports both feature and inference monitoring (easy to setup)

Small set of dependencies: prometheus, numpy, and fluentd