To confidently deploy models to production, you need to know how each model was built, trained, re-trained, and evaluated. That is where tools for model metadata storage and management come in. They give you a central place to store, organize, display, compare, search, review, and access all your models and model-related metadata. You can use tools from this category as an experiment tracking tool, model registry, or both. Read more about what model storage and management is and checkout additional resources below:
The MLOps Community has worked with vendors and community members to profile the major solutions available in the market today, based on our model store evaluation framework.
Stand-alone
Open-source
Open-source
None
Open-source: community (Discord server, GH issues, self-hosted forum, support email, social media)
N/A
None
Stand-alone
Available as a managed cloud service (SaaS) and as a commercial on-premises deployment.
Once you get the package, you can deploy it wherever you want.
See the on-premises installation guide at https://neptune.ai/wp-content/uploads/Neptune-Installation-Guide.pdf.
On-premises plans
Available during working hours 8 - 18 CEST
Available in the Scale plan.
You can manage organization and project-level access in the Scale plan.
Depending on your security needs, you can use SaaS or on-premises versions.
Read more at https://neptune.ai/wp-content/uploads/privacy-policy.pdf.
Stand-alone as an open-source project
Managed service as part of the Databricks platform
Available both as the open-source software and a managed solution as a part of the Databricks platform.
The open-source version is compatible on all clouds
Databricks platform can be deployed on Google Cloud, AWS, and Microsoft Azure
Open-source: None
Databricks platform: there are service levels guarantees that depend on your plan
Open-source: community
Databricks platform: you can get up to 24 x 7 support and response times guarantees that depend on your support plan
Open-source: None
Databricks platform: advanced user management options
Open-source: None
Databricks platform: enterprise level
Standalone platform that is infrastructure-agnostic.
Client library is open source, and the server and UI interface is closed source.
GCP, AWS, Azure an On-prem
Service level guarantees are plan-dependent.
Support is available during regular business hours online, via Slack, and over the phone for all customers.
Additional support is provided for all enterprise customers.
SAML, OIDC, Active Directory, PingFederate or OAuth2 identity providers.
Access controls can be asserted at the team and project levels.
GDPR and SOC 2 Type 2 compliant.
Part of a broader platform
Open-source with closed-source paid features
Ability to be deployed on-prem and all clouds either self hosted with OSS or managed with Enterprise
Standard SLA that can be customized for Enterprise customers
Open-source: community
Enterprise: customizable 24X7 SLA based on needs
ClearML Server supports self user management.
Enterprise offers full user management with SSO and user roles and permissions
SSO and LDAP in Enterprise as well as user management and roles and resource-based permissions
Standalone platform
Open-source or managed
Cloud-native, available in your VPC on public cloud and/or Kubernetes.
For enterprise plans
Community support on Slack, with additional channels and SLA for enterprise plans.
SSO for enterprise plans.
Integrates with cloud-native solutions for ACL.
Depends on deployment type, typically implemented by the customer as data stays in the customer cloud account.
Stand-alone
Open-source
Open-source
None
Open-source: community (Discord server, GH issues, self-hosted forum, support email, social media)
N/A
None
Standalone platform
Open-source, managed control plane, and commercial enterprise features.
All editions (free and paid) allow on-premises deployment.
The community edition is full on-premise and can be deployed on any k8s cluster.
The cloud edition is a hybrid deployment where the control plane is managed and the workload, code, logs, metrics, models, and artifacts are on the customer’s private cloud account or on-prem cluster(s).
The enterprise edition allows to deploy both the control plane and agents (data plane) on the private cloud account or the on-prem cluster(s).
SLAs can be purchased.
Community Edition comes with community support.
All paying customers have access to email and slack support.
Additional support priority and SLAs can be purchased.
No user management in the community edition.
User management is available in all paid versions with organization-level roles, ACL, and RBAC, all users can additionally issue scoped tokens to interact with the APIs on their behalf.
Business and Enterprise plans have access to advanced restrictions, additional team-level roles, and management.
SSO with SAML is available to enterprise customers.
Advanced logical restrictions and native integration with K8S RBAC.
Workload, data, code, artifacts, logs, and models are always on the customer's clusters in all editions.
Part of a broader platform
Primarily managed cloud service
AWS
Azure
GCP
For enterprise clients only (per agreement)
No
SSO available (Okta, SAML, Azure AD)
User access management features
Stand alone tool with self serve deployment option.
The api and sdk are open source.
The product is closed source.
Free version available.
The user can deploy modeld.io on any kubernetes cluster (managed or on prem). The system is managed by the user.
Service level guarantees depends on your plan
24/7 support is available depending on your plan.
There is support in the API. User access is based on kubernetes RBAC.
Based on kubernetes rbac. All objects are native kubernetes resources.
Stand-alone
Open-source with a paid Enterprise edition that contains additional features and commercial support.
On-prem and self-hosted private cloud.
Open-source: best effort.
Enterprise: Support SLAs guaranteed.
For Enterprise.
Enterprise: yes. Full integration with SAML/SCIM.
Open-source: basic user management system.
Enterprise: RBAC.
Standalone platform with community, self-serve, and managed deployment options.
Primarily managed cloud service, with custom delivery as needed for enterprise
Available on public cloud (AWS, GCP, Azure) and full enterprise-level on-prem deployment
Depends on your plan.
Every company in subscription or contract is provided a private slack channel for direct engineering support 24/7.
All users are provided user access management within collaboration features.
Enterprise-level customers are provided customized SSO and authentication features.
User access management features
Stand-alone tool.
Available as a free community product, self-service commercial software, and managed enterprise product.
Any cloud environment.
On-premise.
Comet Enterprise: there are service level guarantees that depend on your plan.
Comet Enterprise: there are service level guarantees that depend on your plan.
Community: None.
Comet Enterprise: advanced user management options.
Community: None.
Comet Enterprise: advanced user management options.
Stand-alone tool
Managed service via tensorboard.dev
Open-source
Managed service via tensorboard.dev
Deploy the open-source version wherever you want
None
None
None
None
Standalone platform for Model Management & Operations
Commercial software for on-premise or private cloud deployment
Managed service in customer cloud or Verta hosted
On-premise (air-gapped)
Private/public cloud platform
Verta hosted offering
SLOs/SLAs are guranteed and can be customized per contract agreement
24x7 support is available based on the contract agreement
RBAC and user management system along with SSO support to integrate into identity systems (e.g SAML, Okta, Active Directory, etc.)
Advanced user management, SSO, RBAC, integrations with enterprise identity providers, audit/access logs, model release approval workflows
Stand-alone tool
Open-source.
Commercial and SaaS offerings come with additional enterprise features.
Any Kubernetes environment, cloud, local
SaaS offering
Pachyderm Enterprise: SLAs available but dependent on the plan.
Two support tiers available.
Open source: Community
Pachyderm Enterprise: 24x7 available depending on plan
Pachyderm Enterprise provides full user and role-based access controls which allows for granular permission setting
Pachyderm Enterprise: advanced user management options
Vendor |
Demo link |
Stand-alone vs. Platform |
Delivery Model |
Clouds Supported |
Pricing Model |
Service Level Guarantees |
Support |
SSO, ACL |
Security and Compliance |
Setup |
Flexibility, Speed, and Accessibility |
Model Versioning, Lineage, and Packaging |
Log and Display of Metadata |
Comparing Experiments and Models |
Organizing and Searching Experiments and Models |
Model Review, Collaboration, and Sharing |
CI/CD/CT Compatibility |
Integrations |
Are you looking to add some metadata storage and management to your ML stack? MLOps Community, with the collaboration of many Experiment tracking, model store and metadata management vendors has created an evaluation framework to help you choose the right product for your needs.
Criteria 1
First, you need to assess whether the product’s commercial characteristics meet your needs. We recommend evaluating the following commercial criteria:
Criteria 2
You will want to make sure that the model store fulfills all the capabilities you need across the operational data workflow. We’ve broken down the capabilities as follows:
Setup
How much work is needed to set up the infrastructure, deploy the tool, maintain it, and connect it to your training workflow?
Flexibility, Speed, and Accessibility
Can you adjust the metadata structure to your needs, is the API and UI fast enough to handle your workload, and can you access models and metadata easily from other tools in your stack?
Log and Display of Metadata
What model and experiment metadata can you log and display in the tool, what gets logged automatically, can you see it live?
Comparing Experiments and Models
What model and experiment metadata can you compare, which comparison visualizations does the tool provide, are there special comparison utilities for your modality (computer vision)?
Organizing and Searching Experiments and Models
How can you organize experiments/models in the tool, how advanced are the search capabilities, can you customize what you see both for a single run and many experiments/models?
Model Review, Collaboration, and Sharing
How does it support model audit, review, approval, and transitions between stages (dev/prod)? Can you lock experiments/models/artifacts downstream for published models?
CI/CD/CT compatibility
How does it support continuous integration and delivery, how does it connect it to continuous training and testing workflows?
Integrations
Which 3rd party data and ML tools does the feature store integrate with?