Metadata Storage and Management

What is Metadata Storage and Management?

To confidently deploy models to production, you need to know how each model was built, trained, re-trained, and evaluated. That is where tools for model metadata storage and management come in. They give you a central place to store, organize, display, compare, search, review, and access all your models and model-related metadata. You can use tools from this category as an experiment tracking tool, model registry, or both. Read more about what model storage and management is and checkout additional resources below:

Metadata Storage and Management Comparison

The MLOps Community has worked with vendors and community members to profile the major solutions available in the market today, based on our model store evaluation framework.

Sort by
    • Stand-alone vs. Platform:

      Stand-alone

    • Delivery Model:

      Open-source

    • Clouds Supported:

      Open-source

    • Service Level Guarantees:

      None

    • Support:

      Open-source: community (Discord server, GH issues, self-hosted forum, support email, social media)

    • SSO, ACL:

      N/A

    • Security and Compliance:

      None

    • Stand-alone vs. Platform:

      Stand-alone

    • Delivery Model:

      Available as a managed cloud service (SaaS) and as a commercial on-premises deployment.

    • Clouds Supported:

      Once you get the package, you can deploy it wherever you want.
      See the on-premises installation guide at https://neptune.ai/wp-content/uploads/Neptune-Installation-Guide.pdf.

    • Service Level Guarantees:

      On-premises plans

    • Support:

      Available during working hours 8 - 18 CEST

    • SSO, ACL:

      Available in the Scale plan.
      You can manage organization and project-level access in the Scale plan.

    • Security and Compliance:

      Depending on your security needs, you can use SaaS or on-premises versions.
      Read more at https://neptune.ai/wp-content/uploads/privacy-policy.pdf.

    • Stand-alone vs. Platform:

      Stand-alone as an open-source project
      Managed service as part of the Databricks platform

    • Delivery Model:

      Available both as the open-source software and a managed solution as a part of the Databricks platform.

    • Clouds Supported:

      The open-source version is compatible on all clouds
      Databricks platform can be deployed on Google Cloud, AWS, and Microsoft Azure

    • Service Level Guarantees:

      Open-source: None
      Databricks platform: there are service levels guarantees that depend on your plan

    • Support:

      Open-source: community
      Databricks platform: you can get up to 24 x 7 support and response times guarantees that depend on your support plan

    • SSO, ACL:

      Open-source: None
      Databricks platform: advanced user management options

    • Security and Compliance:

      Open-source: None
      Databricks platform: enterprise level

    • Stand-alone vs. Platform:

      Standalone platform that is infrastructure-agnostic.

    • Delivery Model:

      Client library is open source, and the server and UI interface is closed source.

    • Clouds Supported:

      GCP, AWS, Azure an On-prem

    • Service Level Guarantees:

      Service level guarantees are plan-dependent.

    • Support:

      Support is available during regular business hours online, via Slack, and over the phone for all customers.
      Additional support is provided for all enterprise customers.

    • SSO, ACL:

      SAML, OIDC, Active Directory, PingFederate or OAuth2 identity providers.
      Access controls can be asserted at the team and project levels.

    • Security and Compliance:

      GDPR and SOC 2 Type 2 compliant.

    • Stand-alone vs. Platform:

      Part of a broader platform

    • Delivery Model:

      Open-source with closed-source paid features

    • Clouds Supported:

      Ability to be deployed on-prem and all clouds either self hosted with OSS or managed with Enterprise

    • Service Level Guarantees:

      Standard SLA that can be customized for Enterprise customers

    • Support:

      Open-source: community
      Enterprise: customizable 24X7 SLA based on needs

    • SSO, ACL:

      ClearML Server supports self user management.
      Enterprise offers full user management with SSO and user roles and permissions

    • Security and Compliance:

      SSO and LDAP in Enterprise as well as user management and roles and resource-based permissions

    • Stand-alone vs. Platform:

      Standalone platform

    • Delivery Model:

      Open-source or managed

    • Clouds Supported:

      Cloud-native, available in your VPC on public cloud and/or Kubernetes.

    • Service Level Guarantees:

      For enterprise plans

    • Support:

      Community support on Slack, with additional channels and SLA for enterprise plans.

    • SSO, ACL:

      SSO for enterprise plans.
      Integrates with cloud-native solutions for ACL.

    • Security and Compliance:

      Depends on deployment type, typically implemented by the customer as data stays in the customer cloud account.

    • Stand-alone vs. Platform:

      Stand-alone

    • Delivery Model:

      Open-source

    • Clouds Supported:

      Open-source

    • Service Level Guarantees:

      None

    • Support:

      Open-source: community (Discord server, GH issues, self-hosted forum, support email, social media)

    • SSO, ACL:

      N/A

    • Security and Compliance:

      None

    • Stand-alone vs. Platform:

      Standalone platform

    • Delivery Model:

      Open-source, managed control plane, and commercial enterprise features.

    • Clouds Supported:

      All editions (free and paid) allow on-premises deployment.
      The community edition is full on-premise and can be deployed on any k8s cluster.
      The cloud edition is a hybrid deployment where the control plane is managed and the workload, code, logs, metrics, models, and artifacts are on the customer’s private cloud account or on-prem cluster(s).
      The enterprise edition allows to deploy both the control plane and agents (data plane) on the private cloud account or the on-prem cluster(s).

    • Service Level Guarantees:

      SLAs can be purchased.

    • Support:

      Community Edition comes with community support.
      All paying customers have access to email and slack support.
      Additional support priority and SLAs can be purchased.

    • SSO, ACL:

      No user management in the community edition.
      User management is available in all paid versions with organization-level roles, ACL, and RBAC, all users can additionally issue scoped tokens to interact with the APIs on their behalf.
      Business and Enterprise plans have access to advanced restrictions, additional team-level roles, and management.
      SSO with SAML is available to enterprise customers.

    • Security and Compliance:

      Advanced logical restrictions and native integration with K8S RBAC.
      Workload, data, code, artifacts, logs, and models are always on the customer's clusters in all editions.

    • Stand-alone vs. Platform:

      Part of a broader platform

    • Delivery Model:

      Primarily managed cloud service

    • Clouds Supported:

      AWS
      Azure
      GCP

    • Service Level Guarantees:

      For enterprise clients only (per agreement)

    • Support:

      No

    • SSO, ACL:

      SSO available (Okta, SAML, Azure AD)

    • Security and Compliance:

      User access management features

    • Stand-alone vs. Platform:

      Stand alone tool with self serve deployment option.

    • Delivery Model:

      The api and sdk are open source.
      The product is closed source.
      Free version available.

    • Clouds Supported:

      The user can deploy modeld.io on any kubernetes cluster (managed or on prem). The system is managed by the user.

    • Service Level Guarantees:

      Service level guarantees depends on your plan

    • Support:

      24/7 support is available depending on your plan.

    • SSO, ACL:

      There is support in the API. User access is based on kubernetes RBAC.

    • Security and Compliance:

      Based on kubernetes rbac. All objects are native kubernetes resources.

    • Stand-alone vs. Platform:

      Stand-alone

    • Delivery Model:

      Open-source with a paid Enterprise edition that contains additional features and commercial support.

    • Clouds Supported:

      On-prem and self-hosted private cloud.

    • Service Level Guarantees:

      Open-source: best effort.
      Enterprise: Support SLAs guaranteed.

    • Support:

      For Enterprise.

    • SSO, ACL:

      Enterprise: yes. Full integration with SAML/SCIM.

    • Security and Compliance:

      Open-source: basic user management system.
      Enterprise: RBAC.

    • Stand-alone vs. Platform:

      Standalone platform with community, self-serve, and managed deployment options.

    • Delivery Model:

      Primarily managed cloud service, with custom delivery as needed for enterprise

    • Clouds Supported:

      Available on public cloud (AWS, GCP, Azure) and full enterprise-level on-prem deployment

    • Service Level Guarantees:

      Depends on your plan.

    • Support:

      Every company in subscription or contract is provided a private slack channel for direct engineering support 24/7.

    • SSO, ACL:

      All users are provided user access management within collaboration features.
      Enterprise-level customers are provided customized SSO and authentication features.

    • Security and Compliance:

      User access management features

    • Stand-alone vs. Platform:

      Stand-alone tool.

    • Delivery Model:

      Available as a free community product, self-service commercial software, and managed enterprise product.

    • Clouds Supported:

      Any cloud environment.
      On-premise.

    • Service Level Guarantees:

      Comet Enterprise: there are service level guarantees that depend on your plan.

    • Support:

      Comet Enterprise: there are service level guarantees that depend on your plan.

    • SSO, ACL:

      Community: None.
      Comet Enterprise: advanced user management options.

    • Security and Compliance:

      Community: None.
      Comet Enterprise: advanced user management options.

  • TensorBoard

    • Stand-alone vs. Platform:

      Stand-alone tool
      Managed service via tensorboard.dev

    • Delivery Model:

      Open-source
      Managed service via tensorboard.dev

    • Clouds Supported:

      Deploy the open-source version wherever you want

    • Service Level Guarantees:

      None

    • Support:

      None

    • SSO, ACL:

      None

    • Security and Compliance:

      None

    • Stand-alone vs. Platform:

      Standalone platform for Model Management & Operations

    • Delivery Model:

      Commercial software for on-premise or private cloud deployment
      Managed service in customer cloud or Verta hosted

    • Clouds Supported:

      On-premise (air-gapped)
      Private/public cloud platform
      Verta hosted offering

    • Service Level Guarantees:

      SLOs/SLAs are guranteed and can be customized per contract agreement

    • Support:

      24x7 support is available based on the contract agreement

    • SSO, ACL:

      RBAC and user management system along with SSO support to integrate into identity systems (e.g SAML, Okta, Active Directory, etc.)

    • Security and Compliance:

      Advanced user management, SSO, RBAC, integrations with enterprise identity providers, audit/access logs, model release approval workflows

    • Stand-alone vs. Platform:

      Stand-alone tool

    • Delivery Model:

      Open-source.
      Commercial and SaaS offerings come with additional enterprise features.

    • Clouds Supported:

      Any Kubernetes environment, cloud, local
      SaaS offering

    • Service Level Guarantees:

      Pachyderm Enterprise: SLAs available but dependent on the plan.
      Two support tiers available.

    • Support:

      Open source: Community
      Pachyderm Enterprise: 24x7 available depending on plan

    • SSO, ACL:

      Pachyderm Enterprise provides full user and role-based access controls which allows for granular permission setting

    • Security and Compliance:

      Pachyderm Enterprise: advanced user management options

Compare providers

Commercial Information

Vendor
Demo link
Stand-alone vs. Platform
Delivery Model
Clouds Supported
Pricing Model
Service Level Guarantees
Support
SSO, ACL
Security and Compliance

Model Store Capabalities

Setup
Flexibility, Speed, and Accessibility
Model Versioning, Lineage, and Packaging
Log and Display of Metadata
Comparing Experiments and Models
Organizing and Searching Experiments and Models
Model Review, Collaboration, and Sharing
CI/CD/CT Compatibility
Integrations

How to choose a solution for Metadata Storage and Management

Are you looking to add some metadata storage and management to your ML stack? MLOps Community, with the collaboration of many Experiment tracking, model store and metadata management vendors has created an evaluation framework to help you choose the right product for your needs.

Criteria 1

Commercial Information

First, you need to assess whether the product’s commercial characteristics meet your needs. We recommend evaluating the following commercial criteria:

  • Delivery Model: Open source or managed service? 
  • Standalone feature store or part of a broader ML platform? 
  • Is the product available on-premises and / or in your public cloud?
  • Is the product delivered as commercial software, open source software, or a managed cloud service?
  • What is the pricing model?  
  • SLOs / SLAs: Does the vendor provide guarantees around service levels?
  • Support: Does the vendor provide 24×7 support?
  • SSO, ACL: Does the vendor provide user access management?
  • Security policy and compliance

Criteria 2

Model Store Capabilities

You will want to make sure that the model store fulfills all the capabilities you need across the operational data workflow. We’ve broken down the capabilities as follows:

Setup
How much work is needed to set up the infrastructure, deploy the tool, maintain it, and connect it to your training workflow?

Flexibility, Speed, and Accessibility
Can you adjust the metadata structure to your needs, is the API and UI fast enough to handle your workload, and can you access models and metadata easily from other tools in your stack?

Log and Display of Metadata
What model and experiment metadata can you log and display in the tool, what gets logged automatically, can you see it live?

Comparing Experiments and Models
What model and experiment metadata can you compare, which comparison visualizations does the tool provide, are there special comparison utilities for your modality (computer vision)?

Organizing and Searching Experiments and Models
How can you organize experiments/models in the tool, how advanced are the search capabilities, can you customize what you see both for a single run and many experiments/models?

Model Review, Collaboration, and Sharing
How does it support model audit, review, approval, and transitions between stages (dev/prod)? Can you lock experiments/models/artifacts downstream for published models?

CI/CD/CT compatibility
How does it support continuous integration and delivery, how does it connect it to continuous training and testing workflows?

Integrations
Which 3rd party data and ML tools does the feature store integrate with?