Skip to main content

Container registry for SCS standard implementation

Introduction​

A container registry is an infrastructure service to enable storing and accessing container images. Images can be pushed to the registry by e.g. Continuous integration pipelines and be pulled from by runtime environments like Kubernetes clusters.

In the standard document ["Requirements for container registries"], requirements for a registry in the context of SCS were introduced. These are based on the principals, that a usable project should be open source, active and feature-rich, especially with regard to security.

Terminology​

Cloud Service Provider (abbr. CSP) Entity that provides scalable computing resources

Cloud Native Computing Foundation (abbr. CNCF) Organization that hosts and develops open source projects for cloud native computing

Motivation​

In order to provide a usable, complete experience for the SCS reference implementation, it must be decided on a registry in accordance with requirements set by the ["Requirements for container registries"] standard as well as other dependencies set by the SCS project, including the OSS requirements.

This document should finally lead to a decision about the container registry used as a reference implementation of the SCS container registry.

Evaluated projects​

A few open source projects were evaluated for this document in order to find suitable candidates for the SCS reference implementation. These projects can be found in the following list of evaluated projects with their classified categories and comments. An initial assessment was done with the checks for OSS healthiness and a general overview of the features described in ["Requirements for container registries"], which enables classifying the projects into one of three categories as follows:

  • ✔️ The project passed all OSS health checks and will be considered further as a valid candidate.

  • ❔ The project passed almost all OSS health checks. There is place for improvement, but the missing points are not crucial from the OSS health check perspective. The project will be considered further as a valid candidate.

  • ❌ The project does not pass the OSS health checks. Some OSS health check showstoppers have been found (e.g. open core software, not actively maintained). The project is filtered at this stage and won't be considered further.

The following list contains these projects with a small assessment listed below them:

  • ✔️ Harbor

    • Harbor project meets all "four opens"
      • Source code is open and available under the Apache 2 license
      • Community is open, structured and well organized via workgroups and various communications channels e.g. Slack, mailing lists, etc. (#harbor Slack channel contains 3k+ members)
      • The development process is open via GitHub issues and well described in the contributing document
      • The design process is open via GitHub issues. Proposals are public. The decision process is well described as well. The project's roadmap is available in the roadmap document
    • Maturity is on the CNCF graduation level. CNCF graduated projects are considered to be stable, widely adopted and production-ready
    • Security
      • The security disclosure and response policy is well described in the project's security document
      • The code is reviewed within a standard PR process
    • Activity
      • 250+ contributors, 4k+ forks, 13k+ GitHub stars
      • The project has been adopted by many companies that run Harbor in their production environments
      • The project collaborates with other communities and projects (see Partners of Harbor section of the project's website)
      • The project is visible and actively contributes to various conferences, e.g. FOSDEM 22, KubeCon Europe, etc.
    • Lock-in risk assessment
      • The project's maintainers document shows that there are a sufficient number of core maintainers/contributors that differ over various companies, we therefore deem the lock-in risk arising from a single point of failure to be low

  • ✔️ Dragonfly

    • Dragonfly project meets all "four opens"
      • Source code is open and available under the Apache 2 license
      • Community is open organized via multiple channels e.g. Slack, mailing lists, etc. (#dragonfly Slack channel contains ~50 members)
      • The development process is open via GitHub issues and well described in the contributing document
      • The design process is open via GitHub issues. The project's roadmap is available in the project's webpage
    • Maturity is on the CNCF incubating level CNCF incubating project is considered stable and used in production by users with the healthy pool of contributors
    • Security
      • The security disclosure is handled via a dedicated email address
      • The code is reviewed within a standard PR process
    • Activity
      • 30+ contributors, 100+ forks, 1k+ GitHub stars
      • The project has been adopted by many companies that run Harbor in their production environments
      • The project is visible and actively contributes to various conferences, e.g. KubeCon North America, KubeCon Europe, etc.
    • Lock-in risk assessment

  • ❔ Project Quay

    • Project Quay meets all "four opens"
      • Source code is open and available under the Apache 2 license
      • Community is open organized via mailing list and IRC
      • Development process is open via JBoss JIRA issues and well described in the governance document
      • Design process is open via JBoss JIRA issues. The project's roadmap is available on the project's webpage
    • Maturity
      • Project Quay is an open-source project that started ~9 years ago. It powers Red Hat enterprise products Red Hat Quay and Quay.io, which are used in a productive way by many. Therefore, the project's maturity is at the good level
    • Security
      • The security disclosure is handled via a dedicated email address
      • The code is reviewed within a standard PR process
    • Activity
      • 50+ contributors, 200+ forks, 2k+ GitHub stars
      • The project has been used by many companies that run Quay in their production environments
    • Lock-in risk assessment
      • The project's owners/maintainers list is not publicly available and is stored in the downstream repository. Therefore, it is hard to distinguish the risk of project failure caused by low diversity across the companies. This should be improved.

  • ❌ Keppel

    • The project seems to be not widely used in a productive way and also the activity around is currently not on a good level (5+ contributors). The development process as well as the design process seem to be open, but neither of them are documented yet.

  • ❌ Nexus

    • Nexus is an open core software that offers paid pro version with advanced features

  • ❌ JFrog

    • JFrog Artifactory is shipped as an open core software with limited features. The software is primarily offered as a paid pro version

  • ❌ Kraken

    • It seems that the project is not actively maintained as is discussed in the related project's issue

  • ❌ Portus

    • It seems that the project is not actively maintained as is discussed in the related project's issue

Deeper look into selected projects​

In the previous section, a wide range of open-source container registry projects (Quay, Harbor, Dragonfly, Keppel, Portus, Kraken, etc.) has been carefully evaluated based on the two main factors: the open-source health and range of supported features.

The open-source software health is crucial and container registry implementation should pass it. It evaluates several important metrics of an open source software like whether the code/community/development/design is fully open or whether the project's maturity, security, and activity are on the desired level. This check also evaluates the lock-in risk due to possible single points of failure or internal project conflicts and several other aspects. Overall, three projects passed the OSS health checks:

The above projects were then evaluated from the "supported features" perspective. The document ["Requirements for container registries"] provides a "Required and desirable features check", which provides desired feature sets for open-source container registry implementations according to SCS requirements (and nice-to-haves). The list of required features is quite long and contains features that are primarily focused on security (authentication, vulnerability scanning, content trust, and validation, etc.), scalability (HA mode, registry replication, p2p integration, etc.) and visibility (monitoring). These requirements should ensure that the selected container registry implementation could be offered by CSPs as a secure and enterprise-ready solution.

The following section compares the selected projects Dragonfly, Quay, and Harbor.

Dragonfly is a healthy open-source project with a growing community and CNCF incubating maturity level. It is considered stable, and widely used by many companies in their production environments. We currently see that it is not as feature-rich as Harbor or Quay, hence it is not considered the best choice here. It seems, that its main aim (currently) is to offer (an efficient, stable, and secure) container distribution solution based on p2p technology. This improves download efficiency and saves bandwidth across CSPs. It also offers integration possibilities that allow one to use it as a p2p distribution network via a "preheat" API. This integration was implemented in the Harbor project via Dragonfly "preheat" adapter, and both parties may benefit from the integration. Harbor profits from Dragonfly's p2p distribution capabilities and on the other hand the Dragonfly project profits from Harbor's feature-rich container registry "frontend".

Quay is an open-source project maintained by Red Hat. Its OSS health is on a good level, the surrounding community is growing, and we consider it to be quite mature as it powers enterprise solutions like Red Hat Quay and Quay.io. Besides this, there is still a place for OSS health improvement. It is hard to distinguish the risk of project failure caused by low diversity across the companies because the project's owners/maintainers list is not publicly available and is stored in the Red Hat private repository. Its feature set is impressive and this project fulfills all must-haves defined in this document. Quay gives you security over your repositories with image vulnerability scanning (Clair integration), content validation (Cosign integration), and access controls. Harbor stands out here as it allows users to use also project Trivy for vulnerability scanning. Project Quay also provides a scalable open-source platform to host container images across any size organization. One drawback in comparison to Harbor is that the proxy cache feature is still marked as a Technology Preview, hence this feature may not be completely production-ready yet. On the other hand, the project Quay supports building Dockerfiles using a set of workers on e.g. Kubernetes. Build triggers, such as GitHub webhooks can be configured to automatically build new versions of repositories when new code is committed. This feature is not supported by the Harbor project.

Harbor is an outstanding open-source, community-led project with fully open and well-documented processes. Its large and thriving community powers the fast-growing feature set and attracts more and more developers and companies to active contributions. Harbor's CNCF graduation in 2020 made it one of the best choices for enterprise customers that want to operate container registries securely and in a large scale. Its community size, landscape, and CNCF graduation make a significant difference in comparison to Quay's open-source health capabilities. The list of supported features is also impressive. This project fulfills all must-haves defined in this document and overcome project Quay with a production-ready proxy cache feature and more options that the user may use in case of image vulnerability scanning. In addition, Harbor profits from p2p distribution capabilities via integration of p2p solutions like Kraken and Dragonfly. It is worth mentioning that Harbor, by design, operates on a single storage backend (e.g. S3). It means that the storage of container images is shared even when the Harbor instance serves multiple tenants. The same approach is used in Quay and Dragonfly projects, but e.g. Keppel uses multi-tenant-aware storage drivers instead so that each customer gets their own separate storage backend. CSP that considers offering container registry "as a service" solution based on Harbor should be aware of this shared storage backend architecture.

In the following table, the feature sets of the evaluated projects that passed the OSS health state are listed and matched against. This enables a better understanding of the decision-making for this document.

FeaturesHarborQuayDragonfly
Audit Logs✓✓✗
Authentication of system identities✓ Robot Accounts✓ Robot Accounts✗
Authentication of users✓ Local database, LDAP, OIDC, UAA✓ Local database, LDAP, Keystone, JWT✓ Local database
Authorization✓✓✓
Automation✓ Webhooks (HTTP, Slack)✓ Webhooks (HTTP, Slack, E-mail ...), building images✗
Vulnerability scanning✓ Trivy, Clair✓ Clair✗
Content Trust and Validation✓ Cosign✓ Cosign✗
Multi-tenancy✓ (not on the storage level)✓ (not on the storage level)✓ (not on the storage level)
Backup and restore✓✓✗
Monitoring✓ Prometheus metrics, Tracing✓ Prometheus metrics, Tracing (only for Clair)✓ Prometheus metrics, Tracing
HA mode✓✓✗
Registry replication✓✓✓
Proxy cache✓✓ Feature is in the technology preview stage (non production ready)✗
Quota management✓ Based on storage consumption✓ Based on storage consumption✗
Garbage collection✓ Non-blocking✓ Non-blocking✗
Retention policy✓ Multiple tag retention rules✓ Only tag expiration rules✗
Additional supported artifacts✗ (only OCI artifacts)✗ (only OCI artifacts)✓ Maven, YUM
Integration possibilities✓ Dragonfly (P2P), Kraken (P2P)✗✓ Harbor, Nydus, eStargz
Deployment capabilities✓ Docker-compose, Helm chart, Operator✓ Docker-compose, Operator✓ Docker-compose, Helm chart
Administration capabilities✓ Terraform, CRDs, Client libraries✓ Ansible, Client libraries✓ Client libraries

Notes:

  • Automation: Harbor should support webhooks following CloudEvents spec in the next release
  • Content Trust and Validation: Harbor announced the deprecation of Notary integration, hence it is not mentioned in the table
  • Multi-tenancy: Harbor, Quay, as well as Dragonfly, operates on a single storage backend (e.g. S3), i.e. the storage of container images is shared between tenants
  • Additional supported artifacts: Harbor announced the deprecation of Chartmuseum integration, hence it is not mentioned in the table

Decision​

Based on the requirements laid out in ["Requirements for container registries"], the OSS health check and the possible software solutions presented in this document, it was decided to use the Harbor project as the container registry for the SCS reference implementation.

"Requirements for container registries" harbor dragonfly projectquay