Skip to content

Discovery Pipeline Documentation

Overview

The SolAr discovery pipeline (solar-discovery) is a standalone component that discovers OCM (Open Component Model) packages in OCI registries and writes them into the SolAr API as ComponentVersion resources.

Discovery is triggered in two ways: by periodically scanning a registry for all repositories, or by receiving push notifications from the registry via webhook. These modes can be used together or independently (scanInterval: 0 disables polling).

The pipeline is composed of a chain of channel-connected stages. Each stage processes events from its input channel and publishes results to its output channel, making the pipeline fully asynchronous and back-pressure-aware.

Pipeline Stages

flowchart LR
    Reg[(OCI Registry)]
    Webhook[Webhook Server]
    Scanner[Registry Scanner]
    Chan([repoEvents channel])

    subgraph Pipeline
        Q[Qualifier]
        F[Filter]
        H[Handler]
        W[APIWriter]
    end

    K8s[(SolAr API)]

    Reg -->|poll interval| Scanner
    Reg -->|push notification| Webhook
    Scanner -->|RepositoryEvent| Chan
    Webhook -->|RepositoryEvent| Chan
    Chan --> Q
    Q -->|ComponentVersionEvent| F
    F -->|ComponentVersionEvent| H
    H -->|WriteAPIResourceEvent| W
    W -->|create/update/annotate| K8s
Hold "Alt" / "Option" to enable pan & zoom

Event Sources

Source Output Responsibility
RegistryScanner RepositoryEvent Periodically scans all repositories in an OCI registry
WebhookServer RepositoryEvent Accepts push notifications from OCI registries and emits events directly

Pipeline Stages

Stage Input Output Responsibility
Qualifier RepositoryEvent ComponentVersionEvent Resolves repository name to namespace + component, looks up all versions via OCM
Filter ComponentVersionEvent ComponentVersionEvent Drops events for ComponentVersions that already exist in the cluster
Handler ComponentVersionEvent WriteAPIResourceEvent Fetches the OCM component descriptor and builds the API resource payload
APIWriter WriteAPIResourceEvent – Creates or updates ComponentVersion resources; on delete events annotates them for an availability re-check

Event Types

classDiagram
    class RepositoryEvent {
        Registry
        Repository
        Version
        Digest
        Type
    }

    class ComponentVersionEvent {
        Namespace
        Component
        Version
    }

    class WriteAPIResourceEvent {
        ComponentSpec
        HelmDiscovery
        Type
    }

    RepositoryEvent --> ComponentVersionEvent : Qualifier
    ComponentVersionEvent --> WriteAPIResourceEvent : Handler
Hold "Alt" / "Option" to enable pan & zoom

Registry Scanner

The RegistryScanner scans a single OCI registry on a configurable interval (ScanInterval, default 24 h). It lists all repositories via the ORAS library and emits a RepositoryEvent for each one. Concurrent scans are prevented by a mutex — if a scan is still running when the next tick fires, the tick is skipped.

OCI registries can also push change notifications via webhooks. The WebhookServer accepts HTTP POST requests on configured paths and converts them directly into RepositoryEvents, bypassing the polling interval.

Qualifier

The Qualifier resolves a raw RepositoryEvent (registry + repository path) into one or more ComponentVersionEvents by:

  1. Splitting the repository path into namespace/component segments.
  2. If the event already carries a specific version (e.g. from a webhook), emitting a single event for that version.
  3. Otherwise, looking up all versions of the component in the OCM repository and emitting one event per version.

Filter

The Filter prevents duplicate work. For EventCreated events it checks whether the corresponding ComponentVersion already exists in the SolAr API. If it does, the event is silently dropped. All other event types (update, delete) pass through unconditionally.

Handler

The Handler fetches the OCM component descriptor for a component version and builds the ComponentVersion payload. Only components that satisfy the OCM packaging contract are processed.

APIWriter

The APIWriter creates or updates ComponentVersion resources in the SolAr API. Each ComponentVersion carries the OCM component name and its source scheme, registry and repository. Each ComponentVersion also records the descriptor's full manifest digest in spec.digest and starts with Available=True; from then on the ComponentVersion controller owns availability.

An OCM component version is immutable, so an update whose descriptor digest differs from the recorded one is refused and only logged: the tag was re-pushed with other content. The ComponentVersion keeps pointing at the descriptor it was discovered from, which the controller then reports as missing. Delete the ComponentVersion to re-discover the new content. The scan path reaches the same outcome because the Filter never updates an existing ComponentVersion.

Sequence Diagrams

Scanner: Periodic poll discovers changes

sequenceDiagram
    participant Timer as Tick (interval)
    participant Scanner as Registry Scanner
    participant Reg as OCI Registry
    participant Chan as repoEvents channel

    Timer->>Scanner: tick
    Scanner->>Reg: List repositories
    Reg-->>Scanner: [ocm-demo, other-component, …]
    loop for each repository
        Scanner->>Chan: RepositoryEvent(repo=ocm-demo, version="")
    end
Hold "Alt" / "Option" to enable pan & zoom

Webhook: Registry pushes a change notification

sequenceDiagram
    participant Dev as Solution Maintainer
    participant Reg as OCI Registry
    participant Webhook as Webhook Server
    participant Chan as repoEvents channel

    Dev->>Reg: ocm transfer ctf ./ocm-demo-ctf localhost/test
    Note over Reg: Pushes opendefense.cloud/ocm-demo v26.4.1
    Reg->>Webhook: POST /webhook/zot-discovery<br/>(repo=…/ocm-demo, tag=v26.4.1)
    Webhook->>Chan: RepositoryEvent(version=v26.4.1)
Hold "Alt" / "Option" to enable pan & zoom

Shared pipeline: qualify, filter & write

Once a RepositoryEvent is on the channel, both the scanner and webhook paths converge into this shared pipeline starting with the Qualifier.

sequenceDiagram
    participant Chan as repoEvents channel
    participant Qualifier as Qualifier
    participant Reg as OCI Registry
    participant Filter as Filter
    participant Handler as Handler
    participant Writer as APIWriter
    participant K8s as SolAr API

    Chan->>Qualifier: RepositoryEvent(ocm-demo, version="")
    Note over Qualifier: No version specified —<br/>look up all versions
    Qualifier->>Reg: ListComponentVersions(ocm-demo)
    Reg-->>Qualifier: [v26.4.1, v26.4.0, …]
    loop for each version
        Qualifier->>Filter: ComponentVersionEvent(ocm-demo, vX.Y.Z)
    end
    Filter->>K8s: Get ComponentVersion "…-v26-4-1"
    K8s-->>Filter: NotFound
    Filter->>Handler: pass event
    Handler->>Reg: LookupComponentVersion(ocm-demo, v26.4.1)
    Reg-->>Handler: ComponentDescriptor (1 Helm resource)
    Handler->>Writer: WriteAPIResourceEvent(ComponentSpec)
    Writer->>K8s: Create ComponentVersion "…-v26-4-1"
Hold "Alt" / "Option" to enable pan & zoom

Component version deleted from registry

sequenceDiagram
    participant Dev as Solution Maintainer
    participant Reg as OCI Registry
    participant Webhook as Webhook Server

    Dev->>Reg: Delete OCI tag v26.4.1
    Reg->>Webhook: POST /webhook/zot-discovery<br/>(type=Deleted, digest=sha256:abc123…)
    Note over Webhook: Only webhooks carry deletion events.<br/>Without one, the controller's periodic availability check notices the miss.
Hold "Alt" / "Option" to enable pan & zoom

Pipeline passthrough & availability re-check

sequenceDiagram
    participant Qualifier as Qualifier
    participant Filter as Filter
    participant Handler as Handler
    participant Writer as APIWriter
    participant K8s as SolAr API

    Note over Qualifier: RepositoryEvent(type=Deleted) —<br/>no OCM lookup needed
    Qualifier->>Filter: ComponentVersionEvent(type=Deleted)
    Note over Filter: Deletions bypass exists-check
    Filter->>Handler: pass through
    Note over Handler: No descriptor fetch needed
    Handler->>Writer: WriteAPIResourceEvent(type=Deleted, digest=sha256:abc123…)
    Writer->>K8s: List CVs (fieldSelector: spec.digest=sha256:abc123…)
    K8s-->>Writer: [ocm-demo-v26-4-1]
    Writer->>K8s: Patch annotation availability-recheck=<timestamp>
    Note over K8s: ComponentVersion controller re-checks the manifest;<br/>Available goes Unknown, then False after the grace period
Hold "Alt" / "Option" to enable pan & zoom

Configuration

The discovery pipeline is configured via the solar-discovery binary flags and a registry configuration file. Per-registry settings include:

Setting Description
scanInterval Polling interval for registry scans (0 = webhook only)
webhookPath HTTP path to register for push notifications (optional)
credentials Username/password for authenticated registries
plainHTTP Whether to use plain HTTP instead of HTTPS