Skip to main content
Version: Next

Data plane signaling upgrade

Upgrade the control plane, HTTP data plane and analytics data plane together. Mixed versions are unsupported.

The signaling migration deletes existing dataflows on all three services and marks existing REQUESTED, STARTED and SUSPENDED DSP transfers as TERMINATED. Existing flow credentials stop working. Completed transfer history and agreements remain. Start new transfers after the upgrade. Rolling back the database schema cannot recover deleted flow state.

Providers now send one DPS start request with the transfer process ID as dataFlowId. Private backend configuration travels in metadata, and data addresses use the DSP DataAddress representation. A 202 response leaves the transfer requested until a started callback arrives. Duplicate readiness notifications do not send another DSP start. The provider persists the request and finishes its HTTP response before dispatching DPS start, including for planes that send an early callback. A failed, malformed or lost start response terminates the transfer and triggers best-effort cleanup. Data-plane startup failures are reported through DSP termination after request acceptance. Failed completion notifications are retried when the plane repeats its completed callback.

Only the party that suspended a transfer may initiate its resume. For consumer-initiated pull resume, the consumer waits for the provider's second start message with a fresh address. Data planes keep the backend configuration captured at start through suspension and resume. Consumer resume requests are recorded before dispatch so an early provider response can be distinguished from an unsolicited resume. A failed resume terminates the transfer with best-effort data-plane cleanup; start a new transfer to try again. This includes a failed DSP start message during consumer resume, even when the cause is a transient network error.

DPS start dispatch after an accepted request runs in the control-plane process once the response is sent. If that process stops in between, the transfer stays REQUESTED; terminate it and request a new transfer.

DPS TCK fixture details and exclusions are in tools/e2e-testing/lib/dps-tck/README.md. Its metadata fallback is test-only. Passing those checks establishes signaling behavior under that fixture, not token renewal or unrestricted third-party interoperability.

Registration now uses PUT /api/tenants/:tenantSlug/dataplanes (full replacement) and DELETE /api/tenants/:tenantSlug/dataplanes/:dataplaneId. A registration contains dataplaneId, endpoint, profiles, and optional labels, title, and healthEndpoint. A private authorization object can supply a separate OAuth client for signaling; when omitted, the control plane uses its configured client. See signaling authentication for credential fields, permissions and peer identities. One analytics deployment advertises both analytics profiles. Health remains unknown when no health endpoint is configured.

The registration migration removes old registrations and their dataset links, and clears data-plane registration caches. Restart data planes to register, then republish datasets; catalog records remain but cannot start transfers until linked again. Configure each data plane's controlPlane.controlplaneId with the receiving control plane's signaling identity and controlEndpoint with its tenant URL. Set controlPlane.tenantSlug to the addressed tenant and preserve controlPlane.dataplaneId across upgrades.

A third-party data plane can instead be configured without self-registration:

initTenants:
  - slug: example
    dataPlanes:
      - dataplaneId: external-http
        endpoint: https://plane.example/signaling
        profiles: [https://w3id.org/dspace-sig/profile/http-pull]
        labels: [eu]

CLI participant configuration accepts the same spec.dataPlanes list. Helm accepts it under config.initTenants[].dataPlanes. API registrations have source api, configured registrations source config.

The supported formats are https://w3id.org/dspace-sig/profile/http-pull, https://tsg.dataspac.es/profile/analytics-execution, and https://tsg.dataspac.es/profile/analytics-data. Unknown profiles are rejected. Incoming tsg:HTTP, tsg:analytics, and tsg:analytics-data are deprecated aliases, with warnings; catalogs publish canonical values. Analytics execution supplies a provider execution endpoint for control requests and application-provisioned bidirectional events. Its neutral name intentionally deviates from the DPS push/pull suffix rule. Analytics data grants access without an endpoint.

HTTP addresses carry a JWT in authorization and authType: bearer, with expiry and refresh properties. Applications should call sdk.transfers.dataAccess(transferId) before sending data requests to obtain current credentials. Management transfer responses omit endpoint properties; dataAccess is the only way to obtain the credentials. See HTTP dataflow execution for renewal and lifecycle behavior.

HTTP JWT startup requires a persistent signing key. The CLI preserves its signing Secret; Helm mounts the Secret named by dataAccessKeySecret. Keep the key and database across restarts, and run one HTTP data-plane replica. When upgrading an existing HTTP plane from opaque tokens without clearing its flows, suspend and resume those flows to obtain JWT credentials. When the signaling migrations described above apply, start new transfers instead.