Tenant-aware analytics execution

Every analytics worker uses the same tenant-aware execution model. Configure one exact execution owner for a single-tenant deployment, or several owners for a shared worker. Both use trusted Docker or Kubernetes jobs, either standalone or through explicitly assigned split runners. There is no dedicated/shared mode switch.

Run at most one executing replica per worker identity. Callback tokens are held by that process, and startup recovery fences jobs from its previous process. CLI and Helm reject multiple executing replicas of the same worker.

Use the authenticated execution API for participant operations and the operator HTTP API for instance management. The operator UI polls authenticated HTTP endpoints. The split runner bridge requires its own authenticated service identity and owner assignments; an ordinary tenant token cannot connect as a runner.

Authority and revocation

Signaling verifies the configured issuer, worker audience, logical control-plane client, canonical tenant_id, dplane-signaling scope and execute:dp.transfer permission. The worker stores ExecutionScope { issuer, controlPlaneId, tenantId } as an unambiguous JSON tuple. Submitted participants, source names and URL paths cannot establish ownership.

The control plane renews each active flow every 10 seconds. Its execution deadline reflects the earliest authorization projection for the owner and any exact locally managed counterparty. The worker caps its lease by that deadline and the signaling JWT expiry. Reading status does not renew a lease. Expiry fences execution until an explicit resume creates new authority.

Jobs retain the exact dataset-grant flow IDs and source snapshots. Submission, start, callbacks, source reads, events, results and queued delivery recheck the execution flow and every captured grant. Owner scope, executing participant, requester, dataspace, dataset and source must agree. Multiple eligible grants for one dataset are rejected. Revoke or finish the obsolete grant before submitting against another one.

A one-second sweep persists expired flows as suspended, revokes serving and retries runtime stops through the Docker or Kubernetes API. A suspended or terminated flow stops its dependent jobs. The worker checks job authority every second. Kubernetes also sets and renews the Job's native activeDeadlineSeconds, measured from its original status.startTime. A restarted worker fences existing jobs because callback tokens are memory-only. Submit a new local instance to run again.

Image references must be administrator-approved; pin digests when qualifying images. Docker checks registry access by pulling the image, even when credentials are omitted and the image is cached. A failed pull never falls back to a cached copy. Kubernetes independently verifies access to the image manifest with exactly the submitted or anonymous credentials before scheduling, then requires an image pull. A kubelet pull alone cannot prove owner authorization because nodes may have ambient registry credentials. For local development, an operator can list exact approved references in orchestration.preloadedImages. This explicitly authorizes every tenant to execute those cached images under the configured pullPolicy. The list must be a subset of allowedImages; Kubernetes compares against each entry's image. Submitted credentials still force registry authorization and a successful pull for preloaded images. The approval list alone does not permit execution from cache.

Submitted command overrides are rejected. Docker preserves the image's default entrypoint and command. Kubernetes uses an explicit approved command configured by the administrator. Jobs run their approved command directly; the worker does not inject a shell wrapper or execute maintenance commands inside running containers. Both runtimes provide the algorithm input through ALGORITHM_INSTANCE_FILE. Docker mounts /run/adp/algorithm-instance.json from a job-specific read-only volume; Kubernetes uses a read-only ConfigMap mount. Registry credentials are excluded from both inputs.

While the worker and runtime API are reachable, the external revocation bound is the remaining authorization projection lifetime, plus the one-second sweep interval and runtime stop latency. Runtime outages change this guarantee as described under Trust limits. Already-open dataset/result streams may finish. Bytes already delivered cannot be recalled. S3 datasets and results stream through the worker, so it does not issue independently usable presigned URLs for them. Event payloads are immutable after their first successful commit. Results from completed jobs remain readable while their flows and grants are active; terminated, cancelled and failed jobs are fenced.

Worker configuration

controlPlane:
  controlplaneId: shared-control-plane
  dataplaneId: shared-analytics
  executionScopes:
    - issuer: https://identity.example
      controlPlaneId: shared-control-plane
      tenantId: 11111111-1111-4111-8111-111111111111
split:
  mode: standalone
orchestration:
  type: docker
  allowedImages:
    - registry.example/approved-algorithm@sha256:<digest>
  socketPath: /var/run/docker.sock
  pullPolicy: Always
  workerContainer: shared-analytics-worker
  limits:
    runAsUser: 1000
    tmpfsSize: 67108864 # 64 MiB for /tmp
    memory: 536870912 # 512 MiB, no swap
    nanoCpus: 1000000000 # one CPU
management:
  operatorPrincipals:
    - issuer: https://identity.example
      subject: operator-subject
datasources:
  sources:
    - sourceId: participant-data
      executionScope:
        issuer: https://identity.example
        controlPlaneId: shared-control-plane
        tenantId: 11111111-1111-4111-8111-111111111111
      type: local
      root: /srv/alice/data

Configure normal strict OAuth/JWKS validation under auth, including an expected management audience. Workers also require authenticated signaling. events.peerEndpoints lists approved remote ADP API bases for outgoing event channels. The worker's own public API base is allowed automatically. Provisioned addresses must use an approved base followed by /event-channels/<id>, and queued delivery rechecks the policy. This prevents a tenant from using event delivery to make the worker contact another job or an arbitrary private endpoint.

To run an approved fixture image built directly into Docker, explicitly grant cached execution to all tenants:

orchestration:
  type: docker
  allowedImages: [approved-analytics:fixture]
  preloadedImages: [approved-analytics:fixture]
  pullPolicy: Never

Retain the worker and isolation settings from the complete configuration above. Kubernetes uses the same preloadedImages list alongside its approved image commands. Preload the image on each eligible node before selecting Never.

Global management accepts only allowlisted human issuer/subject pairs without a tenant grant. An empty operator allowlist denies all instance-wide management. A tenant's manage:* permission does not confer operator access.

Source entries require their exact execution scope. Two owners can use the same source ID. Unscoped global sources are unavailable. Local sources enforce canonical filesystem containment, including symlinks. HTTP sources constrain origin/path, keep upstream authorization in the worker and do not follow redirects. S3 sources use the configured bucket and optional keyPrefix; credentials and clients belong to that binding. Result scratch storage under events.payloads is separate from dataset sources. Its keys contain owner scope, local job ID and event ID.

The control plane owns tenant-local registrations; the worker does not self-register or need a tenant catalog. Under each managed tenancy profile, register the same worker identity and endpoint:

dataPlanes:
  - dataplaneId: shared-analytics
    endpoint: https://analytics.example/api
    executionLease: true
    profiles:
      - https://tsg.dataspac.es/profile/analytics-execution
      - https://tsg.dataspac.es/profile/analytics-data

Use the real reachable worker endpoint and match dataplaneId to its signaling audience. The executing participant comes from the owning flow, never a process-wide catalog cache.

Runtime trust boundary

Run the standalone worker itself in Docker. Set orchestration.workerContainer to its exact container ID or name, give it the adp.tsg.app/component=analytics-worker label and the worker identity label derived from its configured dataplaneId. The runtime checks that the configured running container has the current worker hostname and both labels. The analytics sample runs each Docker worker in its own verified container.

Each local job has a generated UUID, immutable execution scope and worker labels. Runtime lookup, logs, stop, deletion and recovery verify these labels. Docker creates a separate internal bridge per job with gateway_mode_ipv4=isolated and attaches only that job and the existing worker. The worker derives the callback address from its own interface on that network. Isolated mode removes the host bridge address. Jobs have no default route to external services or other job networks. IPv6 is disabled on the bridge and inside each job; the runtime also requires gateway_mode_ipv6=isolated and rejects any assigned IPAM gateway.

Linux capabilities are dropped and no-new-privileges is enabled. Each job is limited to 256 processes and to orchestration.limits.memory (default 512 MiB, no swap) and orchestration.limits.nanoCpus (default one CPU). Jobs run with orchestration.limits.runAsUser as their UID and GID (default 1000) and a read-only root filesystem. Writable /tmp is a tmpfs bounded by orchestration.limits.tmpfsSize (default 64 MiB). The mount disallows executable files. Docker rotates each job's JSON logs across two files of at most 10 MiB each. Images must support that nonroot identity, keep writes in /tmp and declare no Docker volumes. The image's working directory stays unchanged. Read the algorithm input through ALGORITHM_INSTANCE_FILE, which points to /run/adp/algorithm-instance.json for Docker. The JSON structure and file-based image contract remain unchanged.

Jobs receive no Docker socket, host dataset mounts, registry credentials or source credentials. Reads use a job-specific token at the worker proxy. Docker network, callbackUrl and mountFiles settings are unsupported. The runtime selects each job network and its callback address.

Docker requires Linux Docker Engine 28 or newer. The runtime verifies the created bridge options and absence of a host gateway before starting a job. An ordinary internal bridge still exposes host services through its bridge address; isolated gateway mode removes that address. When qualifying a daemon, verify that a host listener reachable through an ordinary internal bridge is blocked from an isolated job bridge, while the direct worker callback remains reachable.

For split Docker runners, the runtime worker identity includes both controlPlane.dataplaneId and split.runnerId. Set the controller container's worker label to the hash of the JSON tuple [dataplaneId, runnerId], using the same s<base64url SHA-256>s encoding as the runtime. This keeps recovery and cleanup separate even when several runners share one logical data-plane identity and daemon.

The worker rechecks authority after pulling the image, before starting the container and after it starts. The approved image runs with its original entrypoint and command. The worker creates a labeled configuration volume and populates it through Docker's archive API using a container that is never started. It removes that staging container before starting the algorithm, which mounts the volume read-only. The JSON file is limited to 1 MiB, matching the former runtime directory bound. Job deletion removes unused configuration volumes; startup also removes owned orphan volumes. Stopped jobs retain their configuration until deletion, alongside their logs. Docker termination depends on a running worker and reachable Docker API; see Trust limits below.

Kubernetes

Kubernetes requires an explicitly selected context, a nondefault job namespace, a separate job service account and a CNI whose NetworkPolicy enforcement has been verified on the actual cluster. Before enabling the worker, probe the actual job network to verify that permitted worker callbacks succeed and unrelated destinations are denied. Kubernetes documents this dependency in its NetworkPolicy requirements.

orchestration:
  type: kubernetes
  context: in-cluster # or an explicit kubeconfig context
  namespace: analytics-jobs
  allowedImages:
    - image: registry.example/approved-algorithm@sha256:<digest>
      command: [node, /app/job.mjs]
  jobServiceAccount: analytics-job
  callbackUrl: http://{POD_IP}:3000
  workerNamespace: analytics-workers
  workerPodSelector:
    app: shared-analytics-worker
  runAsUser: 1000
  resourceRequests: { cpu: 100m, memory: 128Mi }
  resourceLimits: { cpu: "1", memory: 512Mi }

Set the POD_IP environment variable from the pod's status.podIP downward API field. The worker resolves the {POD_IP} host placeholder at startup and brackets IPv6 addresses. An explicit callback URL must contain the running worker's literal Pod IP and API port. The worker verifies that the IP belongs to a selected running pod without host networking. A Service address or DNS name is insufficient for this boundary. Jobs receive no DNS egress. A worker Pod IP change fences existing work; new jobs use the replacement worker address.

Each job gets a NetworkPolicy before its Job is created. It denies ingress and allows egress only to selected worker pods in the exact worker namespace on the callback port. The runtime rejects existing policies that would also select the job and grant broader access. Keep this namespace under administrator control: NetworkPolicies are additive, so subsequently adding a permissive policy can widen access.

Jobs reject a job ServiceAccount with imagePullSecrets, both at startup and before each submission. Only a submission's own pull secret is attached to its pod. Node credential providers and admission configuration may still affect the kubelet pull, but they cannot bypass the independent manifest authorization check. That check uses the Registry v2 manifest endpoint over HTTPS with anonymous access, Basic credentials or a Bearer challenge. The worker must directly reach the registry and its HTTPS token endpoint within ten seconds; redirects, environment proxies, HTTP registries and provider-only credential flows are unsupported. The submitted credential registry must match the image hostname, with Docker Hub aliases accepted. Configure explicitly preloaded images for a disconnected fixture; submitted credentials never skip verification.

Jobs have no mounted service-account token, dataset PVC, host path, source credential or container socket. They run as a nonroot UID with a read-only root filesystem, dropped capabilities, seccomp, resource limits and a writable temporary directory. Algorithm configuration is mounted read-only from a ConfigMap. Source bytes come through the job capability proxy. Lifecycle operations, logs, recovery and cleanup match immutable job, owner and worker labels. The controller retains scoped Kubernetes management privileges; the job account does not inherit them. See Kubernetes' security context reference for the underlying controls.

The initial Job timeout is the remaining lease duration at creation. Once Kubernetes reports status.startTime, the worker adjusts the timeout to the verified absolute lease deadline. Each renewal uses that same start time, so repeated checks do not add execution time. The Job controller enforces the last accepted timeout if the worker stops. A Job that has not started when the worker becomes unavailable can start later with its initial relative timeout; this does not provide an absolute expiry during scheduling outages. Native Job deadlines depend on the Kubernetes control plane and do not guarantee termination on an unreachable node. See the Kubernetes Job deadline documentation.

The Helm chart adds the controller's runtime RBAC and optional tokenless job service account for Kubernetes workers. Set kubernetes.createJobServiceAccount: true to create that account. The job and worker namespaces must already exist. Job namespace metadata reads use a ClusterRole restricted to that exact name. Cross-namespace worker pod reads receive a separate RoleBinding.

Trust limits

The worker retains its external control-plane connectivity; jobs use the worker API for callbacks, data and authorized peer events. Every deployment requires administrator-qualified images. Runtime enforcement uses the Docker and Kubernetes APIs rather than a process inside the algorithm container.

Docker has no native renewable container deadline. If the worker crashes or loses Docker API access, a running container can continue computing until recovery fences it. Callbacks and data access still require unexpired authority when the worker serves them. Kubernetes retains its last accepted native Job timeout, subject to the scheduling and control-plane limits above. Removing the in-container watchdog trades worker-independent expiry for direct image execution, fewer runtime assumptions and no exec privileges.

The remaining orchestration code keeps four behaviors that a bare create/status/delete implementation would lose:

• Immutable worker, owner and job labels protect lifecycle actions and logs from cross-tenant access.
• Per-job networks or NetworkPolicies restrict data access to the selected worker. Image approval, registry authorization and resource limits remain part of admission.
• Serialized job operations, terminal outcomes and retried stops preserve cleanup after partial failures and concurrent lifecycle requests.
• Startup recovery fences surviving jobs and removes owned orphan resources. Removing it would leave containers, policies, credentials or network pools behind after a crash.

Network policy enforcement must still be qualified on the actual cluster. API checks do not prove that the CNI enforces the policy.

These controls do not protect against a compromised controller, runtime, cluster administrator or operator, and do not establish a production HA contract.

Assigned split runners

The server holds original execution/data grants and federated event channels. Each runner holds its local source registry, runtime and separate persistent database. The server maps a verified OAuth issuer/subject (and optional client ID) to one runner ID and exact execution scopes. A scope may belong to only one runner. Job assignment is persisted and immutable; changing configuration does not move existing jobs to another runner.

The client requests an audience-specific service token and connects over HTTPS. The server checks signature, issuer, audience, configured principal and expiry. Each connection receives a fresh session ID. Every bridge message includes protocol version, runner ID, session ID, owner scope and execution revision. Outbound work goes only to its assigned connection. Status, events and payload chunks must match the same persisted assignment and current authority.

The server renews bounded snapshots while original execution and dataset grants remain valid. The runner persists monotonically increasing revisions and deadlines in an authority ledger. Revocation or locally observed expiry permanently fences that local job UUID, including across pruning and reconnect. Retain the ledger with the runner database. Deleting it discards the replay fence. While the runner is running, its authority sweep fences jobs at the last verified deadline even if the bridge disconnects. If the runner itself is unavailable, the runtime limits above apply.

Result payloads travel as bytes, never as runner-selected fetch URLs. Chunk assembly is bound to the connection, session, job, event, transfer and revision. Split payloads default to a 64 MiB limit (split.maxPayloadBytes, configurable downward); buffering is bounded and chunks default to 512 KiB. At most 128 chunks are accepted, so smaller configured chunks also reduce the effective payload limit. Successful delivery requires a receiver commit acknowledgement. An interrupted transfer cannot reuse bytes from an old session. See split deployment for both sides' configuration.

Buffering a full payload needs several times its size in process memory on both sides. Measure memory with the configured payload limit and size both split processes accordingly, or lower split.maxPayloadBytes. The byte cap is not a process-memory limit.

Client identity and migration

Signaling flow IDs remain globally unique on one worker; a duplicate ID is rejected across owners. The submitted algorithm id is the distributed run correlation. The response id is a fresh local instance UUID and sharedRunId preserves the submitted ID. Clients must use the returned local ID for lifecycle calls, event channels, polling and callbacks. The analytics orchestration client already stores this returned ID per participant. Algorithm input contains both fields. External event IDs are unique within a local job, so peers may reuse the same external ID on different owners without collision.

SQLite and PostgreSQL migrations retain old flows, jobs, events, channels, payload references and queued deliveries. Historical ownership remains null; the worker never invents a default tenant. Before upgrading, drain existing work and stop its runtime containers or Jobs. Back up each database separately, configure explicit owner scopes and owner-bound sources, then create fresh flows and jobs. Do not merge participant databases or infer ownership from participant names. Unowned historical work stays fenced. The same steps apply to a deployment serving just one owner.

Retained historical object payloads use the old key format and cannot be read through the new owner-scoped payload APIs. Export needed payloads before cutover, or read them through an operator-controlled old deployment with the original database and storage configuration. Do not assign invented owners or bypass ownership checks to recover them. Deleting or pruning an unowned historical job removes its database metadata and inline payloads only; it leaves old object payloads and unowned runtime resources untouched. Preserve the database backup and payload references before pruning, and remove retained objects or runtime resources only through an operator-controlled cleanup of the verified old deployment.

Remove controlPlane.mode and the former self-registration fields. Add explicit controlplaneId, dataplaneId and controlPlane.executionScopes on standalone workers and split servers. Move Docker sharedLimits to limits and shared.preloadedImages to preloadedImages. Move Kubernetes settings from orchestration.shared directly into orchestration. Docker file mounts, custom networks and fixed callback URLs are no longer supported. Split clients require their runner identity, authenticated bridge and split.executionScopes.

Existing runtime resources with older ownership or worker labels are not adopted. Restarted workers also fence their previous jobs because callback tokens are memory-only. Submit new local instances after restart. Retain the split authority ledger with its database; deleting it removes the replay fence. Rollback requires resolving repeated external event IDs across jobs and any retained authority history or active runner assignments. Rollback does not discard that history automatically.

The local analytics sample keeps one standalone Alfa worker and one Bravo split server/client pair, each restricted to its explicit owner. It uses this execution model with a shared Wallet and Control Plane. Configuration checks do not qualify Docker network isolation, Kubernetes policy enforcement, payload memory limits or lifecycle revocation on a deployment.

Operator audit records

The platform operator audit endpoint includes records across execution owners. Only configured operator principals can read it; tenant execution capabilities cannot access the management audit API.

New instance creation and status changes produce audit records with the local instance ID, owner tenant ID and distributed run correlation. Failed instances have error severity. Rejected execution capabilities also produce an audit record. These entries omit algorithm definitions, data sources, bearer capabilities, dataset contents and result payloads. Reads and event polling remain excluded by default to avoid filling the audit log with polling traffic.

Database audit storage must be enabled for the UI to show records. Existing jobs are not backfilled: records begin with events observed after the update.