Skip to main content
Version: Next

TSG Analytics Orchestration

The Analytics Orchestration application is the coordinating half of a distributed analytics run. Where the Analytics Data Plane executes an algorithm on one participant's data, this application decides which participants take part, under what agreement, and when — and then drives the DSP negotiations and transfers that let each plane do its work.

It owns two channels, deliberately separated:

  • a governance channel between the orchestration applications of the participants, carrying project agreements, signatures and credential offers. Peers are found by resolving their DID document, and every message is authenticated with a JWT signed by the sender's wallet;
  • an execution channel per participant, which is an ordinary DSP negotiation and transfer followed by a submission to that participant's execution endpoint.

No research data passes over the governance or execution channels: data-plane events travel directly between the planes. The participant-local application is nevertheless data-aware when an operator asks it to inspect a configured datasource while authoring catalog metadata.

Operator flow​

Every participant runs its own instance, backed by that participant's control plane and wallet. The normal flow is:

  1. An initiator creates a project agreement and sends signature requests to the other participants' DID-advertised orchestration inboxes.
  2. Each participant signs or rejects the agreement in its own console. The initiator finalizes only after every participant has signed; recipients independently verify the signatures and canonical agreement hash.
  3. Participants publish their orchestration capability and register data that already exists in a participant source. Linking a data dataset to the finalized project adds a credential-gated catalog offer.
  4. The initiator creates a run from the finalized project. Participant identity comes from the agreement; DSP addresses and datasets are discovered from the catalog, with explicit local overrides available for development.
  5. The application negotiates the required transfers, submits the same algorithm instance to every data plane, provisions only the event routes in the run's routing matrix, and reports participant and delivery state.

The console is served at /; the REST API is under /api. Management routes use the common OAuth/ABAC guard when authentication is enabled. /api/inbound is protected separately by the signed governance envelope.

Deployment boundaries​

Dataset registration does not upload bytes. The orchestration app declares read-only access to local, HTTP, or S3 data in its own configuration; the analytics data plane separately declares execution access using the same logical sourceId. A published private asset contains only sourceId and path; its media type remains catalog metadata. CSV description scans the source for an exact record count and keeps only a bounded sample for type inference. Event payloads bypass the orchestration application: the data planes deliver them directly, while the application records their metadata and can fetch a payload on demand from its producing plane.

This makes the orchestration application a data-aware component. In a split DMZ/LAN topology it should normally run in the LAN alongside the restricted runner and its sources. The DMZ data-plane server needs no source configuration. A dedicated source-local metadata service can replace inspection later without changing the catalog or execution contract.

The current implementation is single-process. In-memory start/termination coordination and polling loops are not safe for several replicas sharing one database. Delivery status is observational because the analytics data plane does not expose a recipient-delivery status API.

Reference​