Configuration

In this section, the configuration of the analytics data plane is explained. Configuration is used based on a config.yaml file which should be placed in the apps/backend folder. This config.yaml file is loaded when booting the application. The values get type checked, and it gives a clear error message if there is a configuration field missing or provided incorrectly. Next to the config.yaml file, you can also set environment variables. These override the values that are listed in the config.yaml file.

Databases

By default, the development database is sqlite. We use postgres databases for production type instances of the analytics data planes.

Authentication

Authentication for frontend services can be done via OAuth. This helps users who need to login to several components to authenticate themselves faster. The data plane is tested against the SSO Bridge, with an Helm chart provided alongside the Helm chart of the data plane, but other OAuth services should be usable (e.g. Keycloak, or hosted OAuth services).

Execution sources

A transferred dataset carries only { sourceId, path } in its private asset configuration. The media type stays in catalog metadata. The source ID resolves through this deployment's role-local source configuration. The orchestration app has its own inspection configuration for the same logical sourceId; its physical path and credentials may differ. The control plane receives neither.

datasources:
  sources:
    - sourceId: party-a-lake
      executionScope:
        issuer: https://identity.example
        controlPlaneId: control-plane-api
        tenantId: 11111111-1111-4111-8111-111111111111
      type: s3
      endpoint: http://minio:9000
      bucket: party-a
      region: eu-west-1
      forcePathStyle: true
    - sourceId: local-dataset
      executionScope:
        issuer: https://identity.example
        controlPlaneId: control-plane-api
        tenantId: 11111111-1111-4111-8111-111111111111
      type: local
      root: /data

Relative local roots resolve from config.yaml. With Helm, sources live under config.datasources.sources, so the existing ConfigMap and rollout checksum are sufficient. Put credentials in configFromSecrets.datasources.sources at the same list index; the chart projects them as TSG__... environment overrides. In split mode configure sources only on the LAN client runner, not the DMZ server.

Jobs read source bytes through the owner-scoped worker proxy with a per-job DATA_ACCESS_TOKEN. Local files and S3 objects stay behind that proxy. Jobs receive no dataset mounts, storage credentials or presigned S3 URLs. The ADP does not inspect CSV, author metadata, upload datasets, or maintain a catalog.

Every standalone worker and split server requires explicit controlplaneId, dataplaneId and a nonempty controlPlane.executionScopes list. A deployment serving one owner uses one scope and the same runtime restrictions. Split clients use split.executionScopes and an authenticated assigned runner identity. See tenant-aware execution for complete runtime settings and upgrade steps. Old unowned jobs are fenced; drain them before upgrading and submit fresh flows and jobs with explicit ownership.

Configuration parameters

Key	Required	Type	Description	Default
ManagementConfig
management		ManagementConfig	Worker global management policy
OperatorPrincipal
management.operatorPrincipals		OperatorPrincipal[]	Global human operators allowed to inspect this worker; tenant grants never confer operator access	[]
management.operatorPrincipals[].issuer	Yes	String	Verified operator issuer
management.operatorPrincipals[].subject	Yes	String	Verified operator subject
DatabaseConfig
db	Yes	DatabaseConfig	Database configuration
db.retryAttempts		Number	Connection attempts during application startup, three seconds apart	10
db.type	Yes	"sqlite" | "postgres"	Type of database
db.database	Yes	String	Name of the database
db.synchronize		Boolean	Synchronize database schema	false
db{type=sqlite}	Yes	SQLiteConfig	Database configuration
db{type=sqlite}.retryAttempts		Number	Connection attempts during application startup, three seconds apart	10
db{type=sqlite}.type		"sqlite" | "postgres"	Type of database	"sqlite"
db{type=sqlite}.database	Yes	String	Name of the database
db{type=sqlite}.synchronize		Boolean	Synchronize database schema	false
db{type=postgres}	Yes	PostgresConfig	Database configuration
db{type=postgres}.host	Yes	String	Host of the database
db{type=postgres}.port	Yes	Number	Port of the database
db{type=postgres}.username	Yes	String	Username of the database
db{type=postgres}.password	Yes	String	Password of the database
db{type=postgres}.ssl		Object	SSL configuration of the database	false
db{type=postgres}.retryAttempts		Number	Connection attempts during application startup, three seconds apart	10
db{type=postgres}.type		"sqlite" | "postgres"	Type of database	"postgres"
db{type=postgres}.database	Yes	String	Name of the database
db{type=postgres}.synchronize		Boolean	Synchronize database schema	false
ServerConfig
server		ServerConfig	Server configuration
server.listen		String	IP address the server listens on	"0.0.0.0"
server.port		Number	Port the server listens on	3000
server.publicDomain		String	Public domain of the server	"localhost"
server.publicAddress		String	Public address of the server	"http://localhost:3000"
server.subPath		String	Sub path of the server
AuthConfig
auth	Yes	AuthConfig	Management authentication configuration
auth.signalingId		String	Receiving plane identity for signaling tokens. Defaults to clientId.
auth.expectedIssuer		String	Expected API token issuer; defaults to the discovered OpenID issuer
auth.expectedAudience		String	Expected API token audience; defaults to clientId.
auth.enabled		Boolean	Enable authentication	true
auth.openIdConfigurationURL		String	OpenID configuration URL
auth.callbackURL		URL	Callback URL the auth service will redirect users
auth.redirectURL		URL	Redirect URL to UI after login/logout
auth.clientId		String	Client ID
auth.clientSecret		String	Client secret (required for client_secret_post authentication)
auth.tokenEndpointAuthMethod		"client_secret_post" | "private_key_jwt"	Token endpoint authentication method: client_secret_post (default) or private_key_jwt	"client_secret_post"
auth.privateKeyJwk		Object	Private key in JWK format for private_key_jwt authentication. Alternative to privateKeyJwkFile.
auth.privateKeyJwkFile		String	Path to file containing private key JWK for private_key_jwt authentication. Alternative to privateKeyJwk.
auth.permissionPath		String	JSON path to extract permissions from the token	"$.permissions[*]"
ControlPlaneConfig
controlPlane	Yes	ControlPlaneConfig	Control plane configuration
controlPlane.controlplaneId	Yes	String	Logical control-plane client identity verified on signaling
controlPlane.dataplaneId	Yes	String	Stable worker identity and audience for authenticated signaling
ExecutionScopeConfig
controlPlane.executionScopes		ExecutionScopeConfig[]	Exact execution owners accepted by this worker. Required on signaling servers; split runners use their bridge assignments	[]
controlPlane.executionScopes[].issuer	Yes	String	Verified signaling issuer
controlPlane.executionScopes[].controlPlaneId	Yes	String	Logical control-plane client identity
controlPlane.executionScopes[].tenantId	Yes	String	Canonical tenant UUID
LoggingConfig
logging		LoggingConfig	Logging configuration
logging.debug		Boolean	Enable debug request logging	false
DataSourcesConfig
datasources		DataSourcesConfig	Participant-local sources used for job execution
SourceConfig
datasources.sources	Yes	SourceConfig[]	Configured source bindings; workers require an exact executionScope on each source. Local paths resolve from this configuration file
SourceExecutionScopeConfig
datasources.sources[].executionScope		SourceExecutionScopeConfig	Owner required by analytics executors: issuer, controlPlaneId and tenantId
datasources.sources[].executionScope.issuer	Yes	String	Verified issuer that grants this execution ownership
datasources.sources[].executionScope.controlPlaneId	Yes	String	Logical control-plane identity that owns this execution scope
datasources.sources[].executionScope.tenantId	Yes	String	Canonical tenant UUID that owns this execution scope
datasources.sources[].sourceId	Yes	String	Stable logical source identifier shared across components
datasources.sources[].type	Yes	"local" | "http" | "s3"	Source type: local, http or s3
datasources.sources[].root		String	local: root directory visible to this component
datasources.sources[].baseUrl		String	http: base URL under which dataset paths are resolved
datasources.sources[].bucket		String	s3: bucket containing the datasets
datasources.sources[].keyPrefix		String	s3: configured object-key prefix containing this owner's datasets
datasources.sources[].endpoint		String	s3: endpoint URL for an S3-compatible service
datasources.sources[].region		String	s3: region
datasources.sources[].forcePathStyle		Boolean	s3: use path-style addressing
datasources.sources[].authorization		String	http: upstream Authorization header value
datasources.sources[].accessKeyId		String	s3: access key ID
datasources.sources[].secretAccessKey		String	s3: secret access key
EventsConfig
events		EventsConfig	Algorithm event configuration
events.peerEndpoints		String	approved remote analytics API bases for peer event channels; the local worker API is always allowed	[]
EventPayloadStoreConfig
events.payloads		EventPayloadStoreConfig	Where algorithm event payloads are stored. When unset, payloads are stored in the database, which does not scale past a few hundred megabytes
events.payloads.type		"local" | "s3"	Store type: 'local' or 's3'	"local"
events.payloads.root		String	local: directory that payloads are written under	"./data/event-payloads"
events.payloads.bucket		String	s3: bucket holding the payloads
events.payloads.endpoint		String	s3: endpoint URL, for S3-compatible stores such as MinIO
events.payloads.region		String	s3: region
events.payloads.forcePathStyle		Boolean	s3: use path-style addressing instead of virtual-hosted style
events.payloads.accessKeyId		String	s3: access key id
events.payloads.secretAccessKey		String	s3: secret access key
events.payloads.prefix		String	Key prefix that payloads are written under	"events"
RuntimeConfig
runtime	Yes	RuntimeConfig	Runtime configuration
runtime.color		String	Primary UI color	"#3B8BF6"
runtime.lightThemeUrl		String	Light theme logo URL
runtime.darkThemeUrl		String	Dark theme logo URL
runtime.jobRefreshIntervalMs		Number	Interval in milliseconds for auto-refreshing the jobs table in the UI (0 to disable)	10000
OrchestrationConfigBase
orchestration		OrchestrationConfigBase	Job runtime configuration, unused on a split server
orchestration.type	Yes	"kubernetes" | "docker"	Orchestration type: 'kubernetes' or 'docker'
orchestration{type=kubernetes}		KubernetesConfig	Job runtime configuration, unused on a split server
KubernetesApprovedImage
orchestration{type=kubernetes}.allowedImages	Yes	KubernetesApprovedImage[]	Administrator-qualified cooperative images and their fixed commands
orchestration{type=kubernetes}.allowedImages[].image	Yes	String	Exact qualified image reference; prefer immutable digests
orchestration{type=kubernetes}.allowedImages[].command	Yes	String	Operator-approved executable and arguments, independent of submitted algorithm commands
orchestration{type=kubernetes}.preloadedImages		String	Exact approved images whose cached copies every tenant may execute without registry authorization	[]
orchestration{type=kubernetes}.jobServiceAccount	Yes	String	Pre-created unprivileged job service account without image pull secrets; default is not accepted
orchestration{type=kubernetes}.callbackUrl	Yes	String	Worker Pod IP callback URL on server.port. The {POD_IP} host placeholder resolves from the environment and brackets IPv6 addresses. Must match a running non-hostNetwork worker pod
orchestration{type=kubernetes}.workerNamespace	Yes	String	Namespace containing the selected worker callback pods
orchestration{type=kubernetes}.workerPodSelector	Yes	Object	Exact worker pod labels allowed by job egress policy
orchestration{type=kubernetes}.runAsUser		Number	Nonroot UID for job processes and writable volumes	1000
KubernetesResources
orchestration{type=kubernetes}.resourceRequests		KubernetesResources	CPU and memory requested for each job
orchestration{type=kubernetes}.resourceRequests.cpu	Yes	String	CPU quantity, for example '100m' or '1'
orchestration{type=kubernetes}.resourceRequests.memory	Yes	String	Memory quantity, for example '128Mi' or '512Mi'
KubernetesResources
orchestration{type=kubernetes}.resourceLimits		KubernetesResources	CPU and memory limits for each job
orchestration{type=kubernetes}.resourceLimits.cpu	Yes	String	CPU quantity, for example '100m' or '1'
orchestration{type=kubernetes}.resourceLimits.memory	Yes	String	Memory quantity, for example '128Mi' or '512Mi'
orchestration{type=kubernetes}.context		String	Explicit kubeconfig context for execution, or 'in-cluster' for the current pod service account
orchestration{type=kubernetes}.namespace		String	Kubernetes namespace	"default"
orchestration{type=kubernetes}.pullPolicy		"IfNotPresent" | "Always" | "Never"	Kubernetes image pull policy. Jobs verify registry manifest authorization and always pull unless listed in preloadedImages without submitted credentials. Leave unset to use Kubernetes defaults for approved preloaded images.
orchestration{type=kubernetes}.nodeSelector		Object	Kubernetes Node selector for scheduling jobs. See https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector for format details and usage examples.
orchestration{type=kubernetes}.affinity		Object	Kubernetes Node affinity for scheduling jobs. See https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/ for format details and usage examples.
orchestration{type=kubernetes}.tolerations		Array	Kubernetes Tolerations for scheduling jobs on tainted nodes. See https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/ for format details and usage examples.
orchestration{type=kubernetes}.type		"kubernetes" | "docker"	Orchestration type: 'kubernetes' or 'docker'	"kubernetes"
orchestration{type=docker}		DockerConfig	Job runtime configuration, unused on a split server
orchestration{type=docker}.preloadedImages		String	Approved images whose cached copies every accepted tenant may execute	[]
DockerLimits
orchestration{type=docker}.limits		DockerLimits	nonroot identity, writable temporary space, CPU and memory limits
orchestration{type=docker}.limits.runAsUser		Number	Nonroot UID and GID for jobs and their runtime directory	1000
orchestration{type=docker}.limits.tmpfsSize		Number	Writable /tmp size in bytes per job; the root filesystem is read-only	67108864
orchestration{type=docker}.limits.memory		Number	Memory limit in bytes per job; swap is disabled	536870912
orchestration{type=docker}.limits.nanoCpus		Number	CPU limit per job in units of 1e-9 CPUs	1000000000
orchestration{type=docker}.allowedImages		String	exact administrator-approved image references. Prefer immutable digest references	[]
orchestration{type=docker}.workerContainer		String	Worker container ID or name; jobs use internal networks attached only to this verified standalone worker or split runner
orchestration{type=docker}.socketPath		String	Docker socket path (e.g., /var/run/docker.sock)	"/var/run/docker.sock"
orchestration{type=docker}.platform		String	Platform to use for Docker containers (e.g., 'linux/amd64')
orchestration{type=docker}.pullPolicy		"IfNotPresent" | "Always" | "Never"	Docker image pull policy. Jobs always pull unless explicitly listed in preloadedImages without submitted credentials. Use IfNotPresent or Never for approved preloaded development images.	"Always"
orchestration{type=docker}.type		"kubernetes" | "docker"	Orchestration type: 'kubernetes' or 'docker'	"docker"
SplitConfig
split		SplitConfig	Client/server split configuration
RunnerConfig
split.runners		RunnerConfig[]	Server runner identity and owner assignments	[]
split.runners[].id	Yes	String	Stable runner identity used in persisted job assignments
split.runners[].issuer	Yes	String	Verified OAuth issuer of the runner service principal
split.runners[].subject	Yes	String	Verified OAuth subject of the runner service principal
split.runners[].clientId		String	Optional verified OAuth client ID required for the runner
SourceExecutionScopeConfig
split.runners[].executionScopes	Yes	SourceExecutionScopeConfig[]	Exact verified execution owners assigned exclusively to this runner
split.runners[].executionScopes[].issuer	Yes	String	Verified issuer that grants this execution ownership
split.runners[].executionScopes[].controlPlaneId	Yes	String	Logical control-plane identity that owns this execution scope
split.runners[].executionScopes[].tenantId	Yes	String	Canonical tenant UUID that owns this execution scope
split.runnerId		String	Client's immutable runner identity
SourceExecutionScopeConfig
split.executionScopes		SourceExecutionScopeConfig[]	Client's exact allowed execution ownership scopes	[]
split.executionScopes[].issuer	Yes	String	Verified issuer that grants this execution ownership
split.executionScopes[].controlPlaneId	Yes	String	Logical control-plane identity that owns this execution scope
split.executionScopes[].tenantId	Yes	String	Canonical tenant UUID that owns this execution scope
split.bridgeAudience		String	Audience requested by the runner for its authenticated bridge connection
split.allowInsecureBridge		Boolean	Allow a plain HTTP bridge only in an explicitly trusted local test network	false
split.maxPayloadBytes		Number	Split event payload limit in bytes, at most 64 MiB. Applies before buffering or chunk assembly	67108864
split.mode		"standalone" | "client" | "server"	Runtime mode: standalone (current behavior), client (network-constrained runner), or server (open-network exchanger)	"standalone"
split.bridgePeerWsUrl		String	Peer WebSocket URL for bridge connection (Socket.IO server base URL)
split.bridgeChunkSize		Number	Chunk size in bytes for large event data transfers over WebSocket (default: 512KB)
AuditModuleConfig
audit		AuditModuleConfig	Audit logging configuration
audit.enabled		Boolean	Enable audit logging	true
audit.minSeverity		"debug" | "info" | "warning" | "error" | "critical"	Minimum severity level to log	"info"
audit.logDenied		Boolean	Always log denied access attempts regardless of action type	true
audit.logDelegated		Boolean	Log delegated access attempts when enabled, while still respecting action-specific success filters	true
audit.logMutations		Boolean	Log successful mutation actions (create, update, delete, manage)	true
audit.logExecute		Boolean	Log successful execute actions	false
audit.logReads		Boolean	Log successful read actions — disabled by default as reads are frequent and rarely relevant for auditing	false
audit.sensitiveResources		Array	Resources logged regardless of minimum severity, while still respecting action-specific success filters	["w.key","w.credential","sso.user"]
AuditHandlersConfig
audit.handlers		AuditHandlersConfig	Handler configuration
audit.handlers.console		Boolean	Enable console audit log handler	false
audit.handlers.database		Boolean	Enable database audit log handler	true
audit.handlers.otlp		Boolean	Enable OTLP audit log handler	false
OtlpAuditHandlerConfig
audit.otlp		OtlpAuditHandlerConfig	OTLP audit log handler configuration
audit.otlp.url		String	OTLP HTTP logs endpoint URL	"http://localhost:4318/v1/logs"
audit.otlp.headers		Object	Additional HTTP headers to include in OTLP requests
audit.otlp.timeoutMillis		Number	OTLP export timeout in milliseconds	5000
audit.otlp.concurrencyLimit		Number	Maximum number of concurrent OTLP export requests	1
audit.otlp.serviceName		String	Service name included on OTLP log attributes	"tsg-dsp-api"
audit.otlp.serviceVersion		String	Optional service version included on OTLP log attributes