Configuration

In this section, the configuration of the analytics data plane is explained. Configuration is used based on a config.yaml file which should be placed in the apps/backend folder. This config.yaml file is loaded when booting the application. The values get type checked, and it gives a clear error message if there is a configuration field missing or provided incorrectly. Next to the config.yaml file, you can also set environment variables. These override the values that are listed in the config.yaml file.

Databases

By default, the development database is sqlite. We use postgres databases for production type instances of the analytics data planes.

Authentication

Authentication for frontend services can be done via OAuth. This helps users who need to login to several components to authenticate themselves faster. The data plane is tested against the SSO Bridge, with an Helm chart provided alongside the Helm chart of the data plane, but other OAuth services should be usable (e.g. Keycloak, or hosted OAuth services).

Execution sources

A transferred dataset carries only { sourceId, path } in its private asset configuration. The media type stays in catalog metadata. The source ID resolves through this deployment's role-local source configuration. The orchestration app has its own inspection configuration for the same logical sourceId; its physical path and credentials may differ. The control plane receives neither.

datasources:
  sources:
    - sourceId: party-a-lake
      type: s3
      endpoint: http://minio:9000
      bucket: party-a
      region: eu-west-1
      forcePathStyle: true
      presignTtlSeconds: 3600
    - sourceId: local-dataset
      type: local
      root: /data
      pvcName: analytics-data-plane-uploads

Relative local roots resolve from config.yaml. With Helm, sources live under config.datasources.sources, so the existing ConfigMap and rollout checksum are sufficient. Put credentials in configFromSecrets.datasources.sources at the same list index; the chart projects them as TSG__... environment overrides. In split mode configure sources only on the LAN client runner, not the DMZ server.

Jobs receive the resolved location through the unchanged DATA_TYPE, DATA_FILE and DATA_URL variables. S3 produces a presigned URL. HTTP uses an ADP proxy and a per-job DATA_ACCESS_TOKEN, keeping upstream authorization in the runner. Local sources require Docker file mounting or a Kubernetes PVC. The ADP does not inspect CSV, author metadata, upload datasets, or maintain a catalog.

Configuration parameters

Key	Required	Type	Description	Default
DatabaseConfig
db	Yes	DatabaseConfig	Database configuration
db.type	Yes	"sqlite" | "postgres"	Type of database
db.database	Yes	String	Name of the database
db.synchronize		Boolean	Synchronize database schema
db{type=sqlite}	Yes	SQLiteConfig	Database configuration
db{type=sqlite}.type		"sqlite" | "postgres"	Type of database	"sqlite"
db{type=sqlite}.database	Yes	String	Name of the database
db{type=sqlite}.synchronize		Boolean	Synchronize database schema
db{type=postgres}	Yes	PostgresConfig	Database configuration
db{type=postgres}.host	Yes	String	Host of the database
db{type=postgres}.port	Yes	Number	Port of the database
db{type=postgres}.username	Yes	String	Username of the database
db{type=postgres}.password	Yes	String	Password of the database
db{type=postgres}.ssl		Unknown	SSL configuration of the database
db{type=postgres}.type		"sqlite" | "postgres"	Type of database	"postgres"
db{type=postgres}.database	Yes	String	Name of the database
db{type=postgres}.synchronize		Boolean	Synchronize database schema
ServerConfig
server		ServerConfig	Server configuration
server.listen		String	IP address the server listens on	"0.0.0.0"
server.port		Number	Port the server listens on	3000
server.publicDomain		String	Public domain of the server	"localhost"
server.publicAddress		String	Public address of the server	"http://localhost:3000"
server.subPath		String	Sub path of the server
AuthConfig
auth	Yes	AuthConfig	Management authentication configuration
auth.enabled		Boolean	Enable authentication	true
auth.openIdConfigurationURL		String	OpenID configuration URL
auth.callbackURL		URL	Callback URL the auth service will redirect users
auth.redirectURL		URL	Redirect URL to UI after login/logout
auth.clientId		String	Client ID
auth.clientSecret		String	Client secret (required for client_secret_post authentication)
auth.tokenEndpointAuthMethod		"client_secret_post" | "private_key_jwt"	Token endpoint authentication method: client_secret_post (default) or private_key_jwt	"client_secret_post"
auth.privateKeyJwk		Object	Private key in JWK format for private_key_jwt authentication. Alternative to privateKeyJwkFile.
auth.privateKeyJwkFile		String	Path to file containing private key JWK for private_key_jwt authentication. Alternative to privateKeyJwk.
auth.permissionPath		String	JSON path to extract permissions from the token	"$.permissions[*]"
ControlPlaneConfig
controlPlane	Yes	ControlPlaneConfig	Control plane configuration
controlPlane.dataPlaneEndpoint	Yes	URL	Data plane management endpoint
controlPlane.managementEndpoint	Yes	URL	Control plane management endpoint
controlPlane.walletEndpoint		URL	Wallet endpoint
controlPlane.controlEndpoint	Yes	URL	Public control plane endpoint
controlPlane.tenantSlug		String	Tenant slug to address on a (possibly shared, multi-tenant) control plane. Omit to use the control plane's default tenant.
controlPlane.initializationDelay		Number	Initialization delay in milliseconds	5000
controlPlane.dataPlaneTitle		String	Data Plane title	"Data Plane - v0.0.0"
LoggingConfig
logging		LoggingConfig	Logging configuration
logging.debug		Boolean	Enable debug request logging
DataStorageConfig
storage		DataStorageConfig	Job data storage configuration
storage.pvcName		String	Persistent volume claim name for job data storage
DataSourcesConfig
datasources		DataSourcesConfig	Participant-local sources used for job execution
SourceConfig
datasources.sources	Yes	SourceConfig[]	Role-local sources available to jobs; local paths resolve from this configuration file
datasources.sources[].sourceId	Yes	String	Stable logical source identifier shared across components
datasources.sources[].type	Yes	"local" | "http" | "s3"	Source type: local, http or s3
datasources.sources[].root		String	local: root directory visible to this component
datasources.sources[].baseUrl		String	http: base URL under which dataset paths are resolved
datasources.sources[].bucket		String	s3: bucket containing the datasets
datasources.sources[].endpoint		String	s3: endpoint URL for an S3-compatible service
datasources.sources[].region		String	s3: region
datasources.sources[].forcePathStyle		Boolean	s3: use path-style addressing
datasources.sources[].authorization		String	http: upstream Authorization header value
datasources.sources[].accessKeyId		String	s3: access key ID
datasources.sources[].secretAccessKey		String	s3: secret access key
datasources.sources[].pvcName		String	local: PVC mounted into Kubernetes jobs
datasources.sources[].presignTtlSeconds		Number	s3: lifetime in seconds of generated presigned URLs
EventsConfig
events		EventsConfig	Algorithm event configuration
EventPayloadStoreConfig
events.payloads		EventPayloadStoreConfig	Where algorithm event payloads are stored. When unset, payloads are stored in the database, which does not scale past a few hundred megabytes
events.payloads.type		"local" | "s3"	Store type: 'local' or 's3'	"local"
events.payloads.root		String	local: directory that payloads are written under	"./data/event-payloads"
events.payloads.bucket		String	s3: bucket holding the payloads
events.payloads.endpoint		String	s3: endpoint URL, for S3-compatible stores such as MinIO
events.payloads.region		String	s3: region
events.payloads.forcePathStyle		Boolean	s3: use path-style addressing instead of virtual-hosted style
events.payloads.presignTtlSeconds		Unknown	s3: lifetime in seconds of the presigned URL relayed across a DMZ/LAN split	3600
events.payloads.accessKeyId		String	s3: access key id
events.payloads.secretAccessKey		String	s3: secret access key
events.payloads.prefix		String	Key prefix that payloads are written under	"events"
RuntimeConfig
runtime	Yes	RuntimeConfig	Runtime configuration
runtime.color		String	Primary UI color	"#3B8BF6"
runtime.lightThemeUrl		String	Light theme logo URL
runtime.darkThemeUrl		String	Dark theme logo URL
runtime.jobRefreshIntervalMs		Unknown	Interval in milliseconds for auto-refreshing the jobs table in the UI (0 to disable)	10000
OrchestrationConfigBase
orchestration	Yes	OrchestrationConfigBase	Orchestration configuration
orchestration.type	Yes	"kubernetes" | "docker"	Orchestration type: 'kubernetes' or 'docker'
orchestration{type=kubernetes}	Yes	KubernetesConfig	Orchestration configuration
orchestration{type=kubernetes}.namespace		String	Kubernetes namespace	"default"
orchestration{type=kubernetes}.pullPolicy		"IfNotPresent" | "Always" | "Never"	Kubernetes image pull policy (IfNotPresent, Always, Never). Leave unset to let Kubernetes default per image: Always for ':latest' or untagged, IfNotPresent for a pinned tag/digest.
orchestration{type=kubernetes}.pullSecrets		String	Kubernetes image pull secrets for private registries (array of secret names)
orchestration{type=kubernetes}.nodeSelector		Object	Kubernetes Node selector for scheduling jobs. See https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector for format details and usage examples.
orchestration{type=kubernetes}.affinity		Object	Kubernetes Node affinity for scheduling jobs. See https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/ for format details and usage examples.
orchestration{type=kubernetes}.tolerations		Array	Kubernetes Tolerations for scheduling jobs on tainted nodes. See https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/ for format details and usage examples.
orchestration{type=kubernetes}.type		"kubernetes" | "docker"	Orchestration type: 'kubernetes' or 'docker'	"kubernetes"
orchestration{type=docker}	Yes	DockerConfig	Orchestration configuration
orchestration{type=docker}.socketPath		String	Docker socket path (e.g., /var/run/docker.sock)	"/var/run/docker.sock"
orchestration{type=docker}.network		String	Docker network to use for containers
orchestration{type=docker}.mountFiles		Boolean	Whether to mount files into Docker containers or provide files via HTTP
orchestration{type=docker}.platform		String	Platform to use for Docker containers (e.g., 'linux/amd64')
orchestration{type=docker}.pullPolicy		"IfNotPresent" | "Always" | "Never"	Docker image pull policy (IfNotPresent, Always, Never). Applied by the data plane; Docker has none. Use IfNotPresent for local development, where images are built into the daemon.	"Always"
orchestration{type=docker}.type		"kubernetes" | "docker"	Orchestration type: 'kubernetes' or 'docker'	"docker"
SplitConfig
split		SplitConfig	Client/server split configuration
split.mode		String	Runtime mode: standalone (current behavior), client (network-constrained runner), or server (open-network exchanger)	"standalone"
split.bridgePeerWsUrl		String	Peer WebSocket URL for bridge connection (Socket.IO server base URL)
split.bridgeChunkSize		Unknown	Chunk size in bytes for large event data transfers over WebSocket (default: 512KB)
AuditModuleConfig
audit		AuditModuleConfig	Audit logging configuration
audit.enabled		Boolean	Enable audit logging	true
audit.minSeverity		"debug" | "info" | "warning" | "error" | "critical"	Minimum severity level to log	"info"
audit.logDenied		Boolean	Always log denied access attempts regardless of action type	true
audit.logDelegated		Boolean	Log delegated access attempts when enabled, while still respecting action-specific success filters	true
audit.logMutations		Boolean	Log successful mutation actions (create, update, delete, manage)	true
audit.logExecute	Yes	Boolean	Log successful execute actions
audit.logReads	Yes	Boolean	Log successful read actions — disabled by default as reads are frequent and rarely relevant for auditing
audit.sensitiveResources		Unknown	Resources that are always logged regardless of action-specific success filters	["w.key","w.credential","sso.user"]
AuditHandlersConfig
audit.handlers		AuditHandlersConfig	Handler configuration
audit.handlers.console	Yes	Boolean	Enable console audit log handler
audit.handlers.database		Boolean	Enable database audit log handler	true
audit.handlers.otlp	Yes	Boolean	Enable OTLP audit log handler
OtlpAuditHandlerConfig
audit.otlp		OtlpAuditHandlerConfig	OTLP audit log handler configuration
audit.otlp.url		String	OTLP HTTP logs endpoint URL	"http://localhost:4318/v1/logs"
audit.otlp.headers		Object	Additional HTTP headers to include in OTLP requests
audit.otlp.timeoutMillis		Unknown	OTLP export timeout in milliseconds	5000
audit.otlp.concurrencyLimit		Unknown	Maximum number of concurrent OTLP export requests	1
audit.otlp.serviceName		String	Service name included on OTLP log attributes	"tsg-dsp-api"
audit.otlp.serviceVersion		String	Optional service version included on OTLP log attributes