Skip to main content
Version: v0.20.0

TSG Troubleshooting Guide

This guide helps you diagnose and resolve common issues when deploying and operating TSG (TNO Security Gateway).

General Troubleshooting Steps​

1. Check System Status​

Verify Kubernetes cluster health:

kubectl cluster-info
kubectl get nodes
kubectl get pods --all-namespaces

Check TSG deployment status:

# List all TSG-related pods
kubectl get pods -n tsg-ecosystem

# Check specific namespace
kubectl get all -n tsg-ecosystem

Verify ingress configuration:

kubectl get ingress -A
kubectl describe ingress <ingress-name> -n <namespace>

2. Check Logs​

Application logs:

# Control Plane logs
kubectl logs -l app.kubernetes.io/name=tsg-control-plane -n tsg-ecosystem --tail=100

# Data Plane logs  
kubectl logs -l app.kubernetes.io/name=tsg-http-data-plane-plane -n tsg-ecosystem --tail=100

# Wallet logs
kubectl logs -l app.kubernetes.io/name=tsg-wallet -n tsg-ecosystem --tail=100

System logs:

# Ingress controller logs
kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx

# cert-manager logs
kubectl logs -n cert-manager -l app=cert-manager

CLI Tool Issues​

Installation Problems​

Error: npm install -g @tsg-dsp/cli fails

Solution:

# Clear npm cache
npm cache clean --force

# Update npm
npm install -g npm@latest

# Install with specific Node.js version
nvm use 22
npm install -g @tsg-dsp/cli@latest

Error: tsg: command not found

Solution:

# Check npm global path
npm config get prefix

# Add to PATH (add to ~/.bashrc or ~/.zshrc)
export PATH="$(npm config get prefix)/bin:$PATH"

# Verify installation
which tsg
tsg --version

Configuration Validation Errors​

The CLI validates the whole document before rendering anything, and names the field that failed. Unknown fields are a hard error, so a typo is reported rather than silently ignored.

Error: property ... should not exist

A field name is misspelled, or belongs to an older schema. Check the Configuration Reference, and if the file predates your CLI:

tsg migrate participant.yaml        # prints the migrated document
tsg migrate participant.yaml -w     # rewrites it in place

Error: Unsupported apiVersion

The file was written for a newer CLI than the one installed. tsg --version lists the versions yours serves.

npm install -g @tsg-dsp/cli@latest

Error: host must be a DNS name without a scheme or path

host: dataspace.example.com          # correct
# NOT: https://dataspace.example.com/

Deployment Issues​

Render Failures​

Error: Refusing to replace non-empty directory

tsg render replaces its output directory wholesale, so it refuses any directory that is not already a bundle. Point -o somewhere else, or remove the directory yourself.

Error: matched no resources

A patch target no longer matches anything, usually after a rename. Render without the patch and grep the output for the name you meant, or set optional: true if matching nothing is intended.

Error: Missing public JWK

With private_key_jwt the public half of each client key is a render input, because rendering never reads from a cluster:

tsg keys generate <stack> <client>

Deploy Command Failures​

Error: kubectl: connection refused

Solution:

# Verify kubectl configuration
kubectl config current-context
kubectl cluster-info

# Check cluster connectivity
kubectl get nodes

# Switch context if needed
kubectl config use-context <correct-context>

Error: Unmet cluster prerequisites

The bundle renders something the cluster cannot accept. The message names the CRD and prints the command that installs it, most often CloudNativePG:

helm repo add cnpg https://cloudnative-pg.github.io/charts
helm upgrade --install cnpg cnpg/cloudnative-pg \
  --namespace cnpg-system --create-namespace

These are cluster-admin operations; TSG never performs them.

Error: Missing referenced Secrets

Rendered manifests only reference credentials. The preflight lists every Secret and key it could not find, before anything is applied:

tsg secrets generate participant.yaml --apply   # demo deployments only

Error: has no inventory, but these objects already exist and are not ours

Something with the same names is already in the namespace without TSG's ownership labels. Remove it, or deploy into a different namespace; the CLI will not adopt objects it cannot prove are its own.

Error: Readiness check failed

The apply succeeded but a component did not become ready. Nothing was pruned and the inventory still records both the previous and the desired objects, so fix the cause and re-run — no reset is needed.

kubectl logs -n tsg-ecosystem deploy/<name>
kubectl describe pod -n tsg-ecosystem -l app.kubernetes.io/name=<name>

Error: Inventory for set ... changed while this run was in progress

Another deployment of the same stack is running, or a previous one finished after this one started. Nothing was pruned; re-run once it has finished.

Network and Connectivity Issues​

Ingress Problems​

Error: 502 Bad Gateway or 503 Service Unavailable

Diagnosis:

# Check ingress controller
kubectl get pods -n ingress-nginx
kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx

# Check backend services
kubectl get svc -n tsg-ecosystem
kubectl get endpoints -n tsg-ecosystem

Solution:

# Verify service endpoints
kubectl describe service control-plane -n tsg-ecosystem

# Check pod readiness
kubectl get pods -n tsg-ecosystem -o wide

# Restart ingress controller if needed
kubectl rollout restart deployment/ingress-nginx-controller -n ingress-nginx

Error: 404 Not Found for TSG endpoints

Solution:

# Verify ingress configuration
kubectl describe ingress -n tsg-ecosystem

# Check ingress annotations
kubectl get ingress -n tsg-ecosystem -o yaml

# Verify DNS resolution
nslookup dataspace.example.com

TLS Certificate Issues​

Error: TLS handshake failed

Diagnosis:

# Check certificate status
kubectl get certificates -A
kubectl describe certificate <cert-name> -n <namespace>

# Check cert-manager logs
kubectl logs -n cert-manager -l app=cert-manager

Solution:

# Delete and recreate certificate
kubectl delete certificate <cert-name> -n <namespace>
tsg deploy ecosystem.yaml  # Will recreate certificate

# Check certificate issuer
kubectl describe clusterissuer letsencrypt-prod

Error: Certificate not ready

Solution:

# Wait for certificate provisioning (can take 5-10 minutes)
kubectl get certificate -n tsg-ecosystem -w

# Check ACME challenge
kubectl get challenges -A
kubectl describe challenge <challenge-name>

# Verify DNS is pointing to ingress
dig dataspace.example.com

Application-Specific Issues​

Control Plane Problems​

Error: Database connection failed

Solution:

# Check database pod
kubectl get pods -l app.kubernetes.io/name=postgresql
kubectl logs -l app.kubernetes.io/name=postgresql

# Verify database credentials
kubectl get secrets -n tsg-ecosystem
kubectl describe secret control-plane-db

Data Plane Problems​

Error: Transfer request failed

Solution:

# Check data plane logs
kubectl logs -l app.kubernetes.io/name=tsg-http-data-plane-plane -A

# Verify data plane registration
kubectl exec -it <control-plane-pod> -- curl http://http-data-plane:8080/health

# Check data source connectivity
kubectl exec -it <data-plane-pod> -- curl <your-data-source-url>

Wallet Problems​

Error: DID resolution failed

Solution:

# Check wallet logs
kubectl logs -l app=wallet -n tsg-ecosystem

# Verify DID document accessibility
curl https://wallet.dataspace.example.com/.well-known/did.json

# Check ingress for wallet
kubectl describe ingress wallet -n tsg-ecosystem

Error: Credential verification failed

Solution:

# Check wallet configuration
kubectl describe configmap wallet-config -n tsg-ecosystem

# Verify key storage
kubectl get secrets -l app=wallet -n tsg-ecosystem

# Test credential endpoint
curl https://wallet.dataspace.example.com/credentials

SSO Bridge Problems​

Error: OAuth token invalid

Solution:

# Check SSO Bridge logs
kubectl logs -l app=sso-bridge -n tsg-ecosystem

# Verify OAuth client configuration
kubectl describe configmap sso-bridge-clients -n tsg-ecosystem

# Test OAuth endpoints
curl https://auth.dataspace.example.com/.well-known/openid-configuration

Performance Issues​

High Resource Usage​

CPU/Memory limits reached:

Solution:

# Check resource usage
kubectl top pods -n tsg-ecosystem
kubectl top nodes

Raise the limits on the component in question, in the deployment configuration:

components:
  controlPlane:
    enabled: true
    resources:
      requests: { memory: 1Gi, cpu: 500m }
      limits: { memory: 2Gi, cpu: 1000m }
tsg deploy ecosystem.yaml

The container's --max-old-space-size is derived from the memory limit, so the Node heap follows it without a second setting.

Slow Response Times​

Network latency issues:

Solution:

# Check pod-to-pod connectivity
kubectl exec -it <pod1> -- ping <pod2-ip>

# Verify service discovery
kubectl exec -it <pod> -- nslookup control-plane.tsg-ecosystem.svc.cluster.local

# Check ingress performance
kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx | grep -i latency

Recovery Procedures​

A failed tsg deploy is safe to re-run: the inventory records both the previous and the desired objects until apply and readiness both succeed, so nothing is orphaned and no reset is needed.

See what the cluster believes​

tsg status ecosystem.yaml     # inventory sets, phase, object counts
tsg diff ecosystem.yaml       # what a deploy would change, and prune

A set stuck in the applying phase means a previous run did not reach readiness. Fix the cause and deploy again.

Redeploy one stack​

tsg deploy ecosystem.yaml -s alfa
kubectl rollout restart deployment/alfa-tsg-control-plane -n tsg-ecosystem

A stack-selected run never reconciles, so it cannot remove the stacks it did not render.

Start a stack over, keeping its data​

tsg uninstall ecosystem.yaml -s alfa   # retains databases and volumes
tsg deploy ecosystem.yaml -s alfa

Add --delete-data only when the databases and volumes should go too. Deleting the namespace by hand is the one thing to avoid: it cascades over every other stack in it, including data uninstall would have retained.

Getting Additional Help​

Diagnostic Information Collection​

Before requesting support, collect diagnostic information:

# Create diagnostic bundle
mkdir tsg-diagnostics
cd tsg-diagnostics

# Collect logs
kubectl logs -l app.kubernetes.io/managed-by=tsg-cli --all-containers=true > tsg-logs.txt

# Collect resource status
kubectl get all -n tsg-ecosystem -o yaml > tsg-resources.yaml

# Collect events
kubectl get events -n tsg-ecosystem --sort-by=.metadata.creationTimestamp > tsg-events.txt

# Collect configuration and the rendered manifests
cp ../ecosystem.yaml ./
tsg render ../ecosystem.yaml -o ./rendered

# Collect the inventory the cluster records
kubectl get configmap -n tsg-ecosystem -l app.kubernetes.io/managed-by=tsg-cli -o yaml > tsg-inventory.yaml

# Create archive
cd ..
tar -czf tsg-diagnostics.tar.gz tsg-diagnostics/

Support Channels​

Useful External Resources​