Troubleshooting Kubernetes environments

The following topics contain information that can help you troubleshoot problems when you encounter unexpected behavior installing and using Zowe™ containers in a Kubernetes environment.

ISSUE: /tmp directory is not writable

Problem:

readOnlyRootFilesystem SecurityContext is enabled by default in Deployment object definition. As a result, /tmp is read-only and not writable to zowe runtime user.

Recommended solution:

Adjust your component to check the TMPDIR or TMP environment variable to determine the location of the temporary directory. Zowe runtime customizes those variables and points them to /home/zowe/instance/tmp directory, which is writable.

Alternative solution:

Disabling readOnlyRootFilesystem SecurityContext is not recommended. But you can make /tmp writable by replacing it with a newly mounted volume. Here is an example of defining the /tmp volume.

apiVersion: apps/v1kind: Deploymentspec:  template:    spec:      volumes:        - name: tmp          emptyDir: {}        containers:          - name: <my-component-name>            volumeMounts:              - name: tmp                mountPath: "/tmp"

With this added to your Deployment, your component should be able to write to /tmp directory.

ISSUE: Permission denied showing in pod log

Problem:

You see error messages similar to the following one in your pod log.

cp: cannot create regular file '/home/zowe/instance/workspace/manifest.json': Permission deniedmkdir: cannot create directory '/home/zowe/instance/workspace/api-mediation': Permission deniedmkdir: cannot create directory '/home/zowe/instance/workspace/backups': Permission deniedcp: cannot create regular file '/home/zowe/instance/workspace/active_configuration.cfg': Permission denied/home/zowe/runtime/bin/internal/prepare-instance.sh: line 236: /home/zowe/instance/workspace/active_configuration.cfg: Permission denied/home/zowe/runtime/bin/internal/prepare-instance.sh: line 240: /home/zowe/instance/workspace/active_configuration.cfg: Permission denied/home/zowe/runtime/bin/internal/prepare-instance.sh: line 241: /home/zowe/instance/workspace/active_configuration.cfg: Permission denied

It means zowe user (UID 20000) does not have write permission to your persistent volume. It's very likely the persistent volume is mounted as root user.

Solution:

To solve this issue, you can modify workload files with extra initContainers step like this:

apiVersion: apps/v1kind: Deploymentspec:  template:    spec:      initContainers:        - name: update-workspace-permission          image: busybox:1.28          command: ['sh', '-c', 'OWNER=`stat -c "%u:%g" /home/zowe/instance/workspace` && PERMISSION=`stat -c "%a" /home/zowe/instance/workspace` && echo "Zowe workspace owner is ${OWNER} with ${PERMISSION} permission" && if [ "${OWNER}" != "20000:20000" -a "${PERMISSION}" != "777" ]; then chown -R 20000:20000 /home/zowe/instance/workspace; fi']          imagePullPolicy: Always          resources:            requests:              memory: "64Mi"              cpu: "10m"            limits:              memory: "128Mi"              cpu: "100m"          volumeMounts:            - name: zowe-workspace              mountPath: "/home/zowe/instance/workspace"          securityContext:            readOnlyRootFilesystem: true            allowPrivilegeEscalation: false            capabilities:              drop:                - all              add:                - CHOWN            runAsUser: 0            runAsGroup: 0

ISSUE: Deployment and ReplicaSet failed to create pod

Problem:

If you are using OpenShift and see these error messages in ReplicaSet Events:

Generated from replicaset-controllerError creating: pods "api-catalog-??????????-" is forbidden: unable to validate against any security context constraint: []

That means the Zowe ServiceAccount zowe-sa doesn't have any SecurityContextConstraint attached.

Solution:

You can run this command to grant a certain level of permission, for example, privileged, to zowe-sa ServiceAccount:

oc admin policy add-scc-to-user privileged -z zowe-sa -n zowe

ISSUE: Failed to create services

Problem:

If you are using OpenShift and apply services, you may see this error:

The Service "api-catalog-service" is invalid: spec.ports[0].appProtocol: Forbidden: This field can be enabled with the ServiceAppProtocol feature gate

Solution:

To fix this issue, you can simply find and comment out this line in the Service definition files:

appProtocol: https

With OpenShift, you can define a PassThrough Route to let Zowe handle TLS connections.