Configuring
Configuring
Zowe provides sample configurations that make it easy for you to run Zowe in Kubernetes. You can use them directly or as a reference.
You can customize the configuration or make your own. If you do so, note the following objects that are expected by the container deployments:
Kind | Name | Note |
---|---|---|
Namespace | zowe | |
ServiceAccount | zowe-sa | |
ConfigMap | zowe-certificates-cm | Contains zowe-certificates.env with the same format as seen on z/OS keystore |
Secret | zowe-certificates-secret | Contains the base64 PEM and P12 data for keystore and truststore |
Ingress | discovery-ingress | Used for external access to the Discovery service |
Ingress | gateway-ingress | Used for external access to the Gateway service |
Route | discovery | Used for external access to the Discovery service |
Route | gateway | Used for external access to the Gateway service |
Service | discovery-service | Used for internal or external access to the Discovery service |
Service | gateway-service | Used for external access to the Gateway service |
Service | catalog-service | Used for access to the Catalog service |
PersistentVolumeClaim | zowe-workspace-pvc | |
HorizontalPodAutoscaler | * | Autoscalers exist for the various pods |
PodDisruptionBudget | * | Disruption budgets exist for the various pods |
To configure the Zowe container environment, complete the following procedure.
#
1. Create namespace and service accountRun the following commands to create Zowe's Namespace zowe
with Service Account zowe-sa
.
kubectl apply -f common/zowe-ns.yamlkubectl apply -f common/zowe-sa.yaml
Note that by default, zowe-sa
service account has automountServiceAccountToken
disabled for security purposes.
To verify, check the following configurations.
kubectl get namespaces
should show a Namespacezowe
.This displays the default Namespace zowe, if not set.
kubectl get serviceaccounts --namespace zowe
should show a ServiceAccountzowe-sa
.This displays the default ServiceAccount zowe-sa, if not set.
#
2. Create Persistent Volume Claim (PVC)Zowe's PVC has a default StorageClass value that may not apply to all Kubernetes clusters. Check and customize the storageClassName
value of samples/workspace-pvc.yaml
as needed. You can use kubectl get sc
to confirm which StorageClass you can use.
After you customize the storageClassName
value, apply the result by issuing the following commands:
kubectl apply -f samples/workspace-pvc.yaml
To verify, run the following commands and check if the STATUS
of line item zowe-workspace-pvc
shows as Bound
.
kubectl get pvc --namespace zowe
IMPORTANT, zowe-workspace-pvc
PersistentVolumeClaim
must be declared in access mode ReadWriteMany
to allow the workspace be shared by all Zowe components.
In some Kubernetes environment, you may need to define PeristentVolume
and define volumeName
in PersistentVolumeClaim
instead of defining storageClassName
. Please consult your Kubernetes administrator to confirm the appropriate way for your environment. This is an example to configure PersistentVolumeClaim
with pre-configured zowe-workspace-pv
PeristentVolume
.
apiVersion: v1kind: PersistentVolumeClaimmetadata: name: zowe-workspace-pvc namespace: zowe labels: app.kubernetes.io/name: zowe app.kubernetes.io/instance: zowe app.kubernetes.io/managed-by: manualspec: storageClassName: "" volumeName: zowe-workspace-pv accessModes: - ReadWriteMany resources: requests: storage: 50Mi
#
3. Create and modify ConfigMaps and SecretsSimilarly, to run Zowe services on z/OS, you can use the Zowe zowe.yaml
configuration file to customize Zowe in Kubernetes.
You can modify samples/config-cm.yaml
and samples/certificates-secret.yaml
directly. Or more conveniently, if you have Zowe ZSS/ZIS running on z/OS, the Kubernetes environment can reuse instance and keystore configuration from that installation. Ensure that the verify certificate setting of your existing keystore configuration is set to STRICT
mode. Otherwise, update your zowe.yaml
configuration file to change the setting to STRICT
mode and generate a new set of certificates.
If you want to manually create, or later customize the ConfigMaps and Secrets, see Customizing or manually creating ConfigMaps and Secrets for details.
To create and modify ConfigMaps and Secrets by using the migrate configuration script, complete the following steps:
a. To make Zowe v2 certificates work in Kubernetes, in your zowe.yaml
(in runtime directory), you need to:
set
zowe.verifyCertificate
toSTRICT
mode.set
zowe.setup.certificate.pkcs12.caAlias
. Default alias islocal_ca
.set
zowe.setup.certificate.pkcs12.caPassword
. Default CA password islocal_ca_password
.make sure the certificate that you are using have defined the following domains in certificate Subject Alt Name (SAN):
- your external domains to access Zowe APIML Gateway Service running in Kubernetes cluster
*.<k8s-namespace>.svc.<k8s-cluster-name>
*.discovery-service.<k8s-namespace>.svc.<k8s-cluster-name>
*.gateway-service.<k8s-namespace>.svc.<k8s-cluster-name>
*.<k8s-namespace>.pod.<k8s-cluster-name>
where,
<k8s-namespace>
is the Kubernetes Namespace you installed Zowe into<k8s-cluster-name>
is the Kubernetes cluster name, which usually should becluster.local
. Note that the following command will automatically add the k8s internal domain into SAN.
Next, on z/OS, run the following command:
cd <runtime-dir> ./bin/zwe migrate for kubernetes --config /path/to/my/zowe.yaml --domains "my-k8s-cluster.company.com"
For more detailed explaination of zwe migrate command parameters, see zwe migrate for kubernetes.
As a result, it displays ConfigMaps zowe-config
and Secrets (zowe-certificates-secret
) Kubernetes objects which are based on the Zowe instance and keystore used. The content looks similar to samples/config-cm.yaml
and samples/certificates-secret.yaml
but with real values.
b. Follow the instructions in the script output to copy the output and save it as a YAML file configs.yaml
on your computer where you manage Kubernetes.
c. Apply the file into Kubernetes:
`kubectl apply -f /path/to/your/configs.yaml`
d. Remove the previously saved configs.yaml
file from all systems for security.
To verify:
kubectl get configmaps --namespace zowe
This command must display the two ConfigMaps
zowe-config
andzowe-certificates-cm
.kubectl get secrets --namespace zowe
This command must display a Secret
zowe-certificates-secret
.
#
4. Expose API Mediation Layer componentsThis step makes Zowe's Gateway, Discovery, and API Catalog servers available over a network.
The Gateway is always required to be externally accessible, and depending upon your environment the Discovery service may also need to be externally accessible.
The actions you need to take in this step vary depending upon your Kubernetes cluster configuration. If you are uncertain about this section, please contact your Kubernetes administrator or the Zowe community.
#
4a. Create serviceYou can set up either a LoadBalancer
or NodePort
type Service.
Note: Because NodePort
cannot be used together with NetworkPolicies
, LoadBalancer
and Ingress
is preferred configuration option.
Review the following table for steps you may take depending on the Kubernetes provider you use. If you don't need additional setups, you can skip steps 4b, 4c and jump directly to the Apply zowe section.
Kubernetes provider | Service | Additional setups required |
---|---|---|
minikube | LoadBalancer or NodePort | Port Forward (on next section Starting, stopping, and monitoring) |
docker-desktop | LoadBalancer | none |
bare-metal | LoadBalancer or NodePort | Create Ingress |
cloud-vendors | LoadBalancer | none |
OpenShift | LoadBalancer or NodePort | Create Route |
#
Defining api-catalog serviceapi-catalog-service
is required by Zowe, but not necessarily exposed to external users. Therefore, api-catalog-service
is defined as type ClusterIP
.
To define this service, run the command:
kubectl apply -f samples/api-catalog-service.yaml
Upon success, you should see the following output:
service/api-catalog-service created
Then, you can proceed with creating the Gateway and Discovery services according to your environment.
#
Applying Gateway ServiceIf using LoadBalancer
, run the command:
kubectl apply -f samples/gateway-service-lb.yaml
Or if using NodePort
instead, first check spec.ports[0].nodePort
as this will be the port to be exposed to external. In this case, the default gateway port is not 7554 but 32554. You will need to use https://<your-k8s-node>:32554/
to access APIML Gateway. To apply NodePort
type gateway-service
, run the following command:
kubectl apply -f samples/gateway-service-np.yaml
To verify either case, run the following command and check that the command displays the service gateway-service
.
kubectl get services --namespace zowe
#
Applying Discovery serviceExposing the Discovery service is only required when there is a Zowe service or extension which needs to be registered to the API Mediation Layer but is running outside of Kubernetes, such as on z/OS. Otherwise, the discovery service can remain accessible only within the Kubernetes environment.
Optional: To set up the discovery service without exposing it externally, edit samples/discovery-service-lb.yaml
if using LoadBalancer
type services, or samples/discovery-service-np.yaml
if using NodePort
type services. In either file, specify ClusterIP
as the type, replacing the NodePort
or LoadBalancer
value.
To enable the service externally when using LoadBalancer
services, run the command:
kubectl apply -f samples/discovery-service-lb.yaml
Or if using NodePort
instead, first check spec.ports[0].nodePort
as this will be the port to be exposed to external. In this case, the default discovery port is not 7553 but 32553. And you will need to use https://<your-k8s-node>:32553/
to access APIML Discovery. To apply NodePort
type discovery-service
, run the following command:
kubectl apply -f samples/discovery-service-np.yaml
To verify either case, run the following command and check that this command displays the service discovery-service
:
kubectl get services --namespace zowe
Upon completion of all the preceding steps in this a. Create service section, you may need to run additional setups. Refer to "Additional setups required" in the table. If you don't need additional setups, you can skip 4b, 4c, 4d, and jump directly to Apply Zowe section.
#
4b. Create Ingress (Bare-metal)An Ingress gives Services externally-reachable URLs and may provide other abilities such as traffic load balancing.
To create Ingress, perform the following steps:
a. Edit samples/gateway-ingress.yaml
and samples/discovery-ingress.yaml
before applying them, by uncommenting the lines (19 and 20) for defining spec.rules[0].host
and http:
, and then commenting out the line below, - http:
b. Run the following commands:
kubectl apply -f samples/bare-metal/gateway-ingress.yamlkubectl apply -f samples/bare-metal/discovery-ingress.yaml
To verify, run the following commands:
kubectl get ingresses --namespace zowe
This command must display two Ingresses gateway-ingress
and discovery-ingress
.
Upon completion, you can finish the setup by applying zowe and starting it.
#
4c. Create Route (OpenShift)If you are using OpenShift and choose to use LoadBalancer
services, you may already have an external IP for the service. You can use that external IP to access Zowe APIML Gateway. To verify your service external IP, run:
oc get svc -n zowe
If you see an IP in the EXTERNAL-IP
column, that means your OpenShift is properly configured and can provision external IP for you. If you see <pending>
and it does not change after waiting for a while, that means you may not be able to use LoadBalancer
services with your current configuration. Try ClusterIP
services and define Route
. A Route is a way to expose a service by giving it an externally reachable hostname.
To create a route, perform the following steps:
a. Check and set the value of spec.port.targetPort
in samples/gateway-route.yaml
and samples/discovery-route.yaml
before applying the changes.
b. Run the following commands:
oc apply -f samples/openshift/gateway-route.yamloc apply -f samples/openshift/discovery-route.yaml
To verify, run the following commands:
oc get routes --namespace zowe
This command must display the two Services gateway
and discovery
.
Upon completion, you can finish the setup by applying zowe and starting it.
#
Customizing or manually creating ConfigMaps and SecretsThe z/OS to k8s convert tool can automatically create a config map and secret. However, if you want to customize or create your own, review the instructions in this section.
To make certificates work in Kubernetes, make sure the certificate you are using have defined the following domains in certificate Subject Alt Name (SAN):
- your external domains to access Zowe APIML Gateway Service running in Kubernetes cluster
*.<k8s-namespace>.svc.<k8s-cluster-name>
*.discovery-service.<k8s-namespace>.svc.<k8s-cluster-name>
*.gateway-service.<k8s-namespace>.svc.<k8s-cluster-name>
*.<k8s-namespace>.pod.<k8s-cluster-name>
<k8s-namespace>
is the Kubernetes Namespace you installed Zowe into. And <k8s-cluster-name>
is the Kubernetes cluster name, which usually should be cluster.local
.
Without the additional domains in SAN, you may see warnings/errors related to certificate validation.
caution
It's not recommended to disable zowe.verifyCertificates
.
Notes: When the following conditions are true, this migration script will regenerate a new set of certificates for you with proper domain names listed above.
- You use
zwe init
command to initialize Zowe - You use
PKCS#12
format keystore by definingzowe.setup.certificate.type: PKCS12
- You did not define
zowe.setup.certificate.pkcs12.import.keystore
and letzwe
command to generate PKCS12 keystore for you - You enabled
STRICT
modezowe.verifyCertificates
To manually create the ConfigMaps and Secrets used by Zowe containers, you must create the following objects:
A ConfigMap, with values based upon a Zowe configuration
zowe.yaml
and similar to the examplesamples/config-cm.yaml
with the following differences to the values seen on a z/OS installation:zowe.setup
andhaInstances
are not needed for Zowe running in Kubernetes and will be ignored. You can remove them.java.home
andnode.home
are not usually needed if you are using Zowe base images.zowe.runtimeDirectory
must be set to/home/zowe/runtime
.zowe.externalDomains
is suggested to define as a list of domains you are using to access your Kubernetes cluster.zowe.externalPort
must be the port you expose to end-user. This value is optional if it's same as default APIML Gateway service port7554
. With default settings,- if you choose
LoadBalancer
gateway-service
, this value is optional, or set to7554
, - if you choose
NodePort
gateway-service
and access the service directly, this value should be same asspec.ports[0].nodePort
with default value32554
, - if you choose
NodePort
gateway-service
and access the service through port forwarding, the value should be the forwarded port you set.
- if you choose
components.discovery.replicas
should be set to same value ofspec.replicas
defined inworkloads/discovery-statefulset.yaml
.- All components running in Kubernetes should use default ports:
components.api-catalog.port
is7552
,components.discovery.port
is7553
,components.gateway.port
is7554
,components.caching-service.port
is7555
,components.jobs-api.port
is7600
,components.files-api.port
is7559
,components.app-server.port
is7556
.
components.caching-service.storage.mode
should NOT be set toVSAM
.redis
is suggested. Follow Redis configuration documentation to customize other Redis related variables. Leave the value to empty for debugging purposes.- Must append and customize these 2 values into
zowe.environments
section:ZWED_agent_host=<ZOWE_ZOS_HOST>
ZWED_agent_https_port=<ZOWE_ZSS_SERVER_PORT>
A Secret, with values based upon a Zowe keystore's files, and similar to the example
samples/certificates-secret.yaml
.You need 2 entries under the
data
section:keystore.p12
: which is base64 encoded PKCS#12 keystore,truststore.p12
: which is base64 encoded PKCS#12 truststore.
And 3 entries under
stringData
section:keystore.key
: is the PEM format of certificate private key,keystore.cer
: is the PEM format of the certificate,ca.cer
: is the PEM format of the certificate authority.
PodDisruptionBudget
#
Zowe provides optional PodDisruptionBudget
which can provide high availability during upgrade. By default, Zowe defines minAvailable
to be 1
for all deployments. This configuration is optional but recommended. To apply PodDisruptionBudget
, run this command:
kubectl apply -f samples/pod-disruption-budget/
To verify this step, run:
kubectl get pdb --namespace zowe
This should show you a list of PodDisruptionBudget
like this:
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGEapi-catalog-pdb 1 N/A 0 1dapp-server-pdb 1 N/A 0 1dcaching-pdb 1 N/A 0 1ddiscovery-pdb 1 N/A 0 1dexplorer-jes-pdb 1 N/A 0 1dexplorer-mvs-pdb 1 N/A 0 1dexplorer-uss-pdb 1 N/A 0 1dfiles-api-pdb 1 N/A 0 1dgateway-pdb 1 N/A 0 1djobs-api-pdb 1 N/A 0 1d
HorizontalPodAutoscaler
#
Zowe provides optional HorizontalPodAutoscaler
which can automatically scale Zowe components based on resource usage. By default, each workload has a minimum of 1 replica and a maximum of 3 to 5 replicas based on CPU usage. This configuration is optional but recommended. HorizontalPodAutoscaler
relies on Kubernetes Metrics server monitoring to provide metrics through the Metrics API. To learn how to deploy the metrics-server, see the metrics-server documentation. Please adjust the HorizontalPodAutoscaler
definitions based on your cluster resources, then run this command to apply them to your cluster:
kubectl apply -f samples/horizontal-pod-autoscaler/
To verify this step, run:
kubectl get hpa --namespace zowe
This should show you a list of HorizontalPodAutoscaler
like this:
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGEapi-catalog-hpa Deployment/api-catalog 60%/70% 1 3 1 20mapp-server-hpa Deployment/app-server 2%/70% 1 5 1 9m59scaching-hpa Deployment/caching 7%/70% 1 3 1 9m59sdiscovery-hpa StatefulSet/discovery 34%/70% 1 3 1 8m15sexplorer-jes-hpa Deployment/explorer-jes 10%/70% 1 3 1 9m59sexplorer-mvs-hpa Deployment/explorer-mvs 10%/70% 1 3 1 9m59sexplorer-uss-hpa Deployment/explorer-uss 10%/70% 1 3 1 9m59sfiles-api-hpa Deployment/files-api 8%/70% 1 3 1 9m59sgateway-hpa Deployment/gateway 20%/60% 1 5 1 9m59sjobs-api-hpa Deployment/jobs-api 8%/70% 1 3 1 9m59s
#
Kubernetes v1.21+If you have Kubernetes v1.21+, several optional changes are recommended based on Deprecated API Migration Guide.
- Kind
CronJob
: changeapiVersion: batch/v1beta1
toapiVersion: batch/v1
onworkloads/zowe-yaml/cleanup-static-definitions-cronjob.yaml
andworkloads/instance-env/cleanup-static-definitions-cronjob.yaml
.apiVersion: batch/v1beta1
will stop working on Kubernetes v1.25. - Kind
PodDisruptionBudget
: changeapiVersion: policy/v1beta1
toapiVersion: policy/v1
on all files insamples/pod-disruption-budget/
.apiVersion: policy/v1beta1
will stop working on Kubernetes v1.25.