Troubleshooting Zowe Application Framework
Troubleshooting Zowe Application Framework
The following topics contain information that can help you troubleshoot problems when you encounter unexpected behavior installing and using Zowe™ Application Framework which includes the Zowe Desktop.
Most of the solutions below identify issues by referring to the Zowe logs. To identify and resolve issues, you should be familiar with their names and locations.
The Zowe Application Framework manages issues in GitHub. When you troubleshoot a problem, you can check whether a GitHub issue (open or closed) that covers the problem already exists. For a list of issues, see the zlux repo.
#
Desktop apps fail to loadSymptom:
When you open apps in the Zowe desktop, a page is displayed with the message "The plugin failed to load."
Solution:
This problem might occur when you use Node.js v8.16.1, which performs auto-encoding in a way that breaks Zowe apps. See https://github.com/ibmruntimes/node/issues/142 for details.
To solve the problem, use a different version of Node.js v8, such as v8.17.0, or use Node.js v12. You can obtain them from the Node.js marketplace. Download the ibm-trial-node-v8.17.0-os390-s390x.pax.Z
file.
#
NODEJSAPP disables immediatelySymptom:
You receive the message CEE5207E The signal SIGABRT was received in stderr
.
Solution:
You might have reached the limit for shared message queues on your LPAR. When Node.js applications are terminated by a SIGKILL signal, shared message queues might not be deallocated. For more information, see the If the NODEJSAPP disables immediately section in the Troubleshooting Node.js applications topic on IBM Knowledge Center.
#
Cannot log in to the Zowe DesktopSymptom:
When you attempt to log in to the Zowe Desktop, you receive the following error message that is displayed beneath the Username and Password fields.
Authentication failed for 3 types: Types: ["saf","apiml","zss"]
The Zowe desktop attempts to authenticate the credentials using the types that have been configured, by default the three above of ["saf","apiml","zss"]
. If Zowe has been configured with the LAUNCH_COMPONENT_GROUPS=DESKTOP
where GATEWAY
is not a launch group, then the message will just include the types ["saf","zss"]
.
Solution:
This error may simply mean your password is not correct. If that's not the case, please continue on following troubleshooting guidance.
For the Zowe Desktop to work, the node server that runs under the ZWESVSTC started task must be able to make cross memory calls to the ZWESIS01 load module running under the ZWESISTC started task. If this communication fails, you see the authentication error.
There are three known problems that might cause this error. The Zowe architecture diagram shows the following connections. One of these three connections likely failed.
- The zssServer connection to the
ZWESISTC
started task using cross memory communication. If this fails, see zssServer unable to communicate with X-MEM. The architecture diagram below has been annotated with a (1) to show this connection. - The Zowe Desktop ZLUX server connection to the zssServer across the default port 8542. If this fails, see ZLUX unable to communicate with zssServer. The architecture diagram below has been annotated with a (2) to show this connection.
- The Zowe Desktop ZLUX server cannot connect to API Mediation Layer for authentication. If this fails, see ZLUX unable to communicate with API Mediation Layer.
#
ZSS server unable to communicate with X-MEMOpen the log file
$INSTANCE_DIR/logs/zssServer-yyyy-mm-dd-hh-ss.log
. This file is created each time ZWESVSTC is started and only the last five files are kept.Look for the message that starts with
ZIS status
.
If the communication works, the message includes
Ok
. For example:ZIS status - Ok (name='ZWESIS_STD ', cmsRC=0, description='Ok'
If the communication works, the problem is likely that the ZLUX server is unable to communicate to the zssServer. For more information, see ZLUX unable to communicate with zssServer.
If the communication is not working, the message includes
Failure
. For example:ZIS status - Failure (name='ZWESIS_STD ', cmsRC=39, description='Cross-memory call ABENDed'
or
ZIS status - Failure (name='ZWESIS_STD ', cmsRC=64, description='N/A', clientVersion=`2`)
or
ZIS status - 'Failure' (name='ZWESI_STD ', cmsRC='12', description='N/A', clientVersion='2')
In this case, check that the ZWESISTC started task is running. If not, start it with the TSO command
/S ZWESISTC
If the problem cannot be easily fixed (such as the ZWESISTC task not running), then it is likely that the cross memory server is not running. To check whether the cross memory is running, check the started task
ZWESISTC
log for any errors.If the cross memory server
ZWESISTC
started task is running, check that the program name of the cross memory procedure matches between theZWESISTC
PROBLIB member and theinstance.env
file used to launch Zowe.By default the proc value is
ZWESIS_STD
, and if a new name is chosen then both files need to be updated for the handshake to be successful.The line in the
ZWESISTC
problib that defines the procedure name that cross memory will use is//ZWESISTC PROC NAME='ZWESIS_STD',MEM=00,RGN=0M
The line in the
instance.env
that specifies the cross memory procedure that the zssServer will try to attach to isZOWE_ZSS_XMEM_SERVER_NAME=ZWESIS_STD
If this is the first time you set up Zowe, it is possible that the cross memory server configuration did not complete successfully. To set up and configure the cross memory server, follow steps as described in the topic Installing and configuring the Zowe cross memory server (ZWESISTC). Once
ZWESISTC
is started, if problems persist, check its log to ensure it has been able to correctly locate its load module ZWESIS01 as well as the parmlib ZWESIP00.If there is an authorization problem, the message might include
Permission Denied
. For example:ZIS status - Failure (name='ZWESIS_STD ', cmsRC=33, description='Permission denied'
Check that the user ID of the ZWESVSTC started task is authorized to access the load module. Only authorized code can call ZWESIS01 because it is an APF-authorized load module.
Note: If you are using RACF security manager, a common reason for seeing
Permission Denied
is that the user running the started task ZWESVSTC (typically ZWESVUSR) does not have READ access to the FACILITY class ZWES.IS.If the message includes the following text, the configuration of the Application Framework server may be incomplete:
ZIS status - Failure read failed ret code 1121 reason 0x76650446
If you are using AT/TLS, then the
"attls" : true
statement might be missing from thezluxserver.json
file. For more information, see Configuring Zowe App Server for HTTPS communication with ZSS.
#
ZLUX unable to communicate with zssServerFollow these steps:
Open the log file
$INSTANCE_DIR/logs/appServer-yyyy-mm-dd-hh-ss.log
. This file is created each time ZWESVSTC is started and only the last five files are kept.Look for the message that starts with
GetAddrInfoReqWrap.onlookup
and the log messages below.yyyy-mm-dd hh:mm:ss.ms <ZWED:16842977> ZWESVUSR INFO (_zsf.apiml,apiml.yyyy-mm-dd hh:mm:ss.ms <ZWED:16842977> ZWESVUSR INFO (_zsf.auth,webauthyyyy-mm-dd hh:mm:ss.ms <ZWED:16842977> ZWESVUSR WARN (_zsf.proxy,proxy. at GetAddrInfoReqWrap.onlookup Ýas oncomplete¨ (dns.js:64:26) { errno: 'ENOTFOUND', code: 'ENOTFOUND', syscall: 'getaddrinfo', hostname: 'localhost'
These messages show that the host name
localhost
cannot be reached between the Zowe desktop server and the zssServer becauselocalhost
has not been mapped to an IP address.Map localhost to port 127.0.0.1.
Create an entry in the file
/etc/hosts
that contains the line127.0.0.1 localhost
Restart the
ZWESVSTC
address space.
#
ZLUX unable to communicate with API Mediation LayerFollow these steps:
- Verify whether API Mediation Layer is started or not. If it is started, you can see a service status page with all green check marks by visiting
https://<your-zowe-host>:<gateway-port>
. If there are any red cross marks, follow the instructions in Troubleshooting API ML to identify and solve the issue. - You may need to wait a little longer to allow API Mediation Layer Gateway to complete the environment test. This could be more noticeable with certain z/OSMF level and Zowe v1.20.0 release and later.
#
Server startup problem ret=1115Symptom:
When ZWESVSTC is restarted, the following message is returned in the output of the ZSS server log file, $INSTANCE_DIR/logs/zssServer-yyyy-mm-dd-hh-ss.log
:
server startup problem ret=1115
Solution: This message means that some other process is already listening on port 7542, either at address 127.0.0.1 (localhost) or at 0.0.0.0 (all addresses). This prevents the ZSS server from starting.
One possibility is that a previously running ZSS server did not shut down correctly, and either the operating system has not released the socket after the ZSS server shut down, or the ZSS server is still running.
#
Application plug-in not in Zowe DesktopSymptom:
An application plug-in is not appearing in the Zowe Desktop.
Solution:
To check whether the plug-in loaded successfully, enter the following URL in a browser to display all successfully loaded Zowe plug-ins:
https://my.mainframe.com:8544/plugins?type=application
You can also search the node server logs for the plug-in identifier, for example org.zowe.sample.app
. If the plug-in loaded successfully, you will find the following message:
[2019-08-06 13:54:21.341 _zsf.bootstrap INFO] - Plugin org.zowe.sampleapp at path=zlux\org.zowe.sampleapp loaded.
If the plug-in did not load successfully, you will find the following message:
[2019-08-06 13:54:21.208 _zsf.bootstrap WARNING] - Error: org.zowe.sampleapp
If the identifier is not in the logs, make sure the plug-in's locator file is in the /zlux-app-server/deploy/instance/ZLUX/plugins/
directory. The plug-in locator is a .json
file, usually with same name as the identifier, for example org.zowe.sampleapp.json
. Open the file and make sure that the path that is defined with the pluginLocation
attribute is correct. If the path is relative, make sure it is relative to the zlux-app-server/bin
directory.
For more information on loading plug-ins to the Desktop, see Adding Your App to the Desktop.
#
Error: You must specify MVD_DESKTOP_DIR in your environmentSymptom:
A plug-in that is built in your local environment using npm run start
or npm run build
failed with an error message about a missing MVD_DESKTOP_DIR environment variable.
Solution:
Add the Zowe Desktop directory path to the MVD_DESKTOP_DIR
environment variable. To specify the path, run the following commands in your Windows console or Linux bash shell:
Windows
export MVD_DESKTOP_DIR=<zlux-root-dir>/zlux-app-manager/virtual-desktop
Mac Os/Linux
set MVD_DESKTOP_DIR=<zlux-root-dir>/zlux-app-manager/virtual-desktop
#
Error: Zowe Desktop address space fails to start { ZWED0115E }After launching the started task ZWESVSTC
there are no Zowe desktop ZWE1DS
address space(s).
Symptom: Check the log for the message
ZWED0115E - Unable to retrieve storage object from cluster. This is probably due to a timeout. You may change the default of '5000' ms by setting 'node.cluster.storageTimeout' within the config. Timeout call null/clusterManager/getStorageAll
The timeout value was increased to be 30000
in 1.11.0 release. To check which release of Zowe you are running, see Determining the Zowe release number. To further increase this, or update the value on a previous release you can add an entry to your $INSTANCE_DIR/instance.env
.
ZWED_node_cluster_storageTimeout=30000
where the timeout value is in milliseconds.
#
Warning: Problem making eureka request { Error: connect ECONNREFUSED }Symptom:
The Zowe started task ZWESVSTC
log contains error messages reporting problems connecting
Problem making eureka request { Error: connect ECONNREFUSED 10.1.1.2:7553at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1195:14)errno: 'ECONNREFUSED',code: 'ECONNREFUSED',syscall: 'connect',address: '10.1.1.2',port: 7553 }
Solution:
You can ignore these messages. These messages are timing-related where different Eureka servers come up, try to connect to each other, and warn that the endpoint they are trying to perform a handshake with is not available. When all of the Eurka services have started, these errors will stop being logged.
#
Warning: ZWED0159W - Plugin (org.zowe.zlux.proxy.zosmf) loading failed.Symptom:
The Zowe started task ZWESVSTC
log contains messages
ZWED0159W - Plugin (org.zowe.zlux.proxy.zosmf) loading failed. Message: "ZWED0047E - Proxy (org.zowe.zlux.proxy.zosmf:data) setup failed.Host & Port for proxy destination are required but were missing.
Solution:
You can ignore these messages which should not occur in 1.11 or later releases. To check which release of Zowe you are running, see Determining the Zowe release number.
#
Warning: ZWED0050W - Could not read swagger doc folder (..)Symptom:
The Zowe started task ZWESVSTC
log contains messages ending
ZWED0050W - Could not read swagger doc folder <RUNTIME_DIR>/components/app-server/share/zlux-workflow/doc/swaggerZWED0050W - Could not read swagger doc folder <RUNTIME_DIR>/components/app-server/share/zlux-app-manager/virtual-desktop/doc/swaggerZWED0050W - Could not read swagger doc folder <RUNTIME_DIR>/components/app-server/share/zlux-app-manager/bootstrap/doc/swaggerZWED0050W - Could not read swagger doc folder <RUNTIME_DIR>/components/app-server/share/zlux-server-framework/plugins/terminal-proxy/doc/swaggerZWED0050W - Could not read swagger doc folder <RUNTIME_DIR>/components/app-server/share/tn3270-ng2/doc/swagger
Solution:
You can ignore these messages.
#
Warning: ZWED0047W - Swagger file for server (...) not foundSymptom:
The Zowe started task ZWESVSTC
log contains messages ending
ZWED0047W - Swagger file for service (org.zowe.zosmf.workflows:zosmf) not foundZWED0047W - Swagger file for service (org.zowe.zlux.ng2desktop:browser-preferences) not foundZWED0047W - Swagger file for service (org.zowe.zlux.bootstrap:adminnotificationdata) not foundZWED0047W - Swagger file for service (org.zowe.terminal.proxy:tn3270data) not foundZWED0047W - Swagger file for service (org.zowe.terminal.tn3270:statediscovery) not found
Solution:
You can ignore these messages.
#
Unable to log in to the explorers when using Zowe V1.13 or V1.14Symptom:
You installed Zowe V1.13 or V1.14. When you start the Zowe server, you see the following error message in the appServer
log.
failed to process config TypeError: config.csp.frame-ancestorsÝ0¨.split is not a function
When you log in to the Zowe Desktop, you cannot open the JES, MVS, or USS Explorers. You receive the following error message:
{"messages":[{"messageType":"ERROR","messageNumber":"ZWEAG708E","messageContent":"The request to the URL '/ui/v1/explorer-uss/' has failed after retrying on all known service instances. Caused by: java.net.ConnectException: EDC8128I Connection refused. (errno2=0x74940000) (Connection refused)","messageKey":"org.zowe.apiml.gateway.connectionRefused"}]}
Solution:
A new property ZOWE_EXPLORER_FRAME_ANCESTORS
was introduced in V1.12. This property is required to be present in the instance.env
file with some valid value. When undefined, it is treated as Boolean, which breaks the string split function. To resolve the issue, define the value for this property in the instance.env
file.
#
Warning: Zowe extensions access to ZSS security endpoints failSymptom:
Zowe extensions fail when accessing the ZSS APIs such as the security-mgmt/classes/default-class/profiles
endpoint. The following error is written to the log.
BPXTLS failed: rc=-1, return code=163, reason code=0x0be80820
Solution:
Access to the ZSS endpoints are protected. To access the ZSS endpoints, the user must have READ
access on the OMVSAPPL
resource in the APPL
class.
To fix this permit access, issue the following TSO command, where userID
is the started task ID of the requesting process. The vendor documentation describes which userID to use which might be ZWESVUSR
.
PERMIT OMVSAPPL CLASS(APPL) ACCESS(READ) ID(userID)