OEM 13c - emctl start oms failing with error - java.io.IOException: Cannot run program "null/common/bin/wlst.sh": error=2, No such file or directory

After the abnormal server reboot, we are unable to startup the OEM services and encounter the below error.

Error:

[oracle@krishna bin]$ ./emctl start oms
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation. All rights reserved.
[oracle@krishna bin]$ ./emctl status oms
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation. All rights reserved.
WebTier is Down
[oracle@krishna bin]$ ./emctl stop oms -all -force
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation. All rights reserved.
Stopping Oracle Management Server...
Error Occurred: Cannot run program "null/common/bin/wlst.sh": error=2, No such file or directory
Please check /u01/oracle/product/13.5.0/em/gc_inst/em/EMGC_OMS1/sysman/log/emctl.log for error details
[oracle@krishna bin]$

Solution:

OPTION1:

Restore the file emgc.properties file from valid backup.

OPTION2:

1. A typical emgc.properties in EM13c would have sample entries such as below:

  

COMMON_ORACLE_HOME=<OMS_HOME>/oracle_common

oracle.sysman.emSDK.svlt.ConsoleServerName=<OMS_HOST>\:<PORT>_Management_Service

JBO_POOL_TTL=-1

AS_PORT=0

EM_UPLOAD_HTTP_PORT=<EM_UPLOAD_HTTP_PORT>

EM_NODEMGR_HOME=<GC_INST>/user_projects/domains/GCDomain/nodemanager

MSPORT=<MSPORT>

ADMIN_SERVER_NAME=EMGC_ADMINSERVER

AS_HTTPS_PORT=7102

EM_REPOS_CONNECTDESCRIPTOR=(DESCRIPTION\=(ADDRESS_LIST\=(ADDRESS\=(PROTOCOL\=TCP)(HOST\=<REPO_DB_HOSTNAME>)(PORT\=<REPO_DB_LSNR_PORT>)))(CONNECT_DATA\=(SID\=<REPO_DB_SID>)))

OHS_ADMIN_PORT=<OHS_ADMIN_PORT>

OMS_HEAP_MAX=1740M

EM_CONSOLE_HTTP_PORT=<EM_CONSOLE_HTTP_PORT>

WLS_HOME=<OMS_HOME>/wlserver

JBO_LAZY_LOAD=true

EM_UPLOAD_HTTPS_PORT=<EM_UPLOAD_HTTPS_PORT>

oracle.sysman.emRep.repositoryMode=repository

AS_HOST=<OMS_HOSTNAME>

OMS_PERMGEN_MAX=768M

EM_DOMAIN_HOME=<GC_INST>/user_projects/domains/GCDomain

EM_CONSOLE_HTTPS_PORT=<EM_CONSOLE_HTTPS_PORT>

CURRENT_OMS_MODE=default

EM_NODEMGR_PORT=<EM_NODEMGR_PORT>

EM_DOMAIN_NAME=GCDomain

MS_HTTPS_PORT=<MS_HTTPS_PORT>

EM_INSTANCE_HOST=<OMS_HOSTNAME>

AS_COHERENCE_PORT=<AS_COHERENCE_PORT>

WEBTIER_ORACLE_HOME=<OMS_HOME>/ohs

MW_HOME=<OMS_HOME>

JBO_MIN_POOL_SIZE=1

EM_INSTANCE_HOME=<GC_INST>/em/EMGC_OMSn

JBO_MAX_POOL_SIZE=50

IS_ADMIN_HOST=true

AS_USERNAME=weblogic

OMS_HEAP_MIN=256M

ORACLE_HOME=<OMS_HOME>

NM_USER=nodemanager

OPSS_USER=SYSMAN_OPSS

OMSNAME=EMGC_OMSn

JBO_MAX_CURSORS=5

EM_REPOS_USER=SYSMAN

JBO_RECYC_THRESHOLD=50

OMS_PERMGEN_MIN=128M

OHS_COMP_NAME=ohsn

 

Note: The port numbers and other settings are OMS specific and the above-depicted entries are just a sample.

Recreate the file using the entries from one of the EMGC_OMSn.out* file under <gc_inst>/user_projects/domains/GCDomain/servers/EMGC_OMSn/logs on the respective OMS node.

2. Start the OMS

Reference - EM 12c: Emctl start/stop/status oms Failed With Error Exception in thread "main" java.lang.NoClassDefFoundError: org.apache.log4j.Logger (Doc ID 2065293.1)

OEM 12c: ORA-28001: the password has expired The Connect Descriptor was (DESCRIPTION=(ADDRESS_LIST = (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=)))(CONNECT_DATA=(SID=)(SERVER=DEDICATED)))

This error occurred on DB home page in OEM 12c.
ORA-28001: the password has expired The Connect Descriptor was (DESCRIPTION=(ADDRESS_LIST = (ADDRESS=(PROTOCOL=tcp)(HOST=<host>)(PORT=<port>)))(CONNECT_DATA=(SID=<sid>)(SERVER=DEDICATED)))

Cause:

The cause of this issue is monitoring user "DBSNMP" password expired.

select username, account_status from dba_users where username='DBSNMP';

Solution:

To resolve this issue we need to reset the password and unlock the user account using the below steps.

select 'alter user "'||d.username||'" identified by values '''||u.password||''';' from dba_users d, sys.user$ u where d.username = upper('DBSNMP') andu.user# = d.user_id;

'ALTERUSER"'||D.USERNAME||'"IDENTIFIEDBYVALUES'''||U.PASSWORD||''';'
--------------------------------------------------------------------------------
alter user "DBSNMP" identified by values '31B5G2AQ92890956';

SQL> alter user "DBSNMP" identified by values '31B5G2AQ92890956';

Note: If you want to keep the old password then you no need to do any changes in OEM but if you want to update the new password then follow the below steps in OEM.
1.      Click Targets->Databases.
2.      Select radio button for Database Instance Name and then Click Configure
3.      Change the Monitor Password, Click Next and Submit