Errors while applying 12.2.7 RUP Patch (24690690) in Oracle E-Business Suite 12.2

System got hanged while applying 12.2.7 RUP Patch (24690690) and adop exits with the errors.

Error:

1) All workers are failed with the Java error..
2) Invalid objects count is increased to 56k.

Troubleshooting:

1) Cleared the invalid objects by using utlrp.sql
2) Run the auto config on DB node
3) Start the patching activity again.

In step (3) while running auto config in database node exists with the below error.

ORA-01400: cannot insert NULL into ("APPLSYS"."FND_NODES"."NODE_NAME")

Adop error message: 

ERRORMSG: Cannot identify any valid application tier nodes in ADOP_VALID_NODES table. Ensure AutoConfig has been run on all nodes.

To resolve this error need to follow the below steps:

Step 1:
drop synonym apps.fnd_nodes;
create synonym apps.fnd_nodes for applsys.fnd_nodes;
drop synonym apps.fnd_oam_context_files;
create synonym apps.fnd_oam_context_files for applsys.fnd_oam_context_files;

Step 2:
Change the worker status to restart using adcrtl utility.

Step 3:
Run auto config again in database node. (Auto config completed without any errors)

Step 4:
Restart the patching activity.

adop phase=apply apply_mode=downtime patches=24690690 abandon=no restart=yes workers=16

ORA-04063: package body "SYS.DBMS_STATS" has errors ORA-06508: PL/SQL: could not find program unit being called: "SYS.DBMS_STATS"

After applying the patch (26925311: DATABASE PATCH SET UPDATE 12.1.0.2.180116)  in Oracle Database 12.1.0.2. We have faced below errors while executing the post patch installation steps.

Error:

We try to compile the invalid objects using utlrp.sql after applying the patch got the below error.

Warning: XDB now invalid, could not find xdbconfig
ORDIM INVALID OBJECTS: ORDIMDPCALLOUTS - INVALID - PACKAGE BODY

ERROR at line 1:
ORA-04063: package body "SYS.DBMS_STATS" has errors
ORA-06508: PL/SQL: could not find program unit being called: "SYS.DBMS_STATS"
ORA-06512: at "SYS.UTL_RECOMP", line 875
ORA-06512: at line 4

SQL> select comp_name, version, status from dba_registry;

--------------------------------------------------------------------------------------------------
COMP_NAME VERSION                                                           STATUS
------------------------------ -------------------------------------------------------------------
Oracle Machine Generated Data  12.1.0.2.0                                VALID
Oracle Text  12.1.0.2.0                                                                 VALID
OLAP Catalog 11.2.0.3.0                                                             OPTION OFF
Spatial 12.1.0.2.0                                                                          INVALID
Oracle Multimedia 12.1.0.2.0                                                       INVALID
Oracle XML Database 12.1.0.2.0                                                 INVALID
Oracle Database Catalog Views 12.1.0.2.0                                   VALID
Oracle Database Packages and Types 12.1.0.2.0                          INVALID
Oracle Real Application Clusters 12.1.0.2.0                                OPTION OFF
JServer JAVA Virtual Machine 12.1.0.2.0                                    VALID
Oracle XDK 12.1.0.2.0                                                                 VALID
Oracle Database Java Packages 12.1.0.2.0                                   VALID
OLAP Analytic Workspace 12.1.0.2.0                                         VALID
Oracle OLAP API 12.1.0.2.0                                                       INVALID


Fix:

SQL> shutdown immediate;
SQL> startup restrict
SQL> @?/rdbms/admin/catalog.sql
SQL> @?/rdbms/admin/catproc.sql
SQL> @?/rdbms/admin/utlrp.sql

Shutdown and restart the database in normal mode.

SQL> select comp_name, version, status from dba_registry;

--------------------------------------------------------------------------------------------------
COMP_NAME VERSION                                                           STATUS
------------------------------ -------------------------------------------------------------------
Oracle Machine Generated Data  12.1.0.2.0                                VALID
Oracle Text  12.1.0.2.0                                                                 VALID
OLAP Catalog 11.2.0.3.0                                                             OPTION OFF
Spatial 12.1.0.2.0                                                                         VALID
Oracle Multimedia 12.1.0.2.0                                                     VALID
Oracle XML Database 12.1.0.2.0                                               VALID
Oracle Database Catalog Views 12.1.0.2.0                                   VALID
Oracle Database Packages and Types 12.1.0.2.0                      VALID
Oracle Real Application Clusters 12.1.0.2.0                                OPTION OFF
JServer JAVA Virtual Machine 12.1.0.2.0                                    VALID
Oracle XDK 12.1.0.2.0                                                                 VALID
Oracle Database Java Packages 12.1.0.2.0                                   VALID
OLAP Analytic Workspace 12.1.0.2.0                                         VALID
Oracle OLAP API 12.1.0.2.0                                                      VALID

ORA-29548: Java system class reported: release of Java system classes in the database (12.1.0.2.0 1.6) does not match that of the oracle executable (12.1.0.2.180116 1.6)

We have faced the below error while applying the below patch in Oracle E-Business Suite 12.2.7 (Upgrade).

Patch: adop phase=apply patches=25828573 hotpatch=yes

Issue: 
ERROR at line 1:
ORA-01403: no data found
ORA-06512: at line 6
ORA-29548: Java system class reported: release of Java system classes in the
database (12.1.0.2.0 1.6) does not match that of the oracle executable

(12.1.0.2.180116 1.6)

Fix:
SQL> sho user
USER is "SYS"
SQL> @?/javavm/install/update_javavm_db.sql
SQL> SET FEEDBACK 1
SQL> SET NUMWIDTH 10
SQL> SET LINESIZE 80
SQL> SET TRIMSPOOL ON
SQL> SET TAB OFF
SQL> SET PAGESIZE 100
SQL>
SQL> alter session set "_ORACLE_SCRIPT"=true;

Session altered.

SQL>
SQL> -- If Java is installed, do CJS.
SQL>
SQL> -- If CJS can deal with the SROs inconsistent with the new JDK,
SQL> -- the drop_sros() call here can be removed.
SQL> call initjvmaux.drop_sros();

Call completed.

SQL>
SQL> create or replace java system;
  2  /

Java created.

SQL>
SQL> update dependency$
  2    set p_timestamp=(select stime from obj$ where obj#=p_obj#)
  3    where (select stime from obj$ where obj#=p_obj#)!=p_timestamp and
  4          (select type# from obj$ where obj#=p_obj#)=29  and
  5          (select owner# from obj$ where obj#=p_obj#)=0;

0 rows updated.

SQL>
SQL> commit;

Commit complete.

SQL>
SQL> alter session set "_ORACLE_SCRIPT"=false;

Session altered.

SQL> select dbms_java.get_jdk_version() from dual;

DBMS_JAVA.GET_JDK_VERSION()
--------------------------------------------------------------------------------
1.6.0_181

1 row selected.


SQL> select dbms_java.longname('TEST') from dual;

DBMS_JAVA.LONGNAME('TEST')
--------------------------------------------------------------------------------
TEST

1 row selected.

Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded : bsu.sh

We have faced java memory issue while applying weblogic patches in Oracle E-Business Suite R12.2.7

Error:
[applmgr@test bsu]$ ./bsu.sh -install -patch_download_dir=/u01/app/applmgr/TEST/fs1/FMW_Home/utils/bsu/cache_dir -patchlist=EQDE -prod_dir=/u01/app/applmgr/TEST/fs1/FMW_Home/wlserver_10.3
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
        at org.apache.xmlbeans.impl.schema.SchemaTypeImpl.getProperties(SchemaTypeImpl.java:704)
        at com.bea.cie.common.dao.xbean.XBeanDataHandler.loadPropertyMap(XBeanDataHandler.java:775)
        at com.bea.cie.common.dao.xbean.XBeanDataHandler.<init>(XBeanDataHandler.java:99)
        at com.bea.cie.common.dao.xbean.XBeanDataHandler.createDataHandler(XBeanDataHandler.java:559)
        at com.bea.cie.common.dao.xbean.XBeanDataHandler.getComplexValue(XBeanDataHandler.java:455)
        at com.bea.plateng.patch.dao.cat.PatchCatalogHelper.getPatchDependencies(PatchCatalogHelper.java:442)
        at com.bea.plateng.patch.dao.cat.PatchCatalogHelper.getPatchDependencies(PatchCatalogHelper.java:464)
        at com.bea.plateng.patch.dao.cat.PatchCatalog.getPatchDependencies(PatchCatalog.java:56)
        at com.bea.plateng.patch.dao.cat.PatchCatalogHelper.getInvalidatedPatchMap(PatchCatalogHelper.java:1621)
        at com.bea.plateng.patch.PatchSystem.updatePatchCatalog(PatchSystem.java:436)
        at com.bea.plateng.patch.PatchSystem.refresh(PatchSystem.java:130)
        at com.bea.plateng.patch.PatchSystem.setCacheDir(PatchSystem.java:201)
        at com.bea.plateng.patch.Patch.main(Patch.java:281)
[

Reason:

The default java memory arguments  provided in bsu.sh file are not sufficient for patching activity (./bsu.sh)

Fix:

Increase the Java memory values in bsu.sh file (under $MW_HOME/utils/bsu) by increasing the value MEM_ARGS values.

Default Value:

MEM_ARGS="-Xms512m -Xmx1024m"

Changed Values:

MEM_ARGS="-Xms1024m -Xmx2048m"

Save the bsu.sh file and restart the patching activity.

Kernel of proper version is not found on node "test" [Expected = "2.6.39" ; Found = "2.6.32-300.3.1.el6uek.x86_64"]

The following error occurred while installing Oracle E-Business Suite 12.2.7 (Database pre-install checks failed).

Error:

ERROR: [Result.addErrorDescription:624]  Kernel of proper version is not found on node "test" [Expected = "2.6.39" ; Found = "2.6.32-300.3.1.el6uek.x86_64"]

Fix:

The following are the steps needed to install the rpm using public-yum, with sample output from an Oracle Linux 6 installation:

1. As an authorized user (such as root), retrieve the file that configures repository locations:
# cd /etc/yum.repos.d
# wget http://public-yum.oracle.com/public-yum-ol6.repo
For Oracle Linux 5 or 7, users should replace 'public-yum-ol6.repo' in the command above with 'public-yum-el5.repo' or 'public-yum-ol7.repo' respectively.

2. Using a text editor, change the field 'enabled=0' to 'enabled=1' for the repositories corresponding to the machine's operating system while also enabling the 'addons' channel. Here's an example of a repo file's entries:

[ol6_latest]

name=Oracle Linux $releasever Latest ($basearch)

baseurl=http://public-yum.oracle.com/repo/OracleLinux/OL6/latest/$basearch/

gpgkey=http://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol6

gpgcheck=1

enabled=1


[ol6_addons]

name=Oracle Linux $releasever Add ons ($basearch)

baseurl=http://public-yum.oracle.com/repo/OracleLinux/OL6/addons/$basearch/

gpgkey=http://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol6

gpgcheck=1

enabled=1



[ol6_UEK_latest]

name=Latest Unbreakable Enterprise Kernel for Oracle Linux $releasever ($basearch)

baseurl=http://public-yum.oracle.com/repo/OracleLinux/OL6/UEK/latest/$basearch/

gpgkey=http://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol6

gpgcheck=1

enabled=1


3.Update all packages and then install the pre-install rpm by running the following yum commands:

# yum update


# yum install oracle-ebs-server-R12-preinstall 
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package oracle-ebs-server-R12-preinstall.x86_64 0:1.0-3.el6 will be installed
--> Processing Dependency: libstdc++-devel for package: oracle-ebs-server-R12-preinstall-1.0-3.el6.x86_64
--> Processing Dependency: libXp(x86-32) for package: oracle-ebs-server-R12-preinstall-1.0-3.el6.x86_64
..
Transaction Summary
================================================================================
Install     176 Package(s)
Upgrade      11 Package(s)
Total download size: 168 M
Is this ok [y/N]: y
Downloading Packages:
..

Dependency Updated:
  db4.x86_64 0:4.7.25-18.el6_4                                                 
  db4-utils.x86_64 0:4.7.25-18.el6_4                                           
  glib2.x86_64 0:2.26.1-3.el6                                                 
  glibc.x86_64 0:2.12-1.132.el6                                               
  glibc-common.x86_64 0:2.12-1.132.el6                                         
  libblkid.x86_64 0:2.17.2-12.14.el6                                           
  libgcc.x86_64 0:4.4.7-4.el6                                                 
  libstdc++.x86_64 0:4.4.7-4.el6                                               
  libuuid.x86_64 0:2.17.2-12.14.el6                                           
  nss-softokn-freebl.x86_64 0:3.14.3-9.el6                                     
  util-linux-ng.x86_64 0:2.17.2-12.14.el6                                     
Complete!
 
Versions listed above are an example - actual versions may be different based on updates to the packages.

Detailed log and backup files are available under /var/log/oracle-ebs-server-R12-preinstall.

libmawt.so: libXtst.so.6: cannot open shared object file: No such file or directory at java.lang.ClassLoader$NativeLibrary.load(Native Method)

This error occur while trying to  ./rapidwiz-version in Oracle linux 6, and resulting the Wrong package version.

Issue:

Rapid Install Wizard will now launch the Java Interface.....

Exception in thread "main" java.lang.UnsatisfiedLinkError: /u01/Stage122/startCD/Disk1/rapidwiz/jre/Linux_x64/1.6.0/lib/i386/xawt/libmawt.so: libXtst.so.6: cannot open shared object file: No such file or directory
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(Unknown Source)
at java.lang.ClassLoader.loadLibrary(Unknown Source)
at java.lang.Runtime.load0(Unknown Source)
at java.lang.System.load(Unknown Source)
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(Unknown Source)
at java.lang.ClassLoader.loadLibrary(Unknown Source)
at java.lang.Runtime.loadLibrary0(Unknown Source)
at java.lang.System.loadLibrary(Unknown Source)
at sun.security.action.LoadLibraryAction.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.awt.NativeLibLoader.loadLibraries(Unknown Source)
at sun.awt.DebugHelper.<clinit>(Unknown Source)
at java.awt.Component.<clinit>(Unknown Source)
Could not find the main class: oracle.apps.ad.rapidwiz.RIWizard.  Program will exit.

Fix:

First we need to remove the old package using that resulting the issue :

rpm -e package name

[root@11111111 yum.repos.d]# rpm -qa | grep -i libaio*
libaio-devel-0.3.107-10.el6.x86_64
libaio-0.3.105-2.x86_64
libaio-0.3.107-10.el6.x86_64
[root@11111111 yum.repos.d]# rpm -e libaio-0.3.105-2.x86_64

Install the required package from the below link :

http://rpm.pbone.net/index.php3/stat/4/idpl/5543656/dir/startcom_5/com/libXi-1.0.1-3.1.i386.rpm.html

http://rpm.pbone.net/index.php3/stat/4/idpl/8118337/dir/startcom_5/com/libXi-1.0.1-3.1.x86_64.rpm.html

If Still issue is not resolved follow the below steps:

1. copy repo from https://public-yum.oracle.com/public-yum-ol6.repo to /etc/yum.repos.d/

2. enable addons channel in .repo file.

3. yum install oracle-ebs-server-R12-preinstall -y

above steps will cover dependencies and let you install EBS.