BEA-000383 A critical service failed. The server will shut itself down

I am facing the below issue while starting the weblogic server after configuring the OBIEE application.

Error:
weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.login(LDAPAtnLoginModuleImpl.java:261)
        at com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper.java:110)
        at java.security.AccessController.doPrivileged(Native Method)
        at com.bea.common.security.internal.service.LoginModuleWrapper.login(LoginModuleWrapper.java:106)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        Truncated. see log file for complete stacktrace

May 29, 2016 11:30:55 AM PDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>
<May 29, 2016 11:30:55 AM PDT> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down>
<May 29, 2016 11:30:55 AM PDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>

Fix:

Create the boot.properties file under $DOMAIN_HOME/servers/AdminServer/security/

[obiee@obiee security]$ pwd
/u0/u01/obiee/product/FMW_HOME/user_projects/domains/bifoundation_domain/servers/AdminServer/security

[obiee@obiee security]$ vi boot.properties
[obiee@obiee security]$ cat boot.properties
username=weblogic
password=welcome123

[ERROR]: Some of the ports specified for the patch file system are not available.

When attempting to run "adop phase=fs_clone" command the following error occurs:

ValidatePatchFSPortPool:
[ERROR]: Some of the ports specified for the patch file system are not available.

In my case here 7002 is run file system weblogic console port and 7001 is patch file system port.

This issue is caused by a port used by WLS Admin Server on the run file system which is not free (e.g port 7001).

Note:
The following command gives the current port used by the WLS Admin Server :
$ grep s_wls_adminport $CONTEXT_FILE

Cause:

In my case both Run and Patch file system admin servers are running.

Solution:

In my case both Run and Patch file system weblogic console ports are running.

To implement the solution, please execute the following steps:

1. Shutdown all apps services on RUN and PATCH file systems.

. /EBSapps.env R
cd $INST_TOP/admin/scripts
adstpall.sh apps/apps

. /EBSapps.env P
cd $INST_TOP/admin/scripts
adstpall.sh apps/apps

2. make sure that port used by the WLS Admin Server is free and no processes are running from your Patch file system via the following command

$ netstat -a | grep 7001
$ ps -eaf | grep 7001

replace 7001 by the port listed in the autoconfig s_wls_adminport parameter and in the error message (e.g. 7001, 7002, ...)
Note: The above commands should returns no activity on the related ports

3. On the RUN file system, start up adadmin server via the following command

. /EBSapps.env run
cd $INST_TOP/admin/scripts
$ adadminsrvctl.sh start

4. Start fs_clone from the beginning  

For Single Node
$ adop phase=fs_clone allnodes=no force=yes

For Multi Node
$ adop phase=fs_clone force=yes

ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance

SQL> startup
ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance
ORACLE instance started.

Total System Global Area 2147483648 bytes
Fixed Size                  2926472 bytes
Variable Size            1392511096 bytes
Database Buffers          738197504 bytes
Redo Buffers               13848576 bytes
Database mounted.
Database opened.

SQL> SELECT name FROM v$parameter WHERE isdeprecated = 'TRUE';

NAME
--------------------------------------------------------------------------------
lock_name_space
instance_groups
resource_manager_cpu_allocation
active_instance_count
buffer_pool_keep
buffer_pool_recycle
log_archive_start
standby_archive_dest
parallel_server
parallel_server_instances
fast_start_io_target

NAME
--------------------------------------------------------------------------------
serial_reuse
rdbms_server_dn
max_enabled_roles
remote_os_authent
sec_case_sensitive_logon
global_context_pool_size
cursor_space_for_time
plsql_v2_compatibility
plsql_debug
background_dump_dest
user_dump_dest

NAME
--------------------------------------------------------------------------------
commit_write
sql_trace
parallel_automatic_tuning
parallel_io_cap_enabled

26 rows selected.

SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

[oracle@rysigo04 ~]$ oerr ora 32004
32004, 00000, "obsolete or deprecated parameter(s) specified for %s instance"
// *Cause:  Obsolete or deprecated parameters for this instance type
//          were specified in the SPFILE or the PFILE on the server side.

// *Action: See alert log for a list of parameters that are obsolete
//          or deprecated. Remove them from the SPFILE or the server
//          side PFILE.

How to install and uninstall weblogic bsu patches

Install .bsu patch in weblogic:

bsu.sh -install -patch_download_dir=/u002/app/applmgr/RYSAAV3/fs1/FMW_Home/utils/bsu/cache_dir -patchlist=FCX7 -prod_dir=/u002/app/applmgr/RYSAAV3/fs1/FMW_Home/wlserver_10.3

Uninstall .bsu patch in weblogic:

bsu.sh -remove -patchlist=BLTG -prod_dir=/u002/app/applmgr/RYSAAV3/fs1/FMW_Home/wlserver_10.3

bsu.sh -remove -patchlist=FCX7 -prod_dir=/u002/app/applmgr/RYSAAV3/fs1/FMW_Home/wlserver_10.3

adop utility cannot be used when a previous patching cycle is incomplete

Checking for existing adop sessions.
        Session Id            :   2
        Prepare phase status  :   NOT APPLICABLE
        Apply phase status    :   ATLEAST ONE PATCH IS ALREADY APPLIED
        Cutover  phase status :   NOT APPLICABLE
        Abort phase status    :   NOT APPLICABLE
        Session status        :   FAILED
    [UNEXPECTED]The adop utility cannot be used when a previous patching cycle is incomplete
    [UNEXPECTED]Ensure there is no existing patching cycle before trying to apply more patches
    [UNEXPECTED]Unrecoverable error occurred. Exiting current adop session.

[STATEMENT] Please run adopscanlog utility, using the command

"adopscanlog -latest=yes"

to get the list of the log files along with snippet of the error message corresponding to each log file.


adop exiting with status = 2 (Fail)

$adop phase=abort

$adop phase=cleanup cleanup_mode=full

Run the following statement to find out the session that is in running state:
SQL>select adop_session_id from ad_adop_sessions where status='R';

Set the status to 'C' (Completed) for that session, to re-try the phase that was interrupted:

SQL>update ad_adop_sessions set status='C' where status='R;

SQL>commit;

ADOP apply phase options ABANDON and RESTART

Abandon:

adop phase=apply patches=22123818 hotpatch=yes abandon=yes

Used to specify whether to restart the previous run of adop. May be useful if the previous action had an error.
Note: If there was an error in the previous run, and 'abandon' is not set to 'yes', the same parameters will be re-used that were used in the failed run.

If you give a value for the 'restart' parameter, it cannot be the same as the value given for this parameter.

Restart:

adop phase=apply patches=22123818 hotpatch=yes restart=yes

Used to specify whether to restart the previous run of ADOP. May be useful if the previous action had an error. If you give a value for the 'abandon' parameter, it cannot be the same as the value given for this parameter.

Utility /Script To Check The Techstack Component Versions (Forms, Http Server, JDK, Framework, Database, etc)

Below steps to know How to find the versions of technology stack components (Forms, iAS, Framework, JDK, OJSP, Database, etc.)?

Script:

$ADPERLPRG $FND_TOP/patch/115/bin/TXKScript.pl -script=$FND_TOP/patch/115/bin/txkInventory.pl -txktop=$APPLTMP -contextfile=$CONTEXT_FILE -appspass=apps -outfile=$APPLTMP/Report_App_Inventory.txt -reporttype=text

Output:

Report file location:  /u002/app/applmgr/RYSAAV3/fs1/inst/apps/RYSAAV3_rysigo04/appltmp/Report_App_Inventory.txt gene   rated successfully.

TXK Technology Inventory Report for EBS Release 12

  Date :Thu May 19 21:09:22 2016
  Hostname :krishna.apps
  Enabled Services:Root Service, Web Administration, Web Entry Point
  Services, Web Application Services, Batch Processing Services
  Instance :TEST
  Platform :LINUX 64-bit
  Tier :APPS
  OS release :2.6.32-300.39.2.el5uek
  DB Host :krishna.apps
  Context File
  :/u002/app/applmgr/TEST/fs1/inst/apps/*******/appl/admin/*******_04.xml (120.401.12020000.9.1202010.10)
  Report File
  :/u002/app/applmgr/TEST/fs1/inst/apps/********/appltmp/Report_
  App_Inventory.txt
  XML Definition
  File:/u002/app/applmgr/TEST/fs1/EBSapps/appl/fnd/12.0.0/html/txkInvent
  ory.xml (120.13.12020000.2)

Weblogic Service Properties :
  Oracle Weblogic Server version                   10.3.6.0
  JDK version on Weblogic Server node              1.7.0_85 (64-Bit)
  Oracle Smart Update version on FMW Home          3.3.0.0
  OCM version on FMW Home (utils)                  Not Configured

HTTP Service Properties:
  Oracle HTTP Server version                       11.1.1.9.0
  Sun JDK client version                           1.6.0_27
  JDK version on HTTP server node                  1.7.0_85
  Oracle JDBC driver version using JAVA API        11.2.0.3.0
  (oracle.jdbc.driver.OracleDriver)
  Oracle AOLJ version                              Roll Up Patch J
  Oracle BC4J version                              10.1.3.43.6
  Oracle BiBeans version                           3.1.1.10
  Oracle HTTP client version                       10h
  Java object cache                                11.1.1.2.0
  Oracle JRAD libraries version                    10.1.3
  Oracle MDS version                               9.0.6.0.0_50
  OA Framework version                             12.2.0
  Oracle help web version                          2.0.13
  Oracle XML driver version                      
  Oracle UIX version                               11_1_1_7_0
  OPatch version on Oracle HTTP Server Oracle Home 11.1.0.12.9
  OCM version on Oracle HTTP Server Oracle Home    Not Configured
  OPatch version on Oracle Common Oracle Home      11.1.0.12.9

Forms Service Properties:
  Developer 10g version                            10.1.2.3.0
  DB client (RSF) library version in 10.1.2        10.1.0.5.0
  oracle home
  JDK version used by AD utilities on forms node   1.7.0_85
  Oracle Application Server patchset version       10.1.2.3.0
  OPatch version on 10.1.2 Oracle Home             1.0.0.0.64
  OCM version on 10.1.2 Oracle Home                Not Configured
  JDK version used by Developer 10g                1.7.0_85-b15

Concurrent Processing Service Properties:
  JDK on concurrent processing node                1.7.0_85
  JDK version used by AD utilities on CP node      1.7.0_85

Code Inventory of Weblogic Server Home:
(/u002/app/applmgr/RYSAAV3/fs1/FMW_Home/wlserver_10.3):
  Oracle WebLogic Sever overlay patch for          -
  10.3.6.0.7 which requires WLS10.3.6.0.7 PSU
  (Patch Number : 17572726 , Patch ID : FCX7) in
  the environment
  S8H6                                             -
  1LRI                                             -
  BLTG                                             -
  TTEM                                             -
  CW9T                                             -
  Oracle WebLogic Sever overlay patch for          -
  10.3.6.0.7 which requires WLS 10.3.6.0.7 PSU
  (Patch Number: 17572726 , Patch ID :FCX7) in
  the environment
  7FC9                                             -
  Y5AP                                             -
  YVDZ                                             -
  YHJK                                             -
  32I2                                             -
  FCX7                                             -
  S9MN                                             -
  Oracle WebLogic Sever overlay patch for          -
  10.3.6.0.7 which requires WLS10.3.6.0.7 PSU
  (Patch Number: 17572726 , Patch ID :FCX7) in
  the environment
  VKXF                                             -
  2GYW                                             -
  Oracle WebLogic Sever overlay patch for          -
  10.3.6.0.7 which requires WLS10.3.6.0.7 PSU
  (Patch Number: 17572726 , Patch ID :FCX7) in
  the environment.
  CIH8                                             -

Code Inventory of OHS Oracle Home:
(/u002/app/applmgr/TEST/fs1/FMW_Home/webtier):
  7663342                                          Wed May 18 15:18:23 PDT 2016
  7572595                                          Wed May 18 16:03:27 PDT 2016
  7427144                                          Wed May 18 15:17:57 PDT 2016
  20922780                                         Wed May 18 16:02:23 PDT 2016
  6599470                                          Wed May 18 15:17:49 PDT 2016
  13417321                                         Wed May 18 16:03:04 PDT 2016
  7707476                                          Wed May 18 15:18:13 PDT 2016
  6750400                                          Wed May 18 15:18:56 PDT 2016
  7695070                                          Wed May 18 16:02:48 PDT 2016
  6845838                                          Wed May 18 15:19:12 PDT 2016
  7393921                                          Wed May 18 15:18:32 PDT 2016

Code Inventory of Oracle Common Oracle Home:
(/u002/app/applmgr/TSET/fs1/FMW_Home/oracle_common):
  17284368                                         Wed May 18 16:04:11 PDT 2016
  20484781                                         Wed May 18 16:03:54 PDT 2016

Code Inventory of C-Oracle Home:
(/u002/app/applmgr/TEST/fs1/EBSapps/10.1.2)
  5261796                                          Thu Nov 15 23:30:46 PST 2012
  11780669                                         Thu Nov 15 23:49:04 PST 2012
  12848228                                         Thu Nov 15 23:59:32 PST 2012
  14825718                                         Wed May 18 16:16:44 PDT 2016
  16778382                                         Wed May 18 16:21:11 PDT 2016
  9974899                                          Thu Nov 15 23:46:06 PST 2012
  12898423                                         Thu Nov 15 23:52:39 PST 2012
  13417321                                         Wed May 18 16:26:37 PDT 2016
  16241466                                         Wed May 18 16:11:11 PDT 2016
  15900214                                         Wed May 18 16:23:08 PDT 2016
  7695070                                          Thu Nov 15 23:50:49 PST 2012
  8943095                                          Wed May 18 16:23:38 PDT 2016
  3559326                                          Thu Mar 26 08:42:23 PDT 2009
  5651245                                          Thu Dec 09 15:03:33 PST 2010
  12887068                                         Thu Nov 15 23:52:13 PST 2012
  8339196                                          Thu Nov 15 23:29:45 PST 2012
  5394728                                          Thu Mar 26 08:42:47 PDT 2009
  8557019                                          Thu Nov 15 23:37:31 PST 2012
  8300196                                          Wed May 18 16:20:42 PDT 2016
  5917053                                          Thu Mar 26 08:41:08 PDT 2009
  17653437                                         Wed May 18 16:10:25 PDT 2016
  17645157                                         Wed May 18 16:11:54 PDT 2016
  12379038                                         Thu Nov 15 23:49:45 PST 2012
  12434128                                         Thu Nov 15 23:42:23 PST 2012
  8528194                                          Wed May 18 16:22:16 PDT 2016
  10152652                                         Wed May 18 16:12:43 PDT 2016
  16271876                                         Wed May 18 16:09:41 PDT 2016
  14353879                                         Wed May 18 16:14:13 PDT 2016
  8447875                                          Thu Nov 15 23:45:14 PST 2012
  5604120                                          Wed May 18 16:21:36 PDT 2016
  16209520                                         Wed May 18 16:17:06 PDT 2016
  7140405                                          Thu Mar 26 08:42:04 PDT 2009
  16275529                                         Wed May 18 16:19:01 PDT 2016
  6400501                                          Sun Dec 12 22:26:47 PST 2010
  7016961                                          Thu Mar 26 08:40:17 PDT 2009
  17907988                                         Wed May 18 16:24:17 PDT 2016
  18620223                                         Wed May 18 16:24:54 PDT 2016
  9273888                                          Thu Nov 15 23:45:40 PST 2012
  19434967                                         Wed May 18 16:25:55 PDT 2016
  14374587                                         Wed May 18 16:13:52 PDT 2016
  4047619                                          Wed May 18 16:19:57 PDT 2016
  5659594                                          Wed May 18 16:22:39 PDT 2016
  12828135                                         Thu Nov 15 23:44:19 PST 2012
  6371228                                          Thu Mar 26 08:41:41 PDT 2009

Ref: (Doc ID 601736.1)

How to Completely Remove Forms (formsapp) from WLS_FORMS and Redeploy the forms in WLS_FORMS

There may be times when the existing Forms application in a managed WLS_FORMS server becomes corrupted and cannot be fixed. Removing it and installing the default Forms application can resolve issues like this.

Removing Forms from WLS:

1. Go to the WLS Console and choose deployments.
2. Ensure "Lock and Edit" is chosen.
3. Check the formsapp (11.1.2) and choose Stop.  Choose either "When work completes" or "Force Stop Now" depending on the concern for user impact.
4. Once it is stopped, its "State" will change to "Prepared".
5. Select the formsapp (11.1.2) again, choose "Delete" and confirm the deletion.
6. You will be prompted to "Activate Changes".
7. You will note that formsapp (11.1.2) no longer appears as a Deployment in the deployed applications list.

Redeploying Forms into WLS:

1. Go to the WLS Console and choose deployments.
2. Ensure "Lock and Edit" is chosen.
3. Click on the "Install" Button.
4. Choose your formsapp.ear from ORACLE_HOME\forms\j2ee and choose Next.
5. Take the default option of "Install this deployment as an application" and choose Next.
6. You will next see target options.  In most cases you will want to choose "cluster_forms" as shown but other options are available.
7. Next a screen will appear showing "General", "Security", "Source Accessibility" options.  Take the defaults which should be: (Name:  formsapp, DD Only: Use only roles and policies that are defined in the deployment descriptors, Source accessibility: Use the defaults defined by the deployment's targets).
8. Click on next.
9. The next screen will give "Additional Configuration" options.
10. Take the default selection: "Yes, take me to the deployment's configuration screen. The  "Summary" selection will show the previous choices made.
11. Click "Finish" and then "Save".
12. Click on "Activate Changes".
13. Choose "Deployments" again and you should see "formsapp (11.1.2) in a "Prepared" state.  Check formsapp (11.1.2) and  choose "Start (Servicing All Requests)." It should become "Active".

Update the Deployment Plan for the formsapp (plan.xml):

1. Go to the WLS Console and choose deployments.
2. Ensure "Lock and Edit" is chosen.
3. Check the formsapp (11.1.2) and choose "Update".
4. You will note that the "Deployment plan path:" is not chosen. Click on the "Change Path" button and choose DOMAIN_HOME\deployments-plans\formsapp\11.1.2\plan.xml
5. Click "Next" and choose: "Update this application in place with new deployment plan changes. (A deployment plan must be specified for this option)".
6. Choose "Finish".
7. Click "Activate Changes".
8. Restart WLS_FORMS and test using:  http://hostname:port/forms/frmservlet to bring up the test form and this should work.

Ref: (Doc ID 2030914.1)

oracle.apps.fnd.framework.OAException: Could not load application module 'oracle.apps.fnd.wf.worklist.server.NotificationsAM'.

In Production, all of a sudden users started getting: Could not load application module

In most jsp's users are getting the following error:
oracle.apps.fnd.framework.OAException: Could not load application module 'oracle.apps.fnd.wf.worklist.server.NotificationsAM'.

## Detail 0 ##
oracle.apps.fnd.framework.OAException: Application: FND, Message Name: FND_NO_TRANSACTION_ID.

The users can login ok and are able to navigate through forms as well.

Fix:

icx_transactions table needs to be cleared.

Please follow the below steps to schedule a concurrent request to purge the inactive sessions periodically (once in a week for high traffic instances):

1.  Login using sysadmin
2.  Navigate to Requests ---> Run
3.  In the Form that opens run/schedule "Purge Inactive Sessions" request.

A job should be scheduled to run this request at least weekly.

Ref: (Doc ID 1577877.1)

Steps to change the APPS, APPLSYS, and APPS_NE password using FNDCPASS or AFPASSWD for EBS 12.2

Whenever using FNDCPASS or AFPASSWD to change the APPS, APPLSYS, and APPS_NE password, we must also perform the following actions:

1. Shut down the application tier services using the below script:

$INST_TOP/admin/scripts/adstpall.sh

2. Change the APPLSYS password using

A. FNDCPASS:
For example, the following command changes the APPLSYS password to 'WELCOME':

FNDCPASS apps/<appspwd> 0 Y system/manager SYSTEM APPLSYS WELCOME

B. AFPASSWD:

AFPASSWD is an enhanced version of FNDCPASS, and includes the following features:

• AFPASSWD only prompts for passwords required for the current operation, allowing separation of duties between applications administrators and database administrators. This also improves interoperability with Oracle Database Vault. In contrast, the FNDCPASS utility currently requires specification of the APPS and the SYSTEM usernames and corresponding passwords, preventing separation of duties between applications administrators and database administrators.
• When changing a password with AFPASSWD, the user is prompted to enter the new password twice to confirm.
• In Oracle E-Business Suite Release 12.2.3 and higher, you can also use the AFPASSWD utility to migrate Oracle E-Business Suite user passwords to a password hashing scheme.

Syntax for using AFPASSWD:
 AFPASSWD [-c <APPSUSER>[@<TWO_TASK>]] -s <APPLSYS>

3. Run autoconfig with the newly changed password.

4. Start AdminServer using the $INST_TOP/admin/scripts/adadminsrvctl.sh script. Do not start any other application tier services.

5. Change the "apps" password in WLS Datasource as follows:

• Log in to WLS Administration Console.
• Click Lock & Edit in Change Center.
• In the Domain Structure tree, expand Services, then select Data Sources.
• On the "Summary of JDBC Data Sources" page, select EBSDataSource.
• On the "Settings for EBSDataSource" page, select the Connection Pool tab.
• Enter the new password in the "Password" field.
• Enter the new password in the "Confirm Password" field.
• Click Save.
• Click Activate Changes in Change Center.

6. Start all the application tier services using the below script

$INST_TOP/admin/scripts/adstrtal.sh

7. Verify the WLS Datastore changes as follows:

• Log in to WLS Administration Console.
• In the Domain Structure tree, expand Services, then select Data Sources.
• On the "Summary of JDBC Data Sources" page, select EBSDataSource.
• On the "Settings for EBSDataSource" page, select Monitoring > Testing.
• Select "oacore_server1".
• Click Test DataSource
• Look for the message "Test of EBSDataSource on server oacore_server1 was successful".

Ref: (Doc ID 1674462.1)

How to check schema size in oracle database

Below script to know the schema size in oracle database.

Script:

set linesize 160
set pagesize 10000
col owner for a15
col segment_name for a30
col segment_type for a20
col TABLESPACE_NAME for a30
clear breaks
clear computes
compute sum of SIZE_IN_GB on report
break on report
select OWNER,sum(bytes)/1024/1024/1000 "SIZE_IN_GB" from dba_segments group by owner order by owner;

ORA-01403: no data found / ORA-06512: at "APPS.FND_NET_SERVICES"

We are getting below error when attempting to run autoconfig on DB tier.

Issue:

java.sql.SQLException: ORA-01403: no data found
ORA-06512: at "APPS.FND_NET_SERVICES", line 877
ORA-06512: at line 1

SP2-0575: Use of Oracle SQL feature not in SQL92 Entry Level

Cause:

Table DUAL existed not owned by SYS.

Table DUAL not owned by SYS should not exist as this can cause issues with EBS.

Fix:

Remove table DUAL not owned by SYS

The existence of this can be checked using:

SQL> column OWNER format a15;
SQL> column OBJECT_NAME format a20;
SQL> select OWNER,OBJECT_NAME,OBJECT_TYPE,CREATED from dba_objects where object_name='DUAL';

Ref: (Doc ID 2054876.1)