Skip to content

Posts from the ‘RAC’ Category

Jun 25

ALTER PLUGGABLE DATABASE OPEN RESETLOGS INSTANCES=ALL Fails with ORA-65107 and ORA-29318

SYMPTOMS

ALTER PLUGGABLE DATABASE <pdb-name> OPEN RESETLOGS with INSTANCES=ALL fails with the following errors.SQL> ALTER PLUGGABLE DATABASE <pdb-name> OPEN RESETLOGS INSTANCES=ALL;
*
ERROR at line 1:
ORA-65107: Error encountered when processing the current task on instance:2
ORA-29318: datafile 1015 is online

CAUSE

When opening the PDB with RESETLOGS in the RAC environment, it can be executed with only one instance.

SOLUTION

run ALTER PLUGGABLE DATABASE <pdb-name> OPEN RESETLOGS without INSTANCES= clause.ALTER PLUGGABLE DATABASE <pdb-name> OPEN RESETLOGS;

In Release 12.2 and later, ALTER PLUGGABLE DATABASE <pdb-name> OPEN RESETLOGS INSTANCES=ALL fails with ORA-39896.ORA-39896: INSTANCES clause and RESETLOGS keyword cannot be used together in ALTER PLUGGABLE DATABASE command

Jun 8

19c:SRVCTL status service fails with Java.lang.NullPointerException

Oracle Database – Enterprise Edition – Version 19.3.0.0.0 and later
Information in this document applies to any platform.

SYMPTOMS

The following errors occur while running srvctl status service command.

[oracle@<db_name> ~]$ srvctl status service -d <dbservice_name>
Exception in thread “main” java.lang.NullPointerException
at oracle.ops.opsctl.StatusAction.internalExecuteService(StatusAction.java:2038)
at oracle.ops.opsctl.StatusAction.executeSIHAService(StatusAction.java:1652)
at oracle.ops.opsctl.Action.executeSIHA(Action.java:443)
at oracle.ops.opsctl.OPSCTLDriver.execute(OPSCTLDriver.java:537)
at oracle.ops.opsctl.OPSCTLDriver.main(OPSCTLDriver.java:25) 

[oracle@<db_name> ~]$ srvctl status database -d <db_name>
Database is running.

[oracle@<db_name> ~]$ srvctl config database -d <db_name>
Database unique name: <db_name>
Database name: <db_name>
Oracle home: <rdbms_home>/product/12.2.0.3/db_1
Oracle user: oracle
Spfile: +<diskgroup_name>/<db_name>/spfile<dba_name>.ora
Password file:
Domain: <domain_name>
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Disk Groups: +<diskgroup_name1>,+<diskgroup_name2>
Services:<db_service>
OSDBA group:
OSOPER group:
Database instance: <db_name>

[oracle@<db_name> ~]$ srvctl config service -d <dbservice_name>
Service name:<dbservice_name>
Cardinality: SINGLETON
Service role: PRIMARY
Management policy: AUTOMATIC
DTP transaction: false
AQ HA notifications: false
Global: false
Commit Outcome: false
Failover type:
Failover method:
Failover retries:
Failover delay:
Failover restore: NONE
Connection Load Balancing Goal: LONG
Runtime Load Balancing Goal: NONE
TAF policy specification: NONE
Edition:
Pluggable database name:
Hub service:
Maximum lag time: ANY
SQL Translation Profile:
Retention: 86400 seconds
Replay Initiation Time: 300 seconds
Drain timeout:
Stop option:
Session State Consistency: DYNAMIC
GSM Flags: 0
Service is enabled
Service uses Java: false

 srvm.trc[main] [ 2019-05-22 11:17:58.926 EDT ] [EntityOperations.isEntityRegisteredCRSD:367] entity: ora.<db_name>.<service_name>.svc, type: 1, registered: true
[main] [ 2019-05-22 11:17:58.926 EDT ] [CRSCache.getAttributesFromCRS:155] CRS: name: ora.<db_name>.<service_name>.svc, type 1, node: null
[main] [ 2019-05-22 11:17:58.926 EDT ] [CRSCache.getAttributesFromCRS:156] attrs: [SERVER_POOLS, CARDINALITY, USR_ORA_DISCONNECT, ROLE, MANAGEMENT_POLICY, DTP, AQ_HA_NOTIFICATION, COMMIT_OUTCOME, FAILOVER_TYPE, FAILOVER_METHOD, FAILOVER_RETRIES, FAILOVER_RESTORE, TAF_FAILOVER_DELAY, CLB_GOAL, RLB_GOAL, GLOBAL, MAX_LAG_TIME, RETENTION, REPLAY_INITIATION_TIME, SESSION_STATE_CONSISTENCY, SERVICE_NAME_PQ, SERVICE_TYPE, GSM_FLAGS, TABLE_FAMILY_ID, DRAIN_TIMEOUT, STOP_OPTION, CSS_CRITICAL, TAF_POLICY, EDITION, PLUGGABLE_DATABASE, HUB_SERVICE, MAX_LAG_TIME, SQL_TRANSLATION_PROFILE, ACTIONS, ACTIVE_PLACEMENT]
[main] [ 2019-05-22 11:17:58.989 EDT ] [CRSCache.getAttributesFromCRS:163] CRS: [<ACTIONS:isJavaService,group:”dba”,user:”oracle” rf_action,group:”dba”,user:”oracle”>, <ACTIVE_PLACEMENT:0>, <AQ_HA_NOTIFICATION:0>, <CARDINALITY:1>, <CLB_GOAL:LONG>, <COMMIT_OUTCOME:0>, <CSS_CRITICAL:no>, <DRAIN_TIMEOUT:>, <DTP:0>, <EDITION:>, <FAILOVER_METHOD:>, <FAILOVER_RESTORE:NONE>, <FAILOVER_RETRIES:>, <FAILOVER_TYPE:>, <GLOBAL:false>, <GSM_FLAGS:0>, <HUB_SERVICE:>, <MANAGEMENT_POLICY:AUTOMATIC>, <MAX_LAG_TIME:ANY>, <PLUGGABLE_DATABASE:>, <REPLAY_INITIATION_TIME:300>, <RETENTION:86400>, <RLB_GOAL:NONE>, <ROLE:PRIMARY>, <SERVER_POOLS:>, <SERVICE_NAME_PQ:>, <SERVICE_TYPE:MAIN>, <SESSION_STATE_CONSISTENCY:DYNAMIC>, <SQL_TRANSLATION_PROFILE:>, <STOP_OPTION:>, <TABLE_FAMILY_ID:>, <TAF_FAILOVER_DELAY:>, <TAF_POLICY:NONE>, <USR_ORA_DISCONNECT:false>]
[main] [ 2019-05-22 11:17:58.990 EDT ] [ServiceArgs.setServiceType:230] set serviceType to MAIN
[main] [ 2019-05-22 11:17:58.990 EDT ] [ServiceArgs.setServiceType:230] set serviceType to MAIN
[main] [ 2019-05-22 11:17:58.990 EDT ] [CRSCache.getAttributesFromCRS:155] CRS: name: ora.<db_name>.<service_name>.svc, type 1, node: null
[main] [ 2019-05-22 11:17:58.990 EDT ] [CRSCache.getAttributesFromCRS:156] attrs: [SERVER_POOLS]
[main] [ 2019-05-22 11:17:59.007 EDT ] [CRSCache.getAttributesFromCRS:163] CRS: [<SERVER_POOLS:>]
[main] [ 2019-05-22 11:17:59.007 EDT ] [ServiceImpl.getServiceCardinality:3425] value of SERVER_POOLS attr :
[main] [ 2019-05-22 11:17:59.007 EDT ] [CRSCache.getAttributesFromCRS:155] CRS: name: ora.<db_name>.<service_name>.svc, type 1, node: null
[main] [ 2019-05-22 11:17:59.007 EDT ] [CRSCache.getAttributesFromCRS:156] attrs: [CARDINALITY]
[main] [ 2019-05-22 11:17:59.027 EDT ] [CRSCache.getAttributesFromCRS:163] CRS: [<CARDINALITY:1>]
[main] [ 2019-05-22 11:17:59.027 EDT ] [ServiceImpl.getServiceCardinality:3435] value of CARDINALITY attr : 1
[main] [ 2019-05-22 11:17:59.027 EDT ] [ServiceImpl.getServiceCardinality:3462] this is a SIHA service
[main] [ 2019-05-22 11:17:59.029 EDT ] [CRSCache.getAttributesFromCache:229] CRS cache: ora.<db_name>.db [<DATABASE_TYPE:SINGLE>]
[main] [ 2019-05-22 11:17:59.029 EDT ] [CRSCache.getAttributesFromCRS:155] CRS: name: ora.<db_name>.db, type 1, node: null
[main] [ 2019-05-22 11:17:59.029 EDT ] [CRSCache.getAttributesFromCRS:156] attrs: [HOSTING_MEMBERS]
[main] [ 2019-05-22 11:17:59.046 EDT ] [CRSCache.getAttributesFromCRS:163] CRS: [<HOSTING_MEMBERS:>]
[main] [ 2019-05-22 11:17:59.046 EDT ] [CRSCache.getAttributesFromCRS:155] CRS: name: ora.<db_name>.db, type 1, node: null
[main] [ 2019-05-22 11:17:59.046 EDT ] [CRSCache.getAttributesFromCRS:156] attrs: [SERVER_POOLS]
[main] [ 2019-05-22 11:17:59.068 EDT ] [CRSCache.getAttributesFromCRS:163] CRS: [<SERVER_POOLS:>]
Exception in thread “main”
java.lang.NullPointerException
at oracle.ops.opsctl.StatusAction.internalExecuteService(StatusAction.java:2038)
at oracle.ops.opsctl.StatusAction.executeSIHAService(StatusAction.java:1652)
at oracle.ops.opsctl.Action.executeSIHA(Action.java:443)
at oracle.ops.opsctl.OPSCTLDriver.execute(OPSCTLDriver.java:537)
at oracle.ops.opsctl.OPSCTLDriver.main(OPSCTLDriver.java:252)
[Thread-1] [ 2019-05-22 11:17:59.069 EDT ] [InnerInterruptHandler.run:116] Running the shutdown hook
[Thread-1] [ 2019-05-22 11:17:59.069 EDT ] [InnerInterruptHandler.run:120] Calling Interrupt handler….ctrlCHandler
[Thread-1] [ 2019-05-22 11:17:59.069 EDT ] [CommandLineParser.<init>:506] Is this a cluster? : false
[Thread-1] [ 2019-05-22 11:17:59.070 EDT ] [OPSCTLDriver.ctrlCHandler:306] Stopped srvctl execution
[Thread-1] [ 2019-05-22 11:17:59.070 EDT ] [OPSCTLDriver.ctrlCHandler:307] Invoking interrupt handler
[Thread-1] [ 2019-05-22 11:17:59.070 EDT ] [CmdOptVals.isOptionSet:1076] Calling isOptionSet for verbose
[Thread-1] [ 2019-05-22 11:17:59.070 EDT ] [OptVal.isSet:155] m_value is null
[Thread-1] [ 2019-05-22 11:17:59.070 EDT ] [SRVCTLHandler.getMethodName:122] getMethodName: Method Name = status_service_Handler
[Thread-1] [ 2019-05-22 11:17:59.070 EDT ] [SRVCTLHandler.invokeCtrlCHandler:230] Interrupt handler method status_service_Handler
[Thread-1] [ 2019-05-22 11:17:59.070 EDT ] [SRVCTLHandler.invokeCtrlCHandler:231] Interrupt handler class oracle.ops.opsctl.StatusAction
[Thread-1] [ 2019-05-22 11:17:59.071 EDT ] [SRVCTLHandler.invokeCtrlCHandler:244] oracle.ops.opsctl.StatusAction.status_service_Handler()
java.lang.Class.getMethod(Class.java:1786)
oracle.ops.opsctl.SRVCTLHandler.invokeCtrlCHandler(SRVCTLHandler.java:234)
oracle.ops.opsctl.OPSCTLDriver.ctrlCHandler(OPSCTLDriver.java:308)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
oracle.cluster.common.InnerInterruptHandler.run(InterruptHandler.java:123)
[Thread-1] [ 2019-05-22 11:17:59.086 EDT ] [OPSCTLDriver.ctrlCHandler:311] PRCG-1036 : Internal error.
oracle.ops.opsctl.SRVCTLHandler.invokeCtrlCHandler(SRVCTLHandler.java:245)
oracle.ops.opsctl.OPSCTLDriver.ctrlCHandler(OPSCTLDriver.java:308)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
oracle.cluster.common.InnerInterruptHandler.run(InterruptHandler.java:123)

CAUSE

Due to bug 29839229
 

SOLUTION

As workaround,

srvctl status service -d <dbname> -s <service_name>

Bug 29839229 fixed in a 20.1 version, request/apply patch 29839229 for your platform/version

Jun 4

Remote_listener Getting Reset To Blank After Cluster Startup

APPLIES TO:

Oracle Database – Enterprise Edition – Version 12.2.0.1 to 19.3.0.0.0 [Release 12.2 to 18]
Information in this document applies to any platform.

SYMPTOMS

users get error while connecting to database

ORA-12514: TNS:listener does not currently know of service requested in connect descriptor


remote_listener value got reset to blank after db node patching on all databases of the cluster causing databases not able to connect from outside, and databases are not getting registered with scan_listeners.
crsd_oraagent_oracle.trc inside <oracle base>/diag/crs/<HOSTNAME>/crs/trace location indicate following messages2019-05-11 19:41:49.764 : USRTHRD:3059689216: {2:49138:41584} LsnrRegThd::setRemoteListener 391 tmpSet endp:<scan name>:<PORT>
2019-05-11 19:41:49.764 : USRTHRD:3059689216: {2:49138:41584} LsnrRegThd::setRemoteListener 440 skip remote listener registration m_runRegRLUpdate:0 isForceRemoteListenerSet:0 isRemoteListenerSet:1
2019-05-11 19:41:49.774 : USRTHRD:3059689216: {2:49138:41584} ORA-01405: fetched column value is NULL

CHANGES

CAUSE

This is due to unpublished Bug 29637215 – FAEOPS: SHELL DB DOESN’T HAVE REMOTE_LISTENER VALUE WITH GRID 19.1.0.3
 

SOLUTION

Bug 29637215 fixed in future release. Apply interim fix 29637215, if available for your platform and Oracle version.

If no patch exists for your version, please contact Oracle Support for a backport request

As a workaround,

Manually update the remote_listener values every time bounce the instances from this cluster.

alter system set remote_listener=’racscan.com:1521′ scope=both sid=’*’;


 

Apr 24

1 Comment

How To Configure SSH for a RAC Installation

GOAL

This document will explain how to configure SSH, which is required to run a RAC installation. Following the instructions in the installation guide are also correct, but sometimes this will not work, although the reason for that isn’t clear. Therefore after some investigation it seems to be that the steps below will work too.

Starting with 11gR2 the Oracle Universal Installer the SSH can be setup automatically using the  ‘SSH Connectivity’ button.

 

SOLUTION

To configure SSH you need to perform the following steps on each node in the cluster.

$ cd $HOME
$ mkdir .ssh
$ chmod 700 .ssh
$ cd .ssh
$ ssh-keygen -t rsa

Now accept the default location for the key file
Enter and confirm a passphrase. (you can also press enter twice).

$ ssh-keygen -t dsa

At the prompts, accept the default location for the key file (press Enter).
Then press “Enter” twice to accept no passphrase.

Please note: SSH with passphrase is not supported for Oracle Clusterware 11g release 2 and later releases.

If you provide a passphrase for pre 11.2 release, then you need to do 2 addition steps.

$ exec /usr/bin/ssh-agent $SHELL
$ /usr/bin/ssh-add

These statements will inform the ssh agent to add the keys to the shell used.

 

$ cat *.pub >> authorized_keys.<nodeX> (nodeX could be the nodename to differentiate files later)

Now do the same steps on the other nodes in the cluster.
When all those steps are done on the other nodes, start to copy the authorized_keys.<nodeX> to all the nodes into $HOME/.ssh/

For example if you have 4 nodes you will have after the copy in the .ssh 4 files with the name authorized_keys.<nodeX>
Then on EACH node continue the configuration of SSH by doing the following:

$ cd $HOME/.ssh
$ cat *.node* >> authorized_keys
$ chmod 600 authorized_keys

NOTE: ALL public keys must appear in ALL authorized_keys files, INCLUDING the LOCAL public key for each node.

To test that everything is working correct now execute the commands

$ ssh <hostnameX> date

So on example in a 4 node environment:

$ ssh node1 date
$ ssh node2 date
$ ssh node3 date
$ ssh node4 date

Repeat this 4 times on each node, including ssh back to the node itself. The nodeX is the hostname of the node.

The first time you will be asked to add the node to a file called ‘known_hosts’ this is correct and answer the question with ‘yes’. After that when correctly configured you must be able to get the date returned and you will not be prompted for a password.

 

Please note, ssh will not allow passwordless access if permissions on the home directory of the account you are using  allow write access for everyone.

You will also see permission denied error when the permissions on $HOME are 777 or 775. $HOME should have permission 700 or 755.

 

Disable banner (/etc/banner) on all cluster nodes when you

  • run clusterverify (cluvfy, runcluvfy)
  • install software
  • patch the system

Please work with your System Administrator or contact your Operating System support in case you still have problems setting up ssh.

Apr 24

Rconfig fails: Grid Infrastructure is Either Not Configured or Not Running on Nodes

SYMPTOMS

Running rconfig with verify option fails with below error –

[oraprod@ebsdb1 upgrade]$ rconfig ConvertToRAC_AdminManaged_verify.xml
<?xml version=”1.0″ ?>
<RConfig version=”1.1″ >
<ConvertToRAC>
<Convert>
<Response>
<Result code=”1″ >
Operation Failed
</Result>
<ErrorDetails>
Grid Infrastructure is either not configured or not running on nodes: ebsdb2
</ErrorDetails>
</Response>
</Convert>
</ConvertToRAC></RConfig>

However, cluster services are online on all nodes

[grid@ebsdb1 ~]$ crsctl check cluster -all
**************************************************************
ebsdb1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
ebsdb2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

 

Trace:

[Worker 2] [ 2016-06-23 01:03:14.952 PDT ] [UnixSystem.checkRemoteExecutionSetup:2328] checkRemoteExecutionSetup:: Error checking user equivalence using Secured Shell ‘/usr/bin/ssh’
[Worker 2] [ 2016-06-23 01:03:14.952 PDT ] [UnixSystem.pathExists:971] PRKC-1044 : Failed to check remote command execution setup for node ebsdb2 using shells /usr/bin/ssh and /usr/bin/rsh
File “/usr/bin/rsh” does not exist on node “ebsdb2”
No RSA host key is known for ebsdb2 and you have requested strict checking.Host key verification failed.
oracle.ops.mgmt.nativesystem.UnixSystem.checkRemoteExecutionSetup(UnixSystem.java:2377)
oracle.ops.mgmt.nativesystem.UnixSystem.getRemoteShellCmd(UnixSystem.java:2098)
oracle.ops.mgmt.nativesystem.UnixSystem.pathExists(UnixSystem.java:966)
oracle.ops.mgmt.command.file.PathExistCommand.execute(PathExistCommand.java:69)
oracle.ops.mgmt.cluster.ClusterConfig$ExecuteCommand.run(ClusterConfig.java:3248)
java.lang.Thread.run(Thread.java:682)
[Worker 2] [ 2016-06-23 01:03:14.953 PDT ] [PathExistCommand.execute:74] REMOTEDIREXCEPTION: PRKC-1044 : Failed to check remote command execution setup for node ebsdb2 using shells /usr/bin/ssh and /usr/bin/rsh
File “/usr/bin/rsh” does not exist on node “ebsdb2”
No RSA host key is known for ebsdb2 and you have requested strict checking.Host key verification failed.
[Worker 2] [ 2016-06-23 01:03:14.953 PDT ] [ClusterConfig$ExecuteCommand.returnCommandToClient:3216] returnCommandToClient; fillCount=1 is full=false
[Worker 2] [ 2016-06-23 01:03:14.953 PDT ] [Semaphore.acquire:109] SyncBufferEmpty:Acquire called by thread Worker 2 m_count=99
[Worker 2] [ 2016-06-23 01:03:14.954 PDT ] [Semaphore.release:85] SyncBufferFull:Release called by thread Worker 2 m_count=2
[Worker 2] [ 2016-06-23 01:03:14.954 PDT ] [ClientResource.getListener:157] Calling getListener
[Worker 2] [ 2016-06-23 01:03:14.954 PDT ] [ClusterConfig$ExecuteCommand.run:3311] Owner Thread name of the blocking Semaphore main
[Worker 2] [ 2016-06-23 01:03:14.955 PDT ] [ClusterConfig$ExecuteCommand.run:3319] Obtained Semaphore
[Worker 2] [ 2016-06-23 01:03:14.955 PDT ] [Semaphore.release:85] ClientResource Constructor:Blocking Semaphore owned by main:Release called by thread Worker 2 m_count=1
[Worker 2] [ 2016-06-23 01:03:14.955 PDT ] [ClusterConfig$ExecuteCommand.run:3334] Released Semaphore by worker=Worker 2
[main] [ 2016-06-23 01:03:14.955 PDT ] [ClusterConfig.block:613] block acquired semnum=1
[main] [ 2016-06-23 01:03:14.955 PDT ] [ClusterConfig.submit:562] Out of block
[Worker 2] [ 2016-06-23 01:03:14.955 PDT ] [Semaphore.acquire:109] SyncBufferFull:Acquire called by thread Worker 2 m_count=0
[main] [ 2016-06-23 01:03:14.956 PDT ] [CommandBuffer.checkCommandStatus:83] CommandBuffer:checkCommandStatus returning false, command number= 3 thread name= Thread[main,5,main]
[main] [ 2016-06-23 01:03:14.956 PDT ] [ClusterConfig.submit:579] status=false
[main] [ 2016-06-23 01:03:14.956 PDT ] [ClusterException.setErrorMessage:138] ClusterException.setErrorMessage: commands[0]=oracle.ops.mgmt.command.file.PathExistCommand@6d588325
[main] [ 2016-06-23 01:03:14.956 PDT ] [ClusterException.setErrorMessage:138] ClusterException.setErrorMessage: commands[1]=oracle.ops.mgmt.command.file.PathExistCommand@4e0a2a38
[main] [ 2016-06-23 01:03:14.956 PDT ] [ClusterException.setErrorMessage:165] ClusterException.setErrorMessage: errString is ‘PRKC-1044 : Failed to check remote command execution setup for node ebsdb2 using shells /usr/bin/ssh and /usr/bin/rsh
File “/usr/bin/rsh” does not exist on node “ebsdb2”
No RSA host key is known for ebsdb2 and you have requested strict checking.Host key verification failed.’
[main] [ 2016-06-23 01:03:14.956 PDT ] [ClusterConfig.destroy:457] destroying resources for client thread Thread[main,5,main]
[main] [ 2016-06-23 01:03:14.957 PDT ] [Utils.getString:175] ==========Str is ebsdb2
[main] [ 2016-06-23 01:03:14.957 PDT ] [OracleHome.getNodeNames:406] exception checking oracle home existence on nodesPRKC-PRKC-1027 : Error checking existence of file /u01/oracle/PROD/12.1.0 on ebsdb2 : <null>
[main] [ 2016-06-23 01:03:14.957 PDT ] [OracleHome.getNodeNames:434] no of nodes returned1
[main] [ 2016-06-23 01:03:14.957 PDT ] [ClusterCheck.checkCluster:219] OUI Node Names Length=1
[main] [ 2016-06-23 01:03:14.958 PDT ] [ClusterCheck.checkCluster:222] OUI Node List Length=1
[main] [ 2016-06-23 01:03:14.958 PDT ] [ClusterUtil.getCSSMode:660] Returned from native code
[main] [ 2016-06-23 01:03:14.958 PDT ] [ClusterUtil.getCSSMode:665] Returned from native code
[main] [ 2016-06-23 01:03:14.961 PDT ] [Instance.getSQLEngine:262] sysCreds.getUser()sys
[main] [ 2016-06-23 01:03:14.961 PDT ] [Instance.getSQLEngine:263] sysCred.getRole()sysdba
[main] [ 2016-06-23 01:03:14.961 PDT ] [Instance.getSQLEngine:264] SID PROD
[main] [ 2016-06-23 01:03:14.961 PDT ] [Instance.getSQLEngine:262] sysCreds.getUser()sys
[main] [ 2016-06-23 01:03:14.961 PDT ] [Instance.getSQLEngine:263] sysCred.getRole()sysdba
[main] [ 2016-06-23 01:03:14.962 PDT ] [Instance.getSQLEngine:264] SID PROD
[main] [ 2016-06-23 01:03:14.962 PDT ] [RACConvertStep.execute:193] STEP Result=Grid Infrastructure is either not configured or not running on nodes: ebsdb2

 

CAUSE

Passwordless ssh to remote node failing

SOLUTION

Passwordless-ssh connection has to be configured between all nodes, refer to:

How To Configure SSH for a RAC Installation

 

Once done,  retry rconfig