Rconfig fails: Grid Infrastructure is Either Not Configured or Not Running on Nodes

[oraprod@ebsdb1 upgrade]$ rconfig ConvertToRAC_AdminManaged_verify.xml
<?xml version=”1.0″ ?>
<RConfig version=”1.1″ >
<ConvertToRAC>
<Convert>
<Response>
<Result code=”1″ >
Operation Failed
</Result>
<ErrorDetails>
Grid Infrastructure is either not configured or not running on nodes: ebsdb2
</ErrorDetails>
</Response>
</Convert>
</ConvertToRAC></RConfig>

[grid@ebsdb1 ~]$ crsctl check cluster -all
**************************************************************
ebsdb1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
ebsdb2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

 

[Worker 2] [ 2016-06-23 01:03:14.952 PDT ] [UnixSystem.checkRemoteExecutionSetup:2328] checkRemoteExecutionSetup:: Error checking user equivalence using Secured Shell ‘/usr/bin/ssh’
[Worker 2] [ 2016-06-23 01:03:14.952 PDT ] [UnixSystem.pathExists:971] PRKC-1044 : Failed to check remote command execution setup for node ebsdb2 using shells /usr/bin/ssh and /usr/bin/rsh
File “/usr/bin/rsh” does not exist on node “ebsdb2”
No RSA host key is known for ebsdb2 and you have requested strict checking.Host key verification failed.
oracle.ops.mgmt.nativesystem.UnixSystem.checkRemoteExecutionSetup(UnixSystem.java:2377)
oracle.ops.mgmt.nativesystem.UnixSystem.getRemoteShellCmd(UnixSystem.java:2098)
oracle.ops.mgmt.nativesystem.UnixSystem.pathExists(UnixSystem.java:966)
oracle.ops.mgmt.command.file.PathExistCommand.execute(PathExistCommand.java:69)
oracle.ops.mgmt.cluster.ClusterConfig$ExecuteCommand.run(ClusterConfig.java:3248)
java.lang.Thread.run(Thread.java:682)
[Worker 2] [ 2016-06-23 01:03:14.953 PDT ] [PathExistCommand.execute:74] REMOTEDIREXCEPTION: PRKC-1044 : Failed to check remote command execution setup for node ebsdb2 using shells /usr/bin/ssh and /usr/bin/rsh
File “/usr/bin/rsh” does not exist on node “ebsdb2”
No RSA host key is known for ebsdb2 and you have requested strict checking.Host key verification failed.
[Worker 2] [ 2016-06-23 01:03:14.953 PDT ] [ClusterConfig$ExecuteCommand.returnCommandToClient:3216] returnCommandToClient; fillCount=1 is full=false
[Worker 2] [ 2016-06-23 01:03:14.953 PDT ] [Semaphore.acquire:109] SyncBufferEmpty:Acquire called by thread Worker 2 m_count=99
[Worker 2] [ 2016-06-23 01:03:14.954 PDT ] [Semaphore.release:85] SyncBufferFull:Release called by thread Worker 2 m_count=2
[Worker 2] [ 2016-06-23 01:03:14.954 PDT ] [ClientResource.getListener:157] Calling getListener
[Worker 2] [ 2016-06-23 01:03:14.954 PDT ] [ClusterConfig$ExecuteCommand.run:3311] Owner Thread name of the blocking Semaphore main
[Worker 2] [ 2016-06-23 01:03:14.955 PDT ] [ClusterConfig$ExecuteCommand.run:3319] Obtained Semaphore
[Worker 2] [ 2016-06-23 01:03:14.955 PDT ] [Semaphore.release:85] ClientResource Constructor:Blocking Semaphore owned by main:Release called by thread Worker 2 m_count=1
[Worker 2] [ 2016-06-23 01:03:14.955 PDT ] [ClusterConfig$ExecuteCommand.run:3334] Released Semaphore by worker=Worker 2
[main] [ 2016-06-23 01:03:14.955 PDT ] [ClusterConfig.block:613] block acquired semnum=1
[main] [ 2016-06-23 01:03:14.955 PDT ] [ClusterConfig.submit:562] Out of block
[Worker 2] [ 2016-06-23 01:03:14.955 PDT ] [Semaphore.acquire:109] SyncBufferFull:Acquire called by thread Worker 2 m_count=0
[main] [ 2016-06-23 01:03:14.956 PDT ] [CommandBuffer.checkCommandStatus:83] CommandBuffer:checkCommandStatus returning false, command number= 3 thread name= Thread[main,5,main]
[main] [ 2016-06-23 01:03:14.956 PDT ] [ClusterConfig.submit:579] status=false
[main] [ 2016-06-23 01:03:14.956 PDT ] [ClusterException.setErrorMessage:138] ClusterException.setErrorMessage: commands[0]=oracle.ops.mgmt.command.file.PathExistCommand@6d588325
[main] [ 2016-06-23 01:03:14.956 PDT ] [ClusterException.setErrorMessage:138] ClusterException.setErrorMessage: commands[1]=oracle.ops.mgmt.command.file.PathExistCommand@4e0a2a38
[main] [ 2016-06-23 01:03:14.956 PDT ] [ClusterException.setErrorMessage:165] ClusterException.setErrorMessage: errString is ‘PRKC-1044 : Failed to check remote command execution setup for node ebsdb2 using shells /usr/bin/ssh and /usr/bin/rsh
File “/usr/bin/rsh” does not exist on node “ebsdb2”
No RSA host key is known for ebsdb2 and you have requested strict checking.Host key verification failed.’
[main] [ 2016-06-23 01:03:14.956 PDT ] [ClusterConfig.destroy:457] destroying resources for client thread Thread[main,5,main]
[main] [ 2016-06-23 01:03:14.957 PDT ] [Utils.getString:175] ==========Str is ebsdb2
[main] [ 2016-06-23 01:03:14.957 PDT ] [OracleHome.getNodeNames:406] exception checking oracle home existence on nodesPRKC-PRKC-1027 : Error checking existence of file /u01/oracle/PROD/12.1.0 on ebsdb2 : <null>
[main] [ 2016-06-23 01:03:14.957 PDT ] [OracleHome.getNodeNames:434] no of nodes returned1
[main] [ 2016-06-23 01:03:14.957 PDT ] [ClusterCheck.checkCluster:219] OUI Node Names Length=1
[main] [ 2016-06-23 01:03:14.958 PDT ] [ClusterCheck.checkCluster:222] OUI Node List Length=1
[main] [ 2016-06-23 01:03:14.958 PDT ] [ClusterUtil.getCSSMode:660] Returned from native code
[main] [ 2016-06-23 01:03:14.958 PDT ] [ClusterUtil.getCSSMode:665] Returned from native code
[main] [ 2016-06-23 01:03:14.961 PDT ] [Instance.getSQLEngine:262] sysCreds.getUser()sys
[main] [ 2016-06-23 01:03:14.961 PDT ] [Instance.getSQLEngine:263] sysCred.getRole()sysdba
[main] [ 2016-06-23 01:03:14.961 PDT ] [Instance.getSQLEngine:264] SID PROD
[main] [ 2016-06-23 01:03:14.961 PDT ] [Instance.getSQLEngine:262] sysCreds.getUser()sys
[main] [ 2016-06-23 01:03:14.961 PDT ] [Instance.getSQLEngine:263] sysCred.getRole()sysdba
[main] [ 2016-06-23 01:03:14.962 PDT ] [Instance.getSQLEngine:264] SID PROD
[main] [ 2016-06-23 01:03:14.962 PDT ] [RACConvertStep.execute:193] STEP Result=Grid Infrastructure is either not configured or not running on nodes: ebsdb2