Skip to content

Posts tagged ‘RAC’

RCONFIG – Cannot get SQLEngine

Problem: convert to rac from single instance using rconfig gives error

$ORACLE_HOME/bin/rconfig ConvertToRAC_AdminManaged.xml

oracle.sysman.assistants.util.sqlEngine.SQLFatalErrorException: Cannot get SQLEngine
Operation Failed. Refer logs at /opt/oracle/app/itdoracle/12c/cfgtoollogs/rconfig/rconfig_09_07_21_15_55_33.log for more details.

Cause: ORACLE_HOME was not set properly in xml file

          <n:SourceDBHome>/opt/oracle/ora12201</n:SourceDBHome>
          <n:TargetDBHome>/opt/oracle/ora12c</n:TargetDBHome>

Solution: source and target were on the same server so set it accordingly to new server

          <n:SourceDBHome>/opt/oracle/ora12201</n:SourceDBHome>
          <n:TargetDBHome>/opt/oracle/ora12201</n:TargetDBHome>

How To Configure SSH for a RAC Installation

GOAL

This document will explain how to configure SSH, which is required to run a RAC installation. Following the instructions in the installation guide are also correct, but sometimes this will not work, although the reason for that isn’t clear. Therefore after some investigation it seems to be that the steps below will work too.

Starting with 11gR2 the Oracle Universal Installer the SSH can be setup automatically using the  ‘SSH Connectivity’ button.

 

SOLUTION

To configure SSH you need to perform the following steps on each node in the cluster.

$ cd $HOME
$ mkdir .ssh
$ chmod 700 .ssh
$ cd .ssh
$ ssh-keygen -t rsa

Now accept the default location for the key file
Enter and confirm a passphrase. (you can also press enter twice).

$ ssh-keygen -t dsa

At the prompts, accept the default location for the key file (press Enter).
Then press “Enter” twice to accept no passphrase.

Please note: SSH with passphrase is not supported for Oracle Clusterware 11g release 2 and later releases.

If you provide a passphrase for pre 11.2 release, then you need to do 2 addition steps.

$ exec /usr/bin/ssh-agent $SHELL
$ /usr/bin/ssh-add

These statements will inform the ssh agent to add the keys to the shell used.

 

$ cat *.pub >> authorized_keys.<nodeX> (nodeX could be the nodename to differentiate files later)

Now do the same steps on the other nodes in the cluster.
When all those steps are done on the other nodes, start to copy the authorized_keys.<nodeX> to all the nodes into $HOME/.ssh/

For example if you have 4 nodes you will have after the copy in the .ssh 4 files with the name authorized_keys.<nodeX>
Then on EACH node continue the configuration of SSH by doing the following:

$ cd $HOME/.ssh
$ cat *.node* >> authorized_keys
$ chmod 600 authorized_keys

NOTE: ALL public keys must appear in ALL authorized_keys files, INCLUDING the LOCAL public key for each node.

To test that everything is working correct now execute the commands

$ ssh <hostnameX> date

So on example in a 4 node environment:

$ ssh node1 date
$ ssh node2 date
$ ssh node3 date
$ ssh node4 date

Repeat this 4 times on each node, including ssh back to the node itself. The nodeX is the hostname of the node.

The first time you will be asked to add the node to a file called ‘known_hosts’ this is correct and answer the question with ‘yes’. After that when correctly configured you must be able to get the date returned and you will not be prompted for a password.

 

Please note, ssh will not allow passwordless access if permissions on the home directory of the account you are using  allow write access for everyone.

You will also see permission denied error when the permissions on $HOME are 777 or 775. $HOME should have permission 700 or 755.

 

Disable banner (/etc/banner) on all cluster nodes when you

  • run clusterverify (cluvfy, runcluvfy)
  • install software
  • patch the system

Please work with your System Administrator or contact your Operating System support in case you still have problems setting up ssh.

Rconfig fails: Grid Infrastructure is Either Not Configured or Not Running on Nodes

SYMPTOMS

Running rconfig with verify option fails with below error –

[oraprod@ebsdb1 upgrade]$ rconfig ConvertToRAC_AdminManaged_verify.xml
<?xml version=”1.0″ ?>
<RConfig version=”1.1″ >
<ConvertToRAC>
<Convert>
<Response>
<Result code=”1″ >
Operation Failed
</Result>
<ErrorDetails>
Grid Infrastructure is either not configured or not running on nodes: ebsdb2
</ErrorDetails>
</Response>
</Convert>
</ConvertToRAC></RConfig>

However, cluster services are online on all nodes

[grid@ebsdb1 ~]$ crsctl check cluster -all
**************************************************************
ebsdb1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
ebsdb2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

 

Trace:

[Worker 2] [ 2016-06-23 01:03:14.952 PDT ] [UnixSystem.checkRemoteExecutionSetup:2328] checkRemoteExecutionSetup:: Error checking user equivalence using Secured Shell ‘/usr/bin/ssh’
[Worker 2] [ 2016-06-23 01:03:14.952 PDT ] [UnixSystem.pathExists:971] PRKC-1044 : Failed to check remote command execution setup for node ebsdb2 using shells /usr/bin/ssh and /usr/bin/rsh
File “/usr/bin/rsh” does not exist on node “ebsdb2”
No RSA host key is known for ebsdb2 and you have requested strict checking.Host key verification failed.
oracle.ops.mgmt.nativesystem.UnixSystem.checkRemoteExecutionSetup(UnixSystem.java:2377)
oracle.ops.mgmt.nativesystem.UnixSystem.getRemoteShellCmd(UnixSystem.java:2098)
oracle.ops.mgmt.nativesystem.UnixSystem.pathExists(UnixSystem.java:966)
oracle.ops.mgmt.command.file.PathExistCommand.execute(PathExistCommand.java:69)
oracle.ops.mgmt.cluster.ClusterConfig$ExecuteCommand.run(ClusterConfig.java:3248)
java.lang.Thread.run(Thread.java:682)
[Worker 2] [ 2016-06-23 01:03:14.953 PDT ] [PathExistCommand.execute:74] REMOTEDIREXCEPTION: PRKC-1044 : Failed to check remote command execution setup for node ebsdb2 using shells /usr/bin/ssh and /usr/bin/rsh
File “/usr/bin/rsh” does not exist on node “ebsdb2”
No RSA host key is known for ebsdb2 and you have requested strict checking.Host key verification failed.
[Worker 2] [ 2016-06-23 01:03:14.953 PDT ] [ClusterConfig$ExecuteCommand.returnCommandToClient:3216] returnCommandToClient; fillCount=1 is full=false
[Worker 2] [ 2016-06-23 01:03:14.953 PDT ] [Semaphore.acquire:109] SyncBufferEmpty:Acquire called by thread Worker 2 m_count=99
[Worker 2] [ 2016-06-23 01:03:14.954 PDT ] [Semaphore.release:85] SyncBufferFull:Release called by thread Worker 2 m_count=2
[Worker 2] [ 2016-06-23 01:03:14.954 PDT ] [ClientResource.getListener:157] Calling getListener
[Worker 2] [ 2016-06-23 01:03:14.954 PDT ] [ClusterConfig$ExecuteCommand.run:3311] Owner Thread name of the blocking Semaphore main
[Worker 2] [ 2016-06-23 01:03:14.955 PDT ] [ClusterConfig$ExecuteCommand.run:3319] Obtained Semaphore
[Worker 2] [ 2016-06-23 01:03:14.955 PDT ] [Semaphore.release:85] ClientResource Constructor:Blocking Semaphore owned by main:Release called by thread Worker 2 m_count=1
[Worker 2] [ 2016-06-23 01:03:14.955 PDT ] [ClusterConfig$ExecuteCommand.run:3334] Released Semaphore by worker=Worker 2
[main] [ 2016-06-23 01:03:14.955 PDT ] [ClusterConfig.block:613] block acquired semnum=1
[main] [ 2016-06-23 01:03:14.955 PDT ] [ClusterConfig.submit:562] Out of block
[Worker 2] [ 2016-06-23 01:03:14.955 PDT ] [Semaphore.acquire:109] SyncBufferFull:Acquire called by thread Worker 2 m_count=0
[main] [ 2016-06-23 01:03:14.956 PDT ] [CommandBuffer.checkCommandStatus:83] CommandBuffer:checkCommandStatus returning false, command number= 3 thread name= Thread[main,5,main]
[main] [ 2016-06-23 01:03:14.956 PDT ] [ClusterConfig.submit:579] status=false
[main] [ 2016-06-23 01:03:14.956 PDT ] [ClusterException.setErrorMessage:138] ClusterException.setErrorMessage: commands[0]=oracle.ops.mgmt.command.file.PathExistCommand@6d588325
[main] [ 2016-06-23 01:03:14.956 PDT ] [ClusterException.setErrorMessage:138] ClusterException.setErrorMessage: commands[1]=oracle.ops.mgmt.command.file.PathExistCommand@4e0a2a38
[main] [ 2016-06-23 01:03:14.956 PDT ] [ClusterException.setErrorMessage:165] ClusterException.setErrorMessage: errString is ‘PRKC-1044 : Failed to check remote command execution setup for node ebsdb2 using shells /usr/bin/ssh and /usr/bin/rsh
File “/usr/bin/rsh” does not exist on node “ebsdb2”
No RSA host key is known for ebsdb2 and you have requested strict checking.Host key verification failed.’
[main] [ 2016-06-23 01:03:14.956 PDT ] [ClusterConfig.destroy:457] destroying resources for client thread Thread[main,5,main]
[main] [ 2016-06-23 01:03:14.957 PDT ] [Utils.getString:175] ==========Str is ebsdb2
[main] [ 2016-06-23 01:03:14.957 PDT ] [OracleHome.getNodeNames:406] exception checking oracle home existence on nodesPRKC-PRKC-1027 : Error checking existence of file /u01/oracle/PROD/12.1.0 on ebsdb2 : <null>
[main] [ 2016-06-23 01:03:14.957 PDT ] [OracleHome.getNodeNames:434] no of nodes returned1
[main] [ 2016-06-23 01:03:14.957 PDT ] [ClusterCheck.checkCluster:219] OUI Node Names Length=1
[main] [ 2016-06-23 01:03:14.958 PDT ] [ClusterCheck.checkCluster:222] OUI Node List Length=1
[main] [ 2016-06-23 01:03:14.958 PDT ] [ClusterUtil.getCSSMode:660] Returned from native code
[main] [ 2016-06-23 01:03:14.958 PDT ] [ClusterUtil.getCSSMode:665] Returned from native code
[main] [ 2016-06-23 01:03:14.961 PDT ] [Instance.getSQLEngine:262] sysCreds.getUser()sys
[main] [ 2016-06-23 01:03:14.961 PDT ] [Instance.getSQLEngine:263] sysCred.getRole()sysdba
[main] [ 2016-06-23 01:03:14.961 PDT ] [Instance.getSQLEngine:264] SID PROD
[main] [ 2016-06-23 01:03:14.961 PDT ] [Instance.getSQLEngine:262] sysCreds.getUser()sys
[main] [ 2016-06-23 01:03:14.961 PDT ] [Instance.getSQLEngine:263] sysCred.getRole()sysdba
[main] [ 2016-06-23 01:03:14.962 PDT ] [Instance.getSQLEngine:264] SID PROD
[main] [ 2016-06-23 01:03:14.962 PDT ] [RACConvertStep.execute:193] STEP Result=Grid Infrastructure is either not configured or not running on nodes: ebsdb2

 

CAUSE

Passwordless ssh to remote node failing

SOLUTION

Passwordless-ssh connection has to be configured between all nodes, refer to:

https://wikidba.com/2020/04/24/how-to-configure-ssh-for-a-rac-installation/

 

Once done,  retry rconfig

%d bloggers like this: