The clock on host is not synchronous with the mean cluster time (RAC 11g)

Symptoms
Cluster alert log on one of the nodes shows the following message

[ctssd(804)]CRS-2409:The clock on host or-21 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.

OR

[ctssd(9534)]CRS-2409:The clock on host   is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.


Cause
This is not an error. ctssd is reporting that there is a time difference and it is not doing anything about it as it is running in observer mode.

Solution
The Cluster Time Synchronization Service (CTSS) is one of the new processes in the Oracle Clustware 11gR2 layer , the CTSS is installed during the GRID infrastructure installation.

CTSS Daemon is responsible for the time management within the cluster. It makes sure each node in the cluster is using the same time. First, CTSS will determine if a Network Time Protocol (NTP) daemon is running in the system and if NTP is not running, then CTSS will run in active mode and perform the same task as the NTP daemon. If NTP daemon is running, CTSS will be started in observer mode.

- We need to check if there is a time difference on the nodes.
- We need to make sure ntp is running properly and with -x option

To see that the clock is synchronized properly using Network Time Protocol(NTP) on all servers, run the below

D:\app\11.2.0\grid\BIN>cluvfy comp clocksync -n all -verbose > d:\temp\cluvfy20120411TimeSyc.txt

Verifying Clock Synchronization across the cluster nodes

Checking if Clusterware is installed on all nodes...
Check of Clusterware install passed

Checking if CTSS Resource is running on all nodes...
Check: CTSS Resource running on all nodes
  Node Name                             Status                 
  ------------------------------------  ------------------------
or-22                         passed                 
or-21                         passed                 
Result: CTSS resource check passed

Querying CTSS for time offset on all nodes...
Result: Query of CTSS for time offset passed

Check CTSS state started...
Check: CTSS state
  Node Name                             State                  
  ------------------------------------  ------------------------
  or-22                         Observer               
  or-21                         Observer               
CTSS is in Observer state. Switching over to clock synchronization checks using NTP
Starting Clock synchronization checks using Network Time Protocol(NTP)...
Checking daemon liveness...
Check: Liveness for "W32Time"
  Node Name                             Running?               
  ------------------------------------  ------------------------
  or-22                         yes                    
  or-21                         yes                    
Result: Liveness check passed for "W32Time"
Check for NTP daemon or service alive passed on all nodes

Result: Clock synchronization check using Network Time Protocol(NTP) passed
Oracle Cluster Time Synchronization Services check passed
Verification of Clock Synchronization across the cluster nodes was successful.

The CTSS will generate a CRS-2409 error if the clock difference is greater than 1 ms between this node and the reference node in the cluster.
This is shown even though CTSS is in observer mode and clock synchronization is being done externally (e.g. ntp). As ntp is setup correctly as shown in above cluvfy output you can ignore this warning.

Note 1:You can check the Windows Time Service related info by using following OS command
w32tm /monitor
Note 2:
There is a general requirement for Oracle RAC that the time on all the nodes be the same.  If the Windows Time Service is being used, it MUST be configured to prevent the time from being adjusted backwards. Perform the following steps to ensure the time is NOT adjusted backwards using Windows Time Service perform the following steps on each cluster node

1. Open a command prompt (as the Admin user) and type 'regedit'.
2. Within the registry editor locate the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Config key.
3. Set the value for MaxNegPhaseCorrection to 0 and exit the registry editor.

4. Open a command prompt and execute the following to put the change into effect:
      cmd> W32tm /config /update

Details can be found at
http://support.microsoft.com/kb/884776
http://support.microsoft.com/kb/816043

Ref:1135337.1, 1311163.1