Database Instance 19c Crashes Intermittently

Oracle Database – Enterprise Edition – Version 19.3.0.0.0 to 19.6.0.0.0 [Release 19]
All Platforms

SYMPTOMS

+ This is an Oracle Database 19.3.0.0 on Microsoft Windows 2016 (64-bit). But this problem can occur on any platform for DB version from 19.3 to 19.6.

+ Intermittently, the instance crashes when it archives to the FRA which is configured on a SAN storage. But these are not necessary key factors either.

+ There is no good reason from the DB side for such a crash. There is no clue either from the DB logs why such crashes occurred….
2019-11-22T01:13:52.348399+00:00
ARC2 (PID:4572): Archived Log entry 1425 added for T-1.S-1438 ID 0x58ee09d8
LAD:1
2019-11-22T01:15:17.173560+00:00
Warning: VKTM detected a forward time drift.
Please see the VKTM trace file for more details:
..\trace\<sid>_vktm_796.trc
2019-11-22T01:25:33.718455+00:00
Starting ORACLE instance (normal) (OS id: 17956)
…

2019-11-25T12:00:51.401991+00:00
ARC3 (PID:18228): Archived Log entry 1704 added for T-1.S-1717 ID 0x58ee09d8
LAD:1
2019-11-25T12:16:59.974303+00:00
Warning: VKTM detected a forward time drift.
Please see the VKTM trace file for more details:
..\trace\<sid>_vktm_21108.trc
2019-11-25T12:20:47.426580+00:00
Starting ORACLE instance (normal) (OS id: 10268)
…

2019-12-10T06:45:34.380138+00:00
ARC3 (PID:24552): Archived Log entry 2918 added for T-1.S-2931 ID 0x58ee09d8
LAD:1
2019-12-10T06:52:06.575056+00:00
Resize operation completed for file# 3, old size 20869120K, new size
20879360K
2019-12-10T06:53:09.482879+00:00
Warning: VKTM detected a forward time drift.
Please see the VKTM trace file for more details:
..\trace\<sid>_vktm_24308.trc
2019-12-10T07:11:23.052682+00:00
Starting ORACLE instance (normal) (OS id: 19608)
…

2019-12-13T06:37:30.555846+00:00
ARC1 (PID:23204): Archived Log entry 3145 added for T-1.S-3158 ID 0x58ee09d8
LAD:1
2019-12-13T06:39:26.398601+00:00
<pdb_name>(3):TABLE SYS.WRI$_OPTSTAT_HISTHEAD_HISTORY: ADDED INTERVAL PARTITION
SYS_P921 (43811) VALUES LESS THAN (TO_DATE(‘ 2019-12-14 00:00:00’,
‘SYYYY-MM-DD HH24:MI:SS’, ‘NLS_CALENDAR=GREGORIAN’))
2019-12-13T06:48:17.595483+00:00
Warning: VKTM detected a forward time drift.
Please see the VKTM trace file for more details:
..\trace\<sid>_vktm_21892.trc
2019-12-13T06:56:24.219627+00:00
Starting ORACLE instance (normal) (OS id: 12156)
…

2019-12-17T11:12:04.311930+00:00
ARC1 (PID:22684): Archived Log entry 3416 added for T-1.S-3429 ID 0x58ee09d8
LAD:1
2019-12-17T11:16:07.008944+00:00
Warning: VKTM detected a forward time drift.
Please see the VKTM trace file for more details:
..\trace\<sid>_vktm_16824.trc
2019-12-17T11:18:25.927836+00:00
Starting ORACLE instance (normal) (OS id: 24632)
…

2019-12-19T16:49:17.207714+00:00
ALTER SYSTEM SET _disable_highres_ticks=FALSE SCOPE=SPFILE;
2019-12-19T16:49:17.238967+00:00
ALTER SYSTEM SET _timer_precision=500 SCOPE=SPFILE;
2019-12-19T16:49:20.817415+00:00
ALTER SYSTEM SET event=’10795 trace name context forever, level 2′
SCOPE=SPFILE;
…

2019-12-20T17:01:04.687429+00:00
NET (PID:25104): Archived Log entry 3627 added for T-1.S-3640 ID 0x58ee09d8
LAD:1
2019-12-20T17:33:01.355689+00:00
Resize operation completed for file# 3, old size 22558720K, new size
22568960K
2019-12-20T17:41:55.185792+00:00
Warning: VKTM detected a forward time drift.
Please see the VKTM trace file for more details:
..\trace\<sid>_vktm_12128.trc
2019-12-20T17:51:37.338467+00:00
Starting ORACLE instance (normal) (OS id: 9472)
…

_disable_highres_ticks = FALSE
_timer_precision = 500
event = “10795 trace name context forever, level 2”
…

2020-01-04T07:07:02.971105+00:00
ARC0 (PID:5404): Archived Log entry 4468 added for T-1.S-4481 ID 0x58ee09d8
LAD:1
2020-01-04T07:50:00.791072+00:00
Resize operation completed for file# 3, old size 24872960K, new size
24883200K
2020-01-04T08:00:13.906056+00:00
Thread 1 advanced to log sequence 4483 (LGWR switch)
Current log# 3 seq# 4483 mem# 0: ..\<SID>\REDO03A.LOG
Current log# 3 seq# 4483 mem# 1: ..\<SID>\REDO03B.LOG
2020-01-04T08:00:16.171956+00:00
ARC1 (PID:19276): Archived Log entry 4469 added for T-1.S-4482 ID 0x58ee09d8
LAD:1
2020-01-04T08:37:08.530619+00:00
Starting ORACLE instance (normal) (OS id: 19372)
…

2020-01-06T13:43:32.621640+00:00
ARC0 (PID:5500): Archived Log entry 4603 added for T-1.S-4616 ID 0x58ee09d8
LAD:1
2020-01-06T14:10:47.771835+00:00
Resize operation completed for file# 3, old size 25200640K, new size
25210880K
2020-01-06T14:50:42.867598+00:00
Starting ORACLE instance (normal) (OS id: 2988)
…

2020-01-06T17:09:17.660728+00:00
ARC1 (PID:22160): Archived Log entry 4606 added for T-1.S-4619 ID 0x58ee09d8
LAD:1
2020-01-06T17:10:07.791962+00:00
Starting ORACLE instance (normal) (OS id: 13320)
…

2020-01-10T17:01:01.363704+00:00
NET (PID:17724): Archived Log entry 4877 added for T-1.S-4890 ID 0x58ee09d8
LAD:1
2020-01-10T17:15:02.907458+00:00
Resize operation completed for file# 3, old size 25856000K, new size
25866240K
2020-01-10T18:25:26.947274+00:00
Starting ORACLE instance (normal) (OS id: 24016)
…

+ On Windows platforms, the ORACLE.exe keeps crashing with no errors in the alert log but in Windows event viewer you can always see the following exception:Exception code: 0xc0000409
Fault offset: 0x000000000fc7ff20

 + On Unix based platforms, the ORA-07445 [kwqbmspqueue] error is raised in the alert log file indicating such an overflow. Top call stack may include:.. <- kwqbmspqueue <- kwqbmspmain_switch <- kwqbmspmain <- kwqmnslv <- kwsbsmspm <- kwsbgcbkms <- ksvrdp_int <- ..

CHANGES

Upgrade to 19c

CAUSE

Bug 30777036 – 19c Database Instance Crashes Intermittently

Base Bug 29920804 Getting ora-7445 [kwqbmspqueue()+278] [sigsegv] every minute since upgraded from 18c to 19
 
This base bug affects DB version 19.3, 19.4, 19.5, 19.6 on any platform. So, it is not port specific. It can happen on any platform.

SOLUTION

Please apply the fix for Bug 29920804. This fix is included in 19.7.0.0.200414DBRU onward. For previous releases, please raise an One/Off Backport request.