ORA-15032,ORA-15040,ORA-15042
SCENARIO
In
one afternoon shift DBA received request to shutdown database for OS patching
and a junior DBA has shutdown database having grid infrastructure without
taking pre-requisite like listener running on database server, list of SID’s
running on database server, number of disk group’s present, name and path of
disks allocated to respective disk group, etc. Post OS patching while starting
database it resulted in ORA-15032, ORA-15040, ORA-15042 error pasted below.
ERROR
SQL> ALTER
DISKGROUP ALL MOUNT /* asm agent */
SQL> ALTER
DISKGROUP ALL ENABLE VOLUME ALL /* asm agent */
Thu Feb 16 17:10:51
2017
SQL> ALTER
DISKGROUP DATA MOUNT /* asm agent */
NOTE: cache
registered group DATA number=1 incarn=0x08884fc9
NOTE: cache began
mount (first) of group DATA number=1 incarn=0x08884fc9
NOTE: Loaded library:
/opt/oracle/extapi/64/asm/orcl/1/libasm.so
NOTE: Assigning
number (1,1) to disk (/dev/mapper/asm_disk01)
NOTE: start
heartbeating (grp 1)
kfdp_query(DATA): 3
kfdp_queryBg(): 3
NOTE: Assigning
number (1,2) to disk ()
kfdp_query(DATA): 4
kfdp_queryBg(): 4
NOTE: cache
dismounting (clean) group 1/0x08884FC9 (DATA)
NOTE: dbwr not being
msg'd to dismount
NOTE: lgwr not being
msg'd to dismount
NOTE: cache
dismounted group 1/0x08884FC9 (DATA)
NOTE: cache ending
mount (fail) of group DATA number=1 incarn=0x08884fc9
kfdp_dismount(): 5
kfdp_dismountBg(): 5
NOTE: De-assigning
number (1,1) from disk (/dev/mapper/asm_disk01)
NOTE: De-assigning
number (1,2) from disk ()
ERROR: diskgroup DATA
was not mounted
NOTE: cache deleting context for group DATA 1/143151049
ORA-15032: not all alterations performed
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk "2" is missing from group number "1"
ERROR: ALTER DISKGROUP DATA MOUNT /* asm agent */
Thu Feb 16 17:10:58 2017
ASM Health Checker found 1 new failures
CAUSE
ORA-15032,
ORA-15040, ORA-15042 is primarily caused due to duplicate disks exist so
the diskgroup cannot be mounted as Oracle does not know which disks to use to
mount the diskgroup.
SOLUTION
When
you try to start HAS in above
scenario, the +ASM instance will start without starting database instance.
[root@raeydb ~]# ps -ef | grep pmon
grid 5186 3034 0 04:09 pts/1 00:00:00 +ASM
Check the details of disk using below command:
SQL> set pages
40000 lines 120
col PATH for a30
select
DISK_NUMBER,MOUNT_STATUS,HEADER_STATUS,MODE_STATUS,STATE,PATH FROM V$ASM_DISK;SQL>
SQL>
DISK_NUMBER MOUNT_S HEADER_STATU MODE_ST STATE PATH
-------------------------------------------------------------------------------
DISK_NUMBER MOUNT_S HEADER_STATU MODE_ST STATE PATH
-------------------------------------------------------------------------------
0 IGNORED MEMBER ONLINE
NORMAL /dev/dm-1
2 CLOSED MEMBER
ONLINE NORMAL /dev/mapper/asm_disk01
1 IGNORED MEMBER ONLINE
NORMAL /dev/mapper/asm_disk02
IGNORED status indicates that disk is duplicate.
Now problem faced in this environment was all the disk listed above in
PATH column were valid hence to find out the appropriate disk, we used “blkid” command in Linux environment
which will list out only the appropriate disk which should be used by ASM instance.
Hence
simple way to make ASM not to refer the duplicate disk is change the ownership
of duplicate disk (disks not listing in output of blkid) to "root:root".
[root@raeydb ~]#
oracleasm querydisk -p /dev/dm-1
Device
"/dev/dm-1" is marked an ASM disk with the label "DISK3"
[root@raeydb ~]#
oracleasm querydisk -p /dev/dm-0
Device
"/dev/dm-0" is marked an ASM disk with the label "DISK2"
[root@raeydb ~]#
oracleasm querydisk -p /dev/mapper/asm_disk02
Device
"/dev/mapper/asm_disk02" is marked an ASM disk with the label
"DISK3"
[root@raeydb ~]#
oracleasm querydisk -p /dev/mapper/asm_disk01
Device
"/dev/mapper/asm_disk01" is marked an ASM disk with the label
"DISK2"
[root@raeydb ~]# ls -ltr
/dev/dm*
brw-rw---- 1 oracle
dba 253, 1 Feb 16 17:09 /dev/dm-1
brw-rw---- 1 oracle
dba 253, 3 Feb 16 17:09 /dev/dm-3
brw-rw---- 1
root root 253, 2 Feb 16 17:09 /dev/dm-2
brw-rw---- 1
root root 253, 0 Feb 16 17:09 /dev/dm-0
[root@raeydb ~]# ls
-ltr /dev/mapper/*
crw------- 1
root root 10, 60 Feb 16 17:09 /dev/mapper/control
brw-rw---- 1
root disk 253, 3 Feb 16 17:09 /dev/mapper/backup
brw-rw---- 1 oracle
dba 253,
1 Feb 16 17:09 /dev/mapper/asm_disk02
brw-rw---- 1 root disk 253,
2 Feb 16 17:09 /dev/mapper/archive
brw-rw---- 1 oracle
dba 253,
0 Feb 16 17:10 /dev/mapper/asm_disk01
[root@raeydb ~]# blkid
/dev/cciss/c0d0p5:
LABEL="/1" UUID="a58c8c31-c683-4aac-8dc3-f0ec6ee2142f"
TYPE="ext3"
/dev/cciss/c0d0p3:
LABEL="SW-cciss/c0d0p3" TYPE="swap"
/dev/cciss/c0d0p1:
LABEL="/boot" UUID="e957daff-f1d2-4789-bd4b-b8512e85d377"
TYPE="ext3"
/dev/cciss/c0d0p2:
LABEL="/tmp1" UUID="83309cd0-bbbd-4fc6-a1a4-1b340534f25f"
TYPE="ext3"
/dev/sdi:
LABEL="DISK2" TYPE="oracleasm"
/dev/sdh:
UUID="e6aa0175-bc09-4434-8928-378a94e1f0aa" SEC_TYPE="ext2"
TYPE="ext3"
/dev/sdg:
UUID="e7d473d7-485f-44c8-aa6a-7a7060fc1222" SEC_TYPE="ext2"
TYPE="ext3"
/dev/mapper/archive:
UUID="e7d473d7-485f-44c8-aa6a-7a7060fc1222" TYPE="ext3"
/dev/mapper/backup:
UUID="e6aa0175-bc09-4434-8928-378a94e1f0aa" TYPE="ext3"
/dev/sdj:
LABEL="DISK3" TYPE="oracleasm"
/dev/sdk:
UUID="e7d473d7-485f-44c8-aa6a-7a7060fc1222" SEC_TYPE="ext2"
TYPE="ext3"
/dev/sdl:
UUID="e6aa0175-bc09-4434-8928-378a94e1f0aa" SEC_TYPE="ext2"
TYPE="ext3"
/dev/sdm:
LABEL="DISK2" TYPE="oracleasm"
/dev/sdn:
LABEL="DISK3" TYPE="oracleasm"
/dev/mapper/asm_disk01: LABEL="DISK2"
TYPE="oracleasm"
/dev/mapper/asm_disk02: LABEL="DISK3"
TYPE="oracleasm"
/dev/sda: LABEL="DISK2"
TYPE="oracleasm"
/dev/sdb: LABEL="DISK3"
TYPE="oracleasm"
/dev/sdc:
UUID="e7d473d7-485f-44c8-aa6a-7a7060fc1222" SEC_TYPE="ext2"
TYPE="ext3"
/dev/sdd:
UUID="e6aa0175-bc09-4434-8928-378a94e1f0aa" SEC_TYPE="ext2"
TYPE="ext3"
/dev/sde:
LABEL="DISK2" TYPE="oracleasm"
/dev/sdf: LABEL="DISK3"
TYPE="oracleasm"
/dev/sdo:
UUID="e7d473d7-485f-44c8-aa6a-7a7060fc1222" SEC_TYPE="ext2"
TYPE="ext3"
/dev/sdp:
UUID="e6aa0175-bc09-4434-8928-378a94e1f0aa" SEC_TYPE="ext2"
TYPE="ext3"
[root@raeydb ~]#
[root@raeydb ~]# ls
-ltr /dev/dm*
brw-rw---- 1 oracle
dba 253, 1 Feb 16 17:09 /dev/dm-1
brw-rw---- 1 oracle
dba 253, 3 Feb 16 17:09 /dev/dm-3
brw-rw---- 1
root root 253, 2 Feb 16 17:09 /dev/dm-2
brw-rw---- 1
root root 253, 0 Feb 16 17:09 /dev/dm-0
After
changing permission to root:root for duplicate disks stop
HAS and start again. The issue will be resolved and grid infrastructure will
start.
[root@raeydb ~]#
[root@raeydb ~]#
chown root:root /dev/dm-1
[root@raeydb ~]#
[root@raeydb ~]# ls
-ltr /dev/dm*
brw-rw---- 1
root root 253, 1 Feb 16 17:09 /dev/dm-1
brw-rw---- 1 oracle
dba 253, 3 Feb 16 17:09 /dev/dm-3
brw-rw---- 1
root root 253, 2 Feb 16 17:09 /dev/dm-2
brw-rw---- 1
root root 253, 0 Feb 16 17:09 /dev/dm-0
[root@raeydb ~]#
ASM DISK DETAILS:
SQL> set pages
40000 lines 120
col PATH for a30
select
DISK_NUMBER,MOUNT_STATUS,HEADER_STATUS,MODE_STATUS,STATE,PATH FROM V$ASM_DISK;SQL>
SQL>
DISK_NUMBER MOUNT_S
HEADER_STATU MODE_ST STATE PATH
----------- -------
------------ ------- -------- -------------------------
2 CACHED MEMBER
ONLINE NORMAL /dev/mapper/asm_disk01
1 CACHED MEMBER
ONLINE NORMAL /dev/mapper/asm_disk02
Contents
of ASM Alert log file:
Thu Feb 16 17:34:57
2017
Starting ORACLE
instance (normal)
LICENSE_MAX_SESSION =
0
LICENSE_SESSIONS_WARNING
= 0
Picked latch-free SCN
scheme 3
Using
LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/11.2.0/grid/dbs/arch
Autotune of undo
retention is turned on.
IMODE=BR
ILAT =0
LICENSE_MAX_USERS = 0
SYS auditing is
disabled
Starting up:
Oracle Database 11g
Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Automatic
Storage Management option.
Using parameter
settings in server-side spfile /u01/app/11.2.0/grid/dbs/spfile+ASM.ora
System parameters
with non-default values:
large_pool_size = 12M
instance_type = "asm"
remote_login_passwordfile=
"EXCLUSIVE"
asm_diskstring = "/dev/dm*"
asm_diskstring = "/dev/mapper/asm_disk0*"
asm_diskgroups = ""
asm_power_limit = 1
diagnostic_dest = "/u01/app/oracle"
Thu Feb 16 17:34:57
2017
PMON started with
pid=2, OS id=29719
Thu Feb 16 17:34:57
2017
VKTM started with
pid=3, OS id=29721 at elevated priority
VKTM running at
(10)millisec precision with DBRM quantum (100)ms
Thu Feb 16 17:34:57
2017
GEN0 started with
pid=4, OS id=29725
Thu Feb 16 17:34:57
2017
DIAG started with
pid=5, OS id=29727
Thu Feb 16 17:34:57
2017
PSP0 started with
pid=6, OS id=29729
Thu Feb 16 17:34:57
2017
DIA0 started with
pid=7, OS id=29731
Thu Feb 16 17:34:57
2017
MMAN started with
pid=8, OS id=29733
Thu Feb 16 17:34:57
2017
DBW0 started with
pid=9, OS id=29735
Thu Feb 16 17:34:58
2017
LGWR started with
pid=10, OS id=29737
Thu Feb 16 17:34:58
2017
CKPT started with
pid=11, OS id=29739
Thu Feb 16 17:34:58
2017
SMON started with
pid=12, OS id=29741
Thu Feb 16 17:34:58
2017
RBAL started with
pid=13, OS id=29743
Thu Feb 16 17:34:58
2017
GMON started with
pid=14, OS id=29745
Thu Feb 16 17:34:58
2017
MMON started with
pid=15, OS id=29747
Thu Feb 16 17:34:58
2017
MMNL started with
pid=16, OS id=29749
ORACLE_BASE from
environment = /u01/app/oracle
Thu Feb 16 17:34:58
2017
SQL> ALTER
DISKGROUP ALL MOUNT /* asm agent */
SQL> ALTER
DISKGROUP ALL ENABLE VOLUME ALL /* asm agent */
Thu Feb 16 17:34:58
2017
SQL> ALTER
DISKGROUP DATA MOUNT /* asm agent */
NOTE: cache
registered group DATA number=1 incarn=0x5cfa859b
NOTE: cache began
mount (first) of group DATA number=1 incarn=0x5cfa859b
NOTE: Loaded library:
/opt/oracle/extapi/64/asm/orcl/1/libasm.so
NOTE: Assigning
number (1,2) to disk (/dev/mapper/asm_disk02)
NOTE: Assigning
number (1,1) to disk (/dev/mapper/asm_disk01)
NOTE: start
heartbeating (grp 1)
kfdp_query(DATA): 3
kfdp_queryBg(): 3
NOTE: cache opening
disk 1 of grp 1: DATA_0001 path:/dev/mapper/asm_disk01
NOTE: cache opening
disk 2 of grp 1: DATA_0002 path:/dev/mapper/asm_disk02
NOTE: F1X0 found on
disk 2 au 8289 fcn 0.153533
NOTE: cache mounting
(first) external redundancy group 1/0x5CFA859B (DATA)
NOTE: cache recovered
group 1 to fcn 0.267933
NOTE: LGWR attempting
to mount thread 1 for diskgroup 1 (DATA)
NOTE: LGWR found
thread 1 closed at ABA 29.6432
NOTE: LGWR mounted
thread 1 for diskgroup 1 (DATA)
NOTE: LGWR opening
thread 1 at fcn 0.267933 ABA 30.6433
NOTE: cache mounting
group 1/0x5CFA859B (DATA) succeeded
NOTE: cache ending
mount (success) of group DATA number=1 incarn=0x5cfa859b
kfdp_query(DATA): 4
kfdp_queryBg(): 4
NOTE: Instance
updated compatible.asm to 11.2.0.0.0 for grp 1
SUCCESS: diskgroup
DATA was mounted
SUCCESS: ALTER
DISKGROUP DATA MOUNT /* asm agent */
NOTE: diskgroup
resource ora.DATA.dg is online
Thu Feb 16 17:35:16
2017
Starting
background process ASMB
Thu Feb 16
17:35:16 2017
ASMB started
with pid=18, OS id=29822
DISCLAIMER
The views expressed here are subjected to
www.RAEY.in business environment. Suggestions and comments will be highly
appreciated.
Comments
Post a Comment