ASM instance: ORA-15032 and ORA-15063 errors occur after shutdown and startup

Background: Using ASM with asmlib by multipath software

The Story:

Restarted ASM instance for the first time:

[loki:oracle@+ASM] /etc/init.d >su -
Password:
[root@loki ~]# /etc/init.d/oracleasm stop
Unmounting ASMlib driver filesystem: [ OK ]
Unloading module "oracleasm": [ OK ]
[root@loki ~]# /etc/init.d/oracleasm start
Loading module "oracleasm": [ OK ]
Mounting ASMlib driver filesystem: [ OK ]
Scanning system for ASM disks: [ OK ]
[root@loki ~]# /etc/init.d/oracleasm scandisks
Scanning system for ASM disks: [ OK ]
[root@loki ~]# /etc/init.d/oracleasm listdisks
PRDWH_DATA_1
PRDWH_DATA_2
[root@loki ~]# logout

We got the following Error:

[loki:oracle@+ASM] /etc/init.d >sqlplus "/ as sysdba"

SQL*Plus: Release 10.2.0.3.0 - Production on Mon Feb 26 13:47:13 2007
Copyright (c) 1982, 2006, Oracle. All Rights Reserved.
Connected to an idle instance.
SQL> startup
ASM instance started
Total System Global Area 130023424 bytes
Fixed Size 2071000 bytes
Variable Size 102786600 bytes
ASM Cache 25165824 bytes
ORA-15032: not all alterations performed
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DATA"

init.ora

asm_diskstring = /dev/dm*

alert.log

SQL> ALTER DISKGROUP ALL MOUNT
Mon Feb 26 14:18:46 2007
NOTE: cache registered group DATA number=1 incarn=0x7813374a
Mon Feb 26 14:18:46 2007
Loaded ASM Library - Generic Linux, version 2.0.2 (KABI_V2) library for asmlib interface
Mon Feb 26 14:18:46 2007
ORA-15186: ASMLIB error function = [asm_open], error = [1], mesg = [Operation not p
ermitted]
Mon Feb 26 14:18:46 2007
ORA-15186: ASMLIB error function = [asm_open], error = [1], mesg = [Operation not permitted]
Mon Feb 26 14:18:46 2007
ERROR: no PST quorum in group 1: required 2, found 0

It seemed that there is something wrongly set in the way they are attaching these disks to the OS
ASMLIB is running under root user, so it shouldn't have any access issue
however, there was an access issue "error function = [asm_open], error = [1], mesg = [Operation not permitted]"

We found out how to TROUBLESHOOT asmlib problems by note 353423.1, so we used kfod

[loki:oracle] /software/oracle/ASM10gR2/bin >kfod asm_diskstring='ORCL:*' disks=all
clsc_ginit: CLSD initialization failed, rc 2, ecode 7
KFOD-00301: file not found; arguments: [2] [kgxgncin]
LEM-00031: file not found; arguments: [lempgmh] [lmserr]
LEM-00033: file not found; arguments: [lempgfm] [Couldn't open message file]
LEM-00031: file not found; arguments: [lempgmh] [lmserr]
LEM-00033: file not found; arguments: [lempgfm] [Couldn't open message file]
LEM-00031: file not found; arguments: [lempgmh] [lmserr]
LEM-00033: file not found; arguments: [lempgfm] [Couldn't open message file]
LEM-00031: file not found; arguments: [lempgmh] [lmserr]

the problem solved after setting ORACLEASM_SCANORDER on file /etc/sysconfig/oracleasm to dm

oracleasm:

ORACLEASM_SCANORDER=dm

Good output looks like this:

[loki:oracle@PRDWH] /software/oracle >kfod asm_diskstring='ORCL:*' disks=all
-----------------------------------------------------------
Disk Size Path ===========================================================
1: 98304 Mb ORCL:PRDWH_DATA_1 2: 98304 Mb ORCL:PRDWH_DATA_2
-----------------------------------------------------------
ORACLE_SID ORACLE_HOME ===========================================================
+ASM /software/oracle/ASM10gR2

Related notes:

Note 353423.1 - Unable To Mount Or Drop A Diskgroup, Fails With Ora-15032 And Ora-15063
Note 398622.1 - ORA-15186: ASMLIB error function = [asm_open], error = [1], mesg = [Operation not permitted]