My experiences with ocrmirror, voting discs and stretched clustering by esheppardvc ()

If you are about to transmigrate to Prophet RAC, there is an important thing to cognize about the OCRmirror. Equally far as I cognise, this is documented nowhere in any Prophesier doctor. This station speaks about the failover belongings of the OCRmirror.

We desire high accessibility, so we postulate to attend RAC ( first fault, the utilization of the word `` demand ''. That is what Prophet sayed them, but anyhow that Holds another station ). So allow 's foresee 2 nodes. Nice, we hap to hold 2 server suites, so this is first-class for high availableness: one waiter in each room.

Therein scenario, the thought automatically accost protect you against failure of one of the computer suites. That Holds what everyone turns bent desire in the terminal.

The first job stillly is: where make we setted the storage? Belike you will be able to convert the direction to purchase a 2d SAN. You desire redundancy after all, is n't it?

Good word: Prophet sayed that ASM can be utilise to mirror the data. They tell that ASM is great and it works our jobs. Beep! Incorrect again. Old statement is true and false. ASM is great so, but it makes n't lick the storage jobs. Lotting the SANs over the two waiter suites is O.K., and ASM will mirror the database-data, but the clump ( Prophesier CRS ) passes to hold a register and a balloting saucer too. These can not be mirrored by ASM. The clump must foremost be upwardly before ASM can be gotten. So where make we lay these?

Luckiliy Prophesier10.2 furnishes the option to previse multiple vote discs. One on each san? Incorrect again. You necessitate to hold an uneven figure of vote discs, and a node can simply proceed when it can see a bulk of the ballot saucers. So placing two ballot saucers on SAN1 and the 3rd vote disc on SAN2 does SAN1 individual point of failure for the whole bunch. And belike you ca n't convert the direction to purchase a tertiary SAN and construct a 3rd waiter room.

To lick his, job, you will take a 3rd location anyways to store the 3rd vote saucer. Fortunately it justly need to store the ballot saucer, and the balloting disc is not accessed rattlingly intensively. So you may utilize a simple PC with a disc that you demonstrate undere.g. fiber or iscsi or even NFS. This tertiary waiter should be setted on a location independent from that what you desire to protect yourself against. So if you desire to protect against failure of one of the computer suites, you will need to place the pc/machine outside of both suites. Too make a point that it holds web connexion to each waiter room without passing through the other room.

This licks the job of the balloting saucers, but what about the OCR? Goodly rattlingly simple, Seer sayed that since10.2 you can hold an OCRmirror as a transcript of the OCR. So permit 's pose the OCR on SAN1 and the OCRmirror on SAN2. Simple is n't it? Beep! Incorrect again.

I recently maked the trial again at a client site with two sans. One OCR + 2 vote discs on SAN1, OCRmirror and voting disc 3 on SAN2. Pulling out the fiber connectednesses to SAN2, and supposition what passes. If you desire the elaborated logfile output, I can direct that to you, but this is the sum-up:

[Error: Irreparable invalid markup ('') in entry. Owner must fix manually. Raw contents below.]

If you are about to transmigrate to Prophet RAC, there is an important thing to cognize about the OCRmirror. Equally far as I cognise, this is documented nowhere in any Prophesier doctor. This station speaks about the failover belongings of the OCRmirror. 

 The following is the traditional mentation way of the higher ICT direction: 
 We desire high accessibility, so we postulate to attend RAC ( first fault, the utilization of the word `` demand ''. That is what Prophet sayed them, but anyhow that Holds another station ). So allow 's foresee 2 nodes. Nice, we hap to hold 2 server suites, so this is first-class for high availableness: one waiter in each room. 
 Therein scenario, the thought automatically accost protect you against failure of one of the computer suites. That Holds what everyone turns bent desire in the terminal. 
 The first job stillly is: where make we setted the storage? Belike you will be able to convert the direction to purchase a 2d SAN. You desire redundancy after all, is n't it? 
 Good word: Prophet sayed that ASM can be utilise to mirror the data. They tell that ASM is great and it works our jobs. Beep! Incorrect again. Old statement is true and false. ASM is great so, but it makes n't lick the storage jobs. Lotting the SANs over the two waiter suites is O.K., and ASM will mirror the database-data, but the clump ( Prophesier CRS ) passes to hold a register and a balloting saucer too. These can not be mirrored by ASM. The clump must foremost be upwardly before ASM can be gotten. So where make we lay these? 
 Luckiliy Prophesier10.2 furnishes the option to previse multiple vote discs. One on each san? Incorrect again. You necessitate to hold an uneven figure of vote discs, and a node can simply proceed when it can see a bulk of the ballot saucers. So placing two ballot saucers on SAN1 and the 3rd vote disc on SAN2 does SAN1 individual point of failure for the whole bunch. And belike you ca n't convert the direction to purchase a tertiary SAN and construct a 3rd waiter room. 
 To lick his, job, you will take a 3rd location anyways to store the 3rd vote saucer. Fortunately it justly need to store the ballot saucer, and the balloting disc is not accessed rattlingly intensively. So you may utilize a simple PC with a disc that you demonstrate undere.g. fiber or iscsi or even NFS. This tertiary waiter should be setted on a location independent from that what you desire to protect yourself against. So if you desire to protect against failure of one of the computer suites, you will need to place the pc/machine outside of both suites. Too make a point that it holds web connexion to each waiter room without passing through the other room. 
 This licks the job of the balloting saucers, but what about the OCR? Goodly rattlingly simple, Seer sayed that since10.2 you can hold an OCRmirror as a transcript of the OCR. So permit 's pose the OCR on SAN1 and the OCRmirror on SAN2. Simple is n't it? Beep! Incorrect again. 
 I recently maked the trial again at a client site with two sans. One OCR + 2 vote discs on SAN1, OCRmirror and voting disc 3 on SAN2. Pulling out the fiber connectednesses to SAN2, and supposition what passes. If you desire the elaborated logfile output, I can direct that to you, but this is the sum-up: 
 January 23 11: 07: 18 & lt; hostname & gt; 
 meat: lpfc 0000: 0a:00.0: 3: 1305 Nexus Down Event x2 haved Information: x2 x20 x110 


It turns out that all Io is barred. This depends on OS and package ( I applied Linux x86_64 + multipathd + device plotter in our instance ). Yet, it turns out that the package queues all Io during 10 proceedings ( 60 retries of 10 seconds ) then gives upwards. Notwithstanding during these 10 proceedings, a simple DD bidding on the neglected lun bents. It hangs so hard that even CTRL-C can not disrupt this. The same is true for all database admission that necessitates physical Io to these luns. 
 CSS is really robust, and produces messages really presently, stating that the tertiary vote saucer hangs ( +/- instantly after the job, so it makes n't hang for 10 transactions. One fillip point for CSS! ). 
 [ CSSD ] 2008-01-23 11: 08:59.492 [ 1168148800 ] & gt; WARNING: clssnmDiskPMT: voting device bent at 500.000000atal, expiry in 99480 meter, saucer ( 1//dev/raw/raw22 ) 
 In my instance, the node is not boot, CSS makes not evict it from the clump because it still sees a bulk of the vote saucers ( the 2 on SAN1 which is still available ). 
 [ CSSD ] 2008-01-23 11: 10:38.978 [ 1168148800 ] & gt; Hint: clssnmDiskPMT: stale disc ( 200010 meter ) ( 1//dev/raw/raw22 ) 
 All other dictations that access the neglected lun knack also and ctrl-c makes n't aid: ocrcheck, srvctl MBD service,... Now this is an OS issue exterior of the orbit of Prophet. If an uninterruptable Io call makes not return, the procedure turns bent be blockaded. ( Comment: the 60 retries look to be hard coded in the multipathd package, but the interval can be posed to 1 2d, ensuing in a bent of 1 min alternatively of 10 transactions ). Besides, if employing this multipath package, you will take to diminish the interval, because you can not afford your client applications to hang for 10 transactions. 
 Stillly 10 proceedings afterward, the multipath package gives au fait queuing the Io ( var/log/messages states: January 23 11: 17: 36 & lt; hostname & gt; multipathd: & lt; lunname & gt;: Disable queueing ) and returns an Io mistake to the procedure. As a outcome all dictations are unblocked, and respond on the Io faults. It turns out now that all bunch functionality that necessitates the OCR is not working anymore ( remember, the ocrmirror is unavailable, the ocr is still available ). The most surprising thing passes in the CRSD logfile. After a piece it gets: 
 2008-01-23 11: 22:13.768: [ CRSD ] [ 1518532928 ] [ Affright ] 0Exception caught at cppStart 
 2008-01-23 11: 22:13.768: [ CRSD ] [ 1518532928 ] [ Terror ] 0cluinfo ( memberid ) neglected for & lt; hostname & gt; 
 ( File:caa_Cluster.cpp, line: 115 
 and it dumps with nucleus. It makes n't re-start. Every ocr bid upshot in: 
 [ client ( 20388 ) ] CRS-1011: OCR can not ascertain that the OCR content comprises the up-to-the-minute updates. Items in /opt/oracle/crs/log/ & lt; hostname & gt;/client/css135.log. 
 The only mode to get out is to boot the node. So we boot ( SAN2 is still unpluged ). This intends that OCR and VD1 and VD2 are available, OCRmirror and VD3 are not yet available. And supposes what, CRS doesn not commence anymore. We see: 
 January 23 11: 45: 02 & lt; hostname & gt; lumberman: Bunch Ready Services attend to dependences. Nosologies in/tmp/crsctl.6599 
 and 
 Prophet Database 11g CRS Release10.2.0.3.0 Production Copyright 1996, 2006 Seer. All rights reserved. 2008-01-23 11: 38:33.078: [ OCROSD ] [ 2897458400 ] utopen: 6m ': neglected in stat OCR file/disk /dev/raw/raw91, errno=2, os mistake string=No such file or directory 2008-01-23 11: 38:33.081: [ OCRRAW ] [ 2897458400 ] proprioini: saucer 0 ( /dev/raw/raw101 ) makes n't hold decent votings ( 1, 2 ) 2008-01-23 11: 38:33.081: [ OCRRAW ] [ 2897458400 ] proprinit: Could not open natural device 2008-01-23 11: 38:33.081: [ default ] [ 2897458400 ] a_init: 7!: Backend init unsuccessful: [ 26 ] 2008-01-23 11: 38:33.081: [ CSSCLNT ] [ 2897458400 ] clsssinit: Unable to access OCR device in OCRinit.PROC-26: Fault while accessing the physical storage Justly after doing the lun comprising OCRmirror available again, it gets ( reinsert the cablegrams and reissuing the natural binding dictations ). 
 This trial proves that the OCRmirror is not a failover for the OCR. As a matter of fact, by holding an OCRmirror, you are less available, because if any of the OCR or OCRmirror neglects, your clump wo n't do normally anymore. So what is the exercise of the OCRmirror so? Goodly, it can be utilized to restore an OCR if that would get corrupt in some manner ( logical corruptness,e.g. write naughts on that with DD, or a lun that is accidentally destruct ). It is even possible to make this online. 
 It is nevertheless really important to realise that the faults on the OCR accession can be rattlingly different, dependent on the job ( physical job, logical corruptness, OS device constellation,... ) In some instances I presume the node can go a failure of OCR and OCRmirror, and a mending can be maked online. Yet there are examples where inaccessibility of one of them makes job on the local node ( as the instance I drew above ). For this ground I prefer to lay OCR and OCRmirror on the same SAN, optionally on different raid sets ( say a raid set gets corrupt or is accidentally destruct... ). 
 Bottom lines: Exploitation RAC for protection against site failure is not straightforward. As a issue the whole thought of `` stretched clumps '' with mis in-between the nodes is capable to the same jobs. Look at what haps just in case of site failure: 
<ul><li><div> In each mirrored ASM diskgroup, one mirror will be took when it is physically not accessible anymore. Nonetheless, subsequently this asks manual intercession to first killing everything considering the metadata of the saucers ( sometimes it is not plum withdrawn and the `` bead disc force '' syntax might assist you. You ask to erase manually the contents of the neglected disc before you can add it again ( ok on UNIX with DD, less obvious on windows ). So you ask to reconstruct the mirror from scrape, which takes clip and Io resources. </div>
</li>
<li><div> If you hold OCR at one site and OCRmirror at the other, your clump may crash and belike you will should boot your nodes to get all stable again. If you place OCR and OCRmirror on the same site, so that site is a individual point of failure. If you utilise OS mirroring for the OCR, you will hold manual intercession took to resync the mirror again, and hopefully you make n't copy in the incorrect way. </div>
</li>
<li><div> It occurs to be that site failures usually make n't occur during working hrs, but in weekends or on moments when the most experient people get on vacation. </div>
</li>
</ul>
 Determination: For the time being I prefer not to mirror storage in a RAC environment. I prefer to put in 1 ( one ) highly surplus SAN per site. If you desire to protect against site failure, regard the utilisation of Information Guard. That is a solution planned for separate storage between primary and standby site and for long-distance call between the two sites. And I desire that in the following release of Prophet Clusterware ocr failover capablenesses will be better, so my determinations may modify. 
 I accent that all the above is my ain personal view, experience and recommendation. Too I can sustain that there are examples where inaccessibility of the ocrmirror makes no such jobs ( except fore.g. an `` unity cheque neglected '' message in ocrcheck ) and a fixture can be maked online. It all depends on the sort of failure. 
Related posts: <a href='http://williebowersuc.livejournal.com/637.html'>Ditto, Lionel</a> 
<a href='http://randallhillou.blogspot.com/2009/03/name-away-dogs.html'>name away the dogs</a> 
<a href='http://nnemcculloughls.livejournal.com/521.html'>Accompanied me.</a>

	Home \| Journal tools \| Personal search form \| My account \| Bookmark \| FAQ \| Contact