Home > Backup and Restore, Database, Oracle, Real Application Cluster, RMAN > Oracle: ORA-00245: control file backup operation failed when trying to perform an actively load balanced RMAN backup in Real Application Cluster

Oracle: ORA-00245: control file backup operation failed when trying to perform an actively load balanced RMAN backup in Real Application Cluster

Problem description:

You are running an Oracle Real Application Cluster Version >= 11.2.0.1 and are trying to perform an actively load balanced backup operation on more than one database instance. The backup of the control file fails with the following error message:

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
 
RMAN-03009: failure of backup command on dev_0 channel at 06/24/2011 11:11:24
ORA-00245: control file backup operation failed

 
Complete output:

RMAN> run {
  allocate channel 'dev_0' type 'sbt_tape'
    parms 'SBT_LIBRARY=/opt/omni/lib/libob2oracle8_64bit.so,ENV=(OB2BARTYPE=Oracle8,OB2APPNAME=racdb,OB2BARLIST=VTL-ORA-racdb-rac-scan)'
    connect 'sys@racdb1';
  allocate channel 'dev_1' type 'sbt_tape'
    parms 'SBT_LIBRARY=/opt/omni/lib/libob2oracle8_64bit.so,ENV=(OB2BARTYPE=Oracle8,OB2APPNAME=racdb,OB2BARLIST=VTL-ORA-racdb-rac-scan)'
    connect 'sys@racdb2';
 
  send device type 'sbt_tape' 'OB2BARHOSTNAME=rac-scan.intern.initso.at';
  backup incremental level 0 format 'VTL-ORA-racdb-rac-scan<racdb_%s:%t:%p>.dbf' database;
  backup format 'VTL-ORA-racdb-rac-scan<racdb_%s:%t:%p>.dbf' archivelog all not backed up 2 times;
  backup format 'VTL-ORA-racdb-rac-scan<racdb_%s:%t:%p>.dbf' current controlfile;
 } 
 
using target database control file instead of recovery catalog
allocated channel: dev_0
channel dev_0: SID=51 instance=racdb1 device type=SBT_TAPE
channel dev_0: Data Protector A.06.11/PHSS_41802/PHSS_41803/DPSOL_00435/DPLNX_
 
allocated channel: dev_1
channel dev_1: SID=19 instance=racdb2 device type=SBT_TAPE
channel dev_1: Data Protector A.06.11/PHSS_41802/PHSS_41803/DPSOL_00435/DPLNX_
 
sent command to channel: dev_1
sent command to channel: dev_0
 
Starting backup at 24-JUN-11
channel dev_0: starting incremental level 0 datafile backup set
channel dev_0: specifying datafile(s) in backup set
input datafile file number=00002 name=+DATA01/racdb/datafile/sysaux.335.752504897
input datafile file number=00006 name=+DATA01/racdb/datafile/undotbs2.364.753481801
input datafile file number=00008 name=+DATA01/racdb/datafile/undotbs4.362.753482033
input datafile file number=00005 name=+DATA01/racdb/datafile/owbsys.402.753454499
input datafile file number=00010 name=+DATA01/racdb/datafile/undotbs6.360.753482663
channel dev_0: starting piece 1 at 24-JUN-11
channel dev_1: starting incremental level 0 datafile backup set
channel dev_1: specifying datafile(s) in backup set
input datafile file number=00004 name=+DATA01/racdb/datafile/users.332.752504905
input datafile file number=00001 name=+DATA01/racdb/datafile/system.336.752504891
input datafile file number=00003 name=+DATA01/racdb/datafile/undotbs1.334.752504899
input datafile file number=00007 name=+DATA01/racdb/datafile/undotbs3.363.753481931
input datafile file number=00009 name=+DATA01/racdb/datafile/undotbs5.361.753482293
channel dev_1: starting piece 1 at 24-JUN-11
channel dev_0: finished piece 1 at 24-JUN-11
piece handle=VTL-ORA-racdb-rac-scan<racdb_163:754485037:1>.dbf tag=TAG20110624T111037 comment=API Version 2.0,MMS Version 65.6.11.0
channel dev_0: backup set complete, elapsed time: 00:00:46
channel dev_0: starting incremental level 0 datafile backup set
channel dev_0: specifying datafile(s) in backup set
channel dev_1: finished piece 1 at 24-JUN-11
piece handle=VTL-ORA-racdb-rac-scan<racdb_164:754485038:1>.dbf tag=TAG20110624T111037 comment=API Version 2.0,MMS Version 65.6.11.0
channel dev_1: backup set complete, elapsed time: 00:00:45
channel dev_1: starting incremental level 0 datafile backup set
channel dev_1: specifying datafile(s) in backup set
including current SPFILE in backup set
channel dev_1: starting piece 1 at 24-JUN-11
RMAN-03009: failure of backup command on dev_0 channel at 06/24/2011 11:11:24
ORA-00245: control file backup operation failed
continuing other job steps, job failed will not be re-run
channel dev_1: finished piece 1 at 24-JUN-11
piece handle=VTL-ORA-racdb-rac-scan<racdb_166:754485083:1>.dbf tag=TAG20110624T111037 comment=API Version 2.0,MMS Version 65.6.11.0
channel dev_1: backup set complete, elapsed time: 00:00:04
released channel: dev_0
released channel: dev_1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
 
RMAN-03009: failure of backup command on dev_0 channel at 06/24/2011 11:11:24
ORA-00245: control file backup operation failed
 
RMAN>

 
Cause:

Oracle changed the mechanism on how control file backups are performed. Until 11.2 Oracle requested a control file enqueue before performing the actual backup. Currently with 11.2.0.1 and 11.2.0.2, Oracle performs a control file backup without requesting a control file enqueue, but instead requires that during backup, the snapshot controlfile is shared between all instances. For non-RAC databases this does not change anything and neither it does for RAC databases if you always perform your backup on the same RAC-instance (without allocating any channel on another instance).

But if your are making use of distributing your backup load over several instances and your snapshot control file is located in you non-shared Oracle Home, you will end up with an “ORA-00245: control file backup operation failed”.

 
Problem resolution:

You have to configure your snapshot control file destination to a shared location accessible by all Oracle RAC instances. In our case, an ASM Cluster Filesystem was available due to the storage of external application data.

Change of snapshot controlfile destintation:

RMAN> CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/u01/acfs_app/snapcf/racdb/snapcf_racdb.f';
 
new RMAN configuration parameters:
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/u01/acfs_app/snapcf/racdb/snapcf_racdb.f';
new RMAN configuration parameters are successfully stored
 
RMAN>

 
After changing the snapshot controlfile location to a directory located on ACFS, the backup did execute without any issues:

RMAN> run {
  allocate channel 'dev_0' type 'sbt_tape'
    parms 'SBT_LIBRARY=/opt/omni/lib/libob2oracle8_64bit.so,ENV=(OB2BARTYPE=Oracle8,OB2APPNAME=racdb,OB2BARLIST=VTL-ORA-racdb-rac-scan)'
    connect 'sys@racdb1';
  allocate channel 'dev_1' type 'sbt_tape'
    parms 'SBT_LIBRARY=/opt/omni/lib/libob2oracle8_64bit.so,ENV=(OB2BARTYPE=Oracle8,OB2APPNAME=racdb,OB2BARLIST=VTL-ORA-racdb-rac-scan)'
    connect 'sys@racdb2';
 
  send device type 'sbt_tape' 'OB2BARHOSTNAME=rac-scan.intern.initso.at';
  backup incremental level 0 format 'VTL-ORA-racdb-rac-scan<racdb_%s:%t:%p>.dbf' database;
  backup format 'VTL-ORA-racdb-rac-scan<racdb_%s:%t:%p>.dbf' archivelog all not backed up 2 times;
  backup format 'VTL-ORA-racdb-rac-scan<racdb_%s:%t:%p>.dbf' current controlfile;
 } 
using target database control file instead of recovery catalog
allocated channel: dev_0
channel dev_0: SID=19 instance=racdb1 device type=SBT_TAPE
channel dev_0: Data Protector A.06.11/PHSS_41802/PHSS_41803/DPSOL_00435/DPLNX_
 
allocated channel: dev_1
channel dev_1: SID=19 instance=racdb2 device type=SBT_TAPE
channel dev_1: Data Protector A.06.11/PHSS_41802/PHSS_41803/DPSOL_00435/DPLNX_
 
sent command to channel: dev_1
sent command to channel: dev_0
 
Starting backup at 24-JUN-11
channel dev_0: starting incremental level 0 datafile backup set
channel dev_0: specifying datafile(s) in backup set
input datafile file number=00002 name=+DATA01/racdb/datafile/sysaux.335.752504897
input datafile file number=00006 name=+DATA01/racdb/datafile/undotbs2.364.753481801
input datafile file number=00008 name=+DATA01/racdb/datafile/undotbs4.362.753482033
input datafile file number=00005 name=+DATA01/racdb/datafile/owbsys.402.753454499
input datafile file number=00010 name=+DATA01/racdb/datafile/undotbs6.360.753482663
channel dev_0: starting piece 1 at 24-JUN-11
channel dev_1: starting incremental level 0 datafile backup set
channel dev_1: specifying datafile(s) in backup set
input datafile file number=00004 name=+DATA01/racdb/datafile/users.332.752504905
input datafile file number=00001 name=+DATA01/racdb/datafile/system.336.752504891
input datafile file number=00003 name=+DATA01/racdb/datafile/undotbs1.334.752504899
input datafile file number=00007 name=+DATA01/racdb/datafile/undotbs3.363.753481931
input datafile file number=00009 name=+DATA01/racdb/datafile/undotbs5.361.753482293
channel dev_1: starting piece 1 at 24-JUN-11
channel dev_0: finished piece 1 at 24-JUN-11
piece handle=VTL-ORA-racdb-rac-scan<racdb_167:754485274:1>.dbf tag=TAG20110624T111433 comment=API Version 2.0,MMS Version 65.6.11.0
channel dev_0: backup set complete, elapsed time: 00:00:35
channel dev_0: starting incremental level 0 datafile backup set
channel dev_0: specifying datafile(s) in backup set
including current control file in backup set
channel dev_0: starting piece 1 at 24-JUN-11
channel dev_1: finished piece 1 at 24-JUN-11
piece handle=VTL-ORA-racdb-rac-scan<racdb_168:754485274:1>.dbf tag=TAG20110624T111433 comment=API Version 2.0,MMS Version 65.6.11.0
channel dev_1: backup set complete, elapsed time: 00:00:36
channel dev_1: starting incremental level 0 datafile backup set
channel dev_1: specifying datafile(s) in backup set
including current SPFILE in backup set
channel dev_1: starting piece 1 at 24-JUN-11
channel dev_0: finished piece 1 at 24-JUN-11
piece handle=VTL-ORA-racdb-rac-scan<racdb_169:754485309:1>.dbf tag=TAG20110624T111433 comment=API Version 2.0,MMS Version 65.6.11.0
channel dev_0: backup set complete, elapsed time: 00:00:07
channel dev_1: finished piece 1 at 24-JUN-11
piece handle=VTL-ORA-racdb-rac-scan<racdb_170:754485310:1>.dbf tag=TAG20110624T111433 comment=API Version 2.0,MMS Version 65.6.11.0
channel dev_1: backup set complete, elapsed time: 00:00:07
Finished backup at 24-JUN-11
 
Starting backup at 24-JUN-11
current log archived
skipping archived log file +FRA01/racdb/archivelog/2011_06_21/thread_2_seq_32.373.754437613; already backed up 2 time(s)
skipping archived log file +FRA01/racdb/archivelog/2011_06_21/thread_2_seq_33.388.754437685; already backed up 2 time(s)
skipping archived log file +FRA01/racdb/archivelog/2011_06_22/thread_2_seq_34.1310.754445543; already backed up 2 time(s)
skipping archived log file +FRA01/racdb/archivelog/2011_06_22/thread_2_seq_35.1319.754445549; already backed up 2 time(s)
skipping archived log file +FRA01/racdb/archivelog/2011_06_22/thread_2_seq_36.1323.754445553; already backed up 2 time(s)
skipping archived log file +FRA01/racdb/archivelog/2011_06_22/thread_2_seq_37.1326.754445557; already backed up 2 time(s)
skipping archived log file +FRA01/racdb/archivelog/2011_06_22/thread_2_seq_38.953.754462269; already backed up 2 time(s)
skipping archived log file +FRA01/racdb/archivelog/2011_06_22/thread_2_seq_39.479.754483331; already backed up 2 time(s)
skipping archived log file +FRA01/racdb/archivelog/2011_06_21/thread_3_seq_40.859.754427105; already backed up 2 time(s)
skipping archived log file +FRA01/racdb/archivelog/2011_06_22/thread_3_seq_41.1316.754445547; already backed up 2 time(s)
skipping archived log file +FRA01/racdb/archivelog/2011_06_22/thread_3_seq_42.1328.754445559; already backed up 2 time(s)
skipping archived log file +FRA01/racdb/archivelog/2011_06_22/thread_3_seq_43.1247.754462267; already backed up 2 time(s)
skipping archived log file +FRA01/racdb/archivelog/2011_06_22/thread_3_seq_44.972.754483331; already backed up 2 time(s)
skipping archived log file +FRA01/racdb/archivelog/2011_06_21/thread_4_seq_30.885.754434005; already backed up 2 time(s)
skipping archived log file +FRA01/racdb/archivelog/2011_06_22/thread_4_seq_31.1315.754445545; already backed up 2 time(s)
skipping archived log file +FRA01/racdb/archivelog/2011_06_22/thread_4_seq_32.1332.754445559; already backed up 2 time(s)
skipping archived log file +FRA01/racdb/archivelog/2011_06_22/thread_4_seq_33.791.754462267; already backed up 2 time(s)
skipping archived log file +FRA01/racdb/archivelog/2011_06_22/thread_4_seq_34.276.754483333; already backed up 2 time(s)
skipping archived log file +FRA01/racdb/archivelog/2011_06_21/thread_5_seq_29.1220.754434005; already backed up 2 time(s)
skipping archived log file +FRA01/racdb/archivelog/2011_06_22/thread_5_seq_30.1313.754445543; already backed up 2 time(s)
skipping archived log file +FRA01/racdb/archivelog/2011_06_22/thread_5_seq_31.1329.754445559; already backed up 2 time(s)
skipping archived log file +FRA01/racdb/archivelog/2011_06_22/thread_5_seq_32.723.754462267; already backed up 2 time(s)
skipping archived log file +FRA01/racdb/archivelog/2011_06_22/thread_5_seq_33.901.754483331; already backed up 2 time(s)
skipping archived log file +FRA01/racdb/archivelog/2011_06_21/thread_6_seq_26.386.754437615; already backed up 2 time(s)
skipping archived log file +FRA01/racdb/archivelog/2011_06_22/thread_6_seq_27.1322.754445551; already backed up 2 time(s)
skipping archived log file +FRA01/racdb/archivelog/2011_06_22/thread_6_seq_28.453.754462267; already backed up 2 time(s)
skipping archived log file +FRA01/racdb/archivelog/2011_06_22/thread_6_seq_29.889.754483333; already backed up 2 time(s)
channel dev_0: starting archived log backup set
channel dev_0: specifying archived log(s) in backup set
input archived log thread=2 sequence=40 RECID=212 STAMP=754483597
input archived log thread=3 sequence=45 RECID=208 STAMP=754483595
input archived log thread=5 sequence=34 RECID=209 STAMP=754483595
input archived log thread=6 sequence=30 RECID=210 STAMP=754483596
input archived log thread=4 sequence=35 RECID=211 STAMP=754483596
channel dev_0: starting piece 1 at 24-JUN-11
channel dev_1: starting archived log backup set
channel dev_1: specifying archived log(s) in backup set
input archived log thread=3 sequence=46 RECID=216 STAMP=754485320
input archived log thread=5 sequence=35 RECID=217 STAMP=754485321
input archived log thread=6 sequence=31 RECID=213 STAMP=754485319
channel dev_1: starting piece 1 at 24-JUN-11
channel dev_0: finished piece 1 at 24-JUN-11
piece handle=VTL-ORA-racdb-rac-scan<racdb_171:754485324:1>.dbf tag=TAG20110624T111523 comment=API Version 2.0,MMS Version 65.6.11.0
channel dev_0: backup set complete, elapsed time: 00:00:07
channel dev_0: starting archived log backup set
channel dev_0: specifying archived log(s) in backup set
input archived log thread=4 sequence=36 RECID=214 STAMP=754485319
input archived log thread=2 sequence=41 RECID=215 STAMP=754485320
channel dev_0: starting piece 1 at 24-JUN-11
channel dev_1: finished piece 1 at 24-JUN-11
piece handle=VTL-ORA-racdb-rac-scan<racdb_172:754485324:1>.dbf tag=TAG20110624T111523 comment=API Version 2.0,MMS Version 65.6.11.0
channel dev_1: backup set complete, elapsed time: 00:00:07
channel dev_0: finished piece 1 at 24-JUN-11
piece handle=VTL-ORA-racdb-rac-scan<racdb_173:754485331:1>.dbf tag=TAG20110624T111523 comment=API Version 2.0,MMS Version 65.6.11.0
channel dev_0: backup set complete, elapsed time: 00:00:07
Finished backup at 24-JUN-11
 
 
Starting backup at 24-JUN-11
channel dev_0: starting full datafile backup set
channel dev_0: specifying datafile(s) in backup set
including current control file in backup set
channel dev_0: starting piece 1 at 24-JUN-11
channel dev_0: finished piece 1 at 24-JUN-11
piece handle=VTL-ORA-racdb-rac-scan<racdb_174:754485339:1>.dbf tag=TAG20110624T111539 comment=API Version 2.0,MMS Version 65.6.11.0
channel dev_0: backup set complete, elapsed time: 00:00:07
Finished backup at 24-JUN-11
 
released channel: dev_0
 
released channel: dev_1
 
RMAN>
  1. No comments yet.
  1. No trackbacks yet.

Connect with Facebook