OCI - Roll Forward A Standby Database Using Recover Database From Service (Doc ID 2931070.1)
| Goal |
| Solution |
APPLIES TO:
Oracle Cloud Infrastructure - Database Service - Version N/A to N/A [Release 1.0]Information in this document applies to any platform.
GOAL
The procedure described in this note is applicable for OCI DBaaS environment, Oracle 18c and higher.
Typically, when rolling forward a physical standby database using primary incremental backup, multiple steps are required:
- Identify the Start SCN on Standby for performing incremental backup on primary
- Perform incremental backup on primary with FROM SCN clause
- Move the backup-pieces from primary to standby
- Catalog the backup-piece on Standby
- Perform recovery on standby using recover database noredo
- Refresh standby controlfile again from primary
Starting from 12.1, we could use "RECOVER DATABASE FROM SERVICE" command which will automate a few steps like performing incremental backup on primary, transfer the backup-pieces to standby over network and perform recovery on standby. However, we still had to manually refresh the standby controlfile and manually restore newly-added datafiles. These steps required manual efforts and are error prone especially when standby files are physically located in a path different to that of primary.
Starting with 18.1, we can use a single command to refresh the standby with changes made on primary:
RMAN> RECOVER STANDBY DATABASE FROM SERVICE <primary_connect_identifier>;
This command will internally keep track of standby file locations, refresh standby controlfile from primary, update the new standby controlfile with standby file names, perform incremental backup on primary, transfer the backup-pieces over network to standby and perform recovery on standby
SOLUTION
1. To refresh the standby, ensure that managed recovery is stopped on standby:
NOTE: If you do not stop managed recovery, you will get RMAN-05150 error during execution
SQL> recover managed standby database cancel; ( or )
DGMGRL> EDIT DATABASE '<standby>' SET STATE='APPLY-OFF';
2. If the standby is RAC with more than one instance, make sure only the instance from which recover standby command will be executed is mounted and all other instances are shutdown to avoid RMAN-05157
3. Make a note of RMAN 'SBT_LIBRARY' and ENV configuration details
$ rman target /
Recovery Manager: Release 19.0.0.0.0 - Production on Sat Feb 25 17:50:29 2023
Version 19.16.0.0.0
Copyright (c) 1982, 2019, Oracle and/or its affiliates. All rights reserved.
RMAN> show all;
using target database control file instead of recovery catalog
RMAN configuration parameters for database with db_unique_name DB0801_IAD3FT are:
CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 30 DAYS;
.....
CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' FORMAT '%d_%I_%U_%T_%t' PARMS 'SBT_LIBRARY=/opt/oracle/dcs/commonstore/oss/DB0801_iad2r2/libopc.so ENV= (OPC_PFILE=/opt/oracle/dcs/commonstore/oss/DB0801_iad2r2/0bab1c55-82e5-4455-a44a-2f07445d9faf/opc_DB0801_iad2r2.ora)'; <<<<<<<<<<
......
RMAN> exit
4. Create following directories on OCI standby database If not already exist
mkdir -p /opt/oracle/dcs/commonstore/oss/<primary database db_unique_name>
mkdir -p /opt/oracle/dcs/commonstore/oss/<primary database db_unique_name>/<primary database backup configuration id>/
Use following command on primary database server as root user to find the primary database backup configuration id
# dbcli list-backupconfigs -j
[ {
"id" : "0bab1c55-82e5-4455-a44a-2f07445d9faf", <<<<<< primary database backup configuration id
"name" : "bq1OjPLWxKMD5ouEMwPU_BC",
...
"recoveryTag" : null
} ]
5. Ensure to copy following files from OCI Primary Database to OCI Standby Database
/opt/oracle/dcs/commonstore/oss/<primary database db_unique_name>/libopc.so
/opt/oracle/dcs/commonstore/oss/<primary database db_unique_name>/<primary database backup configuration id>/opc_<primary db_unique_name>.ora
/opt/oracle/dcs/commonstore/oss/<primary database db_unique_name>/<primary database backup configuration id>/cwallet.sso
6. Ensure to set 600 permissions to cwallet.sso file
On standby:
cd /opt/oracle/dcs/commonstore/oss/<primary db_unique_name>/<primary database backup configuration id>/
$ chmod 600 cwallet.sso
Expected file owner and permissions :
$ ll /opt/oracle/dcs/commonstore/oss/DB0801_iad2r2/0bab1c55-82e5-4455-a44a-2f07445d9faf
total 180
-rw------- 1 oracle oinstall 113647 Feb 25 18:15 cwallet.sso
-rwxr-xr-x 1 oracle oinstall 252 Feb 25 17:58 opc_DB0801_iad2r2.ora
$ ll /opt/oracle/dcs/commonstore/oss/DB0801_iad2r2/libopc.so
-rwxr-xr-x 1 oracle oinstall 96406584 Feb 25 17:58 /opt/oracle/dcs/commonstore/oss/DB0801_iad2r2/libopc.so
7. Ensure that Oracle Net connectivity is established between the physical standby database and the primary database by adding an entry corresponding to the primary database in the tnsnames.ora file of the physical standby database. In below example, DB0801_IAD2R2 is the connect identifier for primary.
8. Connect RMAN to standby as target and run "RECOVER STANDBY DATABASE FROM SERVICE <primary_database_connect_identifier>" command. Find below an example run for the command:
On Standby:
$ rman target /
Recovery Manager: Release 19.0.0.0.0 - Production on Sat Feb 25 18:15:58 2023
Version 19.16.0.0.0
Copyright (c) 1982, 2019, Oracle and/or its affiliates. All rights reserved.
connected to target database: DB0801 (DBID=995451477, not open)
RMAN> RECOVER STANDBY DATABASE FROM SERVICE DB0801_IAD2R2; // DB0801_IAD2R2 <- Primary database TNS alias/connect identifier
Starting recover at 25-FEB-23
Oracle instance started
Total System Global Area 14495511760 bytes
Fixed Size 9151696 bytes
Variable Size 1912602624 bytes
Database Buffers 12415139840 bytes
Redo Buffers 158617600 bytes
contents of Memory Script:
{
restore standby controlfile from service 'DB0801_IAD2R2';
alter database mount standby database;
}
executing Memory Script
Starting restore at 25-FEB-23
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=483 device type=DISK
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: using network backup set from service DB0801_IAD2R2
channel ORA_DISK_1: restoring control file
channel ORA_DISK_1: restore complete, elapsed time: 00:00:04
output file name=+RECO/DB0801_IAD3FT/CONTROLFILE/current.258.1111591957
Finished restore at 25-FEB-23
released channel: ORA_DISK_1
Statement processed
contents of Memory Script:
{
set newname for tempfile 1 to
"+DATA/DB0801_IAD3FT/TEMPFILE/temp.274.1111592153";
set newname for tempfile 2 to
"+DATA/DB0801_IAD3FT/E52EEA120CB1049BE0536F00000A7ED0/TEMPFILE/temp.275.1111592155";
set newname for tempfile 4 to
"+DATA";
switch tempfile all;
set newname for datafile 1 to
"+DATA/DB0801_IAD3FT/DATAFILE/system.260.1111591967";
set newname for datafile 3 to
"+DATA/DB0801_IAD3FT/DATAFILE/sysaux.261.1111591967";
set newname for datafile 4 to
"+DATA/DB0801_IAD3FT/DATAFILE/undotbs1.262.1111591967";
set newname for datafile 5 to
"+DATA/DB0801_IAD3FT/E52EEA120CB1049BE0536F00000A7ED0/DATAFILE/system.263.1111591971";
set newname for datafile 6 to
"+DATA/DB0801_IAD3FT/E52EEA120CB1049BE0536F00000A7ED0/DATAFILE/sysaux.264.1111591983";
set newname for datafile 7 to
"+DATA/DB0801_IAD3FT/E52EEA120CB1049BE0536F00000A7ED0/DATAFILE/undotbs1.265.1111591989";
set newname for datafile 8 to
"+DATA/DB0801_IAD3FT/E52F07124F563736E0536F00000A30E4/DATAFILE/system.266.1111591989";
set newname for datafile 9 to
"+DATA/DB0801_IAD3FT/E52F07124F563736E0536F00000A30E4/DATAFILE/sysaux.267.1111591991";
set newname for datafile 10 to
"+DATA/DB0801_IAD3FT/E52F07124F563736E0536F00000A30E4/DATAFILE/undotbs1.268.1111591997";
set newname for datafile 11 to
"+DATA/DB0801_IAD3FT/DATAFILE/users.269.1111591997";
set newname for datafile 12 to
"+DATA/DB0801_IAD3FT/E52F07124F563736E0536F00000A30E4/DATAFILE/users.270.1111591997";
catalog datafilecopy "+DATA/DB0801_IAD3FT/DATAFILE/system.260.1111591967",
"+DATA/DB0801_IAD3FT/DATAFILE/sysaux.261.1111591967",
"+DATA/DB0801_IAD3FT/DATAFILE/undotbs1.262.1111591967",
"+DATA/DB0801_IAD3FT/E52EEA120CB1049BE0536F00000A7ED0/DATAFILE/system.263.1111591971",
"+DATA/DB0801_IAD3FT/E52EEA120CB1049BE0536F00000A7ED0/DATAFILE/sysaux.264.1111591983",
"+DATA/DB0801_IAD3FT/E52EEA120CB1049BE0536F00000A7ED0/DATAFILE/undotbs1.265.1111591989",
"+DATA/DB0801_IAD3FT/E52F07124F563736E0536F00000A30E4/DATAFILE/system.266.1111591989",
"+DATA/DB0801_IAD3FT/E52F07124F563736E0536F00000A30E4/DATAFILE/sysaux.267.1111591991",
"+DATA/DB0801_IAD3FT/E52F07124F563736E0536F00000A30E4/DATAFILE/undotbs1.268.1111591997",
"+DATA/DB0801_IAD3FT/DATAFILE/users.269.1111591997",
"+DATA/DB0801_IAD3FT/E52F07124F563736E0536F00000A30E4/DATAFILE/users.270.1111591997";
switch datafile all;
}
executing Memory Script
executing command: SET NEWNAME
executing command: SET NEWNAME
executing command: SET NEWNAME
Starting implicit crosscheck backup at 25-FEB-23
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=174 device type=DISK
Finished implicit crosscheck backup at 25-FEB-23
Starting implicit crosscheck copy at 25-FEB-23
using channel ORA_DISK_1
Finished implicit crosscheck copy at 25-FEB-23
searching for all files in the recovery area
cataloging files...
cataloging done
List of Cataloged Files
=======================
File Name: +RECO/DB0801_IAD3FT/AUTOBACKUP/2022_08_01/s_1111592603.269.1111592845
.....
File Name: +RECO/DB0801_IAD3FT/ARCHIVELOG/2023_02_17/thread_1_seq_5212.489.1129071715
renamed tempfile 1 to +DATA/DB0801_IAD3FT/TEMPFILE/temp.274.1111592153 in control file
renamed tempfile 2 to +DATA/DB0801_IAD3FT/E52EEA120CB1049BE0536F00000A7ED0/TEMPFILE/temp.275.1111592155 in control file
renamed tempfile 4 to +DATA in control file
executing command: SET NEWNAME
....
executing command: SET NEWNAME
cataloged datafile copy
datafile copy file name=+DATA/DB0801_IAD3FT/DATAFILE/sysaux.261.1111591967 RECID=4 STAMP=1129745846
cataloged datafile copy
datafile copy file name=+DATA/DB0801_IAD3FT/DATAFILE/system.260.1111591967 RECID=5 STAMP=1129745846
cataloged datafile copy
datafile copy file name=+DATA/DB0801_IAD3FT/DATAFILE/undotbs1.262.1111591967 RECID=6 STAMP=1129745853
cataloged datafile copy
datafile copy file name=+DATA/DB0801_IAD3FT/E52EEA120CB1049BE0536F00000A7ED0/DATAFILE/system.263.1111591971 RECID=7 STAMP=1129745853
cataloged datafile copy
datafile copy file name=+DATA/DB0801_IAD3FT/E52EEA120CB1049BE0536F00000A7ED0/DATAFILE/sysaux.264.1111591983 RECID=8 STAMP=1129745859
cataloged datafile copy
datafile copy file name=+DATA/DB0801_IAD3FT/E52EEA120CB1049BE0536F00000A7ED0/DATAFILE/undotbs1.265.1111591989 RECID=9 STAMP=1129745859
cataloged datafile copy
datafile copy file name=+DATA/DB0801_IAD3FT/E52F07124F563736E0536F00000A30E4/DATAFILE/system.266.1111591989 RECID=10 STAMP=1129745865
cataloged datafile copy
datafile copy file name=+DATA/DB0801_IAD3FT/E52F07124F563736E0536F00000A30E4/DATAFILE/sysaux.267.1111591991 RECID=11 STAMP=1129745865
cataloged datafile copy
datafile copy file name=+DATA/DB0801_IAD3FT/E52F07124F563736E0536F00000A30E4/DATAFILE/undotbs1.268.1111591997 RECID=12 STAMP=1129745871
cataloged datafile copy
datafile copy file name=+DATA/DB0801_IAD3FT/DATAFILE/users.269.1111591997 RECID=13 STAMP=1129745871
cataloged datafile copy
datafile copy file name=+DATA/DB0801_IAD3FT/E52F07124F563736E0536F00000A30E4/DATAFILE/users.270.1111591997 RECID=14 STAMP=1129745877
datafile 1 switched to datafile copy
input datafile copy RECID=5 STAMP=1129745846 file name=+DATA/DB0801_IAD3FT/DATAFILE/system.260.1111591967
datafile 3 switched to datafile copy
input datafile copy RECID=4 STAMP=1129745846 file name=+DATA/DB0801_IAD3FT/DATAFILE/sysaux.261.1111591967
datafile 4 switched to datafile copy
input datafile copy RECID=6 STAMP=1129745853 file name=+DATA/DB0801_IAD3FT/DATAFILE/undotbs1.262.1111591967
datafile 5 switched to datafile copy
input datafile copy RECID=7 STAMP=1129745853 file name=+DATA/DB0801_IAD3FT/E52EEA120CB1049BE0536F00000A7ED0/DATAFILE/system.263.1111591971
datafile 6 switched to datafile copy
input datafile copy RECID=8 STAMP=1129745859 file name=+DATA/DB0801_IAD3FT/E52EEA120CB1049BE0536F00000A7ED0/DATAFILE/sysaux.264.1111591983
datafile 7 switched to datafile copy
input datafile copy RECID=9 STAMP=1129745859 file name=+DATA/DB0801_IAD3FT/E52EEA120CB1049BE0536F00000A7ED0/DATAFILE/undotbs1.265.1111591989
datafile 8 switched to datafile copy
input datafile copy RECID=10 STAMP=1129745865 file name=+DATA/DB0801_IAD3FT/E52F07124F563736E0536F00000A30E4/DATAFILE/system.266.1111591989
datafile 9 switched to datafile copy
input datafile copy RECID=11 STAMP=1129745865 file name=+DATA/DB0801_IAD3FT/E52F07124F563736E0536F00000A30E4/DATAFILE/sysaux.267.1111591991
datafile 10 switched to datafile copy
input datafile copy RECID=12 STAMP=1129745871 file name=+DATA/DB0801_IAD3FT/E52F07124F563736E0536F00000A30E4/DATAFILE/undotbs1.268.1111591997
datafile 11 switched to datafile copy
input datafile copy RECID=13 STAMP=1129745871 file name=+DATA/DB0801_IAD3FT/DATAFILE/users.269.1111591997
datafile 12 switched to datafile copy
input datafile copy RECID=14 STAMP=1129745877 file name=+DATA/DB0801_IAD3FT/E52F07124F563736E0536F00000A30E4/DATAFILE/users.270.1111591997
Executing: alter database rename file '+RECO/DB0801_IAD2R2/ONLINELOG/group_1.257.1111584153' to '+RECO/DB0801_IAD3FT/ONLINELOG/group_1.259.1111592009'
Executing: alter database rename file '+RECO/DB0801_IAD2R2/ONLINELOG/group_2.258.1111584153' to '+RECO/DB0801_IAD3FT/ONLINELOG/group_2.260.1111592015'
Executing: alter database rename file '+RECO/DB0801_IAD2R2/ONLINELOG/group_3.259.1111584153' to '+RECO/DB0801_IAD3FT/ONLINELOG/group_3.261.1111592025'
contents of Memory Script:
{
recover database from service 'DB0801_IAD2R2';
}
executing Memory Script
Starting recover at 25-FEB-23
using channel ORA_DISK_1
allocated channel: ORA_SBT_TAPE_1
channel ORA_SBT_TAPE_1: SID=176 device type=SBT_TAPE
channel ORA_SBT_TAPE_1: Oracle Database Backup Service Library VER=21.0.0.1
allocated channel: ORA_SBT_TAPE_2
channel ORA_SBT_TAPE_2: SID=333 device type=SBT_TAPE
channel ORA_SBT_TAPE_2: Oracle Database Backup Service Library VER=21.0.0.1
skipping datafile 5; already restored to SCN 2654157
skipping datafile 6; already restored to SCN 2654157
skipping datafile 7; already restored to SCN 2654157
channel ORA_DISK_1: starting incremental datafile backup set restore
channel ORA_DISK_1: using network backup set from service DB0801_IAD2R2
destination for restore of datafile 00001: +DATA/DB0801_IAD3FT/DATAFILE/system.260.1111591967
channel ORA_SBT_TAPE_1: starting incremental datafile backup set restore
channel ORA_SBT_TAPE_1: using network backup set from service DB0801_IAD2R2
destination for restore of datafile 00003: +DATA/DB0801_IAD3FT/DATAFILE/sysaux.261.1111591967
channel ORA_SBT_TAPE_2: starting incremental datafile backup set restore
channel ORA_SBT_TAPE_2: using network backup set from service DB0801_IAD2R2
destination for restore of datafile 00004: +DATA/DB0801_IAD3FT/DATAFILE/undotbs1.262.1111591967
channel ORA_DISK_1: restore complete, elapsed time: 00:00:03
channel ORA_DISK_1: starting incremental datafile backup set restore
channel ORA_DISK_1: using network backup set from service DB0801_IAD2R2
destination for restore of datafile 00008: +DATA/DB0801_IAD3FT/E52F07124F563736E0536F00000A30E4/DATAFILE/system.266.1111591989
channel ORA_SBT_TAPE_1: restore complete, elapsed time: 00:00:04
channel ORA_SBT_TAPE_1: starting incremental datafile backup set restore
channel ORA_SBT_TAPE_1: using network backup set from service DB0801_IAD2R2
destination for restore of datafile 00009: +DATA/DB0801_IAD3FT/E52F07124F563736E0536F00000A30E4/DATAFILE/sysaux.267.1111591991
channel ORA_SBT_TAPE_2: restore complete, elapsed time: 00:00:05
channel ORA_SBT_TAPE_2: starting incremental datafile backup set restore
channel ORA_SBT_TAPE_2: using network backup set from service DB0801_IAD2R2
destination for restore of datafile 00010: +DATA/DB0801_IAD3FT/E52F07124F563736E0536F00000A30E4/DATAFILE/undotbs1.268.1111591997
channel ORA_DISK_1: restore complete, elapsed time: 00:00:04
channel ORA_DISK_1: starting incremental datafile backup set restore
channel ORA_DISK_1: using network backup set from service DB0801_IAD2R2
destination for restore of datafile 00011: +DATA/DB0801_IAD3FT/DATAFILE/users.269.1111591997
channel ORA_SBT_TAPE_1: restore complete, elapsed time: 00:00:03
channel ORA_SBT_TAPE_1: starting incremental datafile backup set restore
channel ORA_SBT_TAPE_1: using network backup set from service DB0801_IAD2R2
destination for restore of datafile 00012: +DATA/DB0801_IAD3FT/E52F07124F563736E0536F00000A30E4/DATAFILE/users.270.1111591997
channel ORA_SBT_TAPE_2: restore complete, elapsed time: 00:00:03
channel ORA_DISK_1: restore complete, elapsed time: 00:00:02
channel ORA_SBT_TAPE_1: restore complete, elapsed time: 00:00:04
starting media recovery
media recovery complete, elapsed time: 00:00:00
Finished recover at 25-FEB-23
Finished recover at 25-FEB-23
RMAN> exit
9. Make sure standby remote destination is VALID state
On Primary :
SQL> alter system archive log current;
SQL> SELECT thread#, dest_id, gvad.status, error FROM gv$archive_dest gvad, gv$instance gvi WHERE gvad.inst_id = gvi.inst_id AND destination is NOT NULL ORDER BY thread#, dest_id;
THREAD# DEST_ID STATUS ERROR
---------- ---------- --------- -----------------------------------------------------------------
1 1 VALID
1 2 VALID <<<<<<<<<<<<< Expected output
SQL> alter system archive log current;
10. Clear the Standby Redologs
Note: You will see error "ORA-15012: ASM file '+RECO/<db_unique_name>/ONLINELOG/group_xx.xxxx.xxxx' does not exist" in Standby Database alertlog If log file physically doesn't exist at OS/ASM level
On Standby :
SQL> set lines 200 pages 200
col member for a85
select i.instance_name, d.name, d.db_unique_name, d.database_role, d.open_mode from v$instance i, v$database d;
show pdbs;
select l.group#, l.thread#, l.bytes, f.type, f.member, l.status from v$log l, v$logfile f where l.group#=f.group# order by f.type, l.group#;
select l.group#, l.thread#, l.bytes, f.type, f.member, l.status from v$standby_log l, v$logfile f where l.group#=f.group# order by f.type, l.group#;
SQL> alter database recover managed standby database cancel;
SQL> alter database clear logfile group <standby redo group #>; // execute for all standby redo groups
Example :
SQL> select l.group#, l.thread#, l.bytes, f.type, f.member, l.status from v$standby_log l, v$logfile f where l.group#=f.group# order by f.type, l.group#;
GROUP# THREAD# BYTES TYPE MEMBER STATUS
---------- ---------- ---------- ------- ------------------------------------------------------------------------------------- ----------
4 1 1073741824 STANDBY +RECO/DB0801_IAD2R2/ONLINELOG/group_4.266.1111591607 UNASSIGNED <<< Primary db_unique_name - DB0801_IAD2R2
5 1 1073741824 STANDBY +RECO/DB0801_IAD2R2/ONLINELOG/group_5.267.1111591613 UNASSIGNED
6 1 1073741824 STANDBY +RECO/DB0801_IAD2R2/ONLINELOG/group_6.268.1111591623 UNASSIGNED
7 1 1073741824 STANDBY +RECO/DB0801_IAD2R2/ONLINELOG/group_7.269.1111591631 UNASSIGNED
SQL> alter database recover managed standby database cancel;
SQL> alter database clear logfile group 4;
Database altered.
SQL> alter database clear logfile group 5;
Database altered.
SQL> alter database clear logfile group 6;
Database altered.
SQL> alter database clear logfile group 7;
Database altered.
SQL> select l.group#, l.thread#, l.bytes, f.type, f.member, l.status from v$standby_log l, v$logfile f where l.group#=f.group# order by f.type, l.group#;
GROUP# THREAD# BYTES TYPE MEMBER STATUS
---------- ---------- ---------- ------- ------------------------------------------------------------------------------------- ----------
4 1 1073741824 STANDBY +RECO/DB0801_IAD3FT/ONLINELOG/group_4.2271.1133453033 UNASSIGNED <<< Standby db_unique_nmae - DB0801_IAD3FT
5 1 1073741824 STANDBY +RECO/DB0801_IAD3FT/ONLINELOG/group_5.2272.1133453071 UNASSIGNED
6 1 1073741824 STANDBY +RECO/DB0801_IAD3FT/ONLINELOG/group_6.2273.1133453083 UNASSIGNED
7 1 1073741824 STANDBY +RECO/DB0801_IAD3FT/ONLINELOG/group_7.2274.1133453095 UNASSIGNED
SQL>
11. (Optional) - Open Standby Database in read only mode
12. Start MRP process on Standby Database
SQL> alter database recover managed standby database disconnect; ( or )
DGMGRL> EDIT DATABASE '<standby>' SET STATE='APPLY-ON';
Comments
Post a Comment