Standby’s Automatic Failover and Pre/Post Processing Scripts

In 2019, a great new feature was introduced in the Standby™️ product called Automatic Failover. This is where a primary and standby database configuration is monitored by an external process - called the Dbvisit Observer.

This process has the capability to perform specific tasks depending on certain rules being met - these rules are basic system checks like database up or down, but also custom rules that the user may want such as a custom script that is being executed and depending on the exit status of the script, tasks can be performed.  

A failover example can be seen as the following: You have a failure on your primary database server, then if for a period of time the Dbvisit Observer detects that the primary database is not available and the standby one is, then if specified in the configuration - the standby database will be activated; in other words, automatic failover will be initiated where the standby database is converted into a full open read/write database, in effect, becoming the primary database.

This is an amazing new feature with lots of configuration options, but in this blog post, I would like to cover the Pre and Post Processing option of Standby™️ which I think is great to use with the Automatic Failover feature. Now if you want to read more about the Automatic Failover feature please see our Documentation for more detail.

Before I get into the details of how you can combine these two great features, let’s first do a quick overview of what the Pre and Post Processing functionality is.

Pre and Post Processing Overview

This functionality was introduced in earlier versions of Standby™️ and is still available in the latest Standby™️ version 9. In summary, with this option, you can execute your own custom scripts before (pre) or after (post) certain Standby™️ functions are executed via a custom user script. The user script can perform any task - and this is where it gets really powerful! For example, if you have a script that you want to execute post standby Activation (failover) to perform certain application tasks (such as start application on DR site or send out additional notifications etc.) this is really a powerful option.  

Now what I want to show in this post is how you can quickly get started with this functionality combined with Standby™️'s Automatic Failover feature. Basically, if you have automatic failover being activated, you can execute a custom script to perform certain tasks.

As part of the Standby™️ installation, we provide two sample scripts located in the DBVISIT_BASE/standby/doc folder:

• example_pre_post_processing.cmd (Windows)

• example_pre_post_processing.sh (Unix)

These scripts are provided as samples to help get you started and you can use these to help build your own. I will focus on the Unix version as I am going to use a Linux-based configuration. When you run the script you will quickly see that it takes two parameters. The first parameter is one of two values, pre or post, where the second parameter can be a number between 1 and 8.  

A quick overview of this can be seen if you run the script with the “-h” for help flag. Now in my example, I renamed the sample script to pre_post.sh and copied it into the DBVISIT_BASE/standby folder.

When I run it with the "-h" flag I get:

Usage:
   pre_post.sh [pre|post] [1|2|3|4|5|6|7|8]
       [pre|post}        - Specify if Pre or Post processing is being performed
       [1|2|3|4|5|6|7|8] - Indicate Dbvisit operation being performed
        -h               - display the usage

        pre  = Pre Dbvisit Processing step
        post = Post Dbvisit Processing step

        1 = Dbvisit is executed on primary and will be sending logs to standby
        2 = Dbvisit is executed on standby and will be applying logs
        3 = Dbvisit Graceful Switchover is executed on primary
        4 = Dbvisit Graceful Switchover is executed on standby
        5 = Dbvisit Activation is executed on standby
        6 = Standby Database is opened read-only
        7 = Dbvisit Create Standby Database is executed on primary (post only)
        8 = Dbvisit Create Standby Database is executed on standby (post only)

   Example, to perform a task on the standby server following activation:

        pre_post.sh post 5

The next part is to add this script to the DDC configuration on the primary system.  Remember to always update the DDC file on the primary system - it will automatically be copied to the standby system on the next run (note that if you are running the Standby™️ Daemon, you will need to restart it to allow the changes to take effect)

The parameter that needs to be added to the DDC file is “DBV_PRE_POST_PROC” - I just added it to the end of my configuration file as follows:

DBV_PRE_POST_PROC=/usr/dbvisit/standby/pre_post.sh

I do not want to get into the details of the script as the blog post will get way too long, but it is done as basic shell scripting and should be easy to follow and then modify to your needs.

The key in this example is that when the standby database is being activated (due to the Automatic Failover kicking into effect) the script will be executed twice. Yes first for the “pre” processing, meaning before the standby database is activated, the script is run with the parameters “pre” and the number “5” which indicates this is for a standby database activation. Then following the standby activation process, the script will be executed a second time with the parameter “post” and the number “5” again.  

In the script, you can now identify the two areas where “pre” and “post” execution is for step 5 (activation of standby) and then add your specific code or you can call your own function or external code from here. In my example, as part of the pre-processing step, I want to write to a log file and then the same following the post-processing. This is just to show how this works and it is not really a production example, normally you would start or stop applications, send out notifications or update some configuration.

I change the following sections in the sample script (see bolded text):

…
… <- removed code for readability 
…

### Now that values are valid, execute required code block
   case ${1} in
   pre)
           addLog "Start Pre Processing"
           case ${2} in
…
… <- removed code for readability 
…
             5)
                addLog "Standby Database Activate"
                ## add custom code here or call custom function
                echo "Starting Standby Database Activation (pre processing script)..." > /tmp/demo.log
                ;;
…
… <- removed code for readability 
…
           esac
   post)
           addLog "Start post Processing"
           case ${2} in
…
… <- removed code for readability 
…
             5)
                addLog "Standby Database Activate"
                ## add custom code here or call custom function
                echo "Standby database was activated (post processing script)..." >> /tmp/demo.log
                ;;
…
… <- removed code for readability 
…
           esac
           ;;
   esac
…
… <- removed code for readability

To illustrate the end result, when the standby database is activated we can see the log file in /tmp/demo.log has two entries:

[oracle@dbv2 tmp]$ cat demo.log

Starting Standby Database Activation (pre processing script)...

Standby database was activated (post processing script)...

[oracle@dbv2 tmp]$

Further investigation into the Activation trace file (located in the DBVISIT_BASE/standby/trace folder) displays the entries showing the pre and post processing being executed:

…
… <- removed code for readability 
…
20200114 15:58:25 main::UTIL_run_command: "/usr/dbvisit/standby/pre_post.sh" "pre" "5" "" 2>&1
20200114 15:58:25 main::UTIL_run_command: return_code=0
20200114 15:58:25 main::UTIL_run_command: ===>command output start
20200114 15:58:25 main::UTIL_run_command: 14-Jan-2020:15:58:25 : Start Pre Processing
20200114 15:58:25 main::UTIL_run_command: 14-Jan-2020:15:58:25 : Standby Database
Activate
20200114 15:58:25 main::UTIL_run_command: ===>command output end
20200114 15:58:25 main::cmn_print: 14-Jan-2020:15:58:25 : Start Pre Processing
20200114 15:58:25 main::cmn_print: 14-Jan-2020:15:58:25 : Standby Database Activate
20200114 15:58:25 main::cmn_print:
Standby™️ pre processing completed (/usr/dbvisit/standby/pre_post.sh)
20200114 15:58:25 main::ora_activate_stdb: start
…
… <- removed code for readability 
…
20200114 15:58:43 main::UTIL_run_command: "/usr/dbvisit/standby/pre_post.sh" "post"
"5" "" 2>&1
20200114 15:58:43 main::UTIL_run_command: return_code=0
20200114 15:58:43 main::UTIL_run_command: ===>command output start
20200114 15:58:43 main::UTIL_run_command: 14-Jan-2020:15:58:43 : Start post Processing
20200114 15:58:43 main::UTIL_run_command: 14-Jan-2020:15:58:43 : Standby Database
Activate
20200114 15:58:43 main::UTIL_run_command: ===>command output end
20200114 15:58:43 main::cmn_print: 14-Jan-2020:15:58:43 : Start post Processing
20200114 15:58:43 main::cmn_print: 14-Jan-2020:15:58:43 : Standby Database Activate
20200114 15:58:43 main::cmn_print:
Standby™️ post processing completed (/usr/dbvisit/standby/pre_post.sh)

More on Automatic Failover and Pre & Post Processing

For more details on these features please see the following sections in the Standby™️ version 9 documentation:

• Automatic Failover

• Pre and Post Processing