Why are Support Packages important? - Part 2

In my previous post, I started to dive into the contents of the Dbvisit Standby Support Package. It truly is a great source of information and we recommend you always include this with any support calls logged. This post is going to continue the discussion around the Dbvisit Standby Support Package and its contents.

There are two key files I would like to highlight:
• os.txt
• sql.txt

These play a crucial role in investigating the issues related to setup and permissions and are a valuable source to the team in getting a better understanding of the particular environment. These two files are generated as part of Support Package creation on both the Primary and Standby database servers.

The os.txt file contains information such as:
• details of the hostname
• OS user groups
• Filesystem details
• oratab entries (in the case of Linux)
• processes running in the server
• the ARCHSOURCE and ARCHDEST directory contents
• fstab details
• plus netstat output of the default ports used by Dbvisit Standby

We don’t stop there; we also provide the output of “srvctl config database” command if the database is registered with Oracle restart or Oracle Clusterware.

The Windows environment has a similar output with the addition of the following:
• with systeminfo
• service details
• plus, Dbvisit user groups and permissions.

We have seen a number of tickets where the Dbvisit Standby software owner (Windows User) is not part of the ORA_DBA group – the end result is that the Create Standby Database (CSD) fails.

Having all this information helps us to save time in resolving the issue and providing a good solution. With the Support Package, we will have a much better understanding of the environment, investigate the problem and point the customer towards the right solution.

Below is a sample screenshot from Windows os.txt file:

Let’s move on to the sql.txt file. This file contains output from a collection of queries that are run against the Primary and Standby databases. To help provide better context and information, the SQL statements are also provided together with the output. This gives us a very good perspective of the current status of the database - both from the Primary and the Standby.

The file also provides the database key (init.ora) parameters – again values from both the Primary and Standby. The output in this file is extremely valuable – it again helps us to gain a better understanding of the database configurations as each environment is different. Investigating Create Standby Database (CSD), Graceful Switchover (GS) or any other possible failures is much easier with the contents of this file. A quick and easy example is that we have seen issues where some of the database files are offline and we were able to capture these errors right out of the sql.txt files.

Moving onto trace files things get even more interesting… and if I dare say so a little bit more complex. But first let’s look at file naming. Trace file names have the following format: <PID>_dbvctl_<Process/Step>_<DDC>_timestamp<YYYYMMDDHH24MM>.trc

Let’s review this with an example. Below is the screenshot of the Support Package contents from a Graceful Switchover (GS) process:

If you look closely, you can see that under kiwi812-vip (the Primary server in this case), there is a file with name 21816_dbvctl_switchover_GUHAN_201801181821.trc, as explained the PID is 21816. Graceful Switchover is the process/step of an operation performed, GUHAN is the DDC name and the timestamp pertains to 18th January 2018 6:21PM.

For me, the secret sauce is that the process not only gets us the Graceful Switchover trace file from the Primary but it picks up all the relevant files from the Standby that were part of the switchover process - nice! I can create this Support Package either from the Dbvist Standby GUI with a single click or I can run the following command from the Primary and then get the Support Package:

./dbvctl –d GUHAN –f support_package –a pid=21816

As a next step, let’s open a trace file in text editor and see how we can investigate and resolve issues. To show how this actually works, I have simulated an error during the CSD process:

I created a Support Package by clicking the Failed Process in the Active Task List and downloaded the file locally. In this test cast dbv1 is the Primary server and dbv2 is the Standby server. DEV is the Dbvisit Standby Configuration (DDC) name as well as the database name.

I have now opened the trace file 315_dbvctl_csd_DEV_201803120051.trc. The first few lines will provide a lot of detail already: the command that was executed, on which server it was executed, the version of Dbvisit Standby in use and the PID of the process.

The next few lines provide a little bit more detail about the trace file such as where to find the error in the trace file and the location of this trace file. After this we display all the Dbvisit Standby Variables, some of these variables have default values in them and we strongly advise not to change any DDC configuration variables which have not been discussed in the user guide.

After displaying the DDC variables, the trace file will contain a large amount of details regarding the process that was run – in this case the Create Standby Database (CSD) process. Now it is important to note that the actual error message is displayed at the end of the trace file.

If you can see the above screenshot there is reference to a different trace file. Now the excellent part of the Support Package is that we don’t have to go back and ask the customer for this particular trace file again, since the Support Package will have the trace file already included as it formed part of this overall CSD process.

Let’s investigate further and open the trace file 614_dbvctl_f_rman_exec_DEV_201803120053.trc and jump to the error at the end to find out what really happened and who is the culprit here.

We can easily zero in on the error here, the backup file for data file 1 was not copied or not found…and yes, I am the culprit here! I moved the backup of data file 1 to a different directory before it was supposed to be copied over to the Standby to simulate this error. After I corrected the error (copied the file back) and resumed the creation of the Standby database process - it completed successfully!

Children are always curious about what’s happening and they always want to investigate, put up a fight and don’t give up until they get tired then try again the next day. But as we grow older the inclination to investigate and our curiosity of how things work or what went wrong decreases. Let’s not give up the curiosity and be a kid again! At Dbvisit we want to make sure that our customers have the right information and tools to investigate the issues themselves or be able to provide the support team with all the required information in one go. We strive to provide the best solution in less time and enable customers to resolve the issues.

I hope I have provided enough information to wake the curiosity in you, and for you to start investigating issues. I can proudly say that issues rarely occur using our product and if they do, with the correct information we can get it resolved quickly.