Often, we forget how important it is to collect relevant information while troubleshooting an issue. In my experience I always find it difficult when I don’t have enough information such as:

1. What was currently in progress?
2. How did I get the error?
3. Is the error a one-time issue or is it reproducible?
4. Do I have enough traces/logs available to investigate the issue?
5. Is the logging/tracing enough to troubleshoot/resolve the issue?

When it comes to investigating issues reported by a customer in Dbvisit's support team, we have a huge ace up our sleeve in the form of collecting all these traces/logs in a consolidated fashion by generating a support package. The concept of the support package was introduced in Dbvisit Standby since Version 8.0.

How to Generate a Support Package

How do we generate a support package and what are the contents? Support packages can be generated either using the GUI or CLI commands. When it comes to generating a support package from the GUI, it’s really very simple. Click on the failing process from the ACTIVE TASK LIST, click on create support package, then download it to your local machine. Once downloaded locally it can then be uploaded to the support ticket for further investigation.

When creating support packages using CLI, we must ensure that we run the command from the server where the process failed and that the PID of the failing process is known.

The command to do this is: ./dbvctl –d <DDC> -f support_package –a pid=<PID>

The Dbvisit Standby V9 Userguide details all the additional options. Now to the interesting part, what are the contents of the support package?

Generic Support Packages

Let’s start with the contents of a generic support package. Dbvisit's support team usually asks for a generic support package if there is no actual error but we need to know certain information about your environment. The support packages are created under the $DBVISIT_BASE/standby/support directory.

For me, the most awesome thing about the support package is that it provides the files from both the primary and standby server, this means we don’t have to run the command on each individual server to collect logs/traces.

Support Package Contents

Now let’s unpack the contents of the support package and see what they contain. Some of the files gathered as part of the support package are configuration files such as:

1. The DDC file dbv_SRCDB.env which is the main Dbvisit Database Configuration file, similar to our init.ora in the database.
2. The DDR file srcdb.db, this is the sqlite repository which holds the metadata.
3. The dbvnetd.conf file which is the dbvnet configuration file.
4. The dbvagent.conf which has configuration details about the Dbvisit agent.
5. Respective log files for dbvagent and dbvnet.
6. The file dbvhost_srcdb.sh contains the hostname and is referred in the DDC file as parameter HOSTNAME_CMD.

The dbvisit_SRCDB.hist file contains details of Dbvisit processing on the primary and standby servers along with the Timestamp of each action.

Additionally, if you check the dbvisit_SRCDB.hist file on the primary server it will have details about the archivelog sent to the standby server and the archivelog gap. Below is the snippet of the file on the primary server:

Destination database not available (Database is down), cannot run log gap report (pid 3235)
201801121435 - Thread: 1 Archive log gap: 3. Transfer log gap: 167 (pid 3345)
201801121435 - 3 archive log transfers to kiwi702 for SRCDB completed for thread 1. Last sequence was 167 for thread 1. (pid 3427)
201801121439 - Thread: 1 Archive log gap: 0. Transfer log gap: 0 (pid 3837)
201801121440 - Thread: 1 Archive log gap: 0. Transfer log gap: 0 (pid 3928)
201801121440 - Thread: 1 Archive log gap: 0. Transfer log gap: 0 (pid 3928)
201801121440 - scheduler heartbeat message sent (pid 3928)
201801121440 - 1 archive log transfer to kiwi702 for SRCDB completed for thread 1. Last sequence was 168 for thread 1. (pid 3928)
201801121442 - Thread: 1 Archive log gap: 0. Transfer log gap: 0 (pid 4218)
201801121442 - 1 archive log transfer to kiwi702 for SRCDB completed for thread 1. Last sequence was 169 for thread 1. (pid 4218)

The standby server will have details about the archivelog applied and the next sequence required. See snippet from the standby server below:

201801121434 - Standby Database SRCDB started on kiwi702. (pid 2823)
201801121437 - scheduler heartbeat message sent (pid 3401)
201801121437 - Log thread 1 sequence 165 (1_165_952279275.arc) applied to standby database SRCDB. (pid 3401)
201801121437 - Log thread 1 sequence 166 (1_166_952279275.arc) applied to standby database SRCDB. (pid 3401)
201801121437 - Log thread 1 sequence 167 (1_167_952279275.arc) applied to standby database SRCDB. (pid 3401)
201801121437 - Last applied: seq 167 thread 1 (pid 3401)
201801121437 - Next required: SCN=9674446 (2018-01-12:14:24:02 +13:00) thread 1 sequence 168 (pid 3401)
201801121440 - Log thread 1 sequence 168 (1_168_952279275.arc) applied to standby database SRCDB. (pid 4375)
201801121440 - Last applied: seq 168 thread 1 (pid 4375)
201801121440 - Next required: SCN=9690980 (2018-01-12:14:40:30 +13:00) thread 1 sequence 169 (pid 4375)
201801121443 - Log thread 1 sequence 169 (1_169_952279275.arc) applied to standby database SRCDB. (pid 5127)
201801121443 - Last applied: seq 169 thread 1 (pid 5127)
201801121443 - Next required: SCN=9691347 (2018-01-12:14:42:51 +13:00) thread 1 sequence 170 (pid 5127)

Another great advantage of this dbvisit_SRCDB.hist file is, if there is a failed process and we are not sure of the PID to generate a support package, we can easily pick it up from the hist file.

It has the timestamp of the issue along with the PID it is/was using. And from this we can easily generate a support package for the PID and subsequently include the trace files relevant to the issue.

In my next blog post we will check out the contents of other Dbvisit Standby Support Package files, how trace files are named and also how to troubleshoot some basic issues.