OGG Activity Logging Tracing (Doc ID 1204284.1)

In this Document

APPLIES TO:

Oracle GoldenGate - Version 11.1.1.0.0 and later
Information in this document applies to any platform.

GOAL

OGG v11.1.1.0+ has added a new feature, Activity Logging, for tracing OGG processes.

TRACE and TRACE2 are still available (but see Doc ID 1204235.1 as there is a bug), but TLTRACE has been deprecated from v11.1.1.0.0.

SOLUTION

RETROACTIVE LOGGING

In OGG v11.1.1.0.0+ Activity Logging is on by default. Any abend by EXTRACT, REPLICAT, Server, GGSCI, MGR will automatically generate a binary file in the OGG install directory:

gglog-<process>.dmp

ACTIVITY TRACING

To evaluate non-terminal problems such as EXTRACT performance or hanging, an XML file can be added into the OGG install directory that trigger Activity Logging to generate trace info.

Use the sample XML trace files below (also attached to this KM under 'Attachments' below.

Follow the step below under the heading 'Tracing Steps by using XML files with Activity Logging'.

Sample XML trace files:

gglog-traceini.xml generates tracing similar to the output of traceini in ASCII format.
gglog-full.xml generate full tracing and outputs to a binary file
gglog-extract-log.xml generates full tracing including RAC threads
LOGSWITCHMSG_replacement.txt generates tarce similar like pre-v11 parameter LOGSWITCHMSG

1. gglog-traceini.xml -- this will generate the application level tracini logging

<?xml version="1.0" ?>
<configuration reset="true">
<appender name="traceini" class="RollingFileAppender">
<param name="BufferedIO" value="false" />
<param name="Append" value="true" />
<param name="File" value="traceLog_%I_%A" />
<param name="MaxBackupIndex" value="99" />
<param name="MaxFileSize" value="10MB" />
<layout class="PatternLayout">
<param name="Pattern" value="%d{%m/%d %H:%M:%S} [%C{1}:%L] %m%n" />
</layout>
</appender>
<logger name="gglog.std">
<level value="all" />
<appender-ref name="traceini" />
</logger>
<logger name="gglog.std.utility">
<level value="off" />
<appender-ref name="traceini" />
</logger>
</configuration>

2. gglog-full.xml -- this file will produce a full logging and generate very large trace file very quickly so it should only be used where EXTRACT/REPLICAT abend immediately after restart:

<?xml version="1.0"?>
<configuration reset="true">
<appender name="myAppender" class="RollingFileAppender">
<param name="File" value="gglog-%I.log"/>
<param name="MaxBackupIndex" value="9"/>
<param name="BufferedIO" value="false"/>
<layout class="PatternLayout">
<param name="Pattern" value="%d{%Y-%m-%d %H:%M:%S} %-5p|%-30.30c|%t|%5L %-24.24C{2} | %m%n"/>
</layout>
</appender>
<root>
<level value="all"/>
<appender-ref name="myAppender"/>
</root>
</configuration>

3. gglog-extract-log.xml -- this file will produce a "TLTRACE DEBUG" equivalent logging for Oracle redo log reading Extract and large number of trace files very quickly so it should only be used where EXTRACT abends immediately after restart. Please also make sure you have "tranlogoptions nofastreads" added to the Extract paramter file

<?xml version="1.0"?>
<configuration reset="true">



<appender name="myAppender" class="RollingFileAppender">
<param name="File" value="gglog-%I_%A.log"/>
<param name="MaxBackupIndex" value="99"/>
<param name="MaxFileSize" value="1000MB"/>
<param name="BufferedIO" value="false"/>

<layout class="PatternLayout">
<param name="Pattern" value="%t %5L %-24.24C{2} | %m%n"/>
</layout>
</appender>

<logger name="gglog.std.utility">
<level value="off"/>
</logger>
<logger name="gglog.std.utility.calltrace">
<level value="off"/>
</logger>
<logger name="gglog.std.system">
<level value="off"/>
</logger>
<logger name="gglog.std.system.calltrace">
<level value="off"/>
</logger>
<logger name="gglog.std.application">
<level value="off"/>
</logger>
<logger name="gglog.std.application.calltrace">
<level value="off"/>
</logger>
<logger name="ggapp.parmscn">
<level value="off"/>
</logger>
<logger name="ggstd.ptrace">
<level value="off"/>
</logger>
<logger name="er.redo.ora">
<level value="all"/>
</logger>
<logger name="er.redo.ora.data">
<level value="all"/>
</logger>
<logger name="er.redo.ora.rtc">
<level value="off"/>
</logger>
<logger name="er.redo.ora.rtcfm">
<level value="off"/>
</logger>
<logger name="er.redo.ora.thread">
<level value="all"/>
</logger>
<logger name="er.redo.ora.thread.checkpoint">
<level value="off"/>
</logger>
<logger name="er.redo.ora.thread.blocks">
<level value="off"/>
</logger>
<logger name="er.redo.ora.thread.logswitch">
<level value="all"/>
</logger>
<logger name="er.redo.ora.thread.unpack">
<level value="all"/>
</logger>

<root>
<appender-ref name="myAppender"/>
</root>
</configuration>

4. tracing same as pre-v11.1 parameter "TRANLOGOPTIONS LOGSWITCHMSG ON"
The parameter is nolonger supported in v11.1 and up. The redo log switch may be traced with activity logging: download the attach the file "Template to generate logswitch information" and follow the instruction bellow to rename it.

5. tracing Oracle redo error only. This may be useful when the error is gone after restarting the extract, and the error may not happen very often.

<?xml version="1.0"?>
<configuration reset="true">


<appender name="binaryAppender" class="RollingFileAppender">
<param name="File" value="gglog-%I_%A.dump"/>
<param name="MaxBackupIndex" value="10"/>
<param name="MaxFileSize" value="500MB"/>
<param name="BufferedIO" value="false"/>
<layout class="BinaryLayout"/>
</appender>
<logger name="er.redo.ora">
<level value="warn"/>
</logger>
<logger name="er.redo.ora.thread">
<level value="warn"/>
</logger>
<logger name="er.redo.ora.thread.logswitch">
<level value="error"/>
</logger>
<root>
<level value="warn"/>
<appender-ref name="binaryAppender"/>
</root>
</configuration>

6. For a Sql Server specific trace, create an xml file with the following lines:

7. For retrace command for .dmp files

CD $OGGHOME

> retrace <gglog-debug.xml> <gglog-extract.dmp>

**Contents of gglog-debug.retrace**

<?xml version="1.0"?>
<configuration reset="true">
<appender name="traceini" class="RollingFileAppender">
<param name="BufferedIO" value="false"/>
<param name="Append" value="true"/>
<param name="File" value="baswamy_retrace_traceLog_%I_%A.log"/>
<param name="MaxBackupIndex" value="20"/>
<param name="MaxFileSize" value="500MB"/>
<layout class="PatternLayout">
<param name="Pattern" value="%d{%Y-%m/%d %H:%M:%S} %-22t|%-5p|%5L %-24.24C{2} | %m%n"/>
</layout>
</appender>
<root>
<appender-ref name="traceini"/>
<level value="all"/>
</root>
</configuration>

8. For tracing OGG processes (extract, replicat etc) under Micro services architecture, follow below guide lines

# Place the activity logging xml file (e.g.: gglog-extract.xml) into the $OGG_ETC_HOME/conf directory
# The trace file will be generated under $OGG_VAR_HOME/log

9. For tracing Micro service specific processes (Service manager) under Micro services architecture, use below xml content

# Place this gglog.xml.txt into your deployment under {DeploymentHome}/etc/conf/, and rename it as gglog.xml.
# Enter back to service manager.
# Look for the gglog-adminsrvr.log under {DeploymentHome}/var/log/.

<?xml version="1.0"?>
<configuration reset="true">



<appender name="myAppender" class="RollingFileAppender">
    <param name="File"           value="gglog-%I.log"/>
    <param name="MaxBackupIndex" value="9"/>
    <param name="BufferedIO"     value="false"/>

    
    <layout class="PatternLayout">
      <param name="Pattern" value="%d{%Y-%m-%d %H:%M:%S} %-5p|%-30.30c|%5L %-24.24C{2} | %m%n"/>
    </layout>
</appender>


<logger name="gglog.std.utility">
    <level value="warn"/>
</logger>
<logger name="gglog.std.utility.calltrace">
    <level value="off"/>
</logger>

<logger name="gglog.std.system">
    <level value="warn"/>
</logger>
<logger name="gglog.std.system.calltrace">
    <level value="off"/>
</logger>

<logger name="gglog.std.application">
    <level value="debug"/>
</logger>
<logger name="gglog.std.application.calltrace">
    <level value="all"/>
</logger>
<logger name="adminsrvr">
    <level value="all"/>
</logger>
<logger name="sca">
    <level value="all"/>
</logger>
<logger name="ggsca">
    <level value="all"/>
</logger>

<root>
    <appender-ref name="myAppender"/>
</root>
</configuration>

10. For issues specific to Integrated Extract/ Replicate or parallel processes, it might be required to trace the specific process you have identified as problem causing one. In such cases, it is recommended to trace those processes along with the parent Extract or Replicate process

Example:

Replicate is RMAI3

Mapper process will be RMAI3M00, RMAI3M01, RMAI3M02 ..etc

Appliers will be RMAI3A00, RMAI3M01.. etc

To trace sub components, Main process (RMAI3) and sub process (RMAI3M** or RMAI3P**) needs to be traced

eg :

gglog-RMAI3.xml(Main process) and gglog-RMAI3M02.xml (Mapper 02) needs to be present for tracing

Tracing Steps by using XML files with Activity Logging:

1. Save one of the XML files above and move the gglog-<type>.xml to a temporary location on the server.

2. Rename the gglog-<type>.xml to gglog-<OGG process>.xml replacing OGG process with either the extract/replicat/mgr/server process or the specific process name to be traced. When using the specific process name, it must be uppercase. For example:
os> mv gglog-traceini.xml gglog-EX_AA.xml -- to trace the extract named EX_AA
or
os> mv gglog-traceini.xml gglog-extract.xml -- to trace all extract processes
or
os> mv gglog-traceini.xml gglog-RP_AA.xml -- to trace the replicat named RP_AA
or
os> mv gglog-traceini.xml gglog-replicat.xml -- to trace all replicat processes
or
os> mv gglog-traceini.xml gglog-mgr.xml -- to trace the MGR process
or
os> mv gglog-traceini.xml gglog-server.xml -- to trace the server collector process
or
os> mv LOGSWITCHMSG_replacement.txt gglog_EX_AA.xml -- to trace redo log switch for extract EX_AA

> mv gglog-debug.retrace gglog-debug.xml -- to use with retrace exe

3. Copy the renamed xml file to the OGG install directory
os> cp gglog-EX_AA.xml $GGS_HOME/

WARNING: Don't rename the file to gglog.xml and move it to the OGG install directory or tracing will immediately begin on all running OGG processes.
**For Microservices architecture, the xml files must be placed under $OGG_ETC_HOME/conf directory

4. If the process is running, the trace will begin immediately. If the process isn't running, restart it. The trace will start generating a file in <OGG_HOME> with the following format:

gglog-traceini.xml output: traceLog_<OGG process name>_<OGG process>. For example:

traceLog_EX_AA_extract
traceLog_RP_AA_replicat
traceLog_ggsci_ggsci
traceLog_mgr_mgr
traceLog_server_server

gglog-full.xml output: gglog__<OGG process name>.log. For example:
gglog_EX_AA.log
gglog_mgr.log

If the OGG process being traced abends, it will generate a dump file in the GGS install directory as noted above under RETROACTIVE LOGGING named gglog-<OGG process>.dmp. For example:

gglog-extract.dmp
gglog-relicat.dmp
gglog-server.dmp

5. When done with the trace, remove or rename the gglog-<OGG process>.xml so tracing is stopped and won't resume when the process is restarted.

(Do not leave activity log tracing enabled in a production environment, otherwise GG replication performance may be severely impacted.)

Get the traceLog/ActivityLog and the dmp (if generated) from the customer and attach to the BugDB.

Search This Blog

Abdul Hafeez Kalsekar -- Oracle database tuning case studies