Custom Logging

The default core Kinetica log that records database interactions, startup/shutdown, error information, and more is located at /opt/gpudb/core/logs/gpudb.log. This log file can be configured to use a custom format that conforms to a rolling logging standard, enable various appenders and/or loggers, change the logging level, and more using the /opt/gpudb/core/etc/gpudb_logger.conf file.

Customizing the Log

  1. Review the available Log Levels.

    • OFF
    • FATAL
    • ERROR
    • WARN
    • UERR
    • INFO
    • DEBUG
    • TRACE
    • NOTSET

  2. Review the available Appenders.

    • ToConsole
    • ToSysLog
    • ToFile
    • ToRollingFile
    • ToRollingDatedFile
    • ToBacktraceFile

  3. Review the available Loggers.

  4. Open /opt/gpudb/core/etc/gpudb_logger.conf with a text editor.

  5. Update the rootLogger setting with the desired level and appender(s):

    log4cplus.rootLogger=<log-level>, <appender1>[, <appender2>, ..., <appenderN>]

  6. Uncomment the desired appender setting section(s):

    # ---------------------------------------------------------------------------
# Rolling dated log file appender.
# ...

log4cplus.appender.ToRollingDatedFile=log4cplus::RollingDatedFileAppender
...

  7. Adjust the appender settings as necessary, e.g., log size, threshold, file name, etc.

  8. Adjust additional logger levels and/or appender(s) as necessary:

    # ===========================================================================
# Loggers
# ...

# Log memory info, logs at DEBUG.
log4cplus.logger.MEMORY=<log-level>, <appender1>[, <appender2>, ..., <appenderN>]

...

  9. Save and exit the file.

  10. Restart the host manager and the database:

    service restart gpudb_host_manager && service restart gpudb

Log Levels

Log levels can be attributed to the rootLogger (the base logger, default level is INFO), the individual loggers at the bottom of the file, or as the threshold for an appender. The log levels can be ranked in order from least information to most information logged like so: OFF, FATAL, ERROR, WARN, UERR, INFO, DEBUG, TRACE. Some loggers will not provide their information unless a certain level is set.

Level Description
OFF Does not log at all.
FATAL Logs unrecoverable errors that occur and cause the database or a component to exit.
ERROR Logs serious and unexpected errors that may severely impact the database, e.g., network errors, disk full, etc.
WARN Logs unfavorable conditions or states that may impair the database, e.g., network outage, slow disks, etc.
UERR Logs malformed user requests or requests that could not be processed, e.g., invalid table names, bad options, etc.
INFO Logs informational messages recorded during startup and shutdown. There are two INFO logs per request by the head node: one when the request is received and one when the request is completed.
DEBUG Logs verbose messages about program processing and state.
TRACE Logs even more verbose messages about program processing and state.
NOTSET Logs at the level of the next logger in the chain, up to the rootLogger. This is equivalent to commenting out the logger.

Some loggers may have chained sub loggers; these sub loggers will use the same level as the next logger up the chain (if set). If none of the loggers or sub loggers in the chain are set, the logger will use the rootLogger level. For example, if the following sub logger is NOTSET:

log4cplus.logger.GlobalManager.pub.socket=NOTSET

It will use the level of the next logger up the chain:

log4cplus.logger.GlobalManager.pub=INFO

Appenders

The gpudb_logger.conf file comes with preconfigured appenders that can be used to create different log types. Multiple appenders can be used if necessary as long as the appropriate settings are uncommented and the appender name is added to the rootLogger appender list. Default database setups have the ToConsole appender enabled.

Important

For sizing considerations, a request typically logs for about 500 bytes, but joins and multi-step queries will log quite a bit more.

ToConsole

The Console appender pipes the output of the /opt/gpudb/core/bin/gpudb script to a /opt/gpudb/core/logs/gpudb-<hostname>-<date>.log file. This file is symbolically linked to /opt/gpudb/core/logs/gpudb.log.

Setting Description
layout Dictates the type of layout the logging of which the logging will take form.
layout.ConversionPattern The log4cplus pattern to use for the layout. See the PatternLayout class reference documentation for details.

ToSysLog

The SysLog appender sends the log output to the centralized SysLog host.

Setting Description
layout Dictates the type of layout the logging of which the logging will take form.
layout.ConversionPattern The log4cplus pattern to use for the layout. See the PatternLayout class reference documentation for details.
syslogHost Hostname for the SysLog host.
Facility Source of the log messages.
Threshold The level of messages to log. See Log Levels for more information.

ToFile

The File appender sends each Kinetica process' (rank and host manager) logs to a separate file.

Important

This appender is demanding in terms of disk space.

Setting Description
File The name and directory of the files to which to send the logs. The files will have an -r<rank-number> or -hm appended for each rank and the host manager respectively.
Threshold The log level required to log messages. See Log Levels for more information.
Append Enables appending log messages to the end of the files. Leave as true.
ImmediateFlush Enables flushing the output stream to the files with each append operation.
layout Dictates the type of layout the logging of which the logging will take form.
layout.ConversionPattern The log4cplus pattern to use for the layout. See the PatternLayout class reference documentation for details.
UseLockFile

Enables the use of a logger lockfile to create a single log file per host.

Important

Setting this value to true results in slower performance as the system will need to check the file lock for each append. It's recommended this setting be left as false.

ToRollingFile

The Rolling File appender processes logs to a rolling log file where a new log file is created once the previous log file reaches the maximum file size. The maximum file size and number of files to keep is configurable.

Setting Description
File The name and directory of the files to which to send the logs. The files will have an -r<rank-number> or -hm appended for each rank and the host manager respectively. As files are periodically replaced, older files are renamed <filename>.1 to <filename>.n where n is equal to MaxBackupIndex.
Threshold The log level required to log messages. See Log Levels for more information.
MaxFileSize The maximum file size a log file can reach before a new one is created.
MaxBackupIndex The maximum number of log files to keep until older files are deleted to make room for new ones. Note that this setting defines the maximum backup index for each process, e.g., if MaxBackupIndex is set to 4, the host manager can have up to four files, rank 0 can have up to four files, etc.
Append Enables appending log messages to the end of the file. Leave as true.
ImmediateFlush Enables flushing the output stream to the file with each append operation.
layout Dictates the type of layout the logging of which the logging will take form.
layout.ConversionPattern The log4cplus pattern to use for the layout. See the PatternLayout class reference documentation for details.
UseLockFile

Enables the use of a logger lockfile to create a single log file per host.

Important

Setting this value to true results in slower performance as the system will need to check the file lock for each append. It's recommended this setting be left as false.

ToRollingDatedFile

The Rolling Dated File appender processes logs to a dated rolling log file where a new log file is created (with the time the file is created appended to the name) once the previous log file reaches the maximum file size. The maximum file size and number of files to keep is configurable.

Setting Description
File The name and directory of the files to which to send the logs. The files will have an -r<rank-number> or -hm appended for each rank and the host manager respectively. As files are periodically replaced, older files are renamed <filename-datetime>-1 to <filename-datetime>-n where n is equal to MaxBackupIndex.
Threshold The log level required to log messages. See Log Levels for more information.
MaxFileSize The maximum file size a log file can reach before a new one is created.
MaxBackupIndex The maximum number of log files to keep until older files are deleted to make room for new ones. Note that this setting defines the maximum backup index for each process, e.g., if MaxBackupIndex is set to 4, the host manager can have up to four files, rank 0 can have up to four files, etc.
Append Enables appending log messages to the end of the file. Leave as true.
ImmediateFlush Enables flushing the output stream to the file with each append operation.
layout Dictates the type of layout the logging of which the logging will take form.
layout.ConversionPattern The log4cplus pattern to use for the layout. See the PatternLayout class reference documentation for details.
UseLockFile

Enables the use of a logger lockfile to create a single log file per host.

Important

Setting this value to true results in slower performance as the system will need to check the file lock for each append. It's recommended this setting be left as false.

ToBacktraceFile

The Backtrace File appender redirects the backtrace logs generated by the ErrorBacktrace logger so they do not appear in the rootLogger output. Add the ERROR log level and the ToBacktraceFile appender to the ErrorBacktrace like so to properly setup this appender:

log4cplus.ErrorBacktrace=ERROR, ToBacktraceFile

Important

It's advised that this appender also be added to the rootLogger so that logs with a threshold of ERROR are also logged to give context to a backtrace.

Setting Description
File The name and directory of the file to which to send the logs.
Threshold The log level required to log messages. See Log Levels for more information.
Append Enables appending log messages to the end of the file. Leave as true.
ImmediateFlush Enables flushing the output stream to the file with each append operation.
layout Dictates the type of layout the logging of which the logging will take form.
layout.ConversionPattern The log4cplus pattern to use for the layout. See the PatternLayout class reference documentation for details.
additivity If using an alternate appender for the ErrorBacktrace logger, set the additivity to false to avoid duplicated the backtrace logs in the rootLogger output.

Loggers

Loggers control specific types of information. Each logger can have its own log level and appender. If the level or appender are not specified, the rootLogger level and/or appender(s) are used.

Loggers are divided into sections:

  • System and process information -- memory, CPU
  • GlobalManager and GlobalRegistry -- publish and subscribe sockets
  • All ranks (not exclusive to a particular rank) -- filter planner, job manager, security manager
  • Head node -- endpoint manager, graph client, WMS
  • Worker nodes -- bytes and GPU manager, queue, tasks

Auditing

The location of the request auditor logs can also be controlled via the gpudb_logger.conf file. Auditing must be enabled before the logging can be configured; see Auditing for more information.

The auditor uses its own appender to separate audit messages from other log messages. All audit messages are at the INFO or ERROR level; it's recommended the audit log level be kept to at least INFO. If desired, the auditor can use a different appender.

The audit log comes with a few preconfigured appender types that are similar to the rootLogger appenders.

Customizing the Log

  1. Review the available Log Levels.

    • OFF
    • FATAL
    • ERROR
    • WARN
    • UERR
    • INFO
    • DEBUG
    • TRACE
    • NOTSET

  2. Review the available Appenders.

    • ToConsole
    • ToSysLog
    • ToFile
    • ToRollingFile
    • ToRollingDatedFile
    • ToBacktraceFile

  3. Open /opt/gpudb/core/etc/gpudb_logger.conf with a text editor.

  4. Adjust the default logger.Audit setting as necessary:

    log4cplus.logger.Audit=INFO, AuditAppender

  5. Optionally, update the additivity setting if audit information is desired in both the rootLogger and the audit log. In most cases, this will not need to be updated.

  6. Optionally, uncomment one of the other audit appenders (File or Rolling File). If uncommenting a new appender, the Console appender will need to be commented out.

  7. Adjust the appender settings as necessary, e.g., log size, threshold, file name, etc.

  8. Save and exit the file.

  9. Restart the host manager and the database:

    service restart gpudb_host_manager && service restart gpudb

Audit Appenders

Console

The Console appender for the audit log will direct output to the rootLogger file(s).

Setting Description
layout Dictates the type of layout the logging of which the logging will take form.
layout.ConversionPattern The log4cplus pattern to use for the layout. See the PatternLayout class reference documentation for details.

File

The File appender for the audit log will direct output to a file.

Setting Description
File The name and directory of the files to which to send the logs. The files will have an -r<rank-number> or -hm appended for each rank and the host manager respectively.
layout Dictates the type of layout the logging of which the logging will take form.
layout.ConversionPattern The log4cplus pattern to use for the layout. See the PatternLayout class reference documentation for details.

Rolling File

The Rolling File appender for the audit log will direct output to a rolling file.

Setting Description
File The name and directory of the files to which to send the logs. The files will have an -r<rank-number> or -hm appended for each rank and the host manager respectively. As files are periodically replaced, older files are renamed <filename>.1 to <filename>.n where n is equal to MaxBackupIndex.
MaxFileSize The maximum file size a log file can reach before a new one is created.
MaxBackupIndex The maximum number of log files to keep until older files are deleted to make room for new ones. Note that this setting defines the maximum backup index for each process, e.g., if MaxBackupIndex is set to 4, the host manager can have up to four files, rank 0 can have up to four files, etc.
Append Enables appending log messages to the end of the file. Leave as false.
layout Dictates the type of layout the logging of which the logging will take form.
layout.ConversionPattern The log4cplus pattern to use for the layout. See the PatternLayout class reference documentation for details.