Version:

Configuration Reference

Kinetica configuration consists of several user-modifiable system parameters present in /opt/gpudb/core/etc/gpudb.conf that are used to set up and tune the system.

Network setup

Configuration parameter default value description
head_ip_address 127.0.0.1 Head HTTP server IP address. Set to the publicly accessible IP address of the first process, rank 0.
head_port 9191 Head HTTP server port to use for 'head_ip_address'.
use_https false Set to true to use HTTPS; if true then https_key_file and https_cert_file must be provided

https_key_file

https_cert_file

 

Files containing the SSL private Key and the SSL certificate for. If required, a self signed certificate (expires after 10 years) can be generated via the command

openssl req -newkey rsa:2048 -new -nodes -x509 -days 3650 -keyout key.pem -out cert.pem
http_allow_origin
Value to return via Access-Control-Allow-Origin HTTP header (for Cross-Origin Resource Sharing). Set to empty to not return the header and disallow CORS.
enable_httpd_proxy false Start a httpd server as a proxy to handle LDAP and/or Kerberos authentication. Each host will run a httpd server and access to each rank is available through http://host:8082/gpudb-1, where port 8082 is defined by 'httpd_proxy_port'. NOTE: HTTPd external endpoints are not affected by the use_https parameter above. If you wish to enable https on HTTPd, you must edit the /opt/gpudb/httpd/conf/httpd.conf and setup https as per the Apache HTTPd documentation at https://httpd.apache.org/docs/2.2/
httpd_proxy_port 8082 TCP port that the httpd auth proxy server will listen on if 'enable_httpd_proxy' is true.
httpd_proxy_use_https false Set to true if the httpd auth proxy server is configured to use HTTPS.
rank0_ip_address ${gaia.rank0.host} Internal use IP address of the head HTTP server, rank 0. Set to either a second internal network accessible by all ranks or to ${gaia.head_ip_address}.
trigger_port 9001 Trigger ZMQ publisher server port (-1 to disable), uses the 'head_ip_address' interface.
set_monitor_port 9002 Set monitor ZMQ publisher server port (-1 to disable), uses the 'head_ip_address' interface.
set_monitor_proxy_port 9003 Set monitor ZMQ publisher internal proxy server port (-1 to disable), uses the 'head_ip_address' interface. Disabling this port effectively prevents worker nodes from publishing set monitor notifications when multi-head ingest is enabled (see 'enable_worker_http_servers')
enable_odbc_connector true Enable ODBC connector
enable_reveal true Enable Reveal runtime
global_manager_port_one 5552 Internal communication ports
global_manager_pub_port 5553 Host manager synchronization port
host_manager_http_port 9300 Http port for web portal of the host manager
enable_worker_http_servers false Enable worker HTTP servers, each process runs its own server for direct ingest.

rank1.worker_http_server_port

rank2.worker_http_server_port

9192

9193

Optionally, specify the worker HTTP server ports. The default is to use (head_port + rank#) for each worker process where rank number is from 1 to number of ranks in 'rankN.host' below.

rank0.host

rank1.host

rank2.host

127.0.0.1

127.0.0.1

127.0.0.1

Optionally, specify a public URL for each worker HTTP server that clients should use to connect for multi-head operations. Note: If specified for any ranks, a public URL must be specified for all ranks. rank0.public_url = rank1.public_url = rank2.public_url = Specify the hosts to run each rank worker process in the cluster on. For a single machine system you may use '127.0.0.1', but if using two or more machines you must specify a hostname or IP address for each rank that is accessible to the other ranks. See also 'head_ip_address' and 'rank0_ip_address'.
rank0.communicator_port 6555 Specify the TCP ports each rank will use to communicate with each other. If the port for any rankN is not specified the port will be assigned to rank0.communicator_port + N.
compress_network_data false rank1.communicator_port = 6556 rank2.communicator_port = 6557 Enables compression of inter-node network data transfers.
communicator_type ZMQ Communicator type (either MPI or ZMQ)

Security

Configuration parameter default value description
require_authentication false Require authentication.
enable_authorization false Enable authorization checks.
min_password_length 0 Minimum password length.
enable_external_authentication false Enable external (LDAP, Kerberos, etc.) authentication. User IDs of externally-authenticated users must be passed in via the REMOTE_USER HTTP header from the authentication proxy. May be used in conjuntion with the enable_httpd_proxy setting above for an integrated external authentication solution. IMPORTANT: DO NOT ENABLE unless external access to GPUdb ports has been blocked via firewall AND the authentication proxy is configured to block REMOTE_USER HTTP headers passed in from clients.
auto_create_external_users false Automatically create accounts for externally-authenticated users. If enable_external_authentication is false, this setting has no effect. Note that accounts are not automatically deleted if users are removed from the external authentication provider and will be orphaned.
auto_grant_external_roles false Automatically add roles passed in via the KINETICA_ROLES HTTP header to externally-authenticated users. Specified roles that do not exist are ignored. If enable_external_authentication is false, this setting has no effect. IMPORTANT: DO NOT ENABLE unless the authentication proxy is configured to block KINETICA_ROLES HTTP headers passed in from clients.
auto_revoke_external_roles   Comma-separated list of roles to revoke from externally-authenticated users prior to granting roles passed in via the KINETICA_ROLES HTTP header, or * to revoke all roles. Preceding a role name with an ! overrides the revocation (e.g. *,!foo revokes all roles except foo). Leave blank to disable. If either enable_external_authentication or auto_grant_external_roles is false, this setting has no effect.

Auditing

Configuration parameter default value description
enable_audit false This section controls the request auditor, which will audit all requests received by the server in full or in part based on the settings below. The output location of the audited requests is controlled via settings in the Auditing section of gpudb_logger.conf. enable_audit: Controls whether request auditing is enabled. If set to true, the following information is audited for every request: Job ID, URI, User, and Client Address. The settings below control whether additional information about each request is also audited. If set to false, all auditing is disabled.
audit_headers false audit_headers: Controls whether HTTP headers are audited for each request. If enable_audit is false this setting has no effect.
audit_body false audit_body: Controls whether the body of each request is audited (in JSON format). If enable_audit is false this setting has no effect. Note: For requests that insert data records, this setting does not control the auditing of the records being inserted, only the rest of the request body; see audit_data below to control this.
audit_data false audit_data: Controls whether records being inserted are audited (in JSON format) for requests that insert data records. If either enable_audit or audit_body is false, this setting has no effect. Note: Enabling this setting during bulk ingestion of data will rapidly produce very large audit logs and may cause disk space exhaustion; use with caution.
lock_audit false lock_audit: Controls whether the above audit settings can be altered at runtime via the /alter/system/properties endpoint. In a secure environment where auditing is required at all times, this should be set to true to lock the settings to what is set in this file.

Licensing

Configuration parameter default value description
license_key   The license key to authorize running.

Process and thread configuration

Configuration parameter default value description
min_http_threads 2 Set min number of web server threads to spawn. (default: 2)
max_http_threads 64 Set max number of web server threads to spawn. (default: 64)
sm_omp_threads 2 Set the number of parallel jobs to create for multi-child set calculations. Use -1 to use the max number of threads (not recommended).
kernel_omp_threads 4 Set the number of parallel calculation threads to use for data processing. Use -1 to use the max number of threads (not recommended).
max_tbb_threads_per_rank -1 Set the maximum number of threads (both workers and masters) to be passed to TBB on initialization. Generally speaking, max_tbb_threads_per_rank - 1 TBB workers will be created.
toms_per_rank 1 Set the number of TOMs per rank, the number of data container shards per rank.
tps_per_tom 4 Set the number of TaskProcessors per TOM, CPU data processors.
tcs_per_tom 4 Set the number of TaskCalculators per TOM, GPU data processors.
persistence_replication_factor 1 Set the number of Backup TOMs, per TOM.

Hardware configuration

Configuration parameter default value description
rank0.gpu 0 Specify the GPU to use for all calculations on the HTTP server node, rank 0. Note that the rank0 GPU may be shared with another rank.

rank1.taskcalc_gpu

rank2.taskcalc_gpu

 

Set GPU devices for each worker rank's TaskCalculators, see 'tcs_per_tom'. If no gpus are specified, each rank's TaskCalculators will share the same GPU and each rank will round-robin the available gpus on the system. Ideally, each rank should use a single specified GPU to allow data caching. Add rankN.taskcalc_gpu as needed, where N ranges from 1 to the highest index in 'rankN.host' For the example below the successively created taskcalcs use the 0,1,2 GPU devices respectively

rank1.taskcalc_gpu = 0 1 2
rank0.numa_node   Set the head HTTP rank0 numa node(s). If left empty there will be no thread affinity or preferred memory node. The node list may be either a single node number or a range, E.G. 1-5,7,10. If there will be many simultaneous users, specify as many nodes as possible that won't overlap the rank1+ worker numa nodes that the GPUs are on. If there will be few simultaneous users and WMS speed is important, choose the numa node the 'rank0.gpu' is on.

rank1.base_numa_node

rank2.base_numa_node

  Set each worker rank's preferred base numa node for CPU affinity and memory allocation. The 'rankN.base_numa_node' is the node or nodes that non-data intensive threads will run in. These nodes do not have to be the same numa nodes that the GPU specified by the corresponding 'rankN.taskcalc_gpu' is on for best performance, though they should be relatively near to their 'rankN.data_numa_node'. There will be no CPU thread affinity or preferred node for memory allocation if not specified or left empty. The node list may be a single node number or a range, E.G. 1-5,7,10.

rank1.data_numa_node

rank2.data_numa_node

  Set each worker rank's preferred data numa node for CPU affinity and memory allocation. The 'rankN.data_numa_node' is the node or nodes that data intensive threads will run in and should be set to the same numa node that the GPU specified by the corresponding 'rankN.taskcalc_gpu' is on for best performance. If the 'rankN.taskcalc_gpu' is specified the 'rankN.data_numa_node' will be automatically set to the node the GPU is attached to, otherwise there will be no CPU thread affinity or preferred node for memory allocation if not specified or left empty. The node list may be a single node number or a range, E.G. 1-5,7,10.

General configuration

Configuration parameter default value description
enable_job_manager true Enable JobManager to enforce processing of requests in the order received
protected_sets MASTER,_MASTER,_DATASOURCE Tables with these names will not be deleted (comma separated).
default_ttl 20 Time-to-live in minutes of non-protected tables before they are deleted.
memory_ttl 9999999 Time in minutes for an unused data tables to stay in memory.
disable_clear_all false Disallow the 'clear/table' command to clear all tables.
point_render_threshold 100000 Threshold number of points (per-TOM) at which point rendering switches to fast mode.
symbology_render_threshold 10000 Threshold for the number of points (per-TOM) after which symbology rendering falls back to regular rendering
max_heatmap_size 3072 Maximum size (in pixels) heatmap that can be generated. This reserves MHS*MHS*8 bytes of GPU memory at rank0
symbol_resolution 100 The image width/height (in pixels) of svg symbols cached in the OpenGL symbol cache.
symbol_texture_size 4000 The width/height (in pixels) of an OpenGL texture which caches symbol images for OpenGL rendering.
enable_opengl_renderer true Enable (if true) HW-accelerated OpenGL renderer instead of Cairo renderer (software renderer).
enable_lod_rendering false Enable level-of-details rendering for fast interaction with a large WKT polygon data. Only available for the OpenGL renderer (enable_opengl_renderer = true).
lod_data_extent -180 -90 180 90 Longitude and latitude ranges of geospatial data for which level-of-details representations are being generated. The order of parameters are min_longitude, min_latitude, max_longitude, and max_latitude. The default values span over the World, but the level-of-details rendering becomes more efficient when the precise extent of geospatial data is specified.
lod_subregion_num 12 6 The number of subregions in horizontal and vertical geospatial data extent. The default parameters divide the World as subregions of 30 degree (lon.) x 30 degree (lat.)
lod_subregion_resolution 512 512 A base image resolution (width and height in pixels) at which a subregion would be rendered in a global view spanning over the whole dataset. Based on this resolution level-of-details representations are generated for the polygons located in the subregion.
max_lod_level 8 The maximum number of levels in the level of details rendering. As the number increases, level-of-details rendering becomes effective at higher zoom levels, but it may increase memory usage for storing level-of-details representations.
lod_preprocessing_level 5 This parameter determines how much shape data are pre-processed for level-of-details rendering during data insert/load or processed on-the-fly in rendering time. This parameter is a trade off between speed and memory. The higher the value, the faster level-of-details rendering is but uses more memory for storing processed shape data. The maximum level is 10 (most shape data are pre-processed) and the minimum level is 0.
max_memory_for_shape_processing 8192 This is the upper bound of system memory usage (in megabytes) for shape processing to generate level-of-details representations on each rank. This parameter overrides lod_preprocessing_level, i.e. stops shape pre-processing, when the memory usage reaches this upper bound. If the value is set to zero, shape processing is performed without memory usage restriction.
enable_pinned_memory false Enable (if true) or disable pinned memory; disabling pinned memory uses malloc and will result in slower transfer times to the GPU, but allows more system memory to be used.
pinned_memory_pool_size 250000000 Size in bytes of the pinned memory pool per-rank process to speed copying data to the GPU. Set to 0 to disable and 'enable_pinned_memory' must be 'false' if enabled.
memory_allocation_limit_mb 0

Maximum host memory allocation limit per rank, 0 means no limit. Specify the default maximum allowed memory allocation size for each rank and optionally override the limit for individual ranks that are running on machines with either more or less memory than the others. This is hard limit and requests requiring large allocations will be refused. Allow for at least 1GB of extra memory per rank and account for the number of other GPUdb ranks and processes running on each machine.

Example, each rank will allocate ~128GB of memory, but rank three and four will allocate up to ~256GB.

memory_allocation_limit_mb = 128000
rank3.memory_allocation_limit_mb = 256000
rank4.memory_allocation_limit_mb = 256000
concurrent_kernel_execution false Enable (if true) multiple kernels to run concurrently on the same GPU
max_concurrent_kernels 4 Maximum number of kernels that can be running at the same time on a given GPU. Set to 0 for no limit. Only takes effect if concurrent_kernel_execution = true
enable_gpu_allocator true Enable (if true) the GPU memory pre-allocator
max_gpu_memory -1 Maximum amount of GPU memory to allocate in bytes, set to -1 to allocate the maximum. Only works if enable_gpu_allocator is true.
fast_polling false Enable (if true) fast polling on the worker ranks (at the costof 100% cpu usage)
max_get_records_size 20000 Maximum number of records that data retrieval requests such as /get/records and /aggregate/groupby will return per request.
request_timeout 20 Timeout (in minutes) for filter-type requests
max_query_temp_memory 64000000000 Maximum amount of temporary memory (in bytes) a query can allocate (per-tom); if more is needed the query will fail
on_startup_script   Set an optional executable command that will be run once when Kinetica is ready for client requests. This can be used to perform any initialization logic that needs to be run before clients connect. It will be run as the gpudb user, so you must ensure that any required permissions are set on the file to allow it to be executed. If the command cannot be executed or returns a non zero error code, then Kinetica will be stopped. Output from the startup script will be logged to /opt/gpudb/core/logs/gpudb-on-start.log (and its dated relatives). The gpudb_env.sh script is run directly before the command, so the path will be set to include the supplied python runtime. Example: on_startup_script = /home/gpudb/on-start.sh param1 param2 ...
timeout_startup_subsystem 60 Timeout (in seconds) to wait for each database subsystem to startup. Subsystems include the ODBC, stats and httpd servers, as well as external text-search ranks.
timeout_shutdown_subsystem 20 Timeout (in seconds) to wait for each database subsystem to exit gracefully before it is force killed.
timeout_shutdown_rank 300 Timeout (in seconds) to wait for a rank to exit gracefully before it is force killed. Machines with slow disk drives may require longer times and data may be lost if drive is not responsive.

Tomcat Configuration

Configuration parameter default value description
enable_tomcat true  

Text search configuration

Configuration parameter default value description
enable_text_search true Enable text search.
use_external_text_server false Use an external text server instead of an internal one. Be sure to update the rankN.text_index_address and rankN.text_search_address params below.
text_indices_per_tom 2 Number of text indices to start for each rank
text_searcher_refresh_interval 20 Searcher refresh intervals - specifies the maximum delay (in seconds) between writing to the text search index and being able to search for the value just written. A value of 0 insures that writes to the index are immediately available to be searched. A more nominal value of 100 should improve ingest speed at the cost of some delay in being able to text search newly added values.

rank1.text_index_address

rank2.text_index_address

ipc:///tmp/gpudb-text-index-1

ipc:///tmp/gpudb-text-index-2

External text server addresses to use if

'use_external_text_server = true'.

Specify one for each worker rank N, where N ranges from 1 to highest index in 'rankN.host'. Add the appropriate number of rankN.text_index_address for each worker rank as needed. The addresses can be a fully qualified TCP address:port for remote servers or an IPC address for local text index servers. If no addresses are specified the text index servers will use IPC and be started on the machine where the rank is running as shown in the IPC example below. You should either specify all addresses or none to get the defaults. Example for remote or TCP servers:

rank1.text_index_address  = tcp://127.0.0.1:4000
rank2.text_index_address  = tcp://127.0.0.1:4001
... up to rank[N].text_index_address = ...

Example for local IPC servers:

rank1.text_index_address  = ipc:///tmp/gpudb-text-index-1
rank2.text_index_address  = ipc:///tmp/gpudb-text-index-2
... up to rank[N].text_index_address = ...

Where '/tmp/gpudb-text-index-1' is the name of the socket file to create.

Data persistence storage configuration and directories

Configuration parameter default value description
persist_directory /opt/gpudb/persist Specify a base directory to store persistence data files.
data_directory ${gaia.persist_directory} Base directory to store data vectors.
object_directory ${gaia.persist_directory} Base directory to store added objects.
sms_directory ${gaia.persist_directory} Base directory to store hashed strings.
text_index_directory ${gaia.persist_directory} Base directory to store the text search index.
temp_directory /tmp Directory for GPUdb to use to store temporary files. Must be a fully qualified path, have at least 100Mb of free space, and execute permission.
persist_encryption_pass_command   Path to script that echos the password for mounting encrypted persistence dirs. Note that all directories must be encrypted with the same password. Leave blank if the persistence directories are not encrypted or if the directories will be mounted before GPUdb starts.
persist_sync false Synchronous persistence file writing instead of asynchronous writing.
persist_sync_time 5 Force syncing the persistence files every N minutes if out of sync. Note that files are always opportunistically saved, this simply enforces a maximum time a file can be out of date. Set to a very high number to disable.
synchronous_update true If true then updates are applied to IndexDB synchronously (slower) If false then updates are asynchronous (faster but less safe)
load_vectors_on_start always When starting, 'always' load all the data into memory before accepting requests, load the necessary data to start, but load the remainder lazily, or only load the data 'on_demand' as requests use it. load_vectors_on_start = 'always' / 'lazy' / 'on_demand'
indexdb_toc_size 1000000 Table of contents size for IndexedDb object file store
indexdb_max_open_files 128 Maximum number of open files for IndexedDb object file store
sms_max_open_files 128 Maximum number of open files (per-TOM) for the SMS (string) store
chunk_size 8000000 Chunk size (0 disables chunking)
max_join_query_dimensions 2 Max join dimensions is the maximum number of tables in a joined table that can be accessed by a query and are not equated by a foreign-key to primary-key equality predicate.
execution_mode device Determines whether to execute kernels on host (cpu) or device (gpu). Possible values are: 'default' : engine decides 'host' : execute only host 'device' : execute only device <integer> : execute on the host if chunked column has equal or fewer rows, otherwise execute on device.

Stats logging configuration

Configuration parameter default value description
enable_stats_server true Run a stats server to collect information about GPUdb and machines it is running on.

stats_server_ip_address

stats_server_port

${gaia.rank0_ip_address}

2003

Stats server ip address (run on head node) default port is 2003
stats_server_namespace gpudb Status server namepace - should be a machine identifier

Procs

Configuration parameter default value description
enable_procs false Enable procs
proc_directory /opt/gpudb/procs Directory where proc files are stored at runtime. Must be a fully qualified path with execute permission. If not specified, temp_directory will be used.
proc_data_directory /opt/gpudb/procs Directory where data transferred to and from procs is written. Must be a fully qualified path with sufficient free space for required volume of data. If not specified, temp_directory will be used.

KiFS

Configuration parameter default value description
enable_kifs false Enable access to the KiFS file system from within procs (UDFs). This will create a 'filesystem' collection and an internal 'kifs' user in Kinetica and mount the file system at the mount point specified below. By default, the file system is accessible to the gpudb_proc user; to make it accessible to other users, you must ensure that user_allow_other is enabled in /etc/fuse.conf and add the other users to the gpudb_proc group (sudo usermod -a -G gpudb_proc <user>).
kifs_mount_point /opt/gpudb/kifs Parent directory of the mount point for the KiFS file system. Must be a fully qualified path. The actual mount point will be a subdirectory 'mount' below this directory. Note that this folder must have read, write and execute permissions for the gpudb user and the gpudb_proc group, and it cannot be a path on an NFS.

HA

Configuration parameter default value description
enable_ha false At the moment, this section is simply to provide information about if and where HA is running, so that the gpudb startup script can properly create proxy-pass entries for httpd when both HA and httpd are enabled. Enable HA.
ha_use_https false Set to true if HA is configured to use HTTPS.

Alerts

Configuration parameter default value description
enable_alerts true Enable the alerting system.
alert_exe   Executable to run when an alert condition occurs. This executable will only be run on rank 0 and does not need to be present on other nodes.

alert_host_status

alert_host_status_filter

alert_rank_status

alert_rank_status_filter

true

fatal_init_error

true

fatal_init_error, not_responding, terminated

Trigger an alert whenever the status of a host or rank changes. Optionally filter so that only a comma-delimited list of statuses trigger alerts. If a filter is empty every status change will trigger an alert.
alert_rank_cuda_error true Trigger an alert if a CUDA error occurs on a rank.
alert_rank_fallback_allocator true Trigger alerts when we hit a fallback allocator, e.g. host memory is allocated because GPU allocation fails. To prevent a flooding of alerts, if a fallback allocator is hit in bursts, not every single hit will necessarily generate an alert.
alert_error_messages true Trigger generic error message alerts, in cases of various significant runtime errors.

alert_memory_absolute

alert_memory_percentage

20, 10, 5, 1 Trigger an alert if available memory on any given node falls to or below a certain threshold, either absolute (number of bytes) or percentage of total memory. For multiple thresholds, use a comma-delimited list of values.

alert_disk_absolute

alert_disk_percentage

20, 10, 5, 1 Trigger an alert if available disk space on any given node falls to or below a certain threshold, either absolute (number of bytes) or percentage of total disk space. For multiple thresholds, use a comma-delimited list of values.
alert_max_stored_alerts 100 The maximum number of triggered alerts guaranteed to be stored at any given time. When this number of alerts is exceeded, older alerts may be discarded to stay within the limit.
chunk_cache_enabled false True if chunk caching enabled
chunk_cache_size 100000000 The maximum number of bytes in the chunk cache
trace_directory /tmp Directory where the tracer event and summary files are stored. Must be a fully qualified path with sufficient free space for required volume of data. If not specified, temp_directory will be used.
trace_event_buffer_size 1000000 The maximum number of tracer events to be collected

Internal developer options