Version:

Configuration Reference

Kinetica configuration consists of several user-modifiable system parameters present in /opt/gpudb/core/etc/gpudb.conf that are used to set up and tune the system.

Network setup

Configuration parameter default value description
head_ip_address 127.0.0.1 Head HTTP server IP address. Set to the publicly accessible IP address of the first process, rank 0.
head_port 9191 Head HTTP server port to use for 'head_ip_address'.
use_https false Set to true to use HTTPS; if true then https_key_file and https_cert_file must be provided

https_key_file

https_cert_file

 

Files containing the SSL private Key and the SSL certificate for. If required, a self signed certificate (expires after 10 years) can be generated via the command

openssl req -newkey rsa:2048 -new -nodes -x509 -days 3650 -keyout key.pem -out cert.pem
http_allow_origin
Value to return via Access-Control-Allow-Origin HTTP header (for Cross-Origin Resource Sharing). Set to empty to not return the header and disallow CORS.
enable_httpd_proxy false Start a httpd server as a proxy to handle LDAP and/or Kerberos authentication. Each host will run a httpd server and access to each rank is available through http://host:8082/gpudb-1, where port 8082 is defined by 'httpd_proxy_port'. NOTE: HTTPd external endpoints are not affected by the use_https parameter above. If you wish to enable https on HTTPd, you must edit the /opt/gpudb/httpd/conf/httpd.conf and setup https as per the Apache HTTPd documentation at https://httpd.apache.org/docs/2.2/
httpd_proxy_port 8082 TCP port that the httpd auth proxy server will listen on if 'enable_httpd_proxy' is true.
rank0_ip_address ${gaia.rank0.host} Internal use IP address of the head HTTP server, rank 0. Set to either a second internal network accessible by all ranks or to ${gaia.head_ip_address}.
trigger_port 9001 Trigger ZMQ publisher server port (-1 to disable), uses the 'head_ip_address' interface.
set_monitor_port 9002 Set monitor ZMQ publisher server port (-1 to disable), uses the 'head_ip_address' interface.
enable_odbc_connector true Enable ODBC connector
enable_reveal true Enable Reveal runtime
global_manager_port_one 5552 Internal communication ports
global_manager_pub_port 5553 Host manager synchronization port
host_manager_http_port 9300 Http port for web portal of the host manager
enable_worker_http_servers false Enable worker HTTP servers, each process runs its own server for direct ingest.

rank1.worker_http_server_port

rank2.worker_http_server_port

9192

9193

Optionally, specify the worker HTTP server ports. The default is to use (head_port + rank#) for each worker process where rank number is from 1 to number of rowns in 'ranksN.host' below.

rank0.host

rank1.host

rank2.host

127.0.0.1

127.0.0.1

127.0.0.1

Specify the hosts to run each rank worker process in the cluster on. For a single machine system you may use '127.0.0.1', but if using two or more machines you must specify a hostname or IP address for each rank that is accessible to the other ranks. See also 'head_ip_address' and 'rank0_ip_address'.
compress_network_data false Enables compression of inter-node network data transfers.
communicator_type ZMQ Communicator type (either MPI or ZMQ)

Security

Configuration parameter default value description
require_authentication false Require authentication.
enable_authorization false Enable authorization checks.
min_password_length 0 Minimum password length.
enable_external_authentication false Enable external (LDAP, Kerberos, etc.) authentication. User IDs of externally-authenticated users must be passed in via the REMOTE_USER HTTP header from the authentication proxy. May be used in conjuntion with the enable_httpd_proxy setting above for an integrated external authentication solution. IMPORTANT: DO NOT ENABLE unless external access to GPUdb ports has been blocked via firewall AND the authentication proxy is configured to block REMOTE_USER HTTP headers passed in from clients.
auto_create_external_users false Automatically create accounts for externally-authenticated users. If enable_external_authentication is false, this setting has no effect. Note that accounts are not automatically deleted if users are removed from the external authentication provider and will be orphaned.
auto_grant_external_roles false Automatically add roles passed in via the KINETICA_ROLES HTTP header to externally-authenticated users. Specified roles that do not exist are ignored. If enable_external_authentication is false, this setting has no effect. IMPORTANT: DO NOT ENABLE unless the authentication proxy is configured to block KINETICA_ROLES HTTP headers passed in from clients.
auto_revoke_external_roles   Comma-separated list of roles to revoke from externally-authenticated users prior to granting roles passed in via the KINETICA_ROLES HTTP header, or * to revoke all roles. Preceding a role name with an ! overrides the revocation (e.g. *,!foo revokes all roles except foo). Leave blank to disable. If either enable_external_authentication or auto_grant_external_roles is false, this setting has no effect.

Auditing

Configuration parameter default value description
enable_audit false This section controls the request auditor, which will audit all requests received by the server in full or in part based on the settings below. The output location of the audited requests is controlled via settings in the Auditing section of gpudb_logger.conf. enable_audit: Controls whether request auditing is enabled. If set to true, the following information is audited for every request: Job ID, URI, User, and Client Address. The settings below control whether additional information about each request is also audited. If set to false, all auditing is disabled.
audit_headers false audit_headers: Controls whether HTTP headers are audited for each request. If enable_audit is false this setting has no effect.
audit_body false audit_body: Controls whether the body of each request is audited (in JSON format). If enable_audit is false this setting has no effect. Note: For requests that insert data records, this setting does not control the auditing of the records being inserted, only the rest of the request body; see audit_data below to control this.
audit_data false audit_data: Controls whether records being inserted are audited (in JSON format) for requests that insert data records. If either enable_audit or audit_body is false, this setting has no effect. Note: Enabling this setting during bulk ingestion of data will rapidly produce very large audit logs and may cause disk space exhaustion; use with caution.
lock_audit false lock_audit: Controls whether the above audit settings can be altered at runtime via the /alter/system/properties endpoint. In a secure environment where auditing is required at all times, this should be set to true to lock the settings to what is set in this file.

Licensing

Configuration parameter default value description
license_key   The license key to authorize running.

Process and thread configuration

Configuration parameter default value description
min_http_threads 2 Set min number of web server threads to spawn. (default: 2)
max_http_threads 512 Set max number of web server threads to spawn. (default: 512)
sm_omp_threads 2 Set the number of parallel jobs to create for multi-child set calculations. Use -1 to use the max number of threads (not recommended).
kernel_omp_threads 4 Set the number of parallel calculation threads to use for data processing. Use -1 to use the max number of threads (not recommended).
max_tbb_threads_per_rank -1 Set the maximum number of threads (both workers and masters) to be passed to TBB on initialization. Generally speaking, max_tbb_threads_per_rank - 1 TBB workers will be created.
toms_per_rank 4 Set the number of TOMs per rank, the number of data container shards per rank.
tps_per_tom 4 Set the number of TaskProcessors per TOM, CPU data processors.
tcs_per_tom 4 Set the number of TaskCalculators per TOM, GPU data processors.
persistence_replication_factor 1 Set the number of Backup TOMs, per TOM.

Hardware configuration

Configuration parameter default value description
rank0.gpu 0 Specify the GPU to use for all calculations on the HTTP server node, rank 0. Note that the rank0 GPU may be shared with another rank.

rank1.taskcalc_gpu

rank2.taskcalc_gpu

 

Set GPU devices for each worker rank's TaskCalculators, see 'tcs_per_tom'. If no gpus are specified, each rank's TaskCalculators will share the same GPU and each rank will round-robin the available gpus on the system. Ideally, each rank should use a single specified GPU to allow data caching. Add rankN.taskcalc_gpu as needed, where N ranges from 1 to the highest index in 'rankN.host' For the example below the successively created taskcalcs use the 0,1,2 GPU devices respectively

rank1.taskcalc_gpu = 0 1 2
rank0.numa_node   Set the head HTTP rank0 numa node(s). If left empty there will be no thread affinity or preferred memory node. The node list may be either a single node number or a range, E.G. 1-5,7,10. If there will be many simultaneous users, specify as many nodes as possible that won't overlap the rank1+ worker numa nodes that the GPUs are on. If there will be few simultaneous users and WMS speed is important, choose the numa node the 'rank0.gpu' is on.

rank1.base_numa_node

rank2.base_numa_node

  Set each worker rank's preferred base numa node for CPU affinity and memory allocation. The 'rankN.base_numa_node' is the node or nodes that non-data intensive threads will run in. These nodes do not have to be the same numa nodes that the GPU specified by the corresponding 'rankN.taskcalc_gpu' is on for best performance, though they should be relatively near to their 'rankN.data_numa_node'. There will be no CPU thread affinity or preferred node for memory allocation if not specified or left empty. The node list may be a single node number or a range, E.G. 1-5,7,10.

rank1.data_numa_node

rank2.data_numa_node

  Set each worker rank's preferred data numa node for CPU affinity and memory allocation. The 'rankN.data_numa_node' is the node or nodes that data intensive threads will run in and should be set to the same numa node that the GPU specified by the corresponding 'rankN.taskcalc_gpu' is on for best performance. If the 'rankN.taskcalc_gpu' is specified the 'rankN.data_numa_node' will be automatically set to the node the GPU is attached to, otherwise there will be no CPU thread affinity or preferred node for memory allocation if not specified or left empty. The node list may be a single node number or a range, E.G. 1-5,7,10.

General configuration

Configuration parameter default value description
enable_job_manager true Enable JobManager to enforce processing of requests in the order received
protected_sets MASTER,_MASTER,_DATASOURCE Tables with these names will not be deleted (comma separated).
default_ttl 20 Time-to-live in minutes of non-protected tables before they are deleted.
memory_ttl 9999999 Time in minutes for an unused data tables to stay in memory.
disable_clear_all false Disallow the 'clear/table' command to clear all tables.
point_render_threshold 100000 Threshold number of points (per-TOM) at which point rendering switches to fast mode.
symbology_render_threshold 10000 Threshold for the number of points (per-TOM) after which symbology rendering falls back to regular rendering
max_heatmap_size 3072 Maximum size (in pixels) heatmap that can be generated. This reserves MHS*MHS*8 bytes of GPU memory at rank0
enable_opengl_renderer true Enable (if true) HW-accelerated OpenGL renderer instead of Cairo renderer (software renderer).
enable_pinned_memory false Enable (if true) or disable pinned memory; disabling pinned memory uses malloc and will result in slower transfer times to the GPU, but allows more system memory to be used.
pinned_memory_pool_size 250000000 Size in bytes of the pinned memory pool per-rank process to speed copying data to the GPU. Set to 0 to disable and 'enable_pinned_memory' must be 'false' if enabled.
memory_allocation_limit_mb 0

Maximum host memory allocation limit per rank, 0 means no limit. Specify the default maximum allowed memory allocation size for each rank and optionally override the limit for individual ranks that are running on machines with either more or less memory than the others. This is hard limit and requests requiring large allocations will be refused. Allow for at least 1GB of extra memory per rank and account for the number of other GPUdb ranks and processes running on each machine.

Example, each rank will allocate ~128GB of memory, but rank three and four will allocate up to ~256GB.

memory_allocation_limit_mb = 128000
rank3.memory_allocation_limit_mb = 256000
rank4.memory_allocation_limit_mb = 256000
concurrent_kernel_execution true Enable (if true) multiple kernels to run concurrently on the same GPU
enable_gpu_allocator true Enable (if true) the GPU memory pre-allocator
max_gpu_memory -1 Maximum amount of GPU memory to allocate in bytes, set to -1 to allocate the maximum. Only works if enable_gpu_allocator is true.
fast_polling false Enable (if true) fast polling on the worker ranks (at the costof 100% cpu usage)
max_get_records_size 20000 Maximum number of records that data retrieval requests such as /get/records and /aggregate/groupby will return per request.
request_timeout 20 Timeout (in minutes) for filter-type requests
max_query_temp_memory 64000000000 Maximum amount of temporary memory (in bytes) a query can allocate (per-tom); if more is needed the query will fail
on_startup_script   Set an optional executable command that will be run once when Kinetica is ready for client requests. This can be used to perform any initialization logic that needs to be run before clients connect. It will be run as the gpudb user, so you must ensure that any required permissions are set on the file to allow it to be executed. If the command cannot be executed or returns a non zero error code, then Kinetica will be stopped. Output from the startup script will be logged to /opt/gpudb/core/logs/gpudb-on-start.log (and its dated relatives). The gpudb_env.sh script is run directly before the command, so the path will be set to include the supplied python runtime. Example: on_startup_script = /home/gpudb/on-start.sh param1 param2 ...
timeout_startup_subsystem 60 Timeout (in seconds) to wait for each database subsystem to startup. Subsystems include the ODBC, stats and httpd servers, as well as external text-search ranks.
timeout_shutdown_subsystem 20 Timeout (in seconds) to wait for each database subsystem to exit gracefully before it is force killed.
timeout_shutdown_rank 300 Timeout (in seconds) to wait for a rank to exit gracefully before it is force killed. Machines with slow disk drives may require longer times and data may be lost if drive is not responsive.

Tomcat Configuration

Configuration parameter default value description
enable_tomcat true  

Text search configuration

Configuration parameter default value description
enable_text_search true Enable text search.
use_external_text_server false Use an external text server instead of an internal one. Be sure to update the rankN.text_index_address and rankN.text_search_address params below.
text_indices_per_tom 2 Number of text indices to start for each rank
text_searcher_refresh_interval 20 Searcher refresh intervals - specifies the maximum delay (in seconds) between writing to the text search index and being able to search for the value just written. A value of 0 insures that writes to the index are immediately available to be searched. A more nominal value of 100 should improve ingest speed at the cost of some delay in being able to text search newly added values.

rank1.text_index_address

rank2.text_index_address

ipc:///tmp/gpudb-text-index-1

ipc:///tmp/gpudb-text-index-2

External text server addresses to use if

'use_external_text_server = true'.

Specify one for each worker rank N, where N ranges from 1 to highest index in 'rankN.host'. Add the appropriate number of rankN.text_index_address for each worker rank as needed. The addresses can be a fully qualified TCP address:port for remote servers or an IPC address for local text index servers. If no addresses are specified the text index servers will use IPC and be started on the machine where the rank is running as shown in the IPC example below. You should either specify all addresses or none to get the defaults. Example for remote or TCP servers:

rank1.text_index_address  = tcp://127.0.0.1:4000
rank2.text_index_address  = tcp://127.0.0.1:4001
... up to rank[N].text_index_address = ...

Example for local IPC servers:

rank1.text_index_address  = ipc:///tmp/gpudb-text-index-1
rank2.text_index_address  = ipc:///tmp/gpudb-text-index-2
... up to rank[N].text_index_address = ...

Where '/tmp/gpudb-text-index-1' is the name of the socket file to create.

Data persistence storage configuration and directories

Configuration parameter default value description
persist_directory /opt/gpudb/persist Specify a base directory to store persistence data files.
data_directory ${gaia.persist_directory} Base directory to store data vectors.
object_directory ${gaia.persist_directory} Base directory to store added objects.
sms_directory ${gaia.persist_directory} Base directory to store hashed strings.
text_index_directory ${gaia.persist_directory} Base directory to store the text search index.
temp_directory /tmp Directory for GPUdb to use to store temporary files. Must be a fully qualified path, have at least 100Mb of free space, and execute permission.
persist_encryption_pass_command   Path to script that echos the password for mounting encrypted persistence dirs. Note that all directories must be encrypted with the same password. Leave blank if the persistence directories are not encrypted or if the directories will be mounted before GPUdb starts.
persist_sync false Synchronous persistence file writing instead of asynchronous writing.
persist_sync_time 5 Force syncing the persistence files every N minutes if out of sync. Note that files are always opportunistically saved, this simply enforces a maximum time a file can be out of date. Set to a very high number to disable.
synchronous_update true If true then updates are applied to IndexDB synchronously (slower) If false then updates are asynchronous (faster but less safe)
load_vectors_on_start always When starting, 'always' load all the data into memory before accepting requests, load the necessary data to start, but load the remainder lazily, or only load the data 'on_demand' as requests use it. load_vectors_on_start = 'always' / 'lazy' / 'on_demand'
indexdb_toc_size 1000000 Table of contents size for IndexedDb object file store
indexdb_max_open_files 128 Maximum number of open files for IndexedDb object file store
sms_max_open_files 128 Maximum number of open files (per-TOM) for the SMS (string) store
chunk_size 8000000 Chunk size (0 disables chunking)
max_join_query_dimensions 2 Max join dimensions is the maximum number of tables in a joined table that can be accessed by a query and are not equated by a foreign-key to primary-key equality predicate.
execution_mode device Determines whether to execute kernels on host (cpu) or device (gpu). Possible values are: 'default' : engine decides 'host' : execute only host 'device' : execute only device <integer> : execute on the host if chunked column has equal or fewer rows, otherwise execute on device.

Stats logging configuration

Configuration parameter default value description
enable_stats_server true Run a stats server to collect information about GPUdb and machines it is running on.

stats_server_ip_address

stats_server_port

${gaia.rank0_ip_address}

2003

Stats server ip address (run on head node) default port is 2003
stats_server_namespace gpudb Status server namepace - should be a machine identifier

Procs

Configuration parameter default value description
enable_procs false Enable procs
proc_directory ${gaia.temp_directory} Directory where proc files are stored at runtime. Must be a fully qualified path with execute permission. If not specified, temp_directory will be used.
proc_data_directory ${gaia.temp_directory} Directory where data transferred to and from procs is written. Must be a fully qualified path with sufficient free space for required volume of data. If not specified, temp_directory will be used.

KiFS

Configuration parameter default value description
enable_kifs false Enable access to the KiFS file system from within procs. This will create a 'filesystem' collection and an internal 'kifs' user in Kinetica and mount the file system at the mount point specified below.
kifs_mount_point /opt/gpudb/kifs Parent directory of the mount point for the KiFS file system. Must be a fully qualified path. The actual mount point will be a subdirectory 'mount' below this directory.

HA

Configuration parameter default value description
enable_ha false At the moment, this section is simply to provide information about if and where HA is running, so that the gpudb startup script can properly create proxy-pass entries for httpd when both HA and httpd are enabled. Enable HA

ha_proc_ip_address

ha_proc_port

${gaia.rank0_ip_address}

9191

The IP address and port of the machine running HA that is accessible from httpd - used to generate proxy-pass entries. Default of 127.0.0.1 because in most cases, HA-Proc will be running on the same machine as rank0.

Internal developer options