Version:

7.0 Release Notes

Publish Date: 01/31/2019

Features

Active Analytics Workbench

  • The Active Analytics Workbench (AAW) platform is now available. A brand new API and UI are available to improve existing machine learning workflows. For machine learning, Tensorflow and Blackbox (using Docker containers) models are supported. AAW also supports continuous deployments, on-demand deployments (via an inferencing REST endpoint), and audits of both TensorFlow models trained internally and imported models. AAW has a new ingestion interface that allows ingesting data from variety of sources and methods, including Kinetica, PostgreSQL, and Kafka.

Core

  • Kinetica now supports Resource Management. Resource management involves the three following resources:
    • Storage Tiers: Data containment layers within the database (VRAM, RAM, Disk Cache, Cold Storage)
    • Tier Strategies: Data object eviction priorities within each storage tier to help define memory usage and data priorities
    • Resource Groups: Resource fencing -- process scheduling priorities and other limits imposed on specific groups of users
  • The ability to alter multiple columns in a single call is possible via the new /alter/table/columns endpoint.

Geospatial / Network Graph

  • Kinetica now includes a graph server. The graph server provides a generic and extensible design of networks with the aim of being tailored or used for various real-life applications, including transportation, utility, social, and geospatial.
  • Kinetica now offers a Vector Tile Service (VTS) to generate Vector Tiles and support client-side visualization of geospatial data contained within the tiles. Generating Vector Tiles using Kinetica involves passing in the VTS URL to the client-side renderer.
  • The /wms endpoint now supports contour plot visualization functionality

KAgent

  • The Kinetica Agent (KAgent) UI is now available to automate Kinetica installation and configuration. KAgent can automatically install Nvidia drivers for CUDA installations, the Active Analytics Workbench (AAW) and Kubernetes (required for AAW), configure the cluster for SSL and/or external authentication and high availability. KAgent is also cloud ready and able to provision and/or deploy to instances in Amazon Web Services (AWS), Google Cloud Platform (GCP), or Microsoft Azure. KAgent also allows one to configure the location of the head node, AAW, and the graph server within the cluster. Upgrades will now be managed using KAgent.

SQL / OLAP

  • Kinetica is now packaged with a new and improved SQL interpreter.
  • A new /execute/sql endpoint is available. You can now send SQL commands to the native API without an ODBC driver.
  • The SQL engine uses a query planner that analyzes an entire query for the many different ways it can be solved (e.g., performing a filter before a join, performing a filter after a join, etc.) and selecting the plan that is the most efficient. Once this plan is utilized, it’s cached so the plan can be used again without having to re-compute it when the same query is received again.
  • Complex SQL queries that involve multiple operations are now analyzed for interdependency. Any operations that have no dependencies on others are executed in parallel.
  • Distributed operations, such as UNIONs between sharded and replicated tables or distributed JOINs, are now possible using SQL syntax or /execute/sql. The database will automatically re-shard or replicate tables temporarily as necessary to help process the query. Note that the distributed query can be slower and use more memory than a traditional non-distributed operation. The sql.distributed_joins setting in /opt/gpudb/core/etc/gpudb.conf controls the ability to use distributed operations.
  • Correlated sub-queries are now supported.
  • New SQL support for:
    • EXPLAIN
    • Partitions
    • Tier Strategy Definitions
    • Logging Levels
  • Partitioning is now available. Table data that is sharded or replicated can be partitioned to aid storage tiering and data skipping. The two types of supported partitioning schemes are:
    • Range
    • Interval

UI

  • The Kinetica Administration Application (GAdmin) has undergone a visual refresh and now supports functionality for managing resource groups, interfacing with tier strategies and partitions, reviewing graph node and edge counts, and deleting graphs.
  • Kinetica Reveal service has undergone a visual refresh as well as received some usability improvements.

Version 7.0.10

Build Date: 11/13/19

SQL/OLAP

  • Added support for using wildcard characters (*) in place of schema and table names for GRANT and REVOKE statements
  • Added support for exposing detailed join filter plan information when using EXPLAIN ANALYZE VERBOSE with operations that do not explicitly join tables but are processed using joins to achieve the desired result
  • Added support for using escape characters in LIKE statements
  • Added support for the following bitwise operators:
    • & (AND)
    • | (OR)
    • ^ (exclusive OR)
    • << (left shift)
    • >> (right shift)
    • ~ (NOT)
  • Added new ipv4_part function that enables extraction of any of the four parts of an IPV4 address

Geospatial / Network Graph

  • Added support for batch solving several node and destination pairs using the SHORTEST_PATH solver in /solve/graph or using the MATCH_BATCH_SOLVES solver (with the appropriate Match Identifier Combinations) in /match/graph
  • Added a new modify option to /create/graph that enables an existing graph to be updated with new options or new nodes, edges, weights, and/or restrictions from new/updated data in its source tables
  • Increased the maximum number of cells that can be produced by the ST_*GRID functions to 100 million (from 20 million)

APIs/Tools/Connectors

  • Added new assume_no_nulls to the RKinetica connector that optimizes JSON deserialization when the expected result is not expected to contain NULL values
  • Added new offset and limit options to the Spark connector egress processor to enable batch/paginated reading

UI

AAW

  • Added support for JSON input when creating a feature set
  • Added support for an embeddable Map/WMS viewer in the RAPIDS Notebook Workspace

KAgent

  • Added support for cloning the data, users, and authorization settings from an existing cluster to a new one provisioned through KAgent
  • Added support for adding nodes to a new cluster that only have Rabbit MQ installed

GAdmin

  • Added support for session management, allowing users to limit concurrent sessions
  • Added support for managing SQL Procedures through a new Procs interface associated with the Query tool
  • Added a new --s-batch-iterator-column-name flag to the Advanced Import tool, which takes a column and uses it to iterate over data when ingesting into Kinetica from PostgreSQL--using this feature will provide better ingest performance

Reveal

  • Added support for the following to the Map slice:
    • Multi-layer custom query string parameters
    • Multi-layer legend

Version 7.0.9

Build Date: 10/16/19

General

  • Improved performance for repeated group-by queries
  • Added three new options to /insert/records:
    • return_individual_errors -- if set to true, any error that occurred when inserting an individual record will be returned
    • allow_partial_batch -- if set to true, all records that did not error during insertion will be inserted and all records that did error will be rejected and reported
    • dry_run -- if set to true, no records will be inserted and any errors will be returned
  • Added support for flushing column data from memory to persist using the new evict_columns option in the /alter/system/properties endpoint

AAW

  • Improved NVIDIA RAPIDS support thanks to Jupyter Notebook integration, multi-tenancy, and fast workspace persistence
  • Added support for selecting either a CPU- or GPU-bound container when deploying a model in the Active Analytics Workbench (AAW)

SQL/OLAP

  • Added support for SQL procedures. There is currently no support for variables or input/output parameters, but a given procedure may return a single result set or output-producing statement.

Geospatial / Network Graph

  • Improved error message logging for the graph server
  • Aligned the /solve/graph endpoint parameters with the other graph endpoints for consistency
  • Improved the performance and reliability of the match_supply_demand solver in the /match/graph endpoint

APIs/Tools/Connectors

  • Added HA failover support for servers returning a 503 error (offline) to the Java, C++, and Python APIs
  • Improved performance for batch reading from PostgreSQL for ingestion into Kinetica via KIO

UI

AAW

  • Improved the user interface for the NVIDIA RAPIDS notebook/model workflow

KAgent

  • Added support for reserving GPUs to be used by AAW

GAdmin

  • Added querying, solving, and matching graph interfaces; each interface follows a WYSIWYG format with access to statement history (including the option to re-use any previous statements) and the ability to manually select from a map any WKT point that can be used as an identifier throughout a graph's lifecycle
  • Added support for assigning additional roles to a new or existing role via the role configuration interface
  • Added support for selecting and detailing features within the WMS viewer
  • Added support for selecting a cluster when using the SQL tool in a High Availability (HA) environment

Reveal

  • Improved support for the map slice:
    • Class break definition and styling enhancements
    • Row/record identification and detailing enhancements
    • Map layer configuration enhancements
  • Enhanced support for classbreak styling
  • Added option to include legend labels for classbreak
  • Added support for external WMS layers

Version 7.0.8

Build Date: 10/07/19

General

  • Operations that consist of dividing by 0 will now throw an error
  • Enabled support for using the following endpoints with join views and materialized views:
    • /filter/byarea
    • /filter/byarea/geometry
    • /filter/bybox
    • /filter/bybox/geometry
    • /filter/bygeometry
    • /filter/bylist
    • /filter/byradius
    • /filter/byradius/geometry
    • /filter/byrange
    • /filter/byseries
    • /filter/bytable
    • /filter/byvalue
  • Added support for dates containing the year 9999, sometimes used to mark a date as unknown or in the future; however, year values between 2900 and 9999 are not supported

SQL/OLAP

  • Improved performance for count distinct operations involving a dictionary encoded column
  • Improved CSV support for the ODBC/JDBC connector through the following features:
    • Wildcards are now supported in filenames
    • Tab delimiters are the default for TSV files; pipe delimiters are the default for PSV files
    • Comments are now supported
    • Custom null string is now supported
    • Custom quote character is now supported
    • Custom escape character is now supported
    • Option to clear table prior to ingest is now available
    • Option to skip the first n rows of the file is now available
    • Option to limit the ingest to n new rows is now available
    • Option to specify error handing mode is now available:
      • Permissive -- Kinetica will attempt to ingest all rows regardless of error
      • Skip -- Kinetica will attempt to ingest all rows that did not error during validation
      • Abort -- Kinetica will stop ingest upon encountering an error

Infrastructure

  • Added support for user impersonation features when external authentication and/or HTTPD is enabled

Geospatial / Network Graph

  • Improved performance of double precision WKT rendering

APIs/Tools/Connectors

  • Added support for users to select a graceful failure mode when ingesting CSV files via KIO:
    • permissive -- KIO will attempt to ingest all rows regardless of error
    • dropmalformed -- KIO will drop malformed records
  • Added support for unsigned long values to the C# API

UI

GAdmin

  • Added support for tagging a UDF during execution; this tag can be referenced when displaying the status of or killing UDFs
  • Added support for searching through the SQL query history
  • Advanced Import:
    • Added support for PostGIS-related columns
    • Enabled additional SSL support
    • Added support for killing long-running import jobs

Reveal

  • Added support for primary map layer reordering
  • Added support for syncing viewport of multiple maps on a dashboard

Version 7.0.7

Build Date: 08/21/2019

General

  • Enabled a new system user admin role that allows management of user passwords, assignment of roles and groups, and access control of other users
  • Enabled the unsigned long data type

AAW

  • Added support for importing data via Bring Your Own Container (BYOC) ingests

SQL/OLAP

  • Added various new security functions for identifying current user and associated roles
  • Added support for executing queries as a provided user
  • Implemented support for unsigned long values
  • Added support for performing SQL UPDATE and DELETE operations using distributed queries (IN sub-queries)
  • Wider support for is true comparisons
  • Added new hints to force executing a particular SQL statement synchronously or asynchronously on other clusters in a High Availability ring:
    • KI_HINT_REPL_SYNC - when a server receives a SQL statement with this hint included, the HA process will execute the SQL statement across the cluster synchronously
    • KI_HINT_REPL_ASYNC - when a server receives a SQL statement with this hint included, the HA process will execute the SQL statement across the cluster asynchronously
  • Added support for displaying the definition of materialized views using SHOW <materialized_view_name>. Only applies to materialized views created using SQL
  • Various performance improvements to HAVING clauses and /aggregate/unique-related operations
  • Enabled inclusion of post-mortem information to SQL EXPLAIN plans should a query fail
  • Added support for multi-column LIST partition key
  • Added the USER ADMIN permission in support of the new system user admin role
  • Added support for replicating the right table when joining two randomly distributed tables, i.e. neither sharded nor replicated

Infrastructure

  • Various performance improvements to cluster start-up
  • Implemented support for the system user admin role
  • Added support for tagging a UDF during UDF execution; this tag can be referenced when displaying the status of a UDF or killing a UDF

Geospatial / Network Graph

  • Added support for cancelling graph server jobs
  • Implemented a generic multiple supply-demand optimization solver to /match/graph
  • Various performance improvements to class break operations in WMS

APIs/Tools/Connectors

  • Added the system_user_admin permission in support of the new system user admin role
  • Implemented support for unsigned long values in multi-head I/O operations to the native APIs
  • KIO enhancements:
    • Added custom parameter support for ingesting CSV data into Kinetica, e.g., null, quote, and comment characters
    • Added support for transferring data from Kinetica to PostgreSQL via the Spark engine
    • Added support for schemas in PostgreSQL URIs

UI

AAW

  • Added functionality for Bring Your Own Container (BYOC) ingest elements, importable Blackbox containers, entity cloning, model training metrics and logs, and deployment environment variables
  • Lambda function transformations are now handled via database operations
  • Feature fitting functionality is now automatically integrated into the dataset training process.

KAgent

  • Added new interface for previewing KAgent command line interface commands that are run in the background during installation
  • Improved the console logs' show/hide detail functionality

GAdmin

  • Added support for creating a graph
  • Added support for the new system user admin role/permission
  • Added support for skipping the header row in a CSV file and for importing a CSV file from a local directory when using Advanced CSV import

Reveal

  • Enabled the viewport to automatically adjust its size based on the WKT data contained within

Version 7.0.6

Publish Date: 07/24/2019

AAW

  • Added support for persisting training metrics during Model Training for model performance comparisons

SQL / OLAP

  • Added support for grouping by WKT columns

  • Added support for character delimiters and specifying columns in a file or destination table when using insert from CSV functionality, e.g.,

    INSERT INTO employee (first_name, last_name, date_of_birth)
    SELECT fname, lname, dob
    FROM FILE."/home/gpudb/data/emp_2017.csv"
    DELIMITER=',';
    
  • Added support for converting various charN types to other types, e.g., char64 to date

Infrastructure

  • User name and source IP address are now included with job information

Geospatial / Network Graph

  • Added support for preventing conflicting graph component configurations
  • Added new All Paths solver for querying all paths between two nodes
  • Added two new query identifiers to retrieve edges or nodes associated with the given labels:
    • QUERY_NODE_LABEL
    • QUERY_EDGE_LABEL

APIs / Tools / Connectors

  • Added all /get/records options to the RecordRetriever class for the Java, Python, and C++ APIs
  • Added date, datetime, and time parsing convenience support to the Java API

UI

AAW

  • Performance improvements for entity list pages

KAgent

  • Added support for the following:
    • Offline installations
    • Post-installation SSL configuration
    • High Availability configuration for single-head or multi-head clusters
    • Job management interface for command line interface jobs
  • Improved the "Add Only" cluster path validation
  • Improved log parsing

GAdmin

  • Improved the Jobs interface with search and sorting functionality
  • Improved the table search functionality
  • Improved the Graph interface with more detailed graph information
  • Improved the explain plan visualizer with more detailed information
  • Improved CSV import handling for incorrect formatting

Reveal

  • Added a new Bubble Chart slice
  • Improved geo-filtering support for materialized views
  • Improved support for global filtering dashboards with multiple maps
  • Performance improvements for the Column Filter slice

Version 7.0.5

Publish Date: 06/25/2019

AAW

  • Added support for multi-record (row) outputs to Blackbox functionality
  • Enabled cluster multi-tenancy via custom Kubernetes namespaces

SQL / OLAP

  • Added support for ASOF joins
  • Added security features to logical views
  • Added table monitor triggers for DELETE and UPDATE events
  • Added support for temporary table names via the # prefix, e.g., #table_name
  • Added support for renaming joins and views
  • Various performance improvements for using Tableau with Kinetica
  • Various performance improvements to predicate equi-joins (joins on the basis of equality)
  • Added support for converting all numeric, date, time, datetime, and timestamp type values to charN

Geospatial / Network Graph

  • Added new geospatial functions:
    • ST_LINELOCATIONPOINT
    • ST_HEXGRID
    • ST_POINTGRID
    • ST_SQUAREGRID
    • ST_TRIANGLEGRID
  • Added a WMS handler for the /visualize/isochrone endpoint
  • Enabled dictionary-encoded multi-property label support for improving performance when using multiple LABEL identifiers
  • Implemented an origin-destination (OD) pair matching solver to /match/graph
  • Various performance improvements to graph solvers and WKT class break for WMS

APIs / Tools / Connectors

  • Added High Availability (HA) support to multi-head input/output for the C++ API
  • Added support for the init_with_now column property to the Python API

UI

AAW

  • Added interface for multi-in/multi-out record support

KAgent

  • Added support for installing KAgent as a server-side job

GAdmin

  • Added support for additional KIO features and options
  • Added support for visualizing WMS via KiSQL
  • Added support for hash partitioning tables
  • Added a new User Data column to the list of running Jobs that displays the SQL being executed via /execute/sql (directly or via ODBC/JDBC)

Reveal

  • Added support for conditional color styling to the Big Number slice

Version 7.0.4

Publish Date: 06/10/2019

AAW

  • Improved installation diagnostics
  • Fixed timestamps on audit logs to reflect UTC time only

SQL / OLAP

  • Added ROOT and KINETICA to the reserved words list to align with SQL standard referencing of the database catalog and root schema
  • Various optimizations and improvements to window functions and distributed operations
  • Expanded support for the IPV4 type
  • Added support for propagating dictionary encoding to result tables
  • Added support to KiSQL, JDBC, and ODBC for designating the primary processing node in a cluster
  • Added support for string concatenation of numeric columns using the || operator

Geospatial / Network Graph

  • Various improvements and simplifications to /query/graph
  • Added a new QUERY_TARGET_NODE_LABEL query identifier for simplified querying of graph nodes

APIs / Tools / Connectors

  • Various performance improvements to the Kafka connector
  • Added support to JavaScript and NodeJS APIs for converting quoted Infinity, -Infinity, and NaN values to null for double and float type columns
  • Added support for importing geospatial data from SQL Server via Kinetica Input/Output (KIO)

UI

AAW

KAgent

  • Added support for Ring management (adding a cluster, removing a cluster, backing up a ring)
  • Added support for repairing a failed cluster installation

GAdmin

  • Added SQL EXPLAIN plan visualization
  • Added support for adding and removing a partition to the table edit interface
  • Added support for rolling log files
  • Added Spark memory parameters to the Advanced Import interface
  • Various improvements to the resource statistics interface

Reveal

  • Added support for VAL_ATTR parameter in a multi-layer map
  • Added point size autoscaling in a scatterplot

Version 7.0.3

Publish Date: 05/08/2019

AAW

  • Added support for NVIDIA RAPIDS to all three deployment modes
  • Improved configurations for all container assets (for inside-firewall use)
  • Bundled dependencies in the RPM

Core

  • Various performance improvements and bug fixes

SQL / OLAP

  • Added support for RANGE and NULLS FIRST/LAST in window functions
  • Added support for using NOT IN within an UPDATE/DELETE query
  • Added support for managing resource groups
  • Added support for CSV & Multi-head ingestion via ODBC
  • Tuning improvements, including:
    • Faster STXY_* functions, geo-joins, & non-geospatial equi-joins
    • Avoiding creating projections when paging through result sets, to reduce memory

Geospatial / Network Graph

  • Added support for Adaptive LOD Vector Tiles (Client side VTS)
  • Added options to image/chart requests for applying log scaling for min/max request parameters
  • Added ST_ConcaveHull geo function
  • Added solution type support to geo functions:
    • ST_DFullyWithin
    • ST_LongestLine
    • ST_MaxDistance
    • ST_Segmentize (Euclidean)
  • Graphs
    • Improved interface for applying restrictions when querying
    • Added support for constant expressions in all endpoints

APIs / Tools / Connectors

  • Added HA support to multi-head ingest & egress via Java API
  • Added primary cluster identification for the C++ API

DevOps

  • Added support for Docker EGL rendering
  • Included S3 Cold storage in backups

UI

AAW

  • Added Blackbox container creation wizard
  • Added RAPIDS model creation, training, and deployment

KAgent

  • More detailed logging and easier to use logging console
  • Added support for upgrading clusters already in KAgent from previous releases of Kinetica 7.0

GAdmin

  • Added support for new KIO options (Spark timeout & multi-head, etc)
  • Added support for Drag & Drop import of CSV files with Kinetica schema headers
  • Added global cluster system status
  • Added support for init_with_now attribute in table configuration editor
  • Added support for CSV data export from WMS preview
  • Improved application of all stats optimization recommendations

Reveal

  • Added global filter on map slice extent
  • Added data export from data browser slice with authentication
  • Added support for dashboard import/export
  • Added dark theme CSS template

Version 7.0.2

Publish Date: 04/17/2019

AAW

  • Added support for NVIDIA RAPIDS Model Training (Regression) and Deployment/Inference
  • Added facility: BlackBox Container Wizard
  • Added auto-launcher for State Manager
  • Upgraded all endpoints to be capable of asynchronous operation

Core

  • Added support for assigning resource groups to roles; users will inherit resource groups from roles if they have none set explicitly

SQL / OLAP

  • Added support for standard SQL views
  • Added support for SQL UPDATE and DELETE statements that reference other tables via JOIN
  • Improved efficiency of geospatial filters
  • Added JDBC support for multi-head ingest, including from CSV files
  • Various optimizations and bug fixes

Geospatial / Network Graph

  • Added Map Matching feature, which can map a given set of points to an underlying graph
  • Added geodesic distance option to ST_buffer geospatial function

APIs / Tools / Connectors

  • R ODBC connector support for row.names and bind & query parameters
  • Added primary cluster identification for Java & Python APIs
  • KIO
    • Added support for connecting to public S3 buckets
    • Improved transfer time between Kinetica instances via Spark
    • Added support for multi-head ingestion into Kinetica
    • Added support for ingesting into a table within a collection
    • Added ability to tune the JVM parameters for Spark memory limits
    • Improved logging

UI

AAW

  • Added image inference testing support
  • Added support for managing credentials for private docker registries

KAgent

  • Added detailed progress reporting during install and other long running tasks
  • More flexible backup and restore operations

GAdmin

  • Added Drag & Drop inferencing of WKT fields within CSV files
  • Added support for tab-delimited file import
  • Added support for role-based resource groups
  • Added support for list partition configuration

Reveal

  • Improved dashboard global filtering performance
  • Added SDK command for removing slices from build
  • Added support for custom base map server URL for map slice
  • Added info pop-up and auto-fit data feature for scatter plot slice

Version 7.0.1

Publish Date: 03/08/2019

AAW

  • Introduced Batch Model deployments
  • Added Credentials concept to handle reuseable connection profiles
  • Added support on Containerized Ingests to utilize saved Credentials concept
  • Introduced second process (state manager) to manage long-running processes
  • Added more filtering options for audit searches
  • Added Config-Driven K8s Cluster Selection
  • Added capture & searchability of stack traces on individual inferences
  • Faster inventory/entity dashboard operations

Core

  • Added ability to bypass Disk Cache tier for persistent objects when Disk Cache & Persist tiers are similarly performant (tuning option)

KAgent

  • Improved handling of upgrading httpd 2.2 configurations (when upgrading from GPUdb 6.2)
  • Initial support for KAgent creation of Nvidia-enabled Kubernetes clusters

SQL / OLAP

  • Improved Partitioning
    • Added List partitions
    • Added support for Date columns
    • Improved partition syntax
  • Added ability to apply the current date/time as a default for date/time columns
  • Added support for Alteryx
  • Improved support for SQL with HA
  • Various optimizations and bug fixes

Geospatial / Network Graph

  • Added ability to query graphs by edge/node ID constant value
  • Added Probability Rank Solver (Hidden Markov Chain)
  • Added new edge/weight configurations (identifier combos)
  • Added ST_partition geospatial function

APIs / Tools / Connectors

  • New R ODBC connector
  • Added high availability support to the basic client class for C++, Java, & Python and to the multi-head operations for Java & Python
  • KIO
    • Added support for Oracle
    • Added support for Teradata
    • Added support for SybaseIQ
    • Added support for SQLServer

UI

AAW

  • Faster loading dashboard
  • Batch model deployment
  • Entity archiving

KAgent

  • Added configuration of non-core/gpudb cluster nodes for dedicated AAW or Graph server
  • Added display of cluster admin service statuses (Running, Stopped, etc)
  • Added AAW administration commands
  • Improved robustness authentication/login

GAdmin

  • KIO UI support for Oracle, Teradata, Sybase IQ, and SQL Server

Reveal

  • Pivot Table heatmap coloring
  • Map auto-zoom and class-break legend
  • New improved Scatter Plot with pan/zoom