Heimdall Database Proxy
Released: Jan 24, 2021
Aggiornamenti in v21.x
v21.12.21.1
Updated Dec 21, 2021
Features
- Libraries have been updated, including the removal of all libraries under the CDDL, MPL, EPL and GPL w/ CPE licenses. This is to simplify the release process for GCP, due to their strict interpretation of how these licenses need to be handled.
- Added parameter consolidation change to simplify queries with a large number of parameters.
v21.12.14.1
Updated Dec 14, 2021
Features
- Due to a new version of log4j being released to completely fix the security issues found, log4j has been updated to 2.16.0. Further, Nearly every other library has also been updated to ensure that as of now, even the strictest scanners will validate the security of all the libraries used.
- Added code to wipe central manager analytics if memory usage becomes too high automatically, as this is the most likely cause of any issue.
- Added code to condense parameters in queries for logging to a few, avoiding issues where millions of parameters would result in queries not being logged due to their size.
- Removed an extra debug line in the previous build that could cause excessive log entries, even when verbose debug mode was not enabled.
- Added retry logic on auto-driver downloads to avoid issues if there is a transient error.
- AWS Marketplace images - Added the RDS CA certificates to the image by default to allow drivers to validate the CA cert issuing the database cert.
v21.12.11.1
Updated Dec 11, 2021
Features
- log4j version has been updated to 2.15.0 to ensure it is not vulnerable to cve-2021-44228.
- Dramatically improved regression tests, although these didn’t result in any actual code changes.
- Improved warnings if there are overlapping ports for DNS or HTTP monitoring.
- You can now throttle certain warnings that could generate large numbers of messages.
Fixes
- Resolved an issue with AWS autodetection of regional clusters.
v21.11.24.1
Updated Nov 24 2021
Features
- Improved lag detection logic. It now uses a stored procedure or function call on each database as part of the lag detection.
- Added the option to run proxies as systemd services, even when on the same server as the central manager. This allows manager restarts without disrupting traffic through the proxy.
- Improved the AWS RDS detection code.
- Improved the naming of servers when using AWS auto-detect and cluster tracking, so they are in alignment.
- Added the ability to route traffic to the nearest reader node by latency, for global deployments.
- Added more regression test cases.
- Removed (currently) unused MySQL8 driver configuration and files.
Fixes
- Resolved issues with SQL Server using array parameter types.
- Resolved an issue preventing pgbench from working when loading data via the COPY operation.
- Resolved other customer encountered bugs and discrepancies.
v21.10.01.1
Updated Oct 1 2021
Features
- Added integration with and support for RDS Global databases for auto failover.
- Added support for latency based reader selection in multi-az and region configurations.
- Added metrics to track jvm heap allocation rate and hiccups.
- Updated RHEL install logic for newer Redhat versions.
- Added synchronization of TLS certificates so all proxies will use the same certificate and updated TLS certificate management documentation.
- Added a knob to enable TLS 1.0 and 1.1 for older drivers that don’t support TLS 1.2+.
- Added significant improvements to the Cloudformation template to make it more flexible and cover more customer environment configurations.
- Added to DNS redirection mode the ability to redirect to AWS public IP, not just private.
- Added the ability to download drivers from Maven central to avoid packaging the MySQL driver for legal reasons.
- Improved documentation for heimdall sync user functionality.
- Disabled vdbs will now be put at the bottom of the status tab.
- Improved Postgres copy support.
- Removed the “track” option from the admin software tab, now alerts will only be for release versions.
- Alerts added if the cache client isn’t ready.
- Added support to cache MySQL stored procedure calls (they require a table attribute before cache will work).
- Added additional tooltip info for servers on the info tab.
- Added 1GB of swap space by default to AWS images to help avoid out of memory situations.
- Improved nocache reasons, including the ability to report multiple reasons at once.
- Improved compatibility with all database protocol types.
- Added ability to detect when binary values are used for primary key inserts and handle it properly when using client-side prepares at the driver level for MySQL.
- Adjusted images so that initial console logging can be observed with “journalctl -u heimdall.service”.
- Added huge improvements in test infrastructure to improve regression coverage.
Fixes
- Fixed issue with the self-diagnostic http health check where it was using the non-lb url for all checks, and didn’t adjust with changing configurations after start.
- Fixed missing SQL queries for SQL server stored procedure calls.
- Resolved issues with using TLS for proxy to manager communication, and changing the base image to use HTTPS by default for such communication.
- Resolved issues with Hazelcast AWS autodetect logic.
- Resolved issue with DNS LB when more than 10 proxies were available, impacting proper load distribution.
v21.07.30.1
Updated Jul 30 2021
Features
- Added a 1 GB swapfile to avoid "out of memory" errors on the proxy.
- Changed the systemd configuration so that the stdout and stderr logs go to /var/log/messages vs. being thrown out, to assist in debugging issues.
v21.07.23.1
Updated Jul 23 2021
Features
- Adjusted the yellow alert on the status tab to show on 5% heap vs 10% heap free.
- You can now generate gc overhead alerts based on total CPU usage, not just a single core.
- Ensured a self-signed certificate is generated by the management server on first startup and port 8443 opened for secure access.
- Added an option to download a software image backup on a software update, allowing old versions to be easily archived by users for downgrades.
- Improved the overall readability of the status tab.
- Included the ability to dump the Microsoft SQL Server AG status on a cluster detect operation, which allows you to debug if the detection process fails to operate cleanly.
- Significant performance optimizations on the cache path with multiplexing, resulting in 7x improvement in qps for 100% cache hit situations.
Fixes
- Resolved an issue with MySQL with multiple result-set responses and another with prepared statements in certain rare situations.
- Added missing useServerPrepStmts to MySQL property list.
v21.04.05.1
Updated Apr 5 2021
Features
- Improved compatibility with delayed transactions.
- Added logic to trim prepared statements on a connection to no more than 2000 prepares to prevent memory overflow. Ruby trims to 1000, and Java to 256, so this should be plenty for most environments.
- Added logic to forget the last used parameters of a prepared statement after use, to reduce memory overhead.
- Added support for hstore types in binary format for Postgres.
- Added better support for commands like drop, alter, etc. that do not actually touch a table, but other metadata. These will be flagged as modification queries, so that read/write split doesn’t send them to the wrong server.
- Added to the sql server pool logic a default resetQuery of exec sp_reset_connection.
- Improved SQL parsing for many different query formats to improve table extraction.
- Avoid using the /tmp directory created by tomcat for temporary file uploads, as centos and derived systems clean up directories over time, and can break the update and other logic.
- Added a filter on keys used by Heimdall in external grid caches to prevent non-heimdall keys from being tracked in the cache logic, adding to memory overhead.
- Added a limit of 10x the number of objects specified in the cache object limit (if set) for tracking remote keys, again to limit memory usage.
- Improved temporary table detection logic, in particular for SQL Server.
- Improvement in TLS handling, including disabling tls 1.3 with SQL Server, as it doesn’t support it yet, and trying to negotiate it with SQL server libraries often breaks them.
- Improved accuracy of Prepared statement performance in the Analytics tab.
- Prevent alerts from prior to a clean restart of the management server from showing back up after restart.
Fixes
- Resolved issue where commands like commit and rollback would result in a sql log entry for the previous SQL command.
- Fix various issues with prepared SQL statements that include ? as part of the content, which impacted java prepared statement formats.
v21.02.27.1
Updated Feb 27 2021
Features
- Change the downloaded log filename format for .zip files to include yyyyMMddHHmm, previously it was using minutes for the month position.
- Add logging of full query hash in debug logs, along with logging some other error conditions for TLS, etc.
- Removed a limit of 1024k for a single Postgres packet on query read, which limited the size of an insert that could be done as a single operation.
- Added DML detection for generic create and drop commands for non-table oriented SQL. This prevents such queries from being directed to a read node with read/write split.
- Ignore quotes and question marks in comments when looking for unbalanced quotes or parameters to fill in for SQL expansion.
- Add further compatibility with the PHP PDO drivers for MySQL.
Fixes
- Critical: Fix a problem with MySQL when using prepared statements and read/write split (or anything that expands the prepared query to raw SQL) introduced in the last release build. This also impacted SQL logging in some cases as well.
- Critical: Disable TLS 1.3 only for SQL Server–clients often will attempt to negotiate tls 1.3 if the proxy supports it, but they aren’t tested with tls 1.3, and end up breaking. SQL Server itself doesn’t support TLS 1.3; This would be observed as a connection hang on any attempt to connect. This was triggered by tls 1.3 being backported by Oracle into Java 8 in late 2020, which is used by default by Heimdall.
- Resolved issue where a rule disabling multiplexing was not taking effect due to a rule processing optimization.
- Resolved an issue with insert queries with more than 500k fields inserted (across all rows) due to limit in regex processing that limited the number of capture groups that could be created.
- Correct an issue where rows read was not reported correctly for SQL Server at the protocol level.
- Resolved issue where proxy auth being disabled for Postgres still resulted in Proxy auth being enabled in the resulting wizard configuration.
v21.02.13.1
Updated Feb 13 2021
Features
- Minor usability improvements in the wizard.
- Detect Azure marketplace deployments for licensing.
- Improved TLS handling and error logging.
- Support binary uploads with prepared queries for PDO MySQL (PHP).
- Improved handling of temporary tables.
- Improve manual update process in low memory situations.
- Add example configuration for PHP’s MySQL PDO to extract the certificate from the keystore.
Fixes
- Resolve issue with the proxy where it wouldn’t update the code on a restart (requires updating the heimdall-entrypoint.sh script).
- Properly handle table names/aliases with question marks when expanding prepared queries for read/write split or transformation.
v21.01.24.1
Updated Jan 24, 2021
Features
- Improved logic for Postgres failovers in the Azure cloud environment in the parsing of the replication connect string and to support the hostaddr option.
- Updated the default included driver database and driver names.
- Remove the creation of new configuration files if the update server API was used.
- Updated Hazelcast to 4.1.1 and include yaml configuration file support.
- Added an option to allow recursive transform calls, to allow the transformation to be applied more than once on a single query.
- Improved logging (as normal).
Fixes
- Resolve issue with Data Direct MySQL drivers that prevented queries from completing execution.
- Corrected some cases where the defaultCatalog option was not being used properly to establish connections.
- Resolved an infinite loop that could occur if using TLS and the client connection was disconnected unexpectedly.
- Resolve issue with internal commands like show pool with MS-SQL Server proxy configurations.
- Resolve a memory leak when connections were dropped by the client under very high new connection/second loads.