Skip to main content

5 posts tagged with "fixed"

View All Tags

· 3 min read
Jake Cave

3.11 Major Release Items​

What's New​

Database Health Diagnostics are here!​

When a masking run is slow, you no longer have to rely on a DBA to tell you why. Get a fast, clear view inside your database - long-running queries, tablespace usage, and (on supported engines) blocking sessions - so you can investigate slowness and delays yourself. Every check is read-only and safe, and this is just the start, with more diagnostic capabilities coming in future releases.

  • Diagnostic Check

  • Tablespace Utilisation on an Oracle Database

IBM Informix Informix is here!​

The go-to database for high-throughput transactional and embedded workloads now plugs straight into TDM - profiling, masking, and compliance included. Another key platform, fully covered.

Now supporting Ubuntu 24!​

TDM workerbees and our Enov8-built MongoDB instances are now certified on the U24 operating system.

Conditional masking just got smarter.​

Two new ways to build more precise rules:

  • In operator - Check a column against an entire list of values in a single rule. No more writing one rule per value.

  • Multiple criteria - Combine several conditions on the same column for finer, context-aware masking.

    • Example #1: Multiple Conditions on the Same Flag Column

    • Example #2: Multiple Conditions on Multiple Flag Columns

Changes​

Connection setup

  • Added tooltips, placeholder hints, and a new Description field to make connections easier to configure and reduce errors.

Connection name limit

  • Connection names now have a maximum length, preventing failures caused by overly long auto-generated config names.

Fixed Issues​

Compliance Report

  • Sources with no PII now correctly show as Compliant in PDF reports.

Compliance timeline

  • Execution columns now stay Pending when profiling, masking, and validation run out of order, instead of showing a misleading OK.

JSON masking

  • Large files now stream during processing, so files over 20 GB mask reliably.

MSSQL masking

  • Better handling of collation mismatches between source and temp tables.

Oracle masking

  • Fixed a temp-table issue affecting Oracle 10g.

Teradata masking

  • Fixed intermittent table-resolution errors during chunked merges.

Parquet masking

  • ID fields like ACN, ABN, and TFN now stay as text, preserving leading zeros.

File masking reports

  • Entries with invalid file paths or schemas now appear in reports with a clear status.

Custom function masking

  • Config state is now isolated during runs, giving consistent results every time.

3.11.1​

Release Date: 29th July 2026

Fixed Issues​

Fixed Value Masking​

  • Inline validation now correctly handles datasets containing a mix of null and non-null values.

Many-to-One Masking​

  • Right-padded values in fixed-length CHAR columns are now masked correctly.

MSSQL Conditional Masking​

  • Resolved an issue where columns could not self-reference during conditional masking.

Note: When the column used for the condition is also the column being masked, validation will be evaluated against the already-masked value, which may result in inaccurate validation results. Use a separate flag column for validation accuracy.

· 4 min read
Jake Cave

3.10 Major Release Items​

What's New​

Reporting & Compliance Enhancements​

TDM reporting has been redesigned to provide a more in-depth compliance view across onboarding and operational activity. Reports can be generated at any stage of application onboarding, with time and date filters to review database activity for defined periods (for example, quarterly reporting windows). PDF and CSV outputs consolidate profiling, masking, and validation outcomes at the column level across multiple executions. Users can also download reports for all jobs executed within the selected period, with filters applied to target specific results such as TFN matches across a single database or grouped databases.

Enhanced Chunking for Fixed-width File Masking​

Added chunked processing support for fixed-width file masking to improve throughput on large datasets. This update improves scalability and reduces memory overhead for high-volume file inputs.

Enhanced Grouped Connection Support​

Extended grouped data connection capabilities with delta profiling and bulk delta scan execution across related data sources. Masking configurations can now also be created or updated in bulk, improving operational efficiency and consistency in larger environments.

Expanded Database & Feature Support​

Added new capabilities across multiple platforms, including:

  • XML masking and casting support for MSSQL
  • SSL/TLS (including TCPS) support for Oracle and Sybase
  • Enhanced MongoDB GSSAPI authentication with additional configuration options

Changes​

MongoDB Log Rotation​

Automated log rotation is now configured during MongoDB installation to prevent log growth and disk space issues. The installer creates and validates a daily rotation policy that retains 7 compressed logs.

Enhanced TDM Configuration Descriptions​

Field descriptions in profile and mask configurations have been updated to provide clearer, more informative guidance. Users will now see contextual information, including usage considerations and any limitations, helping to improve configuration accuracy and reduce setup errors.

Fixed Issues​

Others_UniqueID Pattern - Database Masking​

  • Updated masking logic for columns with unique constraints so generated values maintain uniqueness and avoid integrity conflicts during masking operations.

Improved Database Profiling Accuracy​

  • Profiling status for columns containing special characters is now reported correctly, including detailed failure information where applicable.

Conditional Masking​

  • Improved conditional masking behavior when the dependent or flag column contains null values. Conditions are now evaluated consistently so rules behave as expected for null-based scenarios.

Database Compatibility Fixes​

Resolved compatibility issues across supported databases, including:

  • Oracle — conditional masking, date handling, and quoting issues
  • PostgreSQL and Aurora variants — validation accuracy and empty table handling
  • MySQL and MariaDB — cross-table masking and merge behaviour
  • MSSQL and Sybase — masking, validation, and function handling
  • Teradata — schema handling and function stability

File Processing Enhancements​

Improved file-based workflow behavior and diagnostics, including:

  • Consistent masking and validation for fixed-width, CSV, and JSON files
  • Improved handling of file paths, including missing files and non-standard path formats
  • Clearer, more informative messages during file processing

3.10.1​

Release Date: 24th April 2026

Fixed Issues​

Teradata​

Resolved cleanup of Enov8_Rn_Tbl post-masking to ensure it is removed after execution.

MSSQL​

  • Introduced chunking support for inserts into temporary tables to improve performance and stability during large operations.

  • Fixed issues with fixed value masking when chunking is enabled, ensuring consistent processing.

  • Resolved issues in cross-table conditional masking to ensure reliable execution.

Compliance Report​

  • Improved UI/CSS styling for the TDM Compliance Report for better readability and presentation.
  • Reintroduced the Forensic Results section within the report, now positioned in the appendix.

Free Text Masking​

Enhanced free text masking capabilities to:

  • Detect and sanitise numeric data within unstructured text.
  • Recognise and mask Australian Medicare numbers.
  • Dynamically adjust text character limits based on available server memory to prevent out-of-memory errors.

3.10.2​

Release Date: 13th May 2026

Fixed Issues​

Oracle and MongoDB Masking​

  • Resolved an issue where chunked distinct value retrieval caused an error during Many-to-One masking.

PostgreSQL Masking​

  • Improved fixed-value masking SQL generation for PostgreSQL when no flag conditions are configured, avoiding an unnecessary flag-column reference in that scenario.

· 4 min read
Jake Cave

3.9 Major Release Items​

What's New​

Introducing the New In-line & Free Text Masking​

Introducing a new Presidio-powered masking capability designed for unstructured fields such as descriptions, comments, and notes. This enhancement brings data protection to free text content, extending masking coverage beyond traditional structured columns and strengthening overall data privacy.

Smart Readiness Indicators​

New status tiles have been added to the Execution Console page, allowing users to quickly verify system readiness before execution.

Pre-Execution Review for Profiling and Masking​

A new confirmation step has been added before running Profiling or Masking jobs. This review screen highlights key configuration details, helping users verify settings and prevent accidental executions.

Expanded Multi-Cloud Data Source Support​

We've broadened our data connectivity options to support more cloud platforms, enabling greater flexibility for hybrid and multi-cloud environments. Newly supported connections include:

  • Google AlloyDB
  • Amazon Aurora PostgreSQL
  • Amazon Aurora MySQL

Validation Revolution​

Validation has been completely re-engineered to run in real time, validating each column immediately after it's masked rather than waiting for the entire job to finish. Results are written live into the report, giving users instant visibility into masking accuracy as the process runs.

The new parallelised validation engine now supports all masking functions - including custom logic - ensuring complete, consistent validation coverage across every data scenario.

Changes​

Centralised Logging and Backup​

All file-based logs and output backups are now stored in a single global logging directory, making it easier to manage and retain backups - especially in recycled or containerised environments where persistent file mounting is required.

Dynamic Bulk Upload Data Sample​

Bulk upload samples now auto-adjust to match the selected Data Source Type, providing the correct format instantly and streamlining multi-source setup.

Fixed-width Data Source​

A new footer configuration option has been added for Fixed-width data source connections, bringing greater consistency across all file-based connection types.

Fixed Issues​

MYSQL Masking​

  • Improved performance for chunked fixed-value masking operations, significantly reducing execution time.

Oracle Profiling​

  • Corrected an issue where row count data was occasionally not captured during profiling.

MongoDB Profiling​

  • Resolved a condition where column statuses could remain in Queued state after scans completed.

MongoDB Masking​

  • Applied low-level stability improvements to masking routines to ensure more consistent and reliable behavior across different environments.

Deprecation Notice​

Parameter Field in Pattern Configuration​

As announced in the previous release, the Parameter field in the Pattern Configuration UI has now been fully deprecated and is no longer supported starting from this release (v3.9). This enforcement is part of our system-wide update to streamline and standardise masking logic via structured configuration options under Additional Configuration.

What You Need To Know​

  • Any existing configurations that still rely on the Parameter field will not function as expected.

  • All masking logic or values previously set under the Parameter field are migrated into the appropriate fields, such as Fixed Value, Chunk Size, or other relevant options.

3.9.1​

Release Date: 3rd December 2025

Fixed​

File Profiling​

  • Made adjustments to improve handling of remote file paths with subdirectories so that profiling consistently follows the file path defined in the Data Source Page.

MSSQL Masking​

  • Improved fixed-value masking for date fields by updating the date comparison logic, ensuring consistent and predictable masking behavior.

Oracle Masking​

  • Refined handling of certain cross-table conditional masking scenarios involving specific join column types, helping these masking operations run more reliably.

Sybase Validation​

  • Enhanced validation behavior for fixed-value masking to ensure more consistent and reliable results.

MongoDB​

  • Improved MongoDB information handling to ensure accuracy and consistency of schema, table, and column names.

Deprecation Notice​

Deprecation Logging​

  • Added a warning in the logs to indicate usage of the deprecated Parameter field.

· 5 min read
Jake Cave

3.8 Major Release Items​

What's New​

Introducing the Data Factory​

Our Data Fabrication and Dataview modules are now unified under the new Data Factory.
This powerful upgrade automatically scans your custom queries to build data generation rules — using lookups, regex, and a new built-in Faker module to generate realistic test data automatically.
This enhancement makes it even easier to create rich, meaningful datasets for development and testing while protecting sensitive information.

Python 3.11 Support​

TDM now runs on Python 3.11, bringing performance improvements, enhanced compatibility, and updated security patches that address known vulnerabilities in older versions.
This upgrade ensures a more secure and efficient runtime environment.

Masked File Directory​

Users can now define the Masked File Directory field for masked files in file-based data connections — giving you more control over where your processed data lands.

Changes​

OracleDB Driver Update​

Replaced the legacy cx_Oracle module with the modern oracledb (v2.3.0), aligning with Oracle's latest recommendations for improved compatibility, supportability, and future-proofing.

Default Post-Mask Validation​

Set post-mask validation to default as True for all manually and auto-created masking configurations, ensuring consistency and helping catch issues early.

Fixed Issues​

Reporting & Logging​

  • Addressed an issue where report and log database sizes could grow disproportionately large relative to the number of scans executed.

File & XML Handling​

  • Improved file masking logic to skip files with no active columns, avoiding unnecessary reads and improving performance.
  • XML Validation now correctly returns exceptions when renaming files, instead of failing the entire operation.

Byte Limit Optimisations​

  • Optimised the byte limit when retrieving table data, preventing memory overflows during Database profiling.

DB2 Profiling String Handling​

  • TDM now handles certain data attributes when profiling DB2 databases. Specifically, the type casting for TYPENAME and LENGTH fields is now correctly treated as strings.

Teradata Masking Resilience​

  • Fixed masking failures on columns with char data types.
  • Enhanced session settings to use the UNICODE charset, resolving issues with untranslatable characters and improving support for multilingual data.

Deprecation Notice​

Parameter Field in Pattern Configuration​

We are deprecating the use of the Parameter field in the Pattern Configuration UI.
Starting from the next Major release (v3.9), configurations using this field will no longer be supported.
This change is part of a system-wide update to streamline and standardise masking logic via structured configuration options under Additional Configuration.

What You Need To Do​

  • Migrate any masking logic or values currently set under the Parameter field into the appropriate fields, such as Fixed Value, Chunk Size, or other relevant options that were previously supported by the Parameter field.
  • Review your saved configurations and update them before the next release date to avoid any disruption.

3.8.1​

Release Date: 30th July 2025

What's New​

Custom Fabrication Functions
You can now define custom Python functions within the Data Library's Fabrication module. This gives you full control over data generation, making it easier to produce complex or highly specific test data for your environment.

Changed​

Health Check API

  • Introduced the /api/health endpoint alias to simplify server readiness checks for external monitoring tools and integrations. See the TDM REST API documentation for detailed information about this endpoint.

Fixed​

Data Factory​

  • Resolved a failure when using the FULLNAME generator with a function-based fabrication type.
  • Generated CSV files now include the correct file extension by default.

DB2 Masking​

  • Masking now gracefully handles BIGINT failures and avoids silent crashes.
  • Fixed failures in conditional masking when using both LIKE and standard conditions.
  • Resolved issues with temp table creation for integer types.

Delimited Files​

  • Masking logs now correctly report inactive columns when present.

MySQL Profiling​

  • Certain unsupported data types are now correctly flagged and skipped.

DB2 Profiling​

  • Improved handling of TYPENAME and LENGTH casting.

Reporting & Validation​

  • Masking report logic now validates inactive columns before applying fail conditions.

3.8.2​

Release Date: 22nd August 2025

Fixed​

MSSQL Profiling​

  • Fixed MSSQL profiling connection issues on RHEL 8 when using Windows Authentication with Kerberos, improving reliability for domain-authenticated servers.

Teradata Connection​

  • Improved handling of special characters in credentials to ensure smoother and more reliable Teradata connections.

Fixed Value Masking​

  • Addressed issues with Other category pattern, specifically Replace_with_Null and Replace_with_empty.

Mask Config Import​

  • Improved handling of importing large masking configurations with over 1,000 patterns, optimizing memory usage to prevent page crashes and ensure smoother processing.

3.8.3​

Release Date: 12th September 2025

Fixed​

MSSQL Profiling​

  • Enhanced database scanning to handle edge cases more smoothly, improving stability and consistency in the following situations:

    • Column names that include special characters such as commas or quotes.

    • Column values that contain invalid or unsupported characters.

Teradata Masking​

  • Resolved an issue where queries involving LATIN TO UNICODE conversion could fail with error 6706: The string contains an untranslatable character, by improving handling of special characters stored in LATIN columns.

· 5 min read
Jake Cave

3.7 Major Release Items​

What's New​

Delta Profiling​

  • The TDM solution now includes Delta Profiling, allowing the system to profile only the changes since the last scan rather than scanning the entire database each time.
  • This significantly reduces profiling time and resource usage for large databases by detecting and processing only new or modified tables and columns.

Run Validation Post-Masking Automatically​

  • A new "Run Validation post-masking" option has been added to the Global Variables section of the masking configuration.
  • When enabled, validation runs automatically after masking, and the result is included in the masking report.

Compliance Report Generator Now Supports All File Types​

  • The Compliance Report Generator now supports file-based sources in addition to databases.
  • Users can now generate reports for all file types available in TDM, allowing seamless reporting for both structured and unstructured data.

Masked File Suffix Configuration​

  • A new "Masked File Suffix" field has been added to Data Connection details and Masking Config.
  • Users can define a custom suffix for masked files.
  • A default suffix is set based on file type (e.g., .masked remains the default, but delimited files will follow _masked.csv format).

Scan Results Now Included in the TDM Summary Report​

  • Users can now optionally include scan results in the TDM Summary Report.
  • Scan results are packaged with the summary report in a ZIP file for easy access and sharing.
  • Utilizes the existing bulk report creation functionality.

Changes​

Multi-Processing for Masked Value Calculation​

  • Previously, TDM used a single CPU for masking operations, leading to slow performance on large datasets.
  • Now, multi-threading distributes masking calculations across multiple CPU cores, significantly improving speed and efficiency.

New Enhancements to Reporting & Configuration​

  • Added a Status Column in Reports – Users can now see the status of each scan at the column level.
  • File Encoding Support for All File Types – Ensures proper encoding detection and handling across all supported file formats.
  • Connection Pooling Parameters for All Supported Databases – Users can now configure connection pooling settings to optimize database connections, including:
    • Connection Pool Size
    • Max Overflow of Pool Size
    • Pool Timeout

Enhanced Data Validation Reporting​

  • More detailed reasons added for why a column was not masked or validated, including:
    • Skipped
    • No data in column
    • Other rule-based exclusions

Fixed Issues​

SQL Server Masking Issue - Conversion Error​

  • Resolved an issue where SQL Server would throw an error when converting nvarchar to numeric in chunk masking.

Oracle Masking Error - 'bool' Object Not Subscriptable​

  • Fixed an Oracle masking issue where a 'bool' object is not subscriptable error occurred when merging tables.

3.7.1​

Release Date: 14th March 2024

Fixed​

General Fixes and Improvements

  • Disabled columns no longer display 'Passed' in the mask report.
  • Failing columns now show the correct status after pausing a masking job.

Sybase Masking​

  • Resolved an issue where some integer columns would fail to mask.
  • Fixed an issue with the fixed-value updating query.
  • Fixed an issue where some masks would fail with chunking enabled.

File Masking​

  • Padded zeros are no longer removed when masking.
  • Improved logging when no columns are found in a file.

Logging​

  • Improved error messaging for Oracle masking functions.

Oracle Cross-Table Conditional Masking​

  • Resolved an issue where some unmet masking conditions were still being masked.

General Profiling Fixes​

  • Skipped columns no longer display 'Queued' in the profiling report.
  • Skipped columns now display 'Failed'.

Delta Profiling​

  • Column counts at the end of delta profiling now only count newly scanned columns.
  • Report improvements.
  • Unsupported data types are now correctly displayed in the profiling report.
  • Significant performance improvements when profiling delimited file types.

Azure Blob​

  • Using a wildcard operator no longer results in a "file not found" error.

3.7.2​

Release Date: 4th April 2025

Fixed​

PostgreSQL Masking​

  • Escape Character Handling: Resolved an issue where escape characters (e.g., “\”) could incorrectly expand to “\” and exceed column length limits, resulting in StringDataRightTruncation errors. We now set standard_conforming_strings to OFF in the session.
  • Undefined Table: Fixed a problem where tables without an explicitly specified schema could cause (psycopg2.errors.UndefinedTable) errors if they were not in the default (public) schema. The fix ensures the schema name is included or the session's search path is updated.
  • Performance with Chunking: Addressed a significant performance issue where chunked masking queries used expensive “IN” subqueries.

3.7.3​

Release Date: 1st May 2025

Fixed​

PostgreSQL Profiling​

  • Data Type Support: Resolved an issue in PostgreSQL profiling where certain data types (e.g., bpchar, numeric, and text) were incorrectly marked as failed due to being unsupported. These types are now added to the list of supported data types.
  • Execution Status: Resolved an issue where columns from empty tables were showing a status of "Queued" despite having completed execution. These columns will now display a status of "Passed", and associated fields (e.g., Rowcount, Random Data) will reflect appropriate default values for empty tables.