Skip to main content

TDM 3.7 Release Notes

· 4 min read
Jake Cave

3.7 Major Release Items​

What's New​

Delta Profiling​

  • The TDM solution now includes Delta Profiling, allowing the system to profile only the changes since the last scan rather than scanning the entire database each time.
  • This significantly reduces profiling time and resource usage for large databases by detecting and processing only new or modified tables and columns.

Run Validation Post-Masking Automatically​

  • A new "Run Validation post-masking" option has been added to the Global Variables section of the masking configuration.
  • When enabled, validation runs automatically after masking, and the result is included in the masking report.

Compliance Report Generator Now Supports All File Types​

  • The Compliance Report Generator now supports file-based sources in addition to databases.
  • Users can now generate reports for all file types available in TDM, allowing seamless reporting for both structured and unstructured data.

Masked File Suffix Configuration​

  • A new "Masked File Suffix" field has been added to Data Connection details and Masking Config.
  • Users can define a custom suffix for masked files.
  • A default suffix is set based on file type (e.g., .masked remains the default, but delimited files will follow _masked.csv format).

Scan Results Now Included in the TDM Summary Report​

  • Users can now optionally include scan results in the TDM Summary Report.
  • Scan results are packaged with the summary report in a ZIP file for easy access and sharing.
  • Utilizes the existing bulk report creation functionality.

Changes​

Multi-Processing for Masked Value Calculation​

  • Previously, TDM used a single CPU for masking operations, leading to slow performance on large datasets.
  • Now, multi-threading distributes masking calculations across multiple CPU cores, significantly improving speed and efficiency.

New Enhancements to Reporting & Configuration​

  • Added a Status Column in Reports – Users can now see the status of each scan at the column level.
  • File Encoding Support for All File Types – Ensures proper encoding detection and handling across all supported file formats.
  • Connection Pooling Parameters for All Supported Databases – Users can now configure connection pooling settings to optimize database connections, including:
    • Connection Pool Size
    • Max Overflow of Pool Size
    • Pool Timeout

Enhanced Data Validation Reporting​

  • More detailed reasons added for why a column was not masked or validated, including:
    • Skipped
    • No data in column
    • Other rule-based exclusions

Fixed Issues​

SQL Server Masking Issue - Conversion Error​

  • Resolved an issue where SQL Server would throw an error when converting nvarchar to numeric in chunk masking.

Oracle Masking Error - 'bool' Object Not Subscriptable​

  • Fixed an Oracle masking issue where a 'bool' object is not subscriptable error occurred when merging tables.

3.7.1​

Release Date: 14th March 2024

Fixed​

General Fixes and Improvements

  • Disabled columns no longer display 'Passed' in the mask report.
  • Failing columns now show the correct status after pausing a masking job.

Sybase Masking​

  • Resolved an issue where some integer columns would fail to mask.
  • Fixed an issue with the fixed-value updating query.
  • Fixed an issue where some masks would fail with chunking enabled.

File Masking​

  • Padded zeros are no longer removed when masking.
  • Improved logging when no columns are found in a file.

Logging​

  • Improved error messaging for Oracle masking functions.

Oracle Cross-Table Conditional Masking​

  • Resolved an issue where some unmet masking conditions were still being masked.

General Profiling Fixes​

  • Skipped columns no longer display 'Queued' in the profiling report.
  • Skipped columns now display 'Failed'.

Delta Profiling​

  • Column counts at the end of delta profiling now only count newly scanned columns.
  • Report improvements.
  • Unsupported data types are now correctly displayed in the profiling report.
  • Significant performance improvements when profiling delimited file types.

Azure Blob​

  • Using a wildcard operator no longer results in a "file not found" error.

3.7.2​

Release Date: 4th April 2025

Fixed​

PostgreSQL Masking​

  • Escape Character Handling: Resolved an issue where escape characters (e.g., “\”) could incorrectly expand to “\” and exceed column length limits, resulting in StringDataRightTruncation errors. We now set standard_conforming_strings to OFF in the session.
  • Undefined Table: Fixed a problem where tables without an explicitly specified schema could cause (psycopg2.errors.UndefinedTable) errors if they were not in the default (public) schema. The fix ensures the schema name is included or the session's search path is updated.
  • Performance with Chunking: Addressed a significant performance issue where chunked masking queries used expensive “IN” subqueries.