Skip to main content

2 posts tagged with "fixed"

View All Tags

· 3 min read
Jake Cave

3.8 Major Release Items​

What's New​

Introducing the Data Factory​

Our Data Fabrication and Dataview modules are now unified under the new Data Factory.
This powerful upgrade automatically scans your custom queries to build data generation rules — using lookups, regex, and a new built-in Faker module to generate realistic test data automatically.
This enhancement makes it even easier to create rich, meaningful datasets for development and testing while protecting sensitive information.

Python 3.11 Support​

TDM now runs on Python 3.11, bringing performance improvements, enhanced compatibility, and updated security patches that address known vulnerabilities in older versions.
This upgrade ensures a more secure and efficient runtime environment.

Masked File Directory​

Users can now define the Masked File Directory field for masked files in file-based data connections — giving you more control over where your processed data lands.

Changes​

OracleDB Driver Update​

Replaced the legacy cx_Oracle module with the modern oracledb (v2.3.0), aligning with Oracle’s latest recommendations for improved compatibility, supportability, and future-proofing.

Default Post-Mask Validation​

Set post-mask validation to default as True for all manually and auto-created masking configurations, ensuring consistency and helping catch issues early.

Fixed Issues​

Reporting & Logging​

  • Addressed an issue where report and log database sizes could grow disproportionately large relative to the number of scans executed.

File & XML Handling​

  • Improved file masking logic to skip files with no active columns, avoiding unnecessary reads and improving performance.
  • XML Validation now correctly returns exceptions when renaming files, instead of failing the entire operation.

Byte Limit Optimisations​

  • Optimised the byte limit when retrieving table data, preventing memory overflows during Database profiling.

DB2 Profiling String Handling​

  • TDM now handles certain data attributes when profiling DB2 databases. Specifically, the type casting for TYPENAME and LENGTH fields is now correctly treated as strings.

Teradata Masking Resilience​

  • Fixed masking failures on columns with char data types.
  • Enhanced session settings to use the UNICODE charset, resolving issues with untranslatable characters and improving support for multilingual data.

Deprecation Notice​

Parameter Field in Pattern Configuration​

We are deprecating the use of the Parameter field in the Pattern Configuration UI.
Starting from the next Major release (v3.9), configurations using this field will no longer be supported.
This change is part of a system-wide update to streamline and standardise masking logic via structured configuration options under Additional Configuration.

What You Need To Do​

  • Migrate any masking logic or values currently set under the Parameter field into the appropriate fields, such as Fixed Value, Chunk Size, or other relevant options that were previously supported by the Parameter field.
  • Review your saved configurations and update them before the next release date to avoid any disruption.

· 5 min read
Jake Cave

3.7 Major Release Items​

What's New​

Delta Profiling​

  • The TDM solution now includes Delta Profiling, allowing the system to profile only the changes since the last scan rather than scanning the entire database each time.
  • This significantly reduces profiling time and resource usage for large databases by detecting and processing only new or modified tables and columns.

Run Validation Post-Masking Automatically​

  • A new "Run Validation post-masking" option has been added to the Global Variables section of the masking configuration.
  • When enabled, validation runs automatically after masking, and the result is included in the masking report.

Compliance Report Generator Now Supports All File Types​

  • The Compliance Report Generator now supports file-based sources in addition to databases.
  • Users can now generate reports for all file types available in TDM, allowing seamless reporting for both structured and unstructured data.

Masked File Suffix Configuration​

  • A new "Masked File Suffix" field has been added to Data Connection details and Masking Config.
  • Users can define a custom suffix for masked files.
  • A default suffix is set based on file type (e.g., .masked remains the default, but delimited files will follow _masked.csv format).

Scan Results Now Included in the TDM Summary Report​

  • Users can now optionally include scan results in the TDM Summary Report.
  • Scan results are packaged with the summary report in a ZIP file for easy access and sharing.
  • Utilizes the existing bulk report creation functionality.

Changes​

Multi-Processing for Masked Value Calculation​

  • Previously, TDM used a single CPU for masking operations, leading to slow performance on large datasets.
  • Now, multi-threading distributes masking calculations across multiple CPU cores, significantly improving speed and efficiency.

New Enhancements to Reporting & Configuration​

  • Added a Status Column in Reports – Users can now see the status of each scan at the column level.
  • File Encoding Support for All File Types – Ensures proper encoding detection and handling across all supported file formats.
  • Connection Pooling Parameters for All Supported Databases – Users can now configure connection pooling settings to optimize database connections, including:
    • Connection Pool Size
    • Max Overflow of Pool Size
    • Pool Timeout

Enhanced Data Validation Reporting​

  • More detailed reasons added for why a column was not masked or validated, including:
    • Skipped
    • No data in column
    • Other rule-based exclusions

Fixed Issues​

SQL Server Masking Issue - Conversion Error​

  • Resolved an issue where SQL Server would throw an error when converting nvarchar to numeric in chunk masking.

Oracle Masking Error - 'bool' Object Not Subscriptable​

  • Fixed an Oracle masking issue where a 'bool' object is not subscriptable error occurred when merging tables.

3.7.1​

Release Date: 14th March 2024

Fixed​

General Fixes and Improvements

  • Disabled columns no longer display 'Passed' in the mask report.
  • Failing columns now show the correct status after pausing a masking job.

Sybase Masking​

  • Resolved an issue where some integer columns would fail to mask.
  • Fixed an issue with the fixed-value updating query.
  • Fixed an issue where some masks would fail with chunking enabled.

File Masking​

  • Padded zeros are no longer removed when masking.
  • Improved logging when no columns are found in a file.

Logging​

  • Improved error messaging for Oracle masking functions.

Oracle Cross-Table Conditional Masking​

  • Resolved an issue where some unmet masking conditions were still being masked.

General Profiling Fixes​

  • Skipped columns no longer display 'Queued' in the profiling report.
  • Skipped columns now display 'Failed'.

Delta Profiling​

  • Column counts at the end of delta profiling now only count newly scanned columns.
  • Report improvements.
  • Unsupported data types are now correctly displayed in the profiling report.
  • Significant performance improvements when profiling delimited file types.

Azure Blob​

  • Using a wildcard operator no longer results in a "file not found" error.

3.7.2​

Release Date: 4th April 2025

Fixed​

PostgreSQL Masking​

  • Escape Character Handling: Resolved an issue where escape characters (e.g., “\”) could incorrectly expand to “\” and exceed column length limits, resulting in StringDataRightTruncation errors. We now set standard_conforming_strings to OFF in the session.
  • Undefined Table: Fixed a problem where tables without an explicitly specified schema could cause (psycopg2.errors.UndefinedTable) errors if they were not in the default (public) schema. The fix ensures the schema name is included or the session's search path is updated.
  • Performance with Chunking: Addressed a significant performance issue where chunked masking queries used expensive “IN” subqueries.

3.7.3​

Release Date: 1st May 2025

Fixed​

PostgreSQL Profiling​

  • Data Type Support: Resolved an issue in PostgreSQL profiling where certain data types (e.g., bpchar, numeric, and text) were incorrectly marked as failed due to being unsupported. These types are now added to the list of supported data types.
  • Execution Status: Resolved an issue where columns from empty tables were showing a status of "Queued" despite having completed execution. These columns will now display a status of "Passed", and associated fields (e.g., Rowcount, Random Data) will reflect appropriate default values for empty tables.