Skip to main content

TDM 3.5 Release Notes

· 4 min read
Jake Cave

3.5 Major Release Items​

What's New​

File Validation​

  • Files can now be validated based on their masking configurations.
    • We recommend using the following workflow for the best results: profiling → masking → validation

File Connection - FTP​

  • Collect files using FTP an connection
  • Users can Profile, Mask, Validate files using FTP login details
  • Connecting to FTP is as simple as connecting to a remote server. Ensure the server type is "FTP".

Profiling - Deep Scan​

  • This feature conducts a thorough scan of each column individually rather than scanning the entire table at once. Particularly effective for tables containing a significant amount of null data. Note: This might increase profiling time by 5 times!

New Front-End Changes​

  • Enhanced user interface:

    • User interface has been rebuilt to ease the configuration process
    • All parameters for masking are now separate grouped fields
    • UDS has extended configuration fields

  • Changes to the execution screen

    • This enables easier access for not only viewing but executing scans on a datasource

  • Config changes

    • Revamped config for more control over scan logic

  • Conditional Masking

    • Separated fields to ease the process of creating and modifying conditional mask statements.

Please see the DCS user guide for more information

Optimised Chunking​

  • The chunking process is expedited by determining the batch size based on the higher value between the total row count and distinct row count.
    • This can still be overridden by user input

Changes and Enhancements​

Database Masking - PostgreSQL, DB2, MariaDB, MSSQL​

  • Skip columns with constraints such as: Primary key Unique key Foreign key
  • By default fixed value masking will now not use chunking. To enable chunking use the parameters field in the masking config

TDM API - Additional Information​

  • Attached additional information to the dcs server check api endpoint to give users the:
    • TDM Version
    • Database start-up time
    • Current time on TDM server Access the endpoint via http://[dcs_server]/dcs/api/dcs_server_check

Masking Report - Additional Information​

  • Report now includes the time taken to mask each column

Custom Functions - Test Function​

  • Can now test look-up and parameter based functions.

Fixed​

Masking - MySQL​

  • Resolved performance issues by optimising the insertion of data

File Masking - JSON​

  • Fixed an issue where minimum and maximum overlap input was not validated

DB Validation​

  • Skip columns with no data with a log message, rather than producing an error
  • Now reducing the lookup of pattern telnum to 1000, so it can be validated accurately

3.5.1​

Release Date: 7th June 2024

Fixed​

PostgreSQL Check Connection​

  • Resolved an edge case where conflicting packages could cause the connection to fail

File Masking​

  • Delimited
    • Fixed an issue where data length was not trimmed correctly if the length was specified
    • Resolved a complication where user defined chunk size would not be updated

Pattern Finder​

  • Solved an issue where an empty data set would raise errors

DB Masking​

  • MSSQL
    • Fixed an error where clustered indexes were not being disabled

3.5.2​

Release Date: 21st June 2024

Fixed​

File Profiling​

  • Fixed an issue where files containing only invisible characters and multiple empty lines were not recognised as empty files.

Conditional Masking​

  • Resolved an issue where the new user interface changes could disrupt the order of conditional masking, leading to a row count error.

Profiling Report​

  • Fixed an issue where the profiling report URL was not updated upon failure.

3.5.3​

Release Date: 8th July 2024

Fixed​

MongoDB Profiling​

  • Resolved an issue where profiling would fail for users lacking permissions to certain tables.

Large File Masking​

  • Resolved a high memory utilisation issue, optimising overall performance and stability.

Large File Profiling​

  • Optimised the profiling process for large files, enhancing speed and efficiency.

Oracle Masking​

  • Fixed a bug where the unpredictable behavior of using rownum could potentially skip rows.

Teradata Insert​

  • Corrected an issue where failed inserts in Teradata could enable debug mode and return the partial failed SQL query.

Teradata Masking​

  • Fixed a problem in Teradata where empty data could cause complications in the generation of masked data.