13 posts tagged with "TDM"

3.6 Major Release Items​

What's New​

File Connection - Azure & S3 Support​

• Users can now connect to and process files stored on both Amazon S3 and Azure.
• Allows for profiling, masking, and validation directly from these cloud storage providers.

Profile Config - Multi-threading with Configurable Thread Count​

• Multi-threading is now available in the profile configuration screen.
• Users can set a thread count for faster profiling, provided the database supports it. This will significantly speed up the profiling process for large datasets.

Compliance Report Generator - Multiple Data Connections in One Report​

• The TDM Report Generator has been enhanced to support multiple data connections within a single report.
• Provides more comprehensive insights and flexibility in reporting.

Conditional Masking 'Like' Query​

• New functionality allows users to apply conditional masking based on SQL 'LIKE' queries.
• Enables the exclusion or inclusion of specific rows based on text patterns.

Group Feature for Data Connections​

• Data connections can now be linked together into groups.
• Allows users to perform group profiling, masking, validation, and reporting. This enables more efficient workflows for large-scale data management.
• Enable the TDM Queue Manager to make use of this new feature!

Data Connection Bulk Upload/Update​

• New feature allows users to upload or update multiple data connections in bulk.
• Streamlines the management of large data environments.

New TDM API Endpoint - Get Logs and Reports​

• A new API endpoint is available to retrieve the JSON version of logs and reports.
• Allows users to programmatically access profile report data.

---

Changes​

Improved Performance for Masking Large Files​

• Optimized chunk size for file masking, particularly for configurations exceeding 10,000 records.
• Results in better performance and scalability for large file masking operations.

File Validation - Unsupported Pattern Handling​

• A new validation mechanism compares the difference between the original and masked files when the pattern of the column is unsupported.
• Ensures data integrity during masking.

---

Fixed Issues​

DOB Masking - ORACLE​

• Addressed issues related to Date of Birth (DOB) masking.
• Ensured more consistent and accurate masking results across various configurations.

Data Generator - Name Field Order​

• Resolved an issue where the Data Generator was incorrectly ordering 'Full Name' with first and last names.

Masking Performance for Large Datasets​

• Improved performance when masking datasets with over 40 million rows.
• Reduced processing times and improved system scalability.

JSON Masking Performance​

• Resolved an issue causing slower-than-expected performance and inconsistent results during the masking of JSON files.

---

---

3.6.1​

Release Date: 15th October 2024

Fixed​

DB Profiling:​

Sybase​

• Fixed issue where profiling would hang if the connection pool is exceeded.

Teradata​

• Addressed masking issue by recollecting sessions after profiling to prevent session limit from exceeding.

Oracle​

• Resolved issue concerning materialised view logs being included in profiling.

Dataview​

• Resolved Dataview not showing the output of stored procedures for Sybase.

DataCatalogue​

• Resolved issue concerning views being included in DataCatalogue for Oracle.

MongoDB Connection Timeout:​

• Implemented more connection retries if it fails to connect to the database.

Bulk Reports:​

• Ensured values in bulk reports are enclosed in double quotes.
• Fixed an issue with conditional masking parameters not showing up in the report.
• Enhanced the displayed error message when generating bulk reports for failed profile scans to provide clearer feedback.

---

---

3.6.2​

Release Date: 30th October 2024

Fixed​

Teradata:​

• Enhanced the accuracy of conditional validation.

Oracle:​

• Resolved an issue where DOB masking for 20th century dates would fail with incompatible nls_date_format.

Sybase:​

• Enhanced performance for forensic scanner.

Couchbase:​

• Updated compatibility with later Couchbase versions.

DB/File:​

• Added conditional masking parameters to validation reports.

File:​

• Fixed an issue with wildcards not working as intended for Azure Blob.

---

---

3.6.3​

Release Date: 15th November, 2024

Fixed​

DB - General​

• Refined authentication handling, so switching types now properly defaults username and password fields to empty strings.
• Removed fabrication config creation for unsupported types.

MSSQL Masking​

• Resolved an issue that could interrupt fixed value masking, ensuring more reliable data processing.

Oracle Masking​

• Added better handling for temporary table and column deletion on exceptions during masking.

Sybase Profiling​

• Updated Profile report to ensure row count displays accurately.

File - General​

• Validate scans now support validation based on data length specifications from masking config.
• Added tailored error messages for unsupported wildcard usage.

Fixed Width Masking​

• Improved parsing of fixed values during masking, now based on data connection-defined column widths.

Validation Report​

• Resolved an issue where Custom Function names were being displayed as an ID in Validation report.

API - MongoDB​

• Resolved an issue with remote MongoDB functionality, allowing for seamless model backup and restore.

3.5 Major Release Items​

What's New​

File Validation​

• Files can now be validated based on their masking configurations.
  • We recommend using the following workflow for the best results: profiling → masking → validation

File Connection - FTP​

• Collect files using FTP an connection
• Users can Profile, Mask, Validate files using FTP login details
• Connecting to FTP is as simple as connecting to a remote server. Ensure the server type is "FTP".

Profiling - Deep Scan​

• This feature conducts a thorough scan of each column individually rather than scanning the entire table at once. Particularly effective for tables containing a significant amount of null data. Note: This might increase profiling time by 5 times!

New Front-End Changes​

• Enhanced user interface:

  • User interface has been rebuilt to ease the configuration process
  • All parameters for masking are now separate grouped fields
  • UDS has extended configuration fields

• Changes to the execution screen

  • This enables easier access for not only viewing but executing scans on a datasource

• Config changes

  • Revamped config for more control over scan logic

• Conditional Masking

  • Separated fields to ease the process of creating and modifying conditional mask statements.

Please see the DCS user guide for more information

Optimised Chunking​

• The chunking process is expedited by determining the batch size based on the higher value between the total row count and distinct row count.
  • This can still be overridden by user input

---

Changes and Enhancements​

Database Masking - PostgreSQL, DB2, MariaDB, MSSQL​

• Skip columns with constraints such as: Primary key Unique key Foreign key
• By default fixed value masking will now not use chunking. To enable chunking use the parameters field in the masking config

TDM API - Additional Information​

• Attached additional information to the dcs server check api endpoint to give users the:
  • TDM Version
  • Database start-up time
  • Current time on TDM server Access the endpoint via http://[dcs_server]/dcs/api/dcs_server_check

Masking Report - Additional Information​

• Report now includes the time taken to mask each column

Custom Functions - Test Function​

• Can now test look-up and parameter based functions.

---

Fixed​

Masking - MySQL​

• Resolved performance issues by optimising the insertion of data

File Masking - JSON​

• Fixed an issue where minimum and maximum overlap input was not validated

DB Validation​

• Skip columns with no data with a log message, rather than producing an error
• Now reducing the lookup of pattern telnum to 1000, so it can be validated accurately

---

---

3.5.1​

Release Date: 7th June 2024

Fixed​

PostgreSQL Check Connection​

• Resolved an edge case where conflicting packages could cause the connection to fail

File Masking​

• Delimited
  • Fixed an issue where data length was not trimmed correctly if the length was specified
  • Resolved a complication where user defined chunk size would not be updated

Pattern Finder​

• Solved an issue where an empty data set would raise errors

DB Masking​

• MSSQL
  • Fixed an error where clustered indexes were not being disabled

---

---

3.5.2​

Release Date: 21st June 2024

Fixed​

File Profiling​

• Fixed an issue where files containing only invisible characters and multiple empty lines were not recognised as empty files.

Conditional Masking​

• Resolved an issue where the new user interface changes could disrupt the order of conditional masking, leading to a row count error.

Profiling Report​

• Fixed an issue where the profiling report URL was not updated upon failure.

---

---

3.5.3​

Release Date: 8th July 2024

Fixed​

MongoDB Profiling​

• Resolved an issue where profiling would fail for users lacking permissions to certain tables.

Large File Masking​

• Resolved a high memory utilisation issue, optimising overall performance and stability.

Large File Profiling​

• Optimised the profiling process for large files, enhancing speed and efficiency.

Oracle Masking​

• Fixed a bug where the unpredictable behavior of using rownum could potentially skip rows.

Teradata Insert​

• Corrected an issue where failed inserts in Teradata could enable debug mode and return the partial failed SQL query.

Teradata Masking​

• Fixed a problem in Teradata where empty data could cause complications in the generation of masked data.

3.4 Major Release Items​

What's New​

SSL Support​

• Users can now connect to databases using custom SSL certificates.
• For MSSQL, MySQL, MariaDB, PostgreSQL

Regex Database Masking​

• Inclusion of an integrated function to manage regex generation.
• A user can place their required regex in the parameter field to create fabricated regex data.

Cross Table Conditional Masking​

• Masking based on cross table relationships. This will join two columns to mask the correct PII. This can be enabled using the parameters field

  Example:

  col|=|value||chunk (optional)|external_table|table_ref|cond_table_ref

Changes and Enhancements​

Database Masking - MySQL​

• Skip columns with constraints such as: Primary key Unique key Foreign key
• By default fixed value masking will now not use chunking. To enable chunking use the parameters field in the masking config

TDM Report - Template Changes​

• Enhancements for better visualisation of the database life cycle.
• Additional graphs and diagrams to view the database life cycle.

Fixed​

Profile MSSQL​

• Resolved an issue where tables where being skipped due to exceeding the holding byte size variable.

DB Masking​

• Oracle Resolved a logs issue where it stated triggers could not be disabled when in fact there was no triggers. Fixed an issue where table or view could not be found due oracle version incompatibility.

• MySQL Resolved an issue where the conditional masking configuration for the 'Address' column in the 'where' clause was ambiguous.

• Custom Functions Fixed an issue where fullname custom function could not be run with other custom functions.

DB Validation​

• Fixed an issue where trailing space was not trimmed before validating the last 4 digits against the lookup value.

Fabrication​

• Resolved an error where column generation failed when the data length was undefined.

Data Source​

• Fixed an error where special characters where being filtered out in the schema name.

Decommissioned​

• Removal of pattern Encrypt One to One

3.4.1​

Release Date: 1st March 2024

Fixed​

Profiling​

• MSSQL Fixed an issue where columns with empty information were not being skipped
• MYSQL Resolved a profile issue in the latest version of MySQL by adding additional support to exclude table type 'views' in databases

DB Masking​

• Oracle Fixed an issue where detailed error logs were not being displayed when using the oracle function 'dbms_parallel'
• MSSQL Resolved an issue where date was out of range due to the way small-datetime behaves It requires data to be the year 1900 and later.

Fabrication​

• MSSQL Resolved a syntax error when creating tables.

TDM Report Generator​

• Corrected logic for folders and output file generation

DataLibrary​

• Others_HashLast4Digits Fixed an issue where masked data generation failed when the source data's last 4 digits were alphabetical values
• Drivers_LicenseManyToOne Corrected an issue wherein pattern name failed to correspond with function name

3.4.2​

Release Date: 18th March 2024

Fixed​

Validation​

• Oracle Resolved an issue where a case sensitive search was being used for schemas

DB Masking​

• Firstname Many to One Fixed an edge case where encoded source values were not being adjusted before masking.
• Parameter Checks Resolved an edge case where conditional masking was attempted due to empty string conditions.
• Oracle Fixed an issue, so constraint types, listed below, are ignored: Primary Key Unique Key Foreign Key
• MySQL Performance improvements by no longer using order methods.

File Masking​

• JSON Resolved an error when using fixed value masking.

3.3 Major Release Items​

What's New​

Fabrication - Profiling/Multi-table Output Support​

• Users can now link and filter by data configuration using the associated profiling output/config.
• Additionally, configurations can be directly generated from profiling results.
• Multiple Tables and Files can now be generated.

TDM Overview Report Generator​

• DCS now offers the capability to produce a summary report, covering profiling, masking, validation, and other scans tailored to a DB data connection.
• The comprehensive report can help streamline analysis, decision-making, and communication for project stakeholders. For the time being, this feature is exclusively available via the API.

Validation - Dual Configuration Support​

• The validation scan has been upgraded to support both the existing validation configuration and the new masking configuration.
• Users now have the ability to choose the masking config instead of the validation config, in order to ensure that the right columns are effectively masked and checked.

API Endpoints​

• Added database load and backup

Changed​

Custom Masking Functions​

• Python scripts can now be deployed directly via the data library in Ecosystem UI.
• Users can update/delete their custom function scripts.

Fabrication - Generate Schema Structure​

• Added the ability to generate schema structures. With this update, a create schema field when checked, will automatically establish tables and their respective structures.

Pattern Finder​

• Outliers have been excluded from the standard deviation calculation to enhance precision.
• Additionally, any duplicate indexes identified have been systematically replaced with uniformly spaced values to ensure accuracy.

Fixed​

Task Manager​

• Fixed an error that caused special characters for URL to be double encoded.

DB Masking​

Oracle

• Fixed escaping backslash for UNISTR function.
• Fixed an inconsistency with masking full names that began with spaces.

MySQL

• Resolved an issue where the conditional masking configuration for the 'Address' column in the 'where' clause was ambiguous.

Teradata

• Improved function definitions and calls for better consistency and reduced errors.

DB Validation​

• Added support for TOP N function with DISTINCT in Teradata.

File Masking​

XML

• Resolved an issue where the XML masking process completed successfully, but did not generate the corresponding masked file based on unique constraints.

API​

• Fixed an error where File Profile report request could return a 400 error.

MSSQL - Performance Update​

• The query responsible for retrieving tables and columns in MSSQL has been optimized. Users can now expect faster and more efficient performance.

Expanded Null Exclusion List​

• The null exclusion list, previously exclusive to delimited file masking, has been expanded to all database types.

Mask Functions​

• Fixed an error where the masked credit card number was being truncated to 5 digits.

Decommissioned​

Data Subsetting​

• Data Subsetting has now been decommissioned.

3.2 Major Release Items​

What's New​

Data Subsetting Module​

• Data Subsetting streamlines data while upholding database rules. Therefore, removing unnecessary data whilst preserving the database integrity.

Profile Config - New "Scan by Column" Configuration Field​

• DCS can now perform a deep scan of each non-null column instead of the whole table.

Wildcard Support​

• Users now have the option to specify a wildcard file format (either as prefix or suffix) in data source connection.

Auto Retrieval of Date Format​

• All DB types now automatically retrieve date format from the database. Also added default date format to auto-generated mask config.

JSON Schema Support​

• Users can now specify a schema file for JSON files in data source connection to be used for profiling and masking.

Custom Functions​

• Ecosystem generated custom functions have now been migrated to the DCS database. Now, users can easily access, edit, and update these functions through the Ecosystem UI in the data library.

Note: While you can create new custom functions, please keep in mind that editing or altering base functions is not supported.

Changed​

DCS API​

• Added new endpoints to allow for backup and restoration of MongoDB data.

Enhanced Masking Logs​

• Added "ERROR" prefix to previously unlabelled failed stages for increased clarity.

Data Subsetting​

• Users can enable the “leave on failure” field which deletes the schema on any error.

Fixed​

Data Catalogue​

• Resolved data catalogue error when scanning Teradata 17.20+.
• Added error handling for unsupported file types.

DB Profiling​

• Added bytes limit for DeepScan profiling.

DB Masking​

• Added error handling for incorrect configs.
• Added chunking to fixed value masking.

Teradata

• Fixed issue of temp column not being deleted in version 17.0.
• Improved error handling when collecting triggers.
• Updated ODBC version to improve Linux compatibility.

Oracle

• Now changing indexes to invisible instead of unusable prior to masking.

MySQL

• Fixed defect where mask report was displaying incorrect pattern name.
• Fixed defect regarding date formatting errors.
• Fixed potential issue of special character replacement.

File Masking​

• Updated partial masking to remove trailing spaces in source values for telephone numbers.
• Fixed intermittent issue of error messages not showing up on XML/JSON mask report.

XML

• Added specific error handling for missing masked files.

PostgreSQL

• Added error messages for when no constraints are found in fixed value masking.

File Profiling​

JSON/XML

• Fixed duplicated messages showing up in the log file.
• Fixed percentage completed being reset for every new file.

Backend​

• Added further error handling for running scans on unsupported data types.

Mask Functions​

• Added further handling for scenarios where source value is the same as masked value.

Decommissioned​

• Nothing has been decommissioned.

3.1 Major Release Items​

What's New​

Pause/Start/Cancel​

• Database Masking executions can now be paused, resumed, and cancelled.

Kerberos Implementation​

• Now supports Kerberos authentication for MSSQL.

New Classes​

• Now supports JSON profiling and masking.
• Now supports Parquet profiling and masking.

Data Generator Regex Functions​

• Data can now be generated using user-generated regex functions.

Advanced REST API​

• New API to obtain data such as:
  • Logs
  • Reports/Outputs
  • Metrics
  • And more

Masking Function​

• New function to mask all strings in a fullname. For example, "James Bob Smith" → "Peter Thomas Jane".

Changed​

File Masking​

• Mismatch in row count between input and output file will now be logged. Since row counts can differ due to bad lines, empty lines, etc.
• Logs now use ASCII text to clearly display breaks such as pauses and restarts.

Database Masking​

Oracle

• Will now utilize a chunking feature to break data inserts into blocks of data to increase performance and reduce transaction log.

File Masking - Delimited​

• Enhanced performance when masking multiple large files, along with optimized memory usage.

Fixed​

Database Masking​

Redshift

• Fixed issue where string values were being skipped if they included quotes.

Decommissioned​

• Nothing has been decommissioned.

2.9.3​

Release Date: 1st April 2023

What's New​

Masking Function​

• New function to mask all strings in a fullname. For example, "James Bob Smith" → "Peter Thomas Jane".

Changed​

File Masking - Fixed Width​

• Fixed issue where malformed first rows were not being masked.
• Fixed issue where unique ID masking was not working for all pattern types.

File Profiling - Fixed Width​

• Fixed format of output file to match other file types.

Masking - Lookup​

• Changes to obtain TFN lookup more effectively.

DataConnection - Redshift​

• When editing the connection, previously saved information was not being displayed or the save was corrupted.

Validation​

• Fixed issue where CSV export for validation reports was displaying 'undefined patterns' when not found.

2.9 Major Release Items​

What's New​

Base DCS - Redshift Support​

• Added support for Redshift in Base DCS.

File Masking​

• Conditional masking has been added for all file types.
• Optimizations made by using chunking when masking delimited files.

Data Compliance Insight​

• View your data operation execution reports across different data sources and types.

Data Connection Insight​

• Centralized view to monitor your data source status and execute different data operations from a single view.

REST API​

• Ecosystem API endpoints for DCS jobs.

Data Bricks​

• New data widgets in custom dashboards.

Changed​

Data Source Management​

• The "Link with System Components" option has been removed and replaced with labels.

PII Splatter Board​

• The redesigned view allows users to search for PII compliance across different data sources.

Execution Queue​

• View the current status of all data operation executions.

Fixed​

Global​

• Extended global timeout for database type Couchbase.

Database Masking​

• Fixed issue where Oracle tables might not be compressed correctly.
• Altered Oracle queries used in masking to allow for larger DDL queries to be saved.
• Resolved bug where masking an empty column did not display the correct error.

File Profiling​

• Log updates for delimited file type now work correctly in all scenarios.

File Masking​

• Fixed issue with delimited files where quotes were not retained if the value contained a comma (e.g., "hello, world").
• Resolved bug where an incorrect engine was used to read delimited files.
• Fixed issue where result count could be incorrect for delimited and fixed-width file types.
• Added better handling for delimited file masking when dealing with bad inputs.

Decommissioned​

• Data Compliance
• Compliance Dashboard
• Profile Dashboard
• Validate Dashboard
• Masking Dashboard

2.9.1​

Release Date: 1st October 2022

Changed​

File Masking - Delimited​

• Enhanced progress updates for delimited file masking. The process now incorporates folders and the size of a file to report an accurate percentage of completion.

File Masking​

• Improved chunking algorithm to strengthen performance. Masking is now based on distinct values found: if greater than 1 million, masking is based on distinct values; otherwise, it is based on row count.

AWS Redshift​

• Reworked logic involving vacuums to maximize performance on masking.

Fixed​

Database Masking​

• Performance improvements for databases with large counts of distinct values. The reworked algorithm constitutes improvements for the following database types:

  • MSSQL
  • MySQL
  • MariaDB
  • PostgreSQL
  • AWS Redshift
  • Teradata

• Resolved a rare bug when masking full names where many-to-one masking returned an index error.

File Masking - XML​

• Improvement in the file gathering process when conducting a folder mask.
• Revised algorithmic changes to update recursion when failing on edge cases.

File Masking - Delimited​

• Resolved issues related to breaking due to unique characters, such as certain Chinese characters.
• Resolved a TFN masking bug on RedHat 8.1 systems.

Masking - Oracle​

• Improvements to masking diagnosis to resolve merging conflicts with special characters.
• Enhanced trigger enabling/disabling when multiple correlated triggers are attached to one table.

Profiling - AWS Redshift​

• Now only retaining tables of the required "type" and excluding types such as views.

DCS Base​

• Fixed temporary table logging issues when failing in rare cases.

2.9.2​

Release Date: 1st February 2023

Changed​

File Masking - Delimited​

• Enhanced performance when masking multiple large files, along with optimized memory usage.

Database Masking - Oracle​

• Will now utilize a chunking feature to break data inserts into blocks of data to increase performance and reduce transaction log.

File Masking​

• Mismatch in row count between input and output file will now be logged. Since row counts can differ due to bad lines, empty lines, etc.

Fixed​

Database Masking - Redshift​

• Fixed issue where string values were being skipped if they included quotes.

File Masking - Delimited​

• Fixed issue where empty space values were being masked.
• Resolved issue where values starting with escape characters or empty space were being skipped.
• Fixed reporting structure to accurately convey when a column is masked multiple times in one scan.

Profile Scanner - Oracle​

• Fixed wildcard functionality to incorporate rules to abide by include/exclude table functionality.

Decommissioned​

• Nothing has been decommissioned.

What's New​

Insights​

• Addition of Data Bricks and Data Booking Insights views.

DataView​

• New Action forms for View Windows and View Configuration that allow bookings to be added and viewed.
• Addition of Manage Data Booking view.

Queue Manager​

• All DCS scripts will now be placed in a queue if the queue manager is enabled.

Profiling, Masking, Validation, Data Fabrication​

• Config Comparison feature added.
• Other minor UI adjustments.

Fixed​

Masking​

• Resolved permission issues when accessing/creating folders for output and logging.
• Fixed issue where masking could fail on compressed tables for database type Oracle.
• Addressed issue when merging tables during masking for database type MySQL.
• Fixed issue where masking could fail when using some functions for type File - Delimited.

Validation​

• Fixed issue where validation patterns could not be created due to incorrect date format.

Data Generator​

• Resolved issue when generating fake credit card numbers.

Exec Scan​

• Fixed script failure under certain conditions for database type Oracle.

What's New​

Masking​

• Support for nullable CLOB and BLOB masking has been added for Oracle.
• Added DCS widgets to custom dashboards.
• Introduced a Meta-only scan, which is triggered if the configuration list is left empty.
• The SQL Server 'Windows Authentication' login method is now available for Windows-hosted instances.
• Data configurations can now be exported and imported as CSV files.

Fixed​

Data Fabricator​

• Improved functions for generating random credit card numbers.

Profiling​

• Enhanced logging for errors, such as failed schema collection.
• Resolved issue where profiling was attempted on 'unprofilable' columns (e.g., Raw and Long Raw types in Oracle).
• Fixed issue where include/exclude rules failed due to mismatched letter cases.

Masking​

• Masking functions with large value lookups now dynamically trim values to fit the column definition.