19 posts tagged with "TDM"

3.2 Major Release Items​

What's New​

Data Subsetting Module​

• Data Subsetting streamlines data while upholding database rules. Therefore, removing unnecessary data whilst preserving the database integrity.

Profile Config - New "Scan by Column" Configuration Field​

• DCS can now perform a deep scan of each non-null column instead of the whole table.

Wildcard Support​

• Users now have the option to specify a wildcard file format (either as prefix or suffix) in data source connection.

Auto Retrieval of Date Format​

• All DB types now automatically retrieve date format from the database. Also added default date format to auto-generated mask config.

JSON Schema Support​

• Users can now specify a schema file for JSON files in data source connection to be used for profiling and masking.

Custom Functions​

• Ecosystem generated custom functions have now been migrated to the DCS database. Now, users can easily access, edit, and update these functions through the Ecosystem UI in the data library.

Note: While you can create new custom functions, please keep in mind that editing or altering base functions is not supported.

Changed​

DCS API​

• Added new endpoints to allow for backup and restoration of MongoDB data.

Enhanced Masking Logs​

• Added "ERROR" prefix to previously unlabelled failed stages for increased clarity.

Data Subsetting​

• Users can enable the “leave on failure” field which deletes the schema on any error.

Fixed​

Data Catalogue​

• Resolved data catalogue error when scanning Teradata 17.20+.
• Added error handling for unsupported file types.

DB Profiling​

• Added bytes limit for DeepScan profiling.

DB Masking​

• Added error handling for incorrect configs.
• Added chunking to fixed value masking.

Teradata

• Fixed issue of temp column not being deleted in version 17.0.
• Improved error handling when collecting triggers.
• Updated ODBC version to improve Linux compatibility.

Oracle

• Now changing indexes to invisible instead of unusable prior to masking.

MySQL

• Fixed defect where mask report was displaying incorrect pattern name.
• Fixed defect regarding date formatting errors.
• Fixed potential issue of special character replacement.

File Masking​

• Updated partial masking to remove trailing spaces in source values for telephone numbers.
• Fixed intermittent issue of error messages not showing up on XML/JSON mask report.

XML

• Added specific error handling for missing masked files.

PostgreSQL

• Added error messages for when no constraints are found in fixed value masking.

File Profiling​

JSON/XML

• Fixed duplicated messages showing up in the log file.
• Fixed percentage completed being reset for every new file.

Backend​

• Added further error handling for running scans on unsupported data types.

Mask Functions​

• Added further handling for scenarios where source value is the same as masked value.

Decommissioned​

• Nothing has been decommissioned.

3.1 Major Release Items​

What's New​

Pause/Start/Cancel​

• Database Masking executions can now be paused, resumed, and cancelled.

Kerberos Implementation​

• Now supports Kerberos authentication for MSSQL.

New Classes​

• Now supports JSON profiling and masking.
• Now supports Parquet profiling and masking.

Data Generator Regex Functions​

• Data can now be generated using user-generated regex functions.

Advanced REST API​

• New API to obtain data such as:
  • Logs
  • Reports/Outputs
  • Metrics
  • And more

Masking Function​

• New function to mask all strings in a fullname. For example, "James Bob Smith" → "Peter Thomas Jane".

Changed​

File Masking​

• Mismatch in row count between input and output file will now be logged. Since row counts can differ due to bad lines, empty lines, etc.
• Logs now use ASCII text to clearly display breaks such as pauses and restarts.

Database Masking​

Oracle

• Will now utilize a chunking feature to break data inserts into blocks of data to increase performance and reduce transaction log.

File Masking - Delimited​

• Enhanced performance when masking multiple large files, along with optimized memory usage.

Fixed​

Database Masking​

Redshift

• Fixed issue where string values were being skipped if they included quotes.

Decommissioned​

• Nothing has been decommissioned.

2.9.3​

Release Date: 1st April 2023

What's New​

Masking Function​

• New function to mask all strings in a fullname. For example, "James Bob Smith" → "Peter Thomas Jane".

Changed​

File Masking - Fixed Width​

• Fixed issue where malformed first rows were not being masked.
• Fixed issue where unique ID masking was not working for all pattern types.

File Profiling - Fixed Width​

• Fixed format of output file to match other file types.

Masking - Lookup​

• Changes to obtain TFN lookup more effectively.

DataConnection - Redshift​

• When editing the connection, previously saved information was not being displayed or the save was corrupted.

Validation​

• Fixed issue where CSV export for validation reports was displaying 'undefined patterns' when not found.

2.9 Major Release Items​

What's New​

Base DCS - Redshift Support​

• Added support for Redshift in Base DCS.

File Masking​

• Conditional masking has been added for all file types.
• Optimizations made by using chunking when masking delimited files.

Data Compliance Insight​

• View your data operation execution reports across different data sources and types.

Data Connection Insight​

• Centralized view to monitor your data source status and execute different data operations from a single view.

REST API​

• Ecosystem API endpoints for DCS jobs.

Data Bricks​

• New data widgets in custom dashboards.

Changed​

Data Source Management​

• The "Link with System Components" option has been removed and replaced with labels.

PII Splatter Board​

• The redesigned view allows users to search for PII compliance across different data sources.

Execution Queue​

• View the current status of all data operation executions.

Fixed​

Global​

• Extended global timeout for database type Couchbase.

Database Masking​

• Fixed issue where Oracle tables might not be compressed correctly.
• Altered Oracle queries used in masking to allow for larger DDL queries to be saved.
• Resolved bug where masking an empty column did not display the correct error.

File Profiling​

• Log updates for delimited file type now work correctly in all scenarios.

File Masking​

• Fixed issue with delimited files where quotes were not retained if the value contained a comma (e.g., "hello, world").
• Resolved bug where an incorrect engine was used to read delimited files.
• Fixed issue where result count could be incorrect for delimited and fixed-width file types.
• Added better handling for delimited file masking when dealing with bad inputs.

Decommissioned​

• Data Compliance
• Compliance Dashboard
• Profile Dashboard
• Validate Dashboard
• Masking Dashboard

2.9.1​

Release Date: 1st October 2022

Changed​

File Masking - Delimited​

• Enhanced progress updates for delimited file masking. The process now incorporates folders and the size of a file to report an accurate percentage of completion.

File Masking​

• Improved chunking algorithm to strengthen performance. Masking is now based on distinct values found: if greater than 1 million, masking is based on distinct values; otherwise, it is based on row count.

AWS Redshift​

• Reworked logic involving vacuums to maximize performance on masking.

Fixed​

Database Masking​

• Performance improvements for databases with large counts of distinct values. The reworked algorithm constitutes improvements for the following database types:

  • MSSQL
  • MySQL
  • MariaDB
  • PostgreSQL
  • AWS Redshift
  • Teradata

• Resolved a rare bug when masking full names where many-to-one masking returned an index error.

File Masking - XML​

• Improvement in the file gathering process when conducting a folder mask.
• Revised algorithmic changes to update recursion when failing on edge cases.

File Masking - Delimited​

• Resolved issues related to breaking due to unique characters, such as certain Chinese characters.
• Resolved a TFN masking bug on RedHat 8.1 systems.

Masking - Oracle​

• Improvements to masking diagnosis to resolve merging conflicts with special characters.
• Enhanced trigger enabling/disabling when multiple correlated triggers are attached to one table.

Profiling - AWS Redshift​

• Now only retaining tables of the required "type" and excluding types such as views.

DCS Base​

• Fixed temporary table logging issues when failing in rare cases.

2.9.2​

Release Date: 1st February 2023

Changed​

File Masking - Delimited​

• Enhanced performance when masking multiple large files, along with optimized memory usage.

Database Masking - Oracle​

• Will now utilize a chunking feature to break data inserts into blocks of data to increase performance and reduce transaction log.

File Masking​

• Mismatch in row count between input and output file will now be logged. Since row counts can differ due to bad lines, empty lines, etc.

Fixed​

Database Masking - Redshift​

• Fixed issue where string values were being skipped if they included quotes.

File Masking - Delimited​

• Fixed issue where empty space values were being masked.
• Resolved issue where values starting with escape characters or empty space were being skipped.
• Fixed reporting structure to accurately convey when a column is masked multiple times in one scan.

Profile Scanner - Oracle​

• Fixed wildcard functionality to incorporate rules to abide by include/exclude table functionality.

Decommissioned​

• Nothing has been decommissioned.

What's New​

Insights​

• Addition of Data Bricks and Data Booking Insights views.

DataView​

• New Action forms for View Windows and View Configuration that allow bookings to be added and viewed.
• Addition of Manage Data Booking view.

Queue Manager​

• All DCS scripts will now be placed in a queue if the queue manager is enabled.

Profiling, Masking, Validation, Data Fabrication​

• Config Comparison feature added.
• Other minor UI adjustments.

Fixed​

Masking​

• Resolved permission issues when accessing/creating folders for output and logging.
• Fixed issue where masking could fail on compressed tables for database type Oracle.
• Addressed issue when merging tables during masking for database type MySQL.
• Fixed issue where masking could fail when using some functions for type File - Delimited.

Validation​

• Fixed issue where validation patterns could not be created due to incorrect date format.

Data Generator​

• Resolved issue when generating fake credit card numbers.

Exec Scan​

• Fixed script failure under certain conditions for database type Oracle.

What's New​

Masking​

• Support for nullable CLOB and BLOB masking has been added for Oracle.
• Added DCS widgets to custom dashboards.
• Introduced a Meta-only scan, which is triggered if the configuration list is left empty.
• The SQL Server 'Windows Authentication' login method is now available for Windows-hosted instances.
• Data configurations can now be exported and imported as CSV files.

Fixed​

Data Fabricator​

• Improved functions for generating random credit card numbers.

Profiling​

• Enhanced logging for errors, such as failed schema collection.
• Resolved issue where profiling was attempted on 'unprofilable' columns (e.g., Raw and Long Raw types in Oracle).
• Fixed issue where include/exclude rules failed due to mismatched letter cases.

Masking​

• Masking functions with large value lookups now dynamically trim values to fit the column definition.

What's New​

Meta Scanning​

• Meta scanning now displays the matched category instead of "found."

Import/Export Configurations​

• Import and export of configurations are now available, facilitating faster bulk updates.

Reporting​

• Added filters to various reports.
• Included row count in profile reports.

Fixed​

MSSQL​

• Resolved issue with column collection when columns contained special characters (e.g., $ or [).

General​

• Fixed bug where values were not correctly replaced with NULL fields.
• Corrected Teradata .csv report issue related to decimal data lengths.
• Fixed issue where CLOB and NCLOB types could be profiled in Oracle.
• Addressed bug where cancelling profiling did not stop the process under certain conditions.
• Fixed issue where disabling triggers did not affect all existing triggers.

What's New​

File Masking and Profiling​

• File masking and profiling now support multiple files and directories.

DCS Masking and Profiling​

• Can now handle multiple headers and footers in a single file.
• Allows the choice of delimiter within quotes.

Memory Management​

• Added memory limit for obtained table data and the ability to skip large data types in profiling.

Database Triggers​

• Database triggers can now be disabled prior to masking.

Fixed​

General​

• Large profiles could previously fail to generate a report.
• Include/exclude rules could incorrectly read tables from other schemas if a table name was duplicated.

XML File Masking​

• Configurations could not be edited for XML file masking if the file path contained backslashes.

DCS Configuration​

• Fixed timeout issue within DCS configuration windows.

Teradata​

• Resolved minor defects in Teradata masking.

MongoDB​

• Profiling and validation could fail when MongoDB was used as the data source.

Data Sources​

• Corrected response issue with the Check Connection function for data sources.

Changed​

Profile Execution Filter​

• The Profile Execution Filter has been changed from a dropdown menu to a calendar widget.

Empty Tables Reporting​

• When removing empty tables in profiling, the number of empty tables is now reported.

Pattern Search Status​

• The Execute Pattern Search Status now refreshes automatically.

What's New​

Data Fabrication Module​

• The Data Fabrication Module has been added! Use this module to synthesize test data in seconds.

Executive Scan​

• Added Executive Scan, allowing selection of either high-level or deep-level scanning.

XML Support​

• DCS now supports XML for file profiling and masking.

Database Support​

• Added support for profiling, masking, and validation for Couchbase and MariaDB database types.

Fixed​

Profile Report​

• Moved profile report back-end creation to Python to prevent multi-threading errors at the end of large profiles.

Teradata​

• Fixed numeric overflow errors when creating temporary tables in Teradata.

Cancel Button​

• Fixed issue where the cancel button did not close the back-end Python process for profiling until a table scan was completed.