16 posts tagged with "TDM"

2.9 Major Release Items​

What's New​

Base DCS - Redshift Support​

• Added support for Redshift in Base DCS.

File Masking​

• Conditional masking has been added for all file types.
• Optimizations made by using chunking when masking delimited files.

Data Compliance Insight​

• View your data operation execution reports across different data sources and types.

Data Connection Insight​

• Centralized view to monitor your data source status and execute different data operations from a single view.

REST API​

• Ecosystem API endpoints for DCS jobs.

Data Bricks​

• New data widgets in custom dashboards.

Changed​

Data Source Management​

• The "Link with System Components" option has been removed and replaced with labels.

PII Splatter Board​

• The redesigned view allows users to search for PII compliance across different data sources.

Execution Queue​

• View the current status of all data operation executions.

Fixed​

Global​

• Extended global timeout for database type Couchbase.

Database Masking​

• Fixed issue where Oracle tables might not be compressed correctly.
• Altered Oracle queries used in masking to allow for larger DDL queries to be saved.
• Resolved bug where masking an empty column did not display the correct error.

File Profiling​

• Log updates for delimited file type now work correctly in all scenarios.

File Masking​

• Fixed issue with delimited files where quotes were not retained if the value contained a comma (e.g., "hello, world").
• Resolved bug where an incorrect engine was used to read delimited files.
• Fixed issue where result count could be incorrect for delimited and fixed-width file types.
• Added better handling for delimited file masking when dealing with bad inputs.

Decommissioned​

• Data Compliance
• Compliance Dashboard
• Profile Dashboard
• Validate Dashboard
• Masking Dashboard

2.9.1​

Release Date: 1st October 2022

Changed​

File Masking - Delimited​

• Enhanced progress updates for delimited file masking. The process now incorporates folders and the size of a file to report an accurate percentage of completion.

File Masking​

• Improved chunking algorithm to strengthen performance. Masking is now based on distinct values found: if greater than 1 million, masking is based on distinct values; otherwise, it is based on row count.

AWS Redshift​

• Reworked logic involving vacuums to maximize performance on masking.

Fixed​

Database Masking​

• Performance improvements for databases with large counts of distinct values. The reworked algorithm constitutes improvements for the following database types:

  • MSSQL
  • MySQL
  • MariaDB
  • PostgreSQL
  • AWS Redshift
  • Teradata

• Resolved a rare bug when masking full names where many-to-one masking returned an index error.

File Masking - XML​

• Improvement in the file gathering process when conducting a folder mask.
• Revised algorithmic changes to update recursion when failing on edge cases.

File Masking - Delimited​

• Resolved issues related to breaking due to unique characters, such as certain Chinese characters.
• Resolved a TFN masking bug on RedHat 8.1 systems.

Masking - Oracle​

• Improvements to masking diagnosis to resolve merging conflicts with special characters.
• Enhanced trigger enabling/disabling when multiple correlated triggers are attached to one table.

Profiling - AWS Redshift​

• Now only retaining tables of the required "type" and excluding types such as views.

DCS Base​

• Fixed temporary table logging issues when failing in rare cases.

2.9.2​

Release Date: 1st February 2023

Changed​

File Masking - Delimited​

• Enhanced performance when masking multiple large files, along with optimized memory usage.

Database Masking - Oracle​

• Will now utilize a chunking feature to break data inserts into blocks of data to increase performance and reduce transaction log.

File Masking​

• Mismatch in row count between input and output file will now be logged. Since row counts can differ due to bad lines, empty lines, etc.

Fixed​

Database Masking - Redshift​

• Fixed issue where string values were being skipped if they included quotes.

File Masking - Delimited​

• Fixed issue where empty space values were being masked.
• Resolved issue where values starting with escape characters or empty space were being skipped.
• Fixed reporting structure to accurately convey when a column is masked multiple times in one scan.

Profile Scanner - Oracle​

• Fixed wildcard functionality to incorporate rules to abide by include/exclude table functionality.

Decommissioned​

• Nothing has been decommissioned.

What's New​

Insights​

• Addition of Data Bricks and Data Booking Insights views.

DataView​

• New Action forms for View Windows and View Configuration that allow bookings to be added and viewed.
• Addition of Manage Data Booking view.

Queue Manager​

• All DCS scripts will now be placed in a queue if the queue manager is enabled.

Profiling, Masking, Validation, Data Fabrication​

• Config Comparison feature added.
• Other minor UI adjustments.

Fixed​

Masking​

• Resolved permission issues when accessing/creating folders for output and logging.
• Fixed issue where masking could fail on compressed tables for database type Oracle.
• Addressed issue when merging tables during masking for database type MySQL.
• Fixed issue where masking could fail when using some functions for type File - Delimited.

Validation​

• Fixed issue where validation patterns could not be created due to incorrect date format.

Data Generator​

• Resolved issue when generating fake credit card numbers.

Exec Scan​

• Fixed script failure under certain conditions for database type Oracle.

What's New​

Masking​

• Support for nullable CLOB and BLOB masking has been added for Oracle.
• Added DCS widgets to custom dashboards.
• Introduced a Meta-only scan, which is triggered if the configuration list is left empty.
• The SQL Server 'Windows Authentication' login method is now available for Windows-hosted instances.
• Data configurations can now be exported and imported as CSV files.

Fixed​

Data Fabricator​

• Improved functions for generating random credit card numbers.

Profiling​

• Enhanced logging for errors, such as failed schema collection.
• Resolved issue where profiling was attempted on 'unprofilable' columns (e.g., Raw and Long Raw types in Oracle).
• Fixed issue where include/exclude rules failed due to mismatched letter cases.

Masking​

• Masking functions with large value lookups now dynamically trim values to fit the column definition.

What's New​

Meta Scanning​

• Meta scanning now displays the matched category instead of "found."

Import/Export Configurations​

• Import and export of configurations are now available, facilitating faster bulk updates.

Reporting​

• Added filters to various reports.
• Included row count in profile reports.

Fixed​

MSSQL​

• Resolved issue with column collection when columns contained special characters (e.g., $ or [).

General​

• Fixed bug where values were not correctly replaced with NULL fields.
• Corrected Teradata .csv report issue related to decimal data lengths.
• Fixed issue where CLOB and NCLOB types could be profiled in Oracle.
• Addressed bug where cancelling profiling did not stop the process under certain conditions.
• Fixed issue where disabling triggers did not affect all existing triggers.

What's New​

File Masking and Profiling​

• File masking and profiling now support multiple files and directories.

DCS Masking and Profiling​

• Can now handle multiple headers and footers in a single file.
• Allows the choice of delimiter within quotes.

Memory Management​

• Added memory limit for obtained table data and the ability to skip large data types in profiling.

Database Triggers​

• Database triggers can now be disabled prior to masking.

Fixed​

General​

• Large profiles could previously fail to generate a report.
• Include/exclude rules could incorrectly read tables from other schemas if a table name was duplicated.

XML File Masking​

• Configurations could not be edited for XML file masking if the file path contained backslashes.

DCS Configuration​

• Fixed timeout issue within DCS configuration windows.

Teradata​

• Resolved minor defects in Teradata masking.

MongoDB​

• Profiling and validation could fail when MongoDB was used as the data source.

Data Sources​

• Corrected response issue with the Check Connection function for data sources.

Changed​

Profile Execution Filter​

• The Profile Execution Filter has been changed from a dropdown menu to a calendar widget.

Empty Tables Reporting​

• When removing empty tables in profiling, the number of empty tables is now reported.

Pattern Search Status​

• The Execute Pattern Search Status now refreshes automatically.

What's New​

Data Fabrication Module​

• The Data Fabrication Module has been added! Use this module to synthesize test data in seconds.

Executive Scan​

• Added Executive Scan, allowing selection of either high-level or deep-level scanning.

XML Support​

• DCS now supports XML for file profiling and masking.

Database Support​

• Added support for profiling, masking, and validation for Couchbase and MariaDB database types.

Fixed​

Profile Report​

• Moved profile report back-end creation to Python to prevent multi-threading errors at the end of large profiles.

Teradata​

• Fixed numeric overflow errors when creating temporary tables in Teradata.

Cancel Button​

• Fixed issue where the cancel button did not close the back-end Python process for profiling until a table scan was completed.