Masking

Overview of Masking

The next step of the process is to Mask your Data Source. Masking obscures data to protect personally identifiable information (PII) and ensures sensitive information is anonymised. After successfully running a profile scan on a data source, a draft mask configuration file is automatically created for review prior to execution.

Mask Configurations

The tables and columns selected for masking and how they should be masked is configured in a Mask Configuration.

For TDM version 3.5:

Navigate to Data Management Hub > Manage Configurations > Mask Configurations. Here you can Add, Edit, Clone or Delete a Masking Configuration.

For TDM versions 3.4 and earlier:

Navigate to Masking > Mask Configurations. Here you can Add, Edit, Clone or Delete a Masking Configuration.

Editing a Mask Configuration

As the profiling process creates a draft Mask Configuration, click the Edit Mask Connection button on the masking configuration you want to edit. This will present the Mask Configuration form.

For TDM version 3.5:

For TDM versions 3.4 and earlier:

On the form you can edit the following configuration information.

Type	Description
Configuration Name	The mask configuration name.
DB Source Connection	The data source the masking configuration is configured for.
Description	The mask configuration description or long name.
Country	The country or language to which the respective configuration may apply. (This field can be null.)
Notification	List of users who will be notified.
Status	Set the masking profile to Active for use or Draft/Inactive to hide it from the Masking Execution area.
Global Chunk Size	Default chunk size for all columns.
Date Format	Default date format for all date columns.
Disable Constraints	Disable constraints for all columns.
Disable Indexes	Disable indexes for all columns.
Disable Triggers	Disable triggers for all columns.
Drop Composite Unique Indexes/Constraints	Drop composite unique indexes and constraints for all columns - currently only supported for MSSQL and Oracle.

Add/Edit Field(s) section: -

Type	Description
Category	The field masking category. (This filters the results visible in the Pattern drop list).
Table Name	The table the column/field is on for masking.
Column Name	The column/field for masking.
PII Level	For viewing/setting the masking pattern as Primary PII, Secondary PII, or Other.
Pattern	The pattern type (Masking function) as defined in the Data Library Administration area.
Active	Indicates whether the column should be considered active for masking. If set to True, the column will be masked; otherwise skipped.
Parameter	Some pattern types/masking functions may allow a parameter to be provided to customize the masking behavior.
Lookup	Some pattern types/masking functions allow a lookup file to be selected to customize the replacement values.
Column Disable Constraints	Disable constraints for specified column.
Column Disable Indexes	Disable indexes for specified column.
Column Disable Triggers	Disable triggers for specified column.
Column Chunk Size	Chunk size for specified column.
Flag Column	Column to be assessed for a condition.
Flag Condition	The comparator that assess the condition. This is either "=" or "!=".
Flag String	The value that the flag column should be compared to.
Flag Table Name	Table to be joined for cross-table conditional masking.
Join Column 1	Column of the source table to be masked. The column to be joined with flag table.
Join Column 2	Column of the flag table to be masked. The column to be joined with source table.
Date Format	Date format for specified column.
Fixed Value	Value to replace all source values.

Once complete, scroll to the bottom of the main window and click Save to Save the configuration or Close to Cancel out without saving.

Note: Please ensure the status is set to Active for the Mask Configuration to be selectable in the Execute Masking screen.

Conditional Masking

Conditional Masking can be utilised by setting up the conditional fields in the masking config. This contains features such as fixed values, flag conditions, chunking, DOB date format, and cross table masking.

For TDM version 3.5:

In TDM3.5, conditional masking can be perfomed either using the parameter field (as done in TDM3.4 and earlier):

OR using the new conditional settings form:

The above forms can be accessed by pressing Edit on a pattern in the Pattern Details section of the Masking config.

For TDM versions 3.4 or earlier:

Field Input Specifications

The following sections highlight how certain fields of the mask configuration requires to be formatted:

Fixed Value

A user specified value that will replace all values in the column. For example, "John" in this case.

For TDM version 3.5:

Fixed value can be specified by either using the parameter field:

OR by using the Fixed Value field in the Additional Configuration form:

For TDM versions 3.4 or earlier:

Example:
John

Flag Conditions

Setup values to either be equal to or not equal to the user specified value. This enables masking to either mask focused data or ignore it.

How to use:

Flag column name|flag condition|flag string

For TDM version 3.5:

Conditional values can be specified by either using the parameter field:

OR by using the Conditional Settings form:

For TDM versions 3.4 or earlier:

Example:
email|=|sam@email WHERE:

• Flag column name is 'email'
• Flag condition is '='
• Flag string is 'sam@email'.

This example translates to masking values only when the value in the 'email' column equals to 'sam@email'.

Conditional Masking using Like(%) Expression

TDM 3.6 now supports conditional masking by using the % character to pattern match values in columns for the flag string.

For example:
email|!=|%domain% WHERE

• Flag column name is 'email'
• Flag condition is !=
• Flag string is '%domain%'

This example translates to masking values only when the value in the 'email' column does not match the string 'domain' in the value.

Like(%) Expression Use Cases:

The following are some use cases for the flag string utilising the % expression:

• Matching substring anywhere:
  • '%text%': Finds values in the column where "text" appears anywhere in the string (before or after other characters).
• Finding values with specific endings:
  • '%son': Matches values that end with specific substring, such as 'Michaelson' or 'Jackson'.
• Finding values with specific beginnings:
  • 'Sam%': Matches values that start with specific substring, such as 'Samantha' or 'Sammy'.
• Matching complex patterns:
  • 'A%B%C': Matches values starting with "A", followed by "B", and ending with "C", where any number of characters may exist between them.

Chunking

Limit the amount of data per query by specifying a chunking value. This limit is beneficial when masking large databases. By default, columns over a million rows will be masked.

For TDM version 3.5: Chunking can be specified by either using the parameter field:

OR by using the Column Chunk Size field in the Additional Configuration form:

For TDM versions 3.4 or earlier:

Example:
John||||10000

Note: Fixed value or conditional values are not required for chunking. The chunking parameter can be used with/without either, for example, '||||1000'.

DOB date format

Specify the date format when performing masking so date format can be either changed or preserved.

For TDM version 3.5:

For TDM versions 3.4 or earlier:

Example:

%Y-%m-%d

Cross Table Masking

Masking based on cross table relationships. This will join two columns to mask the correct PII. However, the conditions will be user specified and enable masking to be completed using information from other tables. This feature can be used by providing external table information in the parameter field.

How to use:

flag column name | flag condition| flag string | chunk size (optional) | flag table name | join column 1 | join column 2

For TDM version 3.5: Cross table masking can be performed using either the parameter field:

OR by using the Conditional Settings form:

For TDM versions 3.4 or earlier:

For example:
email|=|sam@email|personal|OriginalID|ReferenceID WHERE

• Flag column name is 'email'
• Flag condition is '='
• Flag string is 'sam@email'
• Flag table name is 'personal'
• Join column 1 is 'OriginalID'
• Join column 2 is 'ReferenceID'

This example translates to masking values only when the 'email' column in the reference table 'personal' equals 'sam@email', after performing a one-to-one join between the 'OriginalID' column in the original table and the 'ReferenceID' column in the reference table.

Adding a Mask Configuration

On the Mask Configurations screen, click the Add Mask Configuration button. This will present a new Profile Configuration form

Once complete, scroll to the bottom of the main window and click Save to Save the configuration or Close to Cancel out without saving. For TDM3.5, the save/close buttons are at the top of the config window.

Cloning a Mask Configuration

On the Mask Configurations screen, click the Clone button on the Mask Configuration you want to duplicate. This will present the Mask Configuration form with a duplication of all data for saving as a new Mask Configuration.

Once complete, scroll to the bottom of the main window and click Save to Save the configuration or Close to Cancel out without saving. For TDM3.5, the save/close buttons are at the top of the config window.

Deleting a Mask Configuration

On the Mask Configurations screen, click the Delete button on the Mask Configuration you want to remove. This will present a confirmation window confirming that you want to delete the Mask Configuration or not.

Comparing Mask Configurations

Tick on the check boxes in front of the two configurations which you want to compare and then press the Config Comparison button to view the comparison report. The text highlighted in red marks the differences in the two configurations.

Execute Masking - Single Data Source

For TDM version 3.5+:

To execute a new masking request, navigate to Data Management Hub > Execution Console.

Here, you can select your data source then press View. Scroll down to the Execution Details section, select Mask in the first dropdown menu, and your chosen mask configuration in the second dropdown menu. The Execute button can be used to then run a mask job or View Results to simply view the results from the previously ran masking jobs.

For TDM versions 3.4 or earlier:

To execute a new masking request, navigate to Masking > Execute Masking

Here you can select a data source connection and Masking Configuration for execution. Press Run button to run the selected masking configuration. On Run button press, the table list down will be refreshed with past executions and will also show the status of current execution request.

Execute Masking - Connection Groups

For TDM version 3.6:

In order to run a mask scan on a connection group, firstly ensure TDM Queue Manager is turned on. Then, navigate to Data Management Hub > Execution Console > Connection Group.

Here, select your Connection Group from the first dropdown menu, then select Mask for your scan type. After selecting all the fields, click on View.

You can then select the mask configurations from the Data Configuration column for the specific Data Sources.

In order to run a bulk mask scan, select all the data sources that you want to execute the scan on and click on Execute.

Then, scroll to the bottom of the page to the Execution Details section to look at the scan status, and access the logs and reports.

Masking Log

In the table below, Log Button can be used to view logs for the execution.

Note: The Log window refreshes every 15 seconds if the mask is currently executing.

A currently executing Mask can also be Cancelled by clicking the Cancel button.

Masking Report - Single Data Source

Successfully completed masks will generate a Mask Report which can be viewed by clicking the Report button.

This can also be searched on or exported out to Excel or CSV.

Masking Report Table

Type	Description
Connection Name	The name of the data connection.
Schema	The schema under which the table resides.
Table Name	The name of the table containing the data.
Column	The specific column within the table.
Data Type	The type of data stored in the column (e.g., string, integer, date).
Data Length	The maximum length of the data in the column.
Masking Pattern Used	The pattern applied to mask the data in the column.
Masking Lookup Used	Lookup table or source used during the masking process.
Masking Parameters Used	Parameters applied during the masking process. TDM 3.4 or earlier
Flag Column	The column used to indicate a specific condition.
Flag Condition	Condition that determines when the flag is applied.
Flag String	The string value that represents the flag.
Masking Status	The current status of the masking operation (e.g., completed, in progress).
Scan Duration (Seconds)	The duration taken to complete the scan, measured in seconds.
Error Details	Details of any errors encountered for that specific column.

Masking Report - Connection Groups

For grouped connections, each data source in the group will generate a Mask Report if successful, which can be viewed by clicking the Report button.

In order export the reports in bulk, navigate to the Execution Console section in the Connection Group tab, and select the data sources to include in the bulk report.

Click on the Generate Report button, then download the report by clicking on the popup window.

Important: Databases and files have different report structures, therefore their reports cannot be generated together in bulk. When selecting the data sources to include in the bulk report, make sure that files and database connections are not selected together.

Feature Support Table

The table below outlines the masking features supported by various data source connection types:

Connection Type	Conditional Masking	Cross Table Masking	Chunking
Amazon Redshift	✅	✅	✅
CouchDB	❌	❌	❌
DB2	✅	❌	✅
MariaDB	✅	✅	✅
MongoDB	❌	❌	❌
MSSQL	✅	✅	✅
MySQL	✅	✅	✅
Oracle	✅	✅	✅
PostgreSQL	✅	✅	✅
Sybase	❌	❌	❌
Teradata	✅	❌	✅
File - AVRO	✅	N/A	❌
File - Delimited	✅	N/A	✅
File - Fixed Width	✅	N/A	❌
File - JSON	✅	N/A	❌
File - ORC	✅	N/A	❌
File - Parquet	✅	N/A	❌
File - XML	✅	N/A	❌

Note: Normal masking is supported across ALL data source connection types.