Step 1: Upload Dataset

Sections

1. Data size: Determine if your data is suitable for synthetic data generation and handle data size issues.
2. Data security requirements: Eligible Government data and data security measures.

Data size

Synthetic data generation requires sufficient input data to train the model. An adequate number of rows relative to columns is needed for the model to reliably capture statistical attributes. Insufficient data may render the dataset unsuitable for synthetic generation. For low-data cases, try our Support Assistant or raise a ticket.

Summary

Ensuring adequate data

The ratio of rows to columns can be improved by doing the following before uploading your dataset to Mirage:

• Increasing the number of rows
• Reducing the number of columns

If your dataset exceeds Mirage’s file-size or row limits, see Reducing the number of rows.

Increasing the number of rows

If your dataset size is small, consider increasing the number of rows if your dataset is a subset of a larger dataset.

The minimum amount of data Mirage requires is 20 rows.

Reducing the number of rows

If your dataset size is too large, consider reducing the number of rows. Mirage accepts files up to 500 MB and datasets with up to 500,000 rows. Reducing the number of rows can also provide a smoother Mirage experience with faster runtimes and fewer errors.

If your use case requires a dataset above Mirage’s limit, please raise a ticket.

Reducing the number of columns

Choosing the right columns to include is a key step in achieving high-quality synthetic data. A more intentional selection of columns allows the model to learn patterns more effectively, leading to better results and faster training times.

The maximum number of columns Mirage allows is 50. Consider reducing the number of columns in your dataset via the following methods before uploading to Mirage.

1. Drop columns not needed for pattern learning

If a column isn’t important for pattern preservation (e.g., not used in analytics or ML tasks), consider excluding it from SDG and generating it using Mock Data Generation instead.

E.g., For a task like predicting patient insurance charges:

• Include: Diagnosis, Age, Number of hospital visits

• Consider dropping: Date of admission, Residential address

  Alternatively, sensitive columns can be replaced with mock data using Mirage’s Mock Data Generation feature during the following data preparation step. To use this feature, do not drop these columns at the current stage.

2. Remove columns that can be derived from others.

E.g., If you have these columns: Date, Year, Month, Day

• Include: Date
• Consider dropping: Year, Month, Day (can be extracted from Date)

3. Consider dropping high cardinality columns

Columns with too many unique values (e.g., long free-text notes, UUIDs) may add noise, slow model training and reduce data utility.

Recommendations:

• Replace with simpler, semantically meaningful features

  e.g. Replace unstructured “ClinicalNotes” with most meaningful features, like “DiagnosisCode”.

• Use Mirage’s Mock Data Generation feature.

Data security requirements

For Government data, Mirage has met IM8 requirements around Application Development Security and Risk Management for use of Government data classified up to Confidential Cloud-Eligible (CCE), Sensitive-High (SH). Key risk mitigation measures include:

• Mirage does not retain data. Raw data is purged upon completion of data generation.
• All data is encrypted in transit and at rest.
• Monthly review of privileged accounts.
• Quarterly Vulnerability Assessment scan and Yearly Penetration Testing.

You may check out the FAQ page for more information on data security related questions.