Accuracy
This option should be selected if your goal is to create a dataset that more closely resembles your original dataset.
This could apply to the following scenarios:
- Creating a larger dataset from a small dataset, as an augmented dataset for machine learning tasks.
- Creating a test dataset that has similar statistical properties to your original dataset for dashboard testing.
Overview
The model selected is based on the size (number of rows) in your dataset:
| Model | Number of rows in dataset |
|---|---|
| Bayesian Network | Recommended for 500 rows and less; max 5000 rows |
| Denoising Diffusion Probabilistic Models (DDPM) | Recommended for more than 500 rows; max 50000 rows |
| Gaussian Copula | Recommended for more than 50000 rows |
Models
Bayesian Network
Bayesian Network generates data from existing data, by mapping out the relationships between data points in the original data. Synthetic data is generated by following the relationships between data points after training is completed. It is preferred for smaller datasets which other models may overfit on.
Sources:
- Gradient Descending: A detailed walkthrough of using Bayesian Network to generate synthetic data with code (R).
- Science Direct: A case study using Bayesian Networks to generate gene expression data.
- Arxiv: A research paper that uses Bayesian Network to generate synthetic data.
Denoising Diffusion Probabilistic Model (DDPM)
Denoising Diffusion Probabilistic Model (DDPM) generates data by obscuring existing data with the following steps:
Noising: Adding noise to existing data in a controlled manner, to obscure real data.
Denoising: Progressively removes noise to generate new data.
Sources:
- Machine Learning Expedition: A clear introduction to DDPM .
- Medium: A detailed walkthrough of DDPM, with diagrams and code (Python).
- Arxiv: The research paper that introduces DDPM.
Gaussian Copula
Gaussian Copula transforms the existing data into having a normal distribution, before learning the relationships between different columns. The relationships between columns are then used to generate synthetic data. It balances quality with computational resources usage, making it more suitable for larger datasets.
Sources:
- An intuitive, visual guide to copulas: An introduction to copulas with graphs showing the normal transformation of the data distribution.
- Copula: A Very Short Introduction: Detailed mathematical explanation of copulas with interactive diagrams.
