Documentation / ML Studio

Data in ML Studio

Import and profile datasets, immutable versions, preparation, quality, sensitivity with masking, and lineage.

On this page

Almost every bad model starts with a problem that was already in the data. The Data tab helps you see it before training. A dataset is a table: rows (examples) and columns (variables and the column to predict).

Import and profile

Press Import and profile and choose a file or a database.

SourceDetail
FilesCSV, TSV, JSON, JSONL, Parquet and SQLite.
DatabaseA read-only SELECT query on a Database connection.
Large dataWorks with datasets larger than RAM.

When importing, Yuli computes the full profile and shows you the Masked sample: the example rows with the sensitive columns hidden.

Immutable versions

Every change (import, clean, split) creates a new, immutable version, identified by a content hash. A version is never overwritten: a model can always say exactly which data it was trained on.

text
ventas.csv
├── v1  sha256:4f1c…  imported
├── v2  sha256:9a07…  remove duplicates
├── v3  sha256:c2e8…  fill nulls (median)
└── v4  sha256:71bd…  split 70 / 15 / 15

The views of a dataset

ViewWhat it shows
ProfileSchema and per-column statistics.
PrepareCleaning and transformation steps.
CardThe dataset card, with its license (required to approve a model).
DataThe rows, masked where appropriate.
SensitivityWhich columns are sensitive and how they are masked.
QualityAn overall score and its dimensions.
LineageWhere each version comes from and which models use it.

Profile

The column table shows Column, Type, Nulls, Unique, Distribution and Summary. Below it:

  • Class balance: how many examples there are of each value of the column to predict.
  • Strongest correlations: the pairs of columns that move together the most.
  • Detected issues: constant columns, identifiers worth excluding, many nulls, duplicate rows or imbalanced classes.

A near-perfect correlation with the column to predict is usually a leak: information from the answer that slips into the variables. The model will look excellent and fail in production. Exclude that column.

Prepare the data

Each Prepare step produces a new version:

StepWhat it does
Remove duplicatesDeletes repeated rows.
Fill nullsFills in missing values.
One-hotTurns a category into columns of 0 and 1.
StandardizeCenters on 0 with a standard deviation of 1.
Normalize (0–1)Scales to the range 0 to 1.
SplitSeparates into training, validation and test.
0 of 5 done

Quality

The Quality view gives an Overall score and a Grade, and breaks it down into dimensions:

DimensionQuestion
MissingHow many values are missing, and where?
ValidityDo the values have the expected type and format?
Duplicates and rulesAre there repeated rows or broken business rules?
OutliersAre there suspicious extreme values?
DistributionsDo the columns have reasonable shapes?
Time and driftDoes the data change over time?
LabelsIs the column to predict consistent?
BiasIs any group under- or over-represented?

Sensitivity and masking

In Sensitivity, each column has a probability of being sensitive, a category and a suggested masking. The detector combines the column name, patterns and check-digit validators (cards, IBAN, CLABE, CPF, CURP/RFC).

  • You and your local models work with the real values: they don't leave your computer.
  • External agents only see sensitive columns as •••. Aggregates do get through: an agent can reason about which variables matter most without seeing a single customer.
  • Only a person can mark a column as not sensitive. An agent can never unmask data for itself.

Sensitive columns are recorded in yuli-ml.yaml:

yaml
sensitiveColumns:
  - email
  - curp
  - telefono

If the model affects people, also review the Bias dimension in Quality and the Robustness and fairness section of the evaluation. See ML privacy and sandbox.

Lineage

Lineage draws the full chain: source file or query → dataset versions → preparation steps → training runs → model versions. Each model keeps its lineage.json, so you can always answer “exactly which data was the version in production trained on?”.

Next step: Training.