Documentation / ML Studio

Evaluation and models

Independent evaluation, metrics, robustness and fairness, champion versus challenger, the model registry, on-disk versioning and how to publish.

On this page

A model isn't ready because it finished training. It's ready when an independent evaluation says it meets the criteria, a person approves it and its package passes the tests.

Independent evaluation

Yuli evaluates each version separately, on the test partition · held out from training: data the model never saw while training or while choosing parameters. The verdict is clear:

  • “Meets the criteria (n/N)”, or
  • “Doesn't meet: …”, with the criterion that failed.

The criteria are the ones in acceptance in yuli-ml.yaml:

yaml
acceptance:
  minimum_f1: 0.85
  maximum_latency_ms: 200
  maximum_memory_mb: 2048
  require_no_nan: true

Metrics

Problem typeMetrics
AllLoss (train / validation)
ClassificationAccuracy, F1, Macro F1, Precision, Recall, ROC AUC
CalibrationECE, Brier
Regression and seriesRMSE, MAE, R², MAPE
Language modelsPerplexity, Tokens/s
Servingp95 latency (ms)

With imbalanced classes, Accuracy lies: a model that always says “no” is right 95% of the time if only 5% are “yes”. Use F1, Macro F1 or Recall depending on which mistake costs more.

Confusion matrix

In classification, the Confusion matrix shows, for each real class, how many times the model got it right and what it confused it with. It's the quickest way to see where it fails.

Robustness and fairness

Robustness and fairness checks how the model changes with perturbed data and across groups. Each test comes out as robust, sensitive or fragile. If the model affects people, always review it.

Evaluation history

Each evaluation is recorded with When, Suite, Test data, Criteria and Verdict, and with the Improvement vs production: how much it improves (or worsens) compared with the model in service.

The model panels

PanelWhat it does
SummaryMetrics, criteria and verdict.
ExplorerWhat the model learned on the inside.
ExplanationWhich variables weigh in each prediction.
TryPredictions by hand: “What if…?” changes a value and you see the effect, “Live · n ms”.
CardThe model card: what it does, data, metrics and limits.
DeploymentLocal service and typed client.
ContractThe input and output shape your application can expect.

Champion versus challenger

The champion is the version in production. A challenger is a new version that competes with it on the same test data. Promoting it only makes sense if the challenger improves measurably.

Registry and states

text
Development → Evaluation → Approved → Staging → Production
                                    ↘ Rejected      ↘ Archived

Promotion to production is always done by a person. One-click rollback. No agent can promote a model.

On-disk versioning

Each version lives in models/<name>/v<major>.<minor>/ with everything needed to reproduce and audit it:

text
models/bajas-clientes/v1.2/
├── model.yaml
├── lineage.json
├── configs/
├── experiments/
│   └── run-3.json
├── artifacts/
│   ├── weights/
│   ├── checkpoints/
│   ├── metrics/
│   └── reports/
├── evaluation/
│   └── evaluation-report.json
└── exports/

Publish

The Publish tab walks through a chain of steps on the validated version:

0 of 7 done

Consumption modes

ModeWhen
EmbeddedThe model runs inside your application.
StandaloneAn executable of its own.
MicroserviceAn HTTP service that others call.
BatchScore batches of data periodically.
StreamingPredictions over streaming events.

Local service

Serve starts the model on 127.0.0.1 with POST /predict (LLMs, with an OpenAI-compatible API). It only listens on your machine.

bash
curl -s http://127.0.0.1:<port>/predict \
  -H "Content-Type: application/json" \
  -d '{"inputs": [{"antiguedad_meses": 4, "plan": "basico", "tickets_90d": 7}]}'

It responds with {"predictions": [...]}, in the same shape described by the Contract panel (model-contract.json).

Typed client SDK

In Typed client (SDK) you generate a client for TypeScript, Python, Java, Go or C#. To integrate it with a timeout, a circuit breaker and a fallback to your own rules if the model fails, ask the agent.

See also ML privacy and sandbox.