Documentation / ML Studio
Evaluation and models
Independent evaluation, metrics, robustness and fairness, champion versus challenger, the model registry, on-disk versioning and how to publish.
On this page
A model isn't ready because it finished training. It's ready when an independent evaluation says it meets the criteria, a person approves it and its package passes the tests.
Independent evaluation
Yuli evaluates each version separately, on the test partition · held out from training: data the model never saw while training or while choosing parameters. The verdict is clear:
- “Meets the criteria (n/N)”, or
- “Doesn't meet: …”, with the criterion that failed.
The criteria are the ones in acceptance in yuli-ml.yaml:
acceptance:
minimum_f1: 0.85
maximum_latency_ms: 200
maximum_memory_mb: 2048
require_no_nan: trueMetrics
| Problem type | Metrics |
|---|---|
| All | Loss (train / validation) |
| Classification | Accuracy, F1, Macro F1, Precision, Recall, ROC AUC |
| Calibration | ECE, Brier |
| Regression and series | RMSE, MAE, R², MAPE |
| Language models | Perplexity, Tokens/s |
| Serving | p95 latency (ms) |
With imbalanced classes, Accuracy lies: a model that always says “no” is right 95% of the time if only 5% are “yes”. Use F1, Macro F1 or Recall depending on which mistake costs more.
Confusion matrix
In classification, the Confusion matrix shows, for each real class, how many times the model got it right and what it confused it with. It's the quickest way to see where it fails.
Robustness and fairness
Robustness and fairness checks how the model changes with perturbed data and across groups. Each test comes out as robust, sensitive or fragile. If the model affects people, always review it.
Evaluation history
Each evaluation is recorded with When, Suite, Test data, Criteria and Verdict, and with the Improvement vs production: how much it improves (or worsens) compared with the model in service.
The model panels
| Panel | What it does |
|---|---|
| Summary | Metrics, criteria and verdict. |
| Explorer | What the model learned on the inside. |
| Explanation | Which variables weigh in each prediction. |
| Try | Predictions by hand: “What if…?” changes a value and you see the effect, “Live · n ms”. |
| Card | The model card: what it does, data, metrics and limits. |
| Deployment | Local service and typed client. |
| Contract | The input and output shape your application can expect. |
Champion versus challenger
The champion is the version in production. A challenger is a new version that competes with it on the same test data. Promoting it only makes sense if the challenger improves measurably.
Registry and states
Development → Evaluation → Approved → Staging → Production
↘ Rejected ↘ ArchivedPromotion to production is always done by a person. One-click rollback. No agent can promote a model.
On-disk versioning
Each version lives in models/<name>/v<major>.<minor>/ with everything needed to reproduce and audit it:
models/bajas-clientes/v1.2/
├── model.yaml
├── lineage.json
├── configs/
├── experiments/
│ └── run-3.json
├── artifacts/
│ ├── weights/
│ ├── checkpoints/
│ ├── metrics/
│ └── reports/
├── evaluation/
│ └── evaluation-report.json
└── exports/Publish
The Publish tab walks through a chain of steps on the validated version:
Consumption modes
| Mode | When |
|---|---|
| Embedded | The model runs inside your application. |
| Standalone | An executable of its own. |
| Microservice | An HTTP service that others call. |
| Batch | Score batches of data periodically. |
| Streaming | Predictions over streaming events. |
Local service
Serve starts the model on 127.0.0.1 with POST /predict (LLMs, with an OpenAI-compatible API). It only listens
on your machine.
curl -s http://127.0.0.1:<port>/predict \
-H "Content-Type: application/json" \
-d '{"inputs": [{"antiguedad_meses": 4, "plan": "basico", "tickets_90d": 7}]}'It responds with {"predictions": [...]}, in the same shape described by the Contract panel
(model-contract.json).
Typed client SDK
In Typed client (SDK) you generate a client for TypeScript, Python, Java, Go or C#. To integrate it with a timeout, a circuit breaker and a fallback to your own rules if the model fails, ask the agent.
See also ML privacy and sandbox.