Model Selection and AutoML: Hyperparameters, Cross-Validation, the AutoML API and Feature Importance
Hyperparameter search and AutoML speed up model selection when the metric is right and the data has no leakage, but they cannot strengthen a weak signal. On Bookstore's cancellation risk we compare grid and random search for FastTree with cross-validation, the ML.NET AutoML API and the mlnet CLI that searches by accuracy, and show with real output how a leaky column produces an AUC of 1.0, Permutation Feature Importance and a fairness check by age and district.
Version note: The code in this post was built and run with .NET 10, ML.NET 6.0 with the Microsoft.ML 6.0.0 package, Microsoft.ML.FastTree 6.0.0, Microsoft.ML.LightGbm 6.0.0 and Microsoft.ML.AutoML 0.24.0; the mlnet CLI version is 16.18.2. The data is synthetic Bookstore data produced by the series' seeded generator; the numbers here are not the results of a real business.
Guess First
In the seventh post we predicted the order cancellation risk for Bookstore, the online bookstore used as the shared example across this site's series, with logistic regression and scored 0.724 ROC-AUC on the test set. In this post we leave the same problem to automatic model selection. First question: if we sweep FastTree's hyperparameters over a 16-point grid and also give AutoML 20 attempts, how much does the AUC go up?
Second question: ML.NET's command-line tool mlnet tries 91 models on the same data in 30 seconds. How large is the gap between the best of these 91 models and the model that says "nobody cancels"?
Parameters and Hyperparameters
A parameter is a value the model learns from the data during training. In logistic regression each feature's weight is a parameter:
// The parameters of a logistic regression: one learned weight per feature slot
var logistic = Classification.Features(ml).Append(ml.BinaryClassification.Trainers.LbfgsLogisticRegression(
new LbfgsLogisticRegressionBinaryTrainer.Options
{
LabelColumnName = "IsCancelled", FeatureColumnName = "Features", NumberOfThreads = 1,
})).Fit(s.Train);
VBuffer<ReadOnlyMemory<char>> names = default;
logistic.Transform(s.Train).Schema["Features"].GetSlotNames(ref names);
float[] weights = [.. logistic.LastTransformer.Model.SubModel.Weights];
Console.WriteLine("logistic regression weights (largest first):");
foreach ((string name, float w) in names.DenseValues().Select(n => n.ToString()).Zip(weights)
.OrderByDescending(x => Math.Abs(x.Second)).Take(6))
{
Console.WriteLine($" {name,-28}{w,8:F3}");
}dotnet run --project src/Bookstore.ML.Lab -- tunelogistic regression weights (largest first):
Payment.CreditCard -0.729
Payment.BankTransfer -0.305
Backorder 0.204
Method.Standard -0.195
FirstOrder 0.194
OrderTotal 0.166This table shows the finding of the seventh post from another angle: orders paid by credit card or bank transfer have negative weights, so they are less risky than cash on delivery. Payment.CashOnDelivery is not on the list, because one-hot encoding creates a column for all three payment methods and the model chose to encode the risk as "the other two are less risky" instead of "cash on delivery is risky". When columns depend on each other, reading the weights one by one can mislead; we will see a more reliable way shortly.
A hyperparameter is a value you choose before training: the number of trees in FastTree, the number of leaves per tree, the minimum number of examples in a leaf, the learning rate. The model does not learn these. Chosen badly, the model either memorizes the data or does not learn enough; the complexity curve in the third post was an example of this.
Grid Search, Random Search and Cross-Validation
The simplest way to choose hyperparameters is grid search: pick a few values for each hyperparameter and try every combination. We measure each combination not with a single validation set but with 5-fold cross-validation, so the choice does not depend on the luck of one split. The folds are grouped by customer again and the test set is never touched:
private static (double Mean, double Std) CvAuc(MLContext ml, IDataView data, IEstimator<ITransformer> pipeline)
{
double[] auc = [.. ml.BinaryClassification.CrossValidateNonCalibrated(data, pipeline, numberOfFolds: 5,
labelColumnName: "IsCancelled", samplingKeyColumnName: "CustomerId", seed: 3).Select(f => f.Metrics.AreaUnderRocCurve)];
double mean = auc.Average();
return (mean, Math.Sqrt(auc.Average(a => (a - mean) * (a - mean))));
}
private static IEstimator<ITransformer> Tree(MLContext ml, int leaves, int minLeaf, int trees, double rate) =>
Classification.Features(ml).Append(ml.BinaryClassification.Trainers.FastTree(new FastTreeBinaryTrainer.Options
{
LabelColumnName = "IsCancelled", FeatureColumnName = "Features",
NumberOfLeaves = leaves, MinimumExampleCountPerLeaf = minLeaf, NumberOfTrees = trees, LearningRate = rate,
}));// Grid search: every combination, each one scored with 5-fold cross-validation
List<(string Name, double Mean, double Std)> results = [];
Console.WriteLine($"{"FastTree grid (leaves, min leaf, rate)",-42}{"CV AUC",8}{"±",7}");
foreach (int leaves in new[] { 2, 4, 16, 64 })
foreach (int minLeaf in new[] { 10, 200 })
foreach (double rate in new[] { 0.05, 0.2 })
{
(double mean, double std) = CvAuc(ml, s.TrainAndValidation, Tree(ml, leaves, minLeaf, 100, rate));
string name = $"leaves {leaves,2}, min leaf {minLeaf,3}, rate {rate:F2}";
results.Add((name, mean, std));
Console.WriteLine($"{name,-42}{mean,8:F3}{std,7:F3}");
}LogisticRegression, 5-fold CV AUC 0.662 ± 0.015FastTree grid (leaves, min leaf, rate) CV AUC ±
leaves 2, min leaf 10, rate 0.05 0.666 0.013
leaves 2, min leaf 10, rate 0.20 0.660 0.012
leaves 2, min leaf 200, rate 0.05 0.666 0.013
leaves 2, min leaf 200, rate 0.20 0.659 0.014
leaves 4, min leaf 10, rate 0.05 0.666 0.016
leaves 4, min leaf 10, rate 0.20 0.661 0.012
leaves 4, min leaf 200, rate 0.05 0.666 0.011
leaves 4, min leaf 200, rate 0.20 0.664 0.011
leaves 16, min leaf 10, rate 0.05 0.660 0.013
leaves 16, min leaf 10, rate 0.20 0.632 0.023
leaves 16, min leaf 200, rate 0.05 0.657 0.013
leaves 16, min leaf 200, rate 0.20 0.643 0.009
leaves 64, min leaf 10, rate 0.05 0.643 0.011
leaves 64, min leaf 10, rate 0.20 0.580 0.014
leaves 64, min leaf 200, rate 0.05 0.651 0.008
leaves 64, min leaf 200, rate 0.20 0.620 0.008The best point of the grid scores 0.666 AUC with 2 leaves per tree, the simplest model, where each tree asks a single question. The result gets worse as the number of leaves and the learning rate grow: 0.580 with 64 leaves and a learning rate of 0.2. Logistic regression's cross-validation AUC is 0.662. A sum of two-leaf trees is a model that adds up the effect of each feature separately, a structure very similar to logistic regression. Because the cancellation risk behaves like a sum of features on this data, the best tree model returns to that structure too.
The problem with grid search is that the number of combinations grows multiplicatively with the number of hyperparameters. Random search spends the same budget on random points in a wider space. As Bergstra and Bengio showed in their 2012 paper, when most hyperparameters barely affect the result, random points try more distinct values of the hyperparameter that matters:
// Random search: the same budget spent on random points of a wider space
Random rnd = new(42);
Console.WriteLine($"{"FastTree random (leaves, min leaf, trees, rate)",-50}{"CV AUC",8}");
for (int i = 0; i < 8; i++)
{
int leaves = 2 << rnd.Next(0, 6), minLeaf = rnd.Next(5, 400), trees = rnd.Next(20, 300);
double rate = Math.Round(Math.Pow(10, -2.5 + 2 * rnd.NextDouble()), 3);
(double mean, _) = CvAuc(ml, s.TrainAndValidation, Tree(ml, leaves, minLeaf, trees, rate));
Console.WriteLine($"{$"leaves {leaves,2}, min leaf {minLeaf,3}, trees {trees,3}, rate {rate:F3}",-50}{mean,8:F3}");FastTree random (leaves, min leaf, trees, rate) CV AUC
leaves 32, min leaf 60, trees 55, rate 0.035 0.664
leaves 4, min leaf 108, trees 222, rate 0.034 0.665
leaves 4, min leaf 305, trees 85, rate 0.010 0.665
leaves 16, min leaf 131, trees 126, rate 0.010 0.657
leaves 16, min leaf 18, trees 247, rate 0.045 0.651
leaves 8, min leaf 65, trees 45, rate 0.081 0.664
leaves 32, min leaf 218, trees 32, rate 0.084 0.657
leaves 2, min leaf 362, trees 213, rate 0.034 0.667Eight random points reach the same result as the 16-point grid: the best is 0.667, again with two-leaf trees. In the last step we train the winner of the grid once on train and validation together and measure it once on the test set:
best grid point: leaves 2, min leaf 10, rate 0.05 (CV AUC 0.666)
test AUC: FastTree 0.720, LogisticRegression 0.733The answer to the first opening question: the hyperparameter search could not beat logistic regression. The 0.004 difference in cross-validation is within the ±0.013 spread between folds, and on the test set logistic regression is ahead. Something else stands out: the test AUCs (0.72–0.73) are clearly higher than the cross-validation means (0.66). The model did not change; the customers in the test set happen to be an easier group to predict in this split. In the third post we discussed the risk of trusting a single split; here we see the same thing from the other side. Select models with cross-validation and read the test result as a single sample.
The ML.NET AutoML API
The Microsoft.ML.AutoML package automates this search: it searches feature transforms, the trainer and the trainer's hyperparameters together. The package version is still 0.x; it was released as 0.24.0 together with ML.NET 6.0. In the new API we first describe the column types, then build a "sweepable" pipeline and hand it to an experiment:
ColumnInformation columns = new() { LabelColumnName = "IsCancelled" };
foreach (string c in new[] { "PaymentMethod", "ShippingMethod", "Warehouse" }) columns.CategoricalColumnNames.Add(c);
foreach (string c in new[] { "ItemCount", "OrderTotal", "DiscountRate", "PreviousOrders" }) columns.NumericColumnNames.Add(c);
foreach (string c in new[] { "OrderId", "CustomerId", "OrderedAtUtc", "Age", "District" }) columns.IgnoredColumnNames.Add(c);
// TrackingEventCount only exists after shipping: a cancelled order always has 0
if (withLeak) columns.NumericColumnNames.Add("TrackingEventCount");
else columns.IgnoredColumnNames.Add("TrackingEventCount");
SweepablePipeline pipeline = ml.Auto().Featurizer(s.Train, columns)
.Append(ml.Auto().BinaryClassification(labelColumnName: "IsCancelled"));
AutoMLExperiment experiment = ml.Auto().CreateExperiment()
.SetPipeline(pipeline)
.SetDataset(s.Train, s.Validation)
.SetBinaryClassificationMetric(BinaryClassificationMetric.AreaUnderRocCurve, labelColumn: "IsCancelled")
.SetTrainingTimeInSeconds(60)
.SetMaxModelToExplore(withLeak ? 4 : 20);
experiment.SetMonitor(new TrialPrinter(pipeline));
Console.WriteLine(withLeak ? "AutoML with TrackingEventCount:" : "AutoML, 20 trials at most, 60 s at most:");
TrialResult best = await experiment.RunAsync();
Console.WriteLine($"best trial {best.TrialSettings.TrialId}: validation AUC {best.Metric:F4}, " +
$"test AUC {Auc(ml, best.Model, s.Test):F4}");
}dotnet run --project src/Bookstore.ML.Lab -- automlAutoML, 20 trials at most, 60 s at most:
trial 0 AUC 0.6822 0.9 s ReplaceMissingValues=>ConvertType=>OneHotEncoding=>Concatenate=>FastTreeBinary
trial 1 AUC 0.6738 0.2 s ReplaceMissingValues=>ConvertType=>OneHotEncoding=>Concatenate=>FastTreeBinary
trial 2 AUC 0.6823 0.1 s ReplaceMissingValues=>ConvertType=>OneHotEncoding=>Concatenate=>FastTreeBinary
trial 3 AUC 0.6701 0.2 s ReplaceMissingValues=>ConvertType=>OneHotEncoding=>Concatenate=>FastForestBinary
trial 4 AUC 0.6822 0.2 s ReplaceMissingValues=>ConvertType=>OneHotHashEncoding=>Concatenate=>FastTreeBinary
trial 5 AUC 0.6738 0.2 s ReplaceMissingValues=>ConvertType=>OneHotHashEncoding=>Concatenate=>FastTreeBinary
trial 6 AUC 0.6701 0.2 s ReplaceMissingValues=>ConvertType=>OneHotEncoding=>Concatenate=>FastForestBinary
trial 7 AUC 0.6769 0.2 s ReplaceMissingValues=>ConvertType=>OneHotEncoding=>Concatenate=>FastForestBinary
trial 8 AUC 0.6782 0.5 s ReplaceMissingValues=>ConvertType=>OneHotHashEncoding=>Concatenate=>LightGbmBinary
trial 9 AUC 0.6521 0.1 s ReplaceMissingValues=>ConvertType=>OneHotEncoding=>Concatenate=>FastTreeBinary
trial 10 AUC 0.6701 0.3 s ReplaceMissingValues=>ConvertType=>OneHotHashEncoding=>Concatenate=>FastForestBinary
trial 11 AUC 0.6720 0.2 s ReplaceMissingValues=>ConvertType=>OneHotEncoding=>Concatenate=>FastForestBinary
trial 12 AUC 0.6782 0.2 s ReplaceMissingValues=>ConvertType=>OneHotEncoding=>Concatenate=>LightGbmBinary
trial 13 AUC 0.5955 3.0 s ReplaceMissingValues=>ConvertType=>OneHotEncoding=>Concatenate=>LightGbmBinary
trial 14 AUC 0.5133 1.2 s ReplaceMissingValues=>ConvertType=>OneHotHashEncoding=>Concatenate=>SdcaLogisticRegressionBinary
trial 15 AUC 0.6701 0.3 s ReplaceMissingValues=>ConvertType=>OneHotHashEncoding=>Concatenate=>FastForestBinary
trial 16 AUC 0.6769 0.2 s ReplaceMissingValues=>ConvertType=>OneHotHashEncoding=>Concatenate=>FastForestBinary
trial 17 AUC 0.6823 0.2 s ReplaceMissingValues=>ConvertType=>OneHotHashEncoding=>Concatenate=>FastTreeBinary
trial 18 AUC 0.6720 0.2 s ReplaceMissingValues=>ConvertType=>OneHotHashEncoding=>Concatenate=>FastForestBinary
trial 19 AUC 0.6744 0.2 s ReplaceMissingValues=>ConvertType=>OneHotHashEncoding=>Concatenate=>FastForestBinary
best trial 2: validation AUC 0.6823, test AUC 0.7181To keep the code short I used a small IMonitor implementation that prints the trial lines in the output; it is attached to the experiment with SetMonitor. In 20 trials AutoML tried FastTree, FastForest, LightGBM and SDCA with one-hot and hashing encoding. The best trial scored 0.682 AUC on the validation set and 0.718 on the test set. That is below the test result of the logistic regression we built by hand (0.733). AutoML is not a bad tool; but in this problem the hard part is not choosing the algorithm, it is that the signal is weak.
Two practical notes. First, AutoML is not deterministic. With the same seed, 19 of 20 trials came out the same in a second run, but trial 17 tried SDCA this time. According to Microsoft Learn the default tuner is Eci Cost Frugal; the "cost" in its name is the training time of the trials. Since durations differ slightly on every run, even the current load of the machine can change which trial is picked. Second, the platform: like Microsoft.ML.TimeSeries in the eighth post, the AutoML package depends on the Intel MKL package, and it also tries LightGBM. That is why I produced the AutoML output in a Linux x64 container. According to Microsoft Learn, Windows needs no extra installation; I did not run it on Windows.
The mlnet CLI and Model Builder
Two tools run the same search without code. Model Builder is a Visual Studio extension; it trains a model from data you pick in a graphical interface and generates code. It runs only in Visual Studio on Windows. mlnet is a .NET tool. NuGet has separate packages per operating system: mlnet-win-x64 for Windows, mlnet-linux-x64 for Linux. I installed the Linux package in the container into a folder, without a global installation:
dotnet tool install mlnet-linux-x64 --version 16.18.2 --tool-path tools
tools/mlnet classification --dataset orders_clean.csv --label-col IsCancelled --has-header true --ignore-cols OrderId CustomerId OrderedAtUtc City District CustomerNote DeliveryDays TrackingEventCount IsReturned ReturnRequestedUtc TotalWeightKg DistanceKm Carrier --train-time 30 --name CancelModel -o outStart Training
start binary classification
Evaluate Metric: Accuracy
Available Trainers: LGBM,FASTFOREST,FASTTREE,LBFGS,SDCA
Training time in second: 30
Use train validate split with ratio: 0.1
| Trainer Accuracy Duration |
|--------------------------------------------------------------------|
|0 FastTreeBinary 0.9574 0.6930 |
|1 FastTreeBinary 0.9574 0.1690 |
...
|--------------------------------------------------------------------|
| Experiment Results |
|--------------------------------------------------------------------|
| Summary |
|--------------------------------------------------------------------|
|ML Task: binary classification |
|Dataset: /w/cli/orders_clean.csv |
|Label : IsCancelled |
|Total experiment time : 29.0000 Secs |
|Total number of models explored: 91 |
|--------------------------------------------------------------------|
| Top 5 models explored |
|--------------------------------------------------------------------|
| Trainer Accuracy Duration |
|--------------------------------------------------------------------|
|35 LightGbmBinary 0.9578 0.8460 |
|0 FastTreeBinary 0.9574 0.6930 |
|1 FastTreeBinary 0.9574 0.1690 |
|2 FastTreeBinary 0.9574 0.1290 |
|3 FastTreeBinary 0.9574 0.1870 |
|--------------------------------------------------------------------|
save CancelModel.mbconfig to /w/cli/out/CancelModel
Generating a console project for the best pipeline at location : /w/cli/out/CancelModelI shortened the 80 lines in the middle of the output. Here is the answer to the second opening question: the best of 91 models scores 0.9578 accuracy, the model that says nobody cancels scores 0.9574. The gap is 0.0004. The metric the CLI optimizes for binary classification is accuracy, and this version has no option to pick another metric; the Model Builder page on Microsoft Learn also says that the default metric for binary classification is accuracy. As we saw in the seventh post, a tool that optimizes accuracy on imbalanced data is really looking for the model that predicts the majority class. On top of that, the latest version of mlnet on NuGet is 16.18.2, from February 2024; the project it generates uses net8.0 and Microsoft.ML 3.0.1. Use the tool for ideas, and rebuild the generated project with current packages.
Permutation Feature Importance
Once a model is trained, the business team's first question is "what does it base its decision on?" Logistic regression's weights could not answer this reliably because of the dependent columns. PFI, permutation feature importance, works independently of the model: it shuffles one feature's values across rows and measures how much the metric gets worse. If the shuffled feature matters, the AUC drops; if it does not, nothing changes.
// Shuffle one feature slot at a time and measure how much the test AUC drops
var pfi = ml.BinaryClassification.PermutationFeatureImportance(model.LastTransformer, test,
labelColumnName: "IsCancelled", permutationCount: 5);
VBuffer<ReadOnlyMemory<char>> names = default;
test.Schema["Features"].GetSlotNames(ref names);
Console.WriteLine($"{"feature slot",-28}{"AUC drop",10}{"±",8}");
foreach ((string name, BinaryClassificationMetricsStatistics m) in names.DenseValues().Select(n => n.ToString())
.Zip(pfi).OrderBy(x => x.Second.AreaUnderRocCurve.Mean).Take(8))
{
Console.WriteLine($"{name,-28}{-m.AreaUnderRocCurve.Mean,10:F4}{m.AreaUnderRocCurve.StandardError,8:F4}");
}dotnet run --project src/Bookstore.ML.Lab -- explainfeature slot AUC drop ±
Payment.CashOnDelivery 0.0681 0.0066
Backorder 0.0421 0.0011
OrderTotal 0.0172 0.0025
Payment.CreditCard 0.0160 0.0033
FirstOrder 0.0144 0.0026
Method.Standard 0.0141 0.0019
Payment.BankTransfer 0.0138 0.0029
DiscountRate 0.0070 0.0015The most important feature is cash on delivery: shuffling it drops the AUC by 0.068. Second is the out-of-stock book (0.042), then, with much smaller effects, the order total, the other payment methods, being the first order and the shipping method. Payment.CashOnDelivery, missing from the weight table, is first here. This list can be explained to the business team: "Payment method and stock state determine the cancellation risk the most, the basket total and the customer's first order add a little." The table also shows why the two-condition rule in the seventh post gave almost the same result as the model.
PFI has limits too. If two features depend on each other, shuffling one shows little effect, because the model gets the same information from the other. Nor is the result a causal claim: it does not say "cash on delivery causes cancellations", only that the model relies on this information.
Fairness: Sensitive Features and Proxies
In the second post we marked the birth year as a sensitive column. We check in two ways whether a model behaves differently by age: what changes when age is added as a feature, and what share of orders gets flagged in each age group? We also add the district (District); in real life the district is a typical proxy feature that can indirectly stand in for sensitive information such as income level or ethnic origin:
(string Name, IEstimator<ITransformer> Pipeline)[] variants =
[
("order features", Classification.Features(ml).Append(Classification.Logistic(ml))),
("+ Age", Classification.Features(ml, ["Age"]).Append(Classification.Logistic(ml))),
("+ District (one-hot)", ml.Transforms.Categorical.OneHotEncoding("DistrictKey", "District")
.Append(Classification.Features(ml, ["DistrictKey"])).Append(Classification.Logistic(ml))),
];dotnet run --project src/Bookstore.ML.Lab -- fairness test AUC flagged under 30 flagged 30-49 flagged 50+
actual cancel rate 4.0 % 3.9 % 4.6 %
order features 0.733 22.5 % 21.3 % 25.2 %
+ Age 0.731 22.5 % 21.3 % 25.3 %
+ District (one-hot) 0.610 22.2 % 24.9 % 24.6 %Adding age does not change the AUC (0.733 and 0.731), and the flag rates by age group stay almost the same. In this synthetic data age does not affect cancellation, and the model does not use it. Still, the flag rates differ between age groups: 25.2% above 50, 21.3% between 30 and 49. The difference exists even though the model never sees age, because the actual cancellation rate in this group is higher too (4.6%) and features such as the payment method are distributed differently across groups. Removing a sensitive column from the model does not guarantee that the model treats that group no differently. That is why a fairness check is done by measuring results per group, not by looking at the column list.
Adding the district drops the AUC to 0.610. One-hot columns for hundreds of districts, combined with few cancellation examples, make the model learn chance differences between districts. In this post the result is just a bad model; on real data the same model could systematically flag people living in certain districts as "risky". If the cancellation risk score is going to turn into a decision such as disabling cash on delivery for a customer, set the decision rule together with the business and legal teams, and use the score as a signal that goes to human approval instead of an automatic rejection. The details under Turkey's personal data protection law are in the Data Privacy and KVKK series.
Watch Out in Production
Do not reduce model selection to a single number. Next to AUC, look at prediction time, model file size, explainability and platform support. By the results in this post, logistic regression is better, smaller and easier to explain than the models AutoML found; it has no native dependency like LightGBM either.
Be ready for small differences between platforms. I also ran the tune command in a Linux x64 container: the FastTree results were exactly the same, while logistic regression's weights and AUCs differed in the third decimal (a test AUC of 0.731 instead of 0.733). Floating-point calculations can round differently with different processor instructions. Train the model file once and ship the same file to every environment; do not retrain it in each one.
Pin the pipeline AutoML chose in code. If you run AutoML again on every training, it can produce a different model and a different feature transform even when the data does not change; we will see why that is a problem when we discuss model versioning in the next post.
Make the PFI and fairness tables part of the model card: which features the model relies on and the flag rates per group should be recomputed every time the model is retrained.
Common Pitfalls
The first pitfall is giving AutoML data with leakage. TrackingEventCount is the number of tracking events of the order; for a cancelled order it is always 0. Give this column to AutoML and the result is:
dotnet run --project src/Bookstore.ML.Lab -- automl --leakAutoML with TrackingEventCount:
trial 0 AUC 1.0000 0.9 s ReplaceMissingValues=>ConvertType=>OneHotEncoding=>Concatenate=>FastTreeBinary
trial 1 AUC 1.0000 0.2 s ReplaceMissingValues=>ConvertType=>OneHotEncoding=>Concatenate=>FastTreeBinary
trial 2 AUC 1.0000 0.1 s ReplaceMissingValues=>ConvertType=>OneHotEncoding=>Concatenate=>FastTreeBinary
trial 3 AUC 1.0000 0.2 s ReplaceMissingValues=>ConvertType=>OneHotEncoding=>Concatenate=>FastForestBinary
best trial 0: validation AUC 1.0000, test AUC 1.0000All four trials score 1.0000 AUC, and so does the test set, because the leakage is in the test set too. AutoML will happily pick this model. The only protection is asking for every column: "is this information available at prediction time?" A perfect score is almost always the sign of a mistake.
The second is trusting the tool's default metric. mlnet ranked 91 models by accuracy and none of them could be told apart from the majority-class baseline. The problem decides the metric, not the tool.
The third is making the highest score the only criterion. The 0.004 difference in cross-validation was smaller than the spread between folds. When the difference is at noise level, pick the simpler model.
The fourth is thinking that deleting the sensitive column solves the fairness problem. Even the model that never saw age flagged age groups at different rates; proxy features such as the district can carry the same information indirectly.
A Small Experiment
★☆☆ Set SetMaxModelToExplore to 50 and raise SetTrainingTimeInSeconds to 120 in the AutoML experiment. Does the best validation AUC change? Which trainers are tried more often?
★★☆ Run the Explain method for the FastTree model as well. Is the order of the important features the same as for logistic regression? Why might the importance of OrderTotal differ?
★★★ Add a breakdown by payment method to the fairness table and compute the cash-on-delivery share in each age group. Does it explain the higher flag rate above 50? Explain the result in a two-paragraph note to the team that will use the model.
In the next post we take a model to production: we put the delivery-time model into an ASP.NET Core minimal API with PredictionEnginePool, measure how the pool solves the thread-safety problem from the fourth post, version the model file, update the model without a restart by replacing the file and watch for drift in the data.