Feature Engineering: Normalization, One-Hot Encoding, Text Featurization and Missing Values

Version note: The code in this post was built and run with .NET 10, ML.NET 6.0 with the Microsoft.ML 6.0.0 package, and Microsoft.ML.FastTree 6.0.0. The data is synthetic Bookstore data produced by the series' seeded generator.

Guess First

In the previous post we trained a linear model for Bookstore, the online bookstore used as the shared example across this site's series, that predicts delivery time with SDCA, and we scaled the features into the 0–1 range with NormalizeMinMax. How much does the error grow if you remove normalization? What happens if you add the order total (OrderTotal, a number between 150 and 40 thousand TRY) to the feature list without normalization? And what if you run the same experiment with the tree-based FastTree?

In this post we add features group by group and measure every step with cross-validation. The answers are in the first table; some of them will not be what you expect.

The Measurement Setup

Every experiment does the same thing: it adds a feature group, builds the pipeline with both SDCA and FastTree, and prints the mean MAE, the mean absolute error in days, of 5-fold cross-validation grouped by customer. The building blocks are small methods:

/// <summary>The post 04 columns: one-hot categories, booleans as 0/1, two numbers.</summary>
private static IEstimator<ITransformer> Base() =>
    Ml.Transforms.Categorical.OneHotEncoding(
            [new("CarrierKey", "Carrier"), new("MethodKey", "ShippingMethod"), new("WarehouseKey", "Warehouse")])
        .Append(Ml.Transforms.Conversion.ConvertType(
            [new("Campaign", "IsCampaign"), new("Backorder", "HasBackorderItem")], DataKind.Single));
private static readonly string[] BaseColumns =
    ["CarrierKey", "MethodKey", "WarehouseKey", "Campaign", "Backorder", "DistanceKm", "ItemCount"];
public static IEstimator<ITransformer> OrderTime() =>
    Ml.Transforms.CustomMapping<OrderTimeInput, OrderTimeFeatures>(OrderTimeMappingFactory.Map, "OrderTime")
        .Append(Ml.Transforms.Categorical.OneHotEncoding("LocalDayKey", "LocalDay"));
public static IEstimator<ITransformer> Weight() =>
    Ml.Transforms.IndicateMissingValues("WeightMissing", "TotalWeightKg")
        .Append(Ml.Transforms.Conversion.ConvertType("WeightMissing", outputKind: DataKind.Single))
        .Append(Ml.Transforms.ReplaceMissingValues("Weight", "TotalWeightKg",
            MissingValueReplacingEstimator.ReplacementMode.Mean));
public static IEstimator<ITransformer> Note() =>
    Ml.Transforms.Text.FeaturizeText("NoteFeatures", "CustomerNote");

// Every step gets a fresh MLContext with the same seed, so the steps differ only in their features.
void Step(string name, Func<IEstimator<ITransformer>> features, string[] columns, string normalize = "MinMax", bool tree = true)
{
    Ml = new MLContext(seed: 1);
    IEstimator<ITransformer> f = features().Append(Ml.Transforms.Concatenate("Features", columns));
    if (normalize == "MinMax") f = f.Append(Ml.Transforms.NormalizeMinMax("Features"));
    if (normalize == "MeanVariance") f = f.Append(Ml.Transforms.NormalizeMeanVariance("Features"));
    string sdca = Cv(data, f.Append(Sdca())).ToString("F3");
    string ft = tree ? Cv(data, f.Append(Tree())).ToString("F3") : "  -  ";
    Console.WriteLine($"{name,-44} SDCA {sdca}   FastTree {ft}");
}

private static double Cv(IDataView data, IEstimator<ITransformer> pipeline) =>
    Ml.Regression.CrossValidate(data, pipeline, numberOfFolds: 5, labelColumnName: "DeliveryDays",
        samplingKeyColumnName: "CustomerId", seed: 7).Average(f => f.Metrics.MeanAbsoluteError);

Each step gets a new MLContext for this reason: consecutive experiments that share a context affect each other, because they consume the context's random number generator in turn. In my first attempt the same configuration gave two different results, 0.850 and 0.855, on different rows of the table. When each step starts with its own context, the difference between steps comes only from the features. Here is the whole table:

dotnet run --project src/Bookstore.ML.Lab -- features

04 columns, no normalization                 SDCA 0.839   FastTree 0.850
04 columns, NormalizeMinMax                  SDCA 0.844   FastTree 0.850
04 columns, NormalizeMeanVariance            SDCA 0.841   FastTree   -  
04 columns + OrderTotal, no normalization    SDCA 4.729   FastTree 0.862
04 columns + OrderTotal, NormalizeMinMax     SDCA 0.844   FastTree 0.862
+ City (one-hot, 16)                         SDCA 0.849   FastTree 0.850
+ District (one-hot)                         SDCA 0.838   FastTree 0.844
+ District (hash, 32 slots)                  SDCA 0.833   FastTree 0.847
+ LocalDay, LocalHour                        SDCA 0.637   FastTree 0.541
+ HoursUntilProcessing                       SDCA 0.525   FastTree 0.531
+ CustomerNote (FeaturizeText)               SDCA 0.519   FastTree 0.530
+ TotalWeightKg (mean + missing flag)        SDCA 0.520   FastTree 0.535

The table reads from top to bottom: each row contains the features of the previous one, except the normalization rows. Let's look at it group by group.

Numeric Features and Normalization

The first three rows are a surprise: on the seven columns from the previous post, normalization barely affects SDCA (0.839, 0.844, 0.841). The distance is between 20 and 1,600, the number of books between 1 and 120, and the rest are 0 and 1. SDCA can cope with that difference in scale.

In the fourth row the picture changes. Adding OrderTotal to the same columns without normalization sends SDCA's error up to 4.729 days; the model is worse than predicting the mean. When the same columns go through NormalizeMinMax, the error is back to 0.844. The optimization of linear models updates the weight of every feature step by step. If one feature's values are tens of thousands of times larger than the others, these steps get unbalanced and the model either converges very slowly or not at all. Normalization prevents this by bringing all features into a similar range: NormalizeMinMax into 0–1, NormalizeMeanVariance to a mean of 0 and a standard deviation of 1.

Look at the FastTree column: 0.850 and 0.862 with or without normalization. Decision trees split on threshold questions such as "is the distance greater than 650?"; scaling all values of a column by the same factor moves the threshold but does not change which order lands on which side. That is why tree-based models are not affected by scale, while linear models, distance-based algorithms such as k-means and neural networks are. The practical rule: always put normalization in the pipeline; it does no harm to tree-based models.

Categorical: One-Hot and Hashing

Adding the city gains nothing for either model (0.849 and 0.850). That is not surprising: the distance between the city and the warehouse is already a feature, and the city itself carries no extra information.

The district is a different case. There are 312 distinct districts, and one-hot encoding opens a separate slot for each. The error drops to 0.838 with SDCA and 0.844 with FastTree; some districts get systematically later deliveries and the model picks that up. Columns with this many values are called high-cardinality, and they have two problems: they grow the feature vector, and the model cannot learn anything reliable for values with few examples. The alternative is hashing:

Step("+ District (hash, 32 slots)", () => City().Append(
    Ml.Transforms.Categorical.OneHotHashEncoding("DistrictKey", "District", numberOfBits: 5)), district);

OneHotHashEncoding passes each district name through a hash function and places it in one of the 32 slots opened by numberOfBits: 5. Different districts can land in the same slot (a collision). Even so, SDCA's error is 0.833, slightly better than one-hot, and FastTree's is 0.847, slightly worse. The vector shrank from 312 slots to 32 with almost no loss. Hashing has one more advantage: when a new district that never appeared in training shows up, one-hot ignores it completely, while hashing still places it in a slot. For categories that keep growing, such as publishers, city-district pairs or book ISBNs, hashing is a good default.

Date and Time: Features from Domain Knowledge

The real jump comes from the date. Giving the model the raw OrderedAtUtc value is useless. But knowing on which day and at what time an order was placed tells a lot. First we convert to local time (Turkey, UTC+3) and extract the day of the week and the hour. We write this transformation as a CustomMapping inside the pipeline:

public sealed class OrderTimeFeatures
{
    public string LocalDay { get; set; } = "";
    public float LocalHour { get; set; }
    /// <summary>Hours until the warehouse can start on the order (15:00 cut-off, no weekend shifts).</summary>
    public float HoursUntilProcessing { get; set; }
}

[CustomMappingFactoryAttribute("OrderTime")]
public sealed class OrderTimeMappingFactory : CustomMappingFactory<OrderTimeInput, OrderTimeFeatures>
{
    private static readonly TimeSpan TurkeyOffset = TimeSpan.FromHours(3);
    public static void Map(OrderTimeInput input, OrderTimeFeatures output)
    {
        DateTime local = input.OrderedAtUtc + TurkeyOffset;
        output.LocalDay = local.DayOfWeek.ToString()[..3];
        output.LocalHour = local.Hour + local.Minute / 60f;
        DateTime start = local.Hour >= 15 ? local.Date.AddDays(1).AddHours(9) : local;
        while (start.DayOfWeek is DayOfWeek.Saturday or DayOfWeek.Sunday)
        {
            start = start.Date.AddDays(1).AddHours(9);
        }
        output.HoursUntilProcessing = (float)(start - local).TotalHours;
    }
    public override Action<OrderTimeInput, OrderTimeFeatures> GetMapping() => Map;
}

The "+ LocalDay, LocalHour" row in the table brings the error down from 0.838 to 0.637 for SDCA and from 0.844 to 0.541 for FastTree. It is the biggest improvement in this series. FastTree pulling ahead makes sense too: trees learn the combined effect of two features, such as "Friday and after 3 pm", on their own through splits, while a linear model gives each feature a separate weight and cannot capture this interaction.

The third feature, HoursUntilProcessing, is a different kind of information. Imagine we asked the warehouse team and learned two rules: an order that arrives after 15:00 waits until the next morning, and the warehouse does not work at weekends. Using these two rules we calculate how many hours an order will wait before processing starts. Look at the effect: SDCA's error drops from 0.637 to 0.525 and the linear model catches up with the tree. For the tree the gain is small (0.531), because it had already learned this interaction from the day and the hour. This is feature engineering at its most valuable: turning a rule the business knows into a single number the model can use directly. This feature lifted a simple, fast model to the level of a complex one.

The campaign week (IsCampaign) is the same kind of feature; the marketing team knows the campaign calendar. Calendar information such as holidays, month end or payday is often more valuable than the date column itself.

Text: FeaturizeText and Turkish Text

The customer's delivery note is free text. FeaturizeText normalizes the text, splits it into words and character triples and opens a slot for each. Let's see which slots one note falls into:

// A few text features and their values for one note
var note = Note().Fit(data);
IDataView one = note.Transform(Ml.Data.LoadFromEnumerable([new DeliveryInput { CustomerNote = "office address - weekdays only" }]));
VBuffer<ReadOnlyMemory<char>> slots = default;
one.Schema["NoteFeatures"].GetSlotNames(ref slots);
float[] values = one.GetColumn<float[]>("NoteFeatures").First();
string[] names = [.. slots.DenseValues().Select(s => s.ToString())];
Console.WriteLine("non-zero text features for 'office address - weekdays only':");
foreach ((string n, float v) in names.Zip(values).Where(x => x.Second > 0 && x.First.StartsWith("Word.")))
{
    Console.WriteLine($"  {n,-22} {v:F3}");
}
Console.WriteLine("stop word languages: " +
    string.Join(", ", Enum.GetNames<StopWordsRemovingEstimator.Language>()));

dotnet run --project src/Bookstore.ML.Lab -- inspect

non-zero text features for 'office address - weekdays only':
  Word.-                 0.333
  Word.office            0.333
  Word.office|address    0.333
  Word.address           0.333
  Word.address|-         0.333
  Word.-|weekdays        0.333
  Word.weekdays          0.333
  Word.weekdays|only     0.333
  Word.only              0.333
stop word languages: English, French, German, Dutch, Danish, Swedish, Italian, Spanish, Portuguese, Portuguese_Brazilian, Norwegian_Bokmal, Russian, Polish, Czech, Arabic, Japanese

The note "office address - weekdays only" became nine word features: single words and pairs of adjacent words (n-grams), each with the same weight. There are character triples too; we filtered them out of the list. This note has a real effect on delivery time: an office address cannot receive deliveries at the weekend. In the table, the text feature takes SDCA from 0.525 to 0.519 and FastTree from 0.531 to 0.530. A small but consistent gain.

The last line of the output matters if you work with Turkish text: Turkish is not among the languages for which ML.NET has a built-in stop word list, the list of words such as "and" or "a" that carry no meaning. ML.NET has no stemming either, which would reduce a word to its root. In Turkish, "kargo", "kargom", "kargoya" and "kargodan" count as four different words, and with little data the model cannot connect them. Character triples ease this problem partly, because all four words share the pieces "kar", "arg" and "rgo". A stronger alternative is turning the text into a vector that carries its meaning with an embedding model and using that as a feature. We saw what an embedding is in the "Embeddings and Similarity for LLMs" post of the LLM Fundamentals series; here I only note the idea.

Missing Values

The weight is empty for about 3 percent of orders. From the previous post we know that with MissingRealsAsNaNs = true these values arrive as NaN. So what happens if we give a NaN feature to SDCA as it is?

// Missing weight without ReplaceMissingValues: what happens to the rows and to a prediction?
var raw = Ml.Transforms.Concatenate("Features", "DistanceKm", "ItemCount", "TotalWeightKg")
    .Append(Ml.Regression.Trainers.Sdca(new SdcaRegressionTrainer.Options
    {
        LabelColumnName = "DeliveryDays", NumberOfThreads = 1,
    }));
ITransformer rawModel = raw.Fit(data);
var rawEngine = Ml.Model.CreatePredictionEngine<DeliveryInput, DeliveryPrediction>(rawModel);
Console.WriteLine($"weight 1.2 kg -> {rawEngine.Predict(new DeliveryInput { DistanceKm = 450, ItemCount = 2, TotalWeightKg = 1.2f }).PredictedDays:F2} days, " +
                  $"weight missing -> {rawEngine.Predict(new DeliveryInput { DistanceKm = 450, ItemCount = 2, TotalWeightKg = float.NaN }).PredictedDays}");
float[] scores = rawModel.Transform(data).GetColumn<float>("Score").ToArray();
Console.WriteLine($"rows scored NaN in the training data: {scores.Count(float.IsNaN)} of {scores.Length}");

weight 1.2 kg -> 2.09 days, weight missing -> NaN
rows scored NaN in the training data: 720 of 23105

The model trained without complaint, but the prediction for an order with an unknown weight is NaN. The score of 720 rows in the training data is NaN too. In production, this model would show "NaN days" for every order with a missing weight. The fix is in the Weight() builder: ReplaceMissingValues fills the missing value with the mean learned from the training data, and IndicateMissingValues keeps the fact that "this value was actually missing" in a separate 0/1 column. Missingness itself can be a signal; for example, if the weight is always missing for one warehouse, the model learns that from this column.

The last row of the table gives an honest answer for this data: adding the weight gains nothing (SDCA 0.520, FastTree 0.535). The weight's effect on delivery time is very weak in this data. "We have it" is not enough reason to add a feature; the measurement has to support it.

Feature Count and Overfitting

Let's see how big the feature vector grows when we add all the groups:

// How wide is each feature group?
ITransformer full = Full(Sdca()).Fit(data);
IDataView t = full.Transform(data);
foreach (string c in new[] { "CityKey", "DistrictKey", "LocalDayKey", "NoteFeatures", "Features" })
{
    Console.WriteLine($"{c,-14} {t.Schema[c].Type}");
}

CityKey        Vector<Single, 16>
DistrictKey    Vector<Single, 312>
LocalDayKey    Vector<Single, 7>
NoteFeatures   Vector<Single, 173>
Features       Vector<Single, 524>

We started with 7 columns and now have 524 slots: 312 for districts and 173 for the delivery note. For 23 thousand rows this is not a problem yet, but as the ratio gets worse you slide to the right side of the overfitting curve from the third post. Every new feature group also gives the model a new chance to memorize: districts with few examples, rare words, a value specific to a single customer. When you add features, watch two things together: does the cross-validation error go down, and does the gap between folds grow?

Training-Serving Skew

We put all the pieces into a single pipeline. In the sixth post we will compare different trainers on this pipeline:

public static IEstimator<ITransformer> Full(IEstimator<ITransformer> trainer) =>
    Base()
        .Append(Ml.Transforms.Categorical.OneHotEncoding([new("CityKey", "City"), new("DistrictKey", "District")]))
        .Append(OrderTime())
        .Append(Note())
        .Append(Weight())
        .Append(Ml.Transforms.Concatenate("Features", [.. BaseColumns, "CityKey", "DistrictKey", "LocalDayKey", "LocalHour",
            "HoursUntilProcessing", "NoteFeatures", "Weight", "WeightMissing"]))
        .Append(Ml.Transforms.NormalizeMinMax("Features"))
        .Append(trainer);

The most important benefit of this structure is that features are computed by the same code in training and in prediction. The opposite is called training-serving skew: the model is trained on features computed one way and runs in production on features computed another way. Its most common form looks like this. A data scientist computes the features with a script during training, the developer who writes the API rewrites the same logic in their own code, and a small difference slips through. Let's see an example. We save the pipeline, load it with a new context and predict the same order twice; the second time, a bug somewhere else in the system sends the local time as if it were UTC:

// The zip stores the mapping's contract name and assembly name; that assembly must be deployable next to the model
MLContext fresh = new();
ITransformer loaded = fresh.Model.Load("models/delivery-features.zip", out _);
var engine = fresh.Model.CreatePredictionEngine<DeliveryInput, DeliveryPrediction>(loaded);
// Thursday 14:10 in Istanbul = 11:10 UTC. A bug elsewhere sends local time as if it were UTC.
DeliveryInput order = new()
{
    OrderedAtUtc = new DateTime(2026, 10, 15, 11, 10, 0, DateTimeKind.Utc),
    City = "Ankara", District = "ANK-07", Warehouse = "ANK-1", DistanceKm = 20, Carrier = "CarrierA",
    ShippingMethod = "Standard", ItemCount = 1, TotalWeightKg = 0.6f, CustomerNote = "",
};
float correct = engine.Predict(order).PredictedDays;
order.OrderedAtUtc = new DateTime(2026, 10, 15, 14, 10, 0, DateTimeKind.Utc);
float skewed = engine.Predict(order).PredictedDays;
Console.WriteLine($"ordered 14:10 local, sent as UTC 11:10 : {correct:F2} days");
Console.WriteLine($"same order, local time sent as UTC      : {skewed:F2} days");

dotnet run --project src/Bookstore.ML.Lab -- skew

ordered 14:10 local, sent as UTC 11:10 : 1.27 days
same order, local time sent as UTC      : 2.07 days

The correct prediction for an order placed in Ankara on Thursday at 14:10 is 1.27 days. When the same order arrives with its time shifted by three hours, it is 2.07 days, because the model now thinks the order was placed after 15:00. There is no exception; the prediction is a plausible number, and it is wrong. Putting the time zone conversion inside the pipeline prevents half of the errors in this class: the API does not have to compute any feature by hand. The other half, the raw data sent to the model being correct in the first place, is a matter of contracts and tests.

CustomMapping comes with a condition. To be able to save the model, you have to give the mapping a contract name ("OrderTime") and a factory class marked with [CustomMappingFactoryAttribute]. Looking at ML.NET's source code, I saw that the zip file stores this contract name and the name of the assembly that contains the factory, and that on load it loads and registers that assembly itself. That is why the model loaded in the new context without a separate RegisterAssembly call. But the application that uses the model must reference that assembly. So putting the factory class in a small library shared by the projects that train and serve the model is a good idea. Steps such as normalization and filling missing values are inside the pipeline for the same reason: the min, max and mean values are stored in the model's zip file and, as in the third post, learned from the training data only.

Common Pitfalls

The first pitfall is applying normalization to all the data before the split. If the min and max are also learned from the test data, a small leak appears; normalization that stays inside the pipeline prevents this automatically.

The second is ignoring differences in scale. In this post a single unnormalized column raised SDCA's error from 0.84 to 4.73. If a linear model's result makes no sense, look at the feature scales first.

The third is different preprocessing in training and prediction. Do not recompute features by hand in the API; put the transformations in the pipeline and save the model together with its pipeline.

The fourth is not thinking about missing values at all. A NaN feature silently turns the prediction into NaN, and the default loader turns a missing value into 0. Neither throws an exception.

The fifth is adding features without measuring. Weight and city gained nothing in this data. Every feature has a maintenance cost: a data source, a missing-value policy, monitoring.

A Small Experiment

★☆☆ Try hashing for the district with numberOfBits: 3 (8 slots) and numberOfBits: 10 (1,024 slots). How does the error change as collisions increase?

★★☆ Add a feature unrelated to campaigns to OrderTimeMappingFactory: the day of the month the order was placed. Does the measurement show any gain? If not, what does that tell you about the data?

★★★ Compute HoursUntilProcessing with a separate cut-off for Express orders: assume Express orders are processed the same day until 17:00. Does the error drop? If it does, have the business confirm the rule; if it does not, your assumption may be wrong.

In the next post we keep this pipeline fixed and change the trainers: we see how linear regression learns with a small gradient descent simulation, compare SDCA, FastTree, FastForest and LightGBM on the same data, and discuss what MAE, RMSE and R² each measure, residual analysis, the effect of outliers on RMSE and extrapolation.