Handling Text Correctly: Normalization, Diagnosing Mojibake and Base64

Version note: The code in this post was run with .NET 10 (C# 14, ASP.NET Core 10), TypeScript 7.0, Node.js 24 and SQL Server 2025; the C# examples are tested with xUnit v3 4.0.3. Base64 follows RFC 4648.

Guess First

Bookstore's authors table has a unique index on the author name. One editor adds an author called "Ayşe" from the admin panel on Windows; another adds her from the name of a file uploaded from a Mac. Both records look exactly the same on screen, and the unique index accepts both. How?

Second question: a CSV file downloaded from Bookstore's reporting screen shows "Kürk Mantolu Madonna" instead of "Kürk Mantolu Madonna" when double-clicked in Excel. Is the file broken, or is Excel reading it wrong? What does "ü" tell you about which letter was read with which encoding?

Third question: a developer says "we encrypted the token with Base64, it's secure". Is that right?

In the previous post of the series we saw how a character is turned into bytes with the UTF encodings. In this last post the work goes the other way: getting the right text back from bytes, comparing the same text the same way, and carrying binary data as text.

Normalization: Two Strings That Look the Same

In the previous post we saw that "ş" can be written two ways: as the single code point U+015F, or as "s" followed by the combining cedilla U+0327. Unicode treats these two forms as "canonically equivalent", the same in meaning and appearance. But string comparison looks at code units:

static string Hex(string s) => string.Join(" ", s.Select(c => ((int)c).ToString("X4")));

string nfc = "Ay\u015Fe";
string nfd = "Ays\u0327e";
Console.WriteLine($"{nfc} {nfd} {nfc == nfd} {nfc.Length} {nfd.Length}");   // Ayşe Ayşe False 4 5
Console.WriteLine(Hex(nfc));                                                  // 0041 0079 015F 0065
Console.WriteLine(Hex(nfd));                                                  // 0041 0079 0073 0327 0065
Console.WriteLine(nfd.Normalize(NormalizationForm.FormC) == nfc);             // True
Console.WriteLine($"{nfc.IsNormalized(NormalizationForm.FormC)} {nfd.IsNormalized(NormalizationForm.FormC)}");  // True False
Console.WriteLine(Hex("\u0130".Normalize(NormalizationForm.FormD)));          // 0049 0307

Unicode Standard Annex #15 defines four normalization forms. NFC combines characters that can be composed into a single code point, so "ş" becomes U+015F. NFD splits characters into a base letter and combining marks, so it becomes "s" + U+0327. Even "İ" splits into "I" and the dot above it (U+0307) in NFD. The common form for text from the web, from Windows and from most keyboards is NFC.

Where does NFD come from? macOS's old file system, HFS+, stored file names as NFD. Today's APFS stores a name as it was written, but files migrated from HFS+ and some applications still produce NFD names. The result: the name of an "Ayşe.docx" file uploaded by a Mac user can reach the server as NFD. That's the answer to the Guess First question: the two records look the same on screen but their bytes differ, and the unique index looks at bytes.

NFKC and NFKD: Compatibility Forms

The other two forms, NFKC and NFKD, go one step further and also merge characters that "mean the same but look different". These are called compatibility forms:

foreach (string s in new[] { "\uFB01ction", "\u2460", "\uFF21\uFF22\uFF23", "x\u00B2", "\u00BD" })
{
    Console.WriteLine($"{s,-8} NFC={s.Normalize(NormalizationForm.FormC)}  NFKC={s.Normalize(NormalizationForm.FormKC)}");
}

fiction   NFC=fiction  NFKC=fiction
①        NFC=①  NFKC=1
ＡＢＣ      NFC=ＡＢＣ  NFKC=ABC
x²       NFC=x²  NFKC=x2
½        NFC=½  NFKC=1⁄2

The "fi" ligature, a circled 1, full-width letters from Asian keyboards and superscripts turn into their plain counterparts in NFKC. That's very useful for search: a search for "fiction" copied from a PDF finds "fiction". But it loses information: "x²" and "x2" aren't the same thing. So use NFKC for search and matching keys, not for text you display.

public static class TextKeys
{
    // One canonical form before comparing or storing a unique key
    public static string Canonical(string value) =>
        value.Normalize(NormalizationForm.FormC).Trim();

    // Search key: compatibility forms folded, case folded
    public static string SearchKey(string value) =>
        value.Normalize(NormalizationForm.FormKC).ToLowerInvariant();
}

Comparison: Ordinal, Culture and Search

StringComparison.Ordinal compares code units and treats NFC and NFD as different. Culture-sensitive comparison knows about canonical equivalence:

Console.WriteLine(string.Equals(nfc, nfd, StringComparison.Ordinal));          // False
Console.WriteLine(string.Equals(nfc, nfd, StringComparison.InvariantCulture)); // True

var authors = new HashSet<string>(StringComparer.Ordinal) { nfc, nfd };
Console.WriteLine(authors.Count);    // 2
var canonical = new HashSet<string>(StringComparer.Ordinal) { TextKeys.Canonical(nfc), TextKeys.Canonical(nfd) };
Console.WriteLine(canonical.Count);  // 1

Because ordinal comparison is fast and predictable, it's the right choice for unique keys, dictionary keys and database indexes, as long as the data was brought into a single form on the way in. The fix is simple: normalize every piece of text from the outside world (form fields, file names, API requests) at the edge of the system with Normalize(NormalizationForm.FormC); after that, ordinal comparison works safely.

Search is a different need. You want a user who types "aysegul" to find "Ayşegül". CompareOptions.IgnoreNonSpace ignores combining marks, but the result depends on the culture:

CompareInfo tr = CultureInfo.GetCultureInfo("tr-TR").CompareInfo;
CompareInfo inv = CultureInfo.InvariantCulture.CompareInfo;
string shelf = "Ay\u015Feg\u00FCl Kitap\u00E7\u0131";
CompareOptions loose = CompareOptions.IgnoreCase | CompareOptions.IgnoreNonSpace;
Console.WriteLine($"{inv.IndexOf(shelf, "aysegul", loose)} {inv.IndexOf(shelf, "kitapci", loose)} {inv.IndexOf(shelf, "kitapc\u0131", loose)}");
// 0 -1 8
Console.WriteLine($"{tr.IndexOf(shelf, "aysegul", loose)} {tr.IndexOf(shelf, "kitapci", loose)} {tr.IndexOf(shelf, "kitapc\u0131", loose)}");
// -1 -1 -1

In the invariant culture, "ş" consists of an "s" and a cedilla and "ü" of a "u" and a diaeresis, so with the marks ignored "aysegul" matches. "ı", however, doesn't decompose; it's a letter on its own, which is why "kitapci" isn't found. In the Turkish culture, ş, ç, ü and ğ are separate letters of the alphabet, so none of them is treated as a mark and all three searches fail. If you want users to search without typing Turkish characters, build the search key yourself, with an explicit mapping table (ş → s, ı → i and so on), and run both the content and the query through the same function. How the Turkish I behaves under cultures was covered in the C# Fundamentals series post "Strings in C#: Interpolation, StringBuilder and the Turkish I Trap".

Diagnosing Mojibake: What "ÅŸ" Tells You

Mojibake is the broken characters you get when text is read with a different encoding than the one it was written in. It looks random, but it isn't; each broken character tells you which mistake was made. To use the old code pages in .NET, you first have to register the provider:

public static class Mojibake
{
    static Mojibake() => Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);

    public static Encoding Windows1252 => Encoding.GetEncoding(1252);

    public static Encoding Windows1254 => Encoding.GetEncoding(1254);
}

string original = "Kitap\u00E7\u0131 \u015Fiir \u011F\u00FC\u00F6 \u0130 \u00C7";
byte[] utf8 = Encoding.UTF8.GetBytes(original);
Console.WriteLine(original);                                                          // Kitapçı şiir ğüö İ Ç
Console.WriteLine(Mojibake.Windows1252.GetString(utf8));                              // Kitapçı şiir ğüö İ Ç
Console.WriteLine(Encoding.UTF8.GetString(Mojibake.Windows1254.GetBytes(original)));  // Kitap�� �iir ��� � �

Two different mistakes, two different pictures. In the first, UTF-8 bytes were read with a single-byte code page: every Turkish character is 2 bytes in UTF-8, so it turns into 2 characters on screen. The UTF-8 bytes of "ş" are C5 9F; in Windows-1252, C5 is "Å" and 9F is "Ÿ". So the moment you see "ÅŸ", the diagnosis is ready: "written as UTF-8, read with a Latin code page". Common pairs:

ç=C3A7=ç  ğ=C49F=ÄŸ  ı=C4B1=ı  ö=C3B6=ö  ş=C59F=ÅŸ  ü=C3BC=ü
Ç=C387=Ç  Ğ=C49E=Äž  İ=C4B0=Ä°  Ö=C396=Ö  Ş=C59E=Åž  Ü=C39C=Ãœ

In the second mistake the reverse happened: bytes written with Windows-1254 were read as UTF-8. In Windows-1254 "ş" is a single byte, FE; that byte can't be part of any valid UTF-8 sequence and is replaced with U+FFFD, that is, �. If you see "�", the data isn't UTF-8, and the original bytes may have been lost at that point.

Diagnosing by looking at the bytes rather than the characters on screen is more reliable. If the broken text is saved once more as UTF-8, you get "double-encoded" data, which shows up clearly in a hex dump:

Console.WriteLine(Convert.ToHexString(Mojibake.Windows1254.GetBytes(original)));
// 4B69746170E7FD20FE69697220F0FCF620DD20C7
Console.WriteLine(Convert.ToHexString(Encoding.UTF8.GetBytes(Mojibake.Windows1252.GetString(utf8))));
// 4B69746170C383C2A7C384C2B120C385C5B869697220C384C5B8C383C2BCC383C2B620C384C2B020C383E280A1

In the first line every Turkish character is a single byte (E7, FD, FE): this is a code page file. In the second, "ç" is C3 83 C2 A7: the two bytes of C3 A7 were each converted to UTF-8 separately. If a file is full of pairs starting with C3 83 and C2, you're looking at double-encoded UTF-8.

Data like this can be recovered: turn the broken text back into the code page it was wrongly read with, and read it correctly as UTF-8. But only do it when you're sure the conversion is lossless:

// UTF-8 bytes that were decoded as Windows-1252: re-encode, decode correctly
public static bool TryRepair(string garbled, out string repaired)
{
    Encoding strict1252 = Encoding.GetEncoding(1252, EncoderFallback.ExceptionFallback, DecoderFallback.ExceptionFallback);
    Encoding strictUtf8 = new UTF8Encoding(false, throwOnInvalidBytes: true);
    try
    {
        repaired = strictUtf8.GetString(strict1252.GetBytes(garbled));
        return true;
    }
    catch (Exception ex) when (ex is EncoderFallbackException or DecoderFallbackException)
    {
        repaired = garbled;
        return false;
    }
}

True Kitapçı şiir ğüö İ Ç
False Kitapçı şiir ğüö İ Ç

The first call repairs the broken text; the second leaves the already correct text alone, because "ı" doesn't exist in Windows-1252 and the strict encoder throws. This line shows why we chose the exception fallback explicitly: the default Windows-1252 encoder turns "ş" into "s" without any error (best-fit mapping):

Console.WriteLine(Convert.ToHexString(Mojibake.Windows1252.GetBytes("\u015F")));   // 73

The Journey of an Encoding: CSV, HTTP and SQL Server

Every time text passes from one system to another, the encoding information either travels with the bytes or gets lost. Three common stops:

CSV files have no metadata field that states the encoding. On Windows, Excel opens a double-clicked CSV without a leading UTF-8 BOM not as UTF-8 but with the system's default code page; on Turkish Windows that's Windows-1254. That's the answer to the second question: the file is fine, Excel reads it with the wrong encoding, and the two bytes of "ü" show up as "ü". Excel's own "CSV UTF-8" save option writes a BOM for exactly this reason. The fix is to write the export with a BOM:

string csv = "Id,Title,Author\n1,K\u00FCrk Mantolu Madonna,Sabahattin Ali\n";
File.WriteAllText("books-nobom.csv", csv);
File.WriteAllText("books-bom.csv", csv, new UTF8Encoding(encoderShouldEmitUTF8Identifier: true));
Console.WriteLine(Mojibake.Windows1254.GetString(File.ReadAllBytes("books-nobom.csv")).Split('\n')[1]);
// 1,Kürk Mantolu Madonna,Sabahattin Ali

$ xxd -l 16 books-bom.csv
00000000: efbb bf49 642c 5469 746c 652c 4175 7468  ...Id,Title,Auth
$ xxd -l 16 books-nobom.csv
00000000: 4964 2c54 6974 6c65 2c41 7574 686f 720a  Id,Title,Author.

The BOM has a cost too: tools that don't expect it treat it as part of the data. When Python's csv module opens the BOM file with utf-8, the first column name becomes '\ufeffId'; with utf-8-sig it's 'Id'. Decide based on who will read the file: with a BOM for exports meant for Excel users, without one for data exchanged between machines.

In HTTP, the encoding travels in the charset parameter of the Content-Type header. ASP.NET Core adds it itself with Results.Text:

app.MapGet("/export/books.csv", () => Results.Text(csv, "text/csv", Encoding.UTF8));
app.MapGet("/legacy/books.csv", () => Results.Bytes(Encoding.GetEncoding(1254).GetBytes(csv), "text/csv; charset=windows-1254"));
app.MapGet("/broken/books.csv", () => Results.Bytes(Encoding.UTF8.GetBytes(csv), "text/csv; charset=iso-8859-1"));

$ curl -i http://localhost:5732/export/books.csv
HTTP/1.1 200 OK
Content-Length: 33
Content-Type: text/csv; charset=utf-8

The client decodes according to the charset in the header. Reading the three endpoints with .NET's HttpClient:

/export/books.csv    1,Kürk Mantolu Madonna
/legacy/books.csv    1,Kürk Mantolu Madonna
/broken/books.csv    1,Kürk Mantolu Madonna

Windows-1254 bytes are read without trouble because they're labelled correctly; UTF-8 bytes with the wrong label produce mojibake. The broken characters come from the wrong label, not from the bytes.

In SQL Server, the encoding depends on the column type and the collation. A varchar column uses the collation's code page. The example database's collation is SQL_Latin1_General_CP1_CI_AS, that is, Windows-1252. On Windows, the default collation at install time is chosen by the system language; on a Turkish system it's Turkish_CI_AS (Windows-1254) and Turkish letters fit in varchar, but the same problem remains for every character that isn't in Windows-1254. nvarchar holds UTF-16. Since SQL Server 2019, collations ending in _UTF8 let varchar hold UTF-8 as well. I wrote the same name into three columns, once with the N prefix and once without:

CREATE TABLE dbo.Authors
(
    Id int IDENTITY PRIMARY KEY,
    NameVarchar varchar(20) NOT NULL,
    NameNvarchar nvarchar(20) NOT NULL,
    NameUtf8 varchar(20) COLLATE Latin1_General_100_CI_AS_SC_UTF8 NOT NULL
);
INSERT INTO dbo.Authors (NameVarchar, NameNvarchar, NameUtf8)
VALUES (N'Ayşe Kılıç', N'Ayşe Kılıç', N'Ayşe Kılıç'),
       ('Ayşe Kılıç', 'Ayşe Kılıç', 'Ayşe Kılıç');

Id|NameVarchar|NameNvarchar|NameUtf8|VarcharBytes|NvarcharBytes|Utf8Bytes
--|-----------|------------|--------|------------|-------------|---------
1|Ayse Kiliç|Ayşe Kılıç|Ayşe Kılıç|10|20|14
2|Ayse Kiliç|Ayse Kiliç|Ayse Kiliç|10|20|11

In the first row, the Windows-1252 varchar column couldn't hold "ş" and "ı", so best-fit wrote "s" and "i"; "ç" survived because it exists in Windows-1252. The nvarchar and UTF-8 columns are correct. The second row is sneakier: a string literal without the N prefix is converted to the database's code page before it reaches the column, so the nvarchar and UTF-8 columns got the broken value too. Even with the right column type, data is lost if the literal is wrong. Strings sent from application code as parameters (ADO.NET, Entity Framework Core, Dapper) go as nvarchar parameters, so this problem usually shows up in hand-written scripts and migrations.

Base64: Carrying Binary Data as Text

Text channels (JSON, URLs, email, HTTP headers) can't carry every byte value. Base64 splits every 3 bytes (24 bits) into four 6-bit groups and turns each group into a character from a 64-character alphabet (A–Z, a–z, 0–9, +, /). Since 3 bytes become 4 characters, the size grows by 33 percent; if the last group is short, it's padded with =:

Console.WriteLine(Convert.ToBase64String("Bookstore"u8));   // Qm9va3N0b3Jl
foreach (string s in new[] { "a", "ab", "abc", "abcd" })
{
    Console.WriteLine($"{s,-4} {Convert.ToBase64String(Encoding.ASCII.GetBytes(s))}");
}
// a    YQ==
// ab   YWI=
// abc  YWJj
// abcd YWJjZA==

Because + and / have special meaning in URLs and file names, RFC 4648 also defines a URL-safe alphabet: - instead of +, _ instead of /, and usually no padding. System.Buffers.Text.Base64Url, which came with .NET 9, does this directly:

byte[] tricky = [0xFB, 0xFF, 0xBF, 0x3E];
Console.WriteLine(Convert.ToBase64String(tricky));                         // +/+/Pg==
Console.WriteLine(Base64Url.EncodeToString(tricky));                       // -_-_Pg
Console.WriteLine(Convert.ToHexString(Base64Url.DecodeFromChars("-_-_Pg"))); // FBFFBF3E

byte[] cover = File.ReadAllBytes("cover.png");
string dataUri = $"data:image/png;base64,{Convert.ToBase64String(cover)}";
Console.WriteLine($"{cover.Length} {dataUri.Length - "data:image/png;base64,".Length} {(double)(dataUri.Length - 22) / cover.Length:F3}");
// 58708 78280 1.333

A data URI is the way to embed a small image inside HTML or CSS. But the 58 KB PNG turns into 78 KB of text, and the browser can't cache it as a separate file; it only makes sense for very small icons.

On the TypeScript side, btoa and atob only accept characters in the Latin-1 range; a character like "ı" makes them fail. You have to turn the text into UTF-8 bytes first. In Node.js, Buffer supports that and Base64Url:

console.log(btoa("Bookstore"), atob("Qm9va3N0b3Jl"));     // Qm9va3N0b3Jl Bookstore
try {
  btoa("Kitap\u00E7\u0131");
} catch (e) {
  console.log((e as Error).name);                         // InvalidCharacterError
}
console.log(Buffer.from("Kitap\u00E7\u0131").toString("base64"));            // S2l0YXDDp8Sx
console.log(Buffer.from([0xfb, 0xff, 0xbf, 0x3e]).toString("base64url"));   // -_-_Pg

Base64 Is Not Encryption: A JWT Example

Base64 is an encoding; it has no key, and anyone can reverse it. That's the answer to the third question: a token "encrypted" with Base64 is plain text to anyone who reads it. The most common example is the JWT, a token format that joins a header, a payload and a signature with Base64Url:

string header = Base64Url.EncodeToString("{\"alg\":\"HS256\",\"typ\":\"JWT\"}"u8);
string payload = Base64Url.EncodeToString("{\"sub\":\"42\",\"name\":\"Ada\",\"role\":\"customer\"}"u8);
byte[] key = RandomNumberGenerator.GetBytes(32);
string signature = Base64Url.EncodeToString(HMACSHA256.HashData(key, Encoding.ASCII.GetBytes($"{header}.{payload}")));
string token = $"{header}.{payload}.{signature}";

string[] parts = token.Split('.');
Console.WriteLine(Encoding.UTF8.GetString(Base64Url.DecodeFromChars(parts[1])));
// {"sub":"42","name":"Ada","role":"customer"}

string forged = Base64Url.EncodeToString("{\"sub\":\"42\",\"name\":\"Ada\",\"role\":\"admin\"}"u8);
byte[] expected = HMACSHA256.HashData(key, Encoding.ASCII.GetBytes($"{parts[0]}.{forged}"));
Console.WriteLine(CryptographicOperations.FixedTimeEquals(expected, Base64Url.DecodeFromChars(parts[2])));
// False

Reading the payload needed no key. A JWT's security comes from the signature, not from secrecy: the forged payload that sets the role to "admin" can't match the HMAC signature. Decoding a token and saying "I cracked it" is different from verifying its signature; in production, always process tokens with a library that verifies them. The difference between encoding, hashing and encryption was covered in the Web Security (OWASP Top 10) series post "OWASP A04: Cryptographic Failures (1) — Encoding, Hashing, Encryption and HTTPS"; the details of HMAC and signatures are in the Cryptography for Developers series.

In Production

Normalize text once at the edge of the system (NFC) and work with that form everywhere inside. Unique keys, search keys and file name comparisons all rest on this single rule.

Carry the encoding with metadata: charset in HTTP, a BOM or a documented contract in files, nvarchar or a UTF-8 collation in the database. If you exchange files with an external system, write the encoding into the integration document.

Put the N prefix on Unicode literals in hand-written SQL. Check migration and seed scripts for this in code review; corrupted data can't be brought back later.

Use a strict decoder when importing data from an old system. With DecoderFallback.ExceptionFallback you catch the wrong encoding on the first line; the default behavior carries on with � and pushes the problem all the way to the reports.

Common Pitfalls

Guessing the encoding. In Windows-1252 every byte value maps to a character, so every byte sequence is "valid" Windows-1252 text. A file that is valid UTF-8 is also a valid Windows-1252 file; the bytes can't decide which it is. Encoding detection libraries guess statistically and are often wrong on short texts. Learn the encoding from the source; don't guess.

Base64-decoding a JWT and trusting what's inside. Using the role or sub value from the payload without verifying the signature means anyone who sends a request can give themselves any role.

Using standard Base64 in a URL. A + in a query string is decoded as a space under form encoding rules:

var query = HttpUtility.ParseQueryString("token=" + Convert.ToBase64String(tricky));
Console.WriteLine(query["token"]);   //  / /Pg==

+/+/Pg== reached the server as / /Pg== and can't be decoded. Use Base64Url in URLs, cookies and file names.

Silent data loss when converting to a code page. Best-fit conversions, like the Windows-1252 encoder turning "ş" into "s", don't throw. If you send data to an old system, use an encoder with an exception fallback and report characters that can't be converted explicitly.

A Small Experiment

★☆☆ Create a file whose name is in NFD ("Ays\u0327e.txt"), then read it back with Directory.GetFiles and compare it with the NFC form using ==. Repeat the comparison with Normalize.

★★☆ Extend Mojibake.TryRepair to also repair text that was broken twice (read wrongly twice): repeat as long as the repair succeeds and the result changes. Write tests for the inputs "ÅŸ" and "ÅŸ".

★★★ Add a ?target=excel parameter to Bookstore's CSV export endpoint: return UTF-8 with a BOM for Excel and UTF-8 without a BOM otherwise, with the right charset in Content-Type in both cases. Write a script that checks the first bytes of both responses with curl and xxd.

With this post the Number Systems and Data Representation series comes to an end. Starting from binary and hex, through bitwise operations, integer and floating point representation, to Unicode and text handling, we saw how a computer holds data at the bit level. If you want to know how these bits are processed in the CPU, the next stop is the How Computers Work series: registers, the pipeline and the cache show the bytes we've seen here on real hardware.