Really Knowing JSON: RFC 8259, Number Precision, Duplicate Keys, JSONC, JSON5 and JSON Lines

Version note: The examples in this post were run with .NET 10 (SDK 10.0.404), Node.js 26.13, Python 3.14, jq 1.8.2 and json5 2.2.3. The JSON specification used is the current standard from December 2017.

Guess First

We will read one record from the ordering service of Bookstore, the online bookstore used as the shared example across this site's series, with four different tools. The file is short but carries two traps: an order number larger than 2^53 and a status key written twice.

{
  "orderId": 9007199254740993,
  "status": "Paid",
  "total": 24.90,
  "status": "Cancelled"
}

What do orderId, status and total come out as when you read this file with JSON.parse in Node.js, json.load in Python, jq on the command line and System.Text.Json in .NET? Do all four agree? And one more: what happens if you hand the same file to the .NET configuration provider, the way appsettings.json is read?

What RFC 8259 Says

Today's definition of JSON is RFC 8259. It was published in December 2017 and is an Internet Standard as STD 90. The specification is short: four primitive types (string, number, boolean, null) and two structured types (object, array). A JSON text can be any value; the old RFC 4627 rule that the outermost value must be an object or an array is gone, so a lone 42 or "Dune" is a valid JSON text.

The string rules are clear: the quotation mark, the backslash and the control characters U+0000–U+001F must be escaped, and any other character may optionally be written as \uXXXX. That "optionally" means different output from different serializers:

$ jq -cn --arg title "Çalıkuşu" --arg note $'The "Hobbit"\tdraft' '{title: $title, note: $note}'
{"title":"Çalıkuşu","note":"The \"Hobbit\"\tdraft"}
$ python -c 'import json; print(json.dumps({"title": "Çalıkuşu"}))'
{"title": "\u00c7al\u0131ku\u015fu"}
$ dotnet run escape.cs
{"Title":"\u00C7al\u0131ku\u015Fu","Note":"The \u0022Hobbit\u0022\tdraft"}
{"Title":"Çalıkuşu","Note":"The \u0022Hobbit\u0022\tdraft"}

All of these outputs are valid and carry the same value. jq writes Turkish letters as they are; Python's json.dumps escapes everything outside ASCII by default (turn it off with ensure_ascii=False); .NET with default settings escapes both the Turkish letters and the double quote as ". The reason is that .NET's default encoder is cautious so that JSON embedded in HTML stays safe. The encoder in the second line leaves letters alone but keeps escaping characters that are risky in HTML:

#:property PublishAot=false
using System.Text.Encodings.Web;
using System.Text.Json;

var book = new { Title = "Çalıkuşu", Note = "The \"Hobbit\"\tdraft" };

Console.WriteLine(JsonSerializer.Serialize(book));

// non-ASCII letters are written as-is; HTML-sensitive characters stay escaped
JsonSerializerOptions relaxed = new() { Encoder = JavaScriptEncoder.Create(System.Text.Unicode.UnicodeRanges.All) };
Console.WriteLine(JsonSerializer.Serialize(book, relaxed));

You notice this difference when you grep a log for a Turkish book title or compare the JSON two services produce byte by byte. Encoder settings are covered in detail in the System.Text.Json posts of the I/O in .NET series.

On encoding the RFC is strict: JSON text exchanged outside a closed ecosystem must be UTF-8, and the sender must not add a BOM. The reader may ignore a BOM; "may", not "must". We will see the consequence below.

What the Spec Leaves Open: Duplicate Keys and Number Precision

RFC 8259 asks for names within an object to be unique with a SHOULD, a recommendation rather than a requirement. It does not define what happens when a name appears twice; it only says the behavior is unpredictable, that some parsers take the last value and others report an error. Numbers are similar: the spec sets no range or precision limit, it only notes that software using IEEE 754 doubles is widespread and that integers between -(2^53)+1 and 2^53-1 will be read the same everywhere. The .NET program from the guess:

#:package Microsoft.Extensions.Configuration.Json@10.0.15
#:property PublishAot=false
using System.Text.Json;
using Microsoft.Extensions.Configuration;

string json = File.ReadAllText("order.json");

// 1) System.Text.Json with the web defaults
Order order = JsonSerializer.Deserialize<Order>(json, JsonSerializerOptions.Web)!;
Console.WriteLine($"Serializer: {order.OrderId} {order.Status} {order.Total}");

// 2) The same with duplicates rejected (.NET 10)
JsonSerializerOptions strict = new(JsonSerializerOptions.Web) { AllowDuplicateProperties = false };
try
{
    JsonSerializer.Deserialize<Order>(json, strict);
}
catch (JsonException ex)
{
    Console.WriteLine($"Strict: {ex.Message}");
}

// 3) Configuration JSON provider (appsettings.json is read this way)
try
{
    IConfiguration config = new ConfigurationBuilder()
        .SetBasePath(Directory.GetCurrentDirectory())
        .AddJsonFile("order.json")
        .Build();
    Console.WriteLine($"Configuration: {config["status"]}");
}
catch (InvalidDataException ex)
{
    Console.WriteLine($"Configuration: {ex.InnerException?.Message}");
}

record Order(long OrderId, string Status, decimal Total);

The results from the four tools:

$ node read-order.mjs
9007199254740992 Cancelled 24.9
{"orderId":9007199254740992,"status":"Cancelled","total":24.9}
$ python read_order.py
9007199254740993 Cancelled 24.9
{"orderId": 9007199254740993, "status": "Cancelled", "total": 24.9}
$ jq -c . order.json
{"orderId":9007199254740993,"status":"Cancelled","total":24.90}
$ dotnet run read-order.cs
Serializer: 9007199254740993 Cancelled 24.90
Strict: Duplicate property 'status' encountered during deserialization of type 'Order'.
Configuration: A duplicate key 'status' was found.

For the duplicate key, all four took the last value by default: the order shows as "Cancelled", not "Paid", and none of them warned. That is enough for two systems to read the same message differently. Differences like this between parsers are reported as security vulnerabilities from time to time, because the layer that validates can see one value while the layer that does the work sees another. In .NET 10, AllowDuplicateProperties = false puts the serializer into a strict mode. The configuration provider is strict already: write the same key twice in an appsettings.json and the application fails at startup with a FormatException.

On numbers Node.js stands apart: JavaScript's only number type is a double, so 9007199254740993 silently became 9007199254740992. The bit-level explanation is in the Integer Representation post of the Number Systems and Data Representation series; from the format's point of view two things matter here. First, total: Node and Python wrote 24.90 back as 24.9. The value is the same but the text changed, while .NET's decimal and jq kept the original text. Second, jq 1.8's behavior: it keeps the original text of a number, but converts it to a double as soon as you do arithmetic on it:

$ jq '.orderId, .orderId + 1' order.json
9007199254740993
9007199254740992

On the JavaScript side the fix is either to carry the ID as a string in the API or to use the reviver's third parameter, standardized in ES2026. It gives you the raw text of the value in the JSON:

import { readFileSync } from "node:fs";

const text = readFileSync("order.json", "utf8");

// the reviver's third argument carries the original source text (ES2026 JSON.parse source text access)
const order = JSON.parse(text, (key, value, context) =>
  key === "orderId" ? BigInt(context.source) : value);

console.log(order.orderId, typeof order.orderId);
console.log(JSON.stringify({ orderId: order.orderId.toString() }));

$ node big-id.mjs
9007199254740993n bigint
{"orderId":"9007199254740993"}

The lasting fix is in the contract: carry IDs that can exceed 2^53 (Snowflake-style IDs, bigint columns in some databases) as strings in the API. And don't process money as doubles; use decimal in .NET and parse_float=Decimal in Python.

Dates, NaN and "0012": Values JSON Does Not Have

JSON has no date type. The common agreement is to carry a date as an ISO 8601 string; .NET writes a UTC DateTime as "2026-11-14T09:30:00Z". It is only a convention: the receiver knows that the string is a date from a schema or a contract, not from JSON itself. Always include the time zone; a timestamp without Z may be interpreted in the reader's local time.

NaN and Infinity have no representation in JSON; the spec excludes them explicitly. Tools fill the gap differently:

$ python -c 'import json; print(json.dumps({"avgRating": float("nan")}))'
{"avgRating": NaN}
$ node -e 'console.log(JSON.stringify({ avgRating: NaN }))'
{"avgRating":null}
$ dotnet run nan.cs
.NET number values such as positive and negative infinity cannot be written as valid JSON. To make it work when using 'JsonSerializer', consider specifying 'JsonNumberHandling.AllowNamedFloatingPointLiterals' (see https://learn.microsoft.com/dotnet/api/system.text.json.serialization.jsonnumberhandling).
{"AvgRating":"NaN"}

Python produces invalid JSON (NaN as a bare word), JavaScript silently turns the value into null, and .NET throws an exception and, if you ask it to, writes the string "NaN". In Bookstore the average rating of a book that has no ratings yet could be NaN; the clearest contract is then to send the field as null or not to send it at all.

Leading zeros are forbidden too: 0012 is not a valid JSON number. The real problem is that such values should never be numbers. An order code, postal code, ISBN or phone number is a "string that looks like a number"; carried as a number it loses its leading zero:

$ echo '{"code": 0012}' | python -c 'import json, sys; json.load(sys.stdin)' 2>&1 | tail -1
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 11 (char 10)
$ echo '{"code": 0012}' | node -e 'JSON.parse(require("node:fs").readFileSync(0, "utf8"))' 2>&1 | grep SyntaxError
SyntaxError: Unexpected number in JSON at position 10 (line 1 column 11)
$ echo '{"code": 0012, "avgRating": NaN}' | jq -c .
{"code":12,"avgRating":null}

Python and Node reject it. jq 1.8.2, however, accepts both 0012 and NaN without an error and writes 12 and null. So jq being able to read a file does not prove that the file is valid JSON; for validation use a strict parser or JSON Schema (the fifth post of this series).

BOM: Three Invisible Bytes at the Start of the File

On Windows some editors, and the -Encoding UTF8 parameter of Windows PowerShell 5.1, add the bytes EF BB BF to the start of a UTF-8 file. You don't see them on screen, but the parser does:

$ printf '\xef\xbb\xbf{"isbn": "9780441013593", "stock": 12}\n' > stock-bom.json
$ xxd stock-bom.json | head -1
00000000: efbb bf7b 2269 7362 6e22 3a20 2239 3738  ...{"isbn": "978
$ node -e 'JSON.parse(require("node:fs").readFileSync("stock-bom.json", "utf8"))' 2>&1 | grep SyntaxError
SyntaxError: Unexpected token '', "{"isbn": "... is not valid JSON
$ python -c 'import json; json.load(open("stock-bom.json", encoding="utf-8"))' 2>&1 | tail -1
json.decoder.JSONDecodeError: Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)
$ jq -c . stock-bom.json
{"isbn":"9780441013593","stock":12}

Node.js and Python's json module rejected the file, jq accepted it. Python's error message even tells you the fix: the utf-8-sig encoding skips the BOM. In .NET the result depends on how you read the file:

#:property PublishAot=false
using System.Text.Json;

// File.ReadAllText detects and drops the BOM
using (JsonDocument doc = JsonDocument.Parse(File.ReadAllText("stock-bom.json")))
{
    Console.WriteLine($"ReadAllText: stock={doc.RootElement.GetProperty("stock")}");
}

// a stream: the serializer skips the BOM itself
await using (FileStream stream = File.OpenRead("stock-bom.json"))
{
    JsonElement root = await JsonSerializer.DeserializeAsync<JsonElement>(stream);
    Console.WriteLine($"Stream: stock={root.GetProperty("stock")}");
}

// raw bytes: the BOM reaches the parser
try
{
    using JsonDocument doc = JsonDocument.Parse(File.ReadAllBytes("stock-bom.json"));
}
catch (JsonException ex)
{
    Console.WriteLine($"ReadAllBytes: {ex.Message}");
}

$ dotnet run bom.cs
ReadAllText: stock=12
Stream: stock=12
ReadAllBytes: '0xEF' is an invalid start of a value. LineNumber: 0 | BytePositionInLine: 0.

File.ReadAllText recognizes and drops the BOM, and the serializer skips it when reading from a stream; but if you pass the byte array straight to JsonDocument.Parse, the BOM reaches the parser. In Bookstore this means a stock file that a supplier prepared on Windows is read fine by one service and fails in another. The rule is simple: write the JSON you produce without a BOM (PowerShell 7 already defaults to UTF-8 without a BOM), and be tolerant of a BOM when reading what comes from outside.

JSON with Comments: JSONC and appsettings.json

RFC 8259 has no comments. Yet we write commented "JSON" every day: tsconfig.json, Visual Studio Code's settings.json and appsettings.json. The format of these files is JSONC, JSON that allows // and /* */ comments. The editor opens them in the "JSON with Comments" mode; according to its documentation trailing commas are also accepted in this mode, with a warning. Bookstore.Api's appsettings.json has both:

{
  // Settings shared by every environment; secrets never go here
  "ConnectionStrings": {
    "BookstoreDb": ""
  },
  "Catalog": {
    "PageSize": 20,
    "MaxPageSize": 100,
  },
  "Ordering": {
    /* Institutional orders: at least 20 copies, 10% discount */
    "InstitutionalMinQuantity": 20,
    "InstitutionalDiscountRate": 0.10
  },
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "AllowedHosts": "*"
}

.NET's JSON configuration provider reads this file without a problem, because internally it parses with JsonCommentHandling.Skip and AllowTrailingCommas. Hand the same file to the serializer with default options, to jq or to another language's JSON parser, and things change:

#:package Microsoft.Extensions.Configuration.Json@10.0.15
#:package Microsoft.Extensions.Configuration.Binder@10.0.15
#:property PublishAot=false
using System.Text.Json;
using Microsoft.Extensions.Configuration;

string path = Path.GetFullPath("../../bookstore/src/Bookstore.Api/appsettings.json");

// the configuration provider skips comments and accepts the trailing comma
IConfiguration config = new ConfigurationBuilder().AddJsonFile(path).Build();
CatalogOptions catalog = config.GetSection("Catalog").Get<CatalogOptions>()!;
Console.WriteLine($"Configuration: PageSize={catalog.PageSize}, MaxPageSize={catalog.MaxPageSize}");

// the serializer with default options does not
try
{
    JsonSerializer.Deserialize<JsonElement>(File.ReadAllText(path));
}
catch (JsonException ex)
{
    Console.WriteLine($"Serializer: {ex.Message}");
}

// the same options the provider uses
JsonDocumentOptions jsonc = new() { CommentHandling = JsonCommentHandling.Skip, AllowTrailingCommas = true };
using JsonDocument doc = JsonDocument.Parse(File.ReadAllText(path), jsonc);
Console.WriteLine($"JsonDocument + options: {doc.RootElement.GetProperty("Ordering").GetProperty("InstitutionalDiscountRate")}");

record CatalogOptions(int PageSize, int MaxPageSize);

$ dotnet run jsonc.cs
Configuration: PageSize=20, MaxPageSize=100
Serializer: '/' is invalid after a value. Expected either ',', '}', or ']'. Path: $ | LineNumber: 1 | BytePositionInLine: 2.
JsonDocument + options: 0.10
$ jq . ../../bookstore/src/Bookstore.Api/appsettings.json
jq: parse error: Invalid numeric literal at line 2, column 5

So appsettings.json is really a JSONC file. A CI script that tries to read a value from it with jq stops at the first comment. If you write such a script, either don't use comments in the file or read it with a tool that understands JSONC. Comments have one more trap: a tool that reads the settings as JSON and writes them back (a deploy script, for example) drops the comments.

JSON5: Extended JSON for Hand-Written Config

JSON5 goes a step further and adds most of the ECMAScript 5.1 object literal syntax to JSON: unquoted keys, single-quoted strings, trailing commas, hexadecimal numbers, a leading plus sign, a trailing decimal point, comments and strings split with an escaped line break. According to the project site, projects such as Chromium, Next.js and Babel use it, and Apple platforms support it natively. A JSON5 file holding Bookstore's shipping rates (the script that reads it uses the json5 package; run npm install in the folder first):

// Shipping rates for the Bookstore checkout (JSON5)
{
  currency: 'TRY',
  freeShippingFrom: 500.,
  carrierCode: 0x1F,
  rates: [
    { region: 'domestic', price: +49.90 },
    { region: 'international', price: 349.90, },
  ],
  note: 'Prices include VAT; \
updated weekly',
}

$ node read-json5.mjs
{
  "currency": "TRY",
  "freeShippingFrom": 500,
  "carrierCode": 31,
  "rates": [
    {
      "region": "domestic",
      "price": 49.9
    },
    {
      "region": "international",
      "price": 349.9
    }
  ],
  "note": "Prices include VAT; updated weekly"
}

JSON5 is comfortable for config files that people write and edit often. It is not a good fit for moving data between systems: its parser is not in every language's standard library, and a JSON5 file handed to a plain JSON parser fails on the first line. Before picking JSON5 for config, compare it with YAML and TOML in the next posts of this series.

JSON Lines: Line-by-Line JSON for Exports and Logs

The rules of JSON Lines, also known as NDJSON, fit in three points: UTF-8, a valid JSON value on each line, and \n as the line separator. Let's produce the end-of-day order export for Bookstore's back office in both forms:

#:property PublishAot=false
using System.Text.Json;

Order[] orders =
[
    new("BS-1001", "Ada", new DateTime(2026, 11, 14, 9, 30, 0, DateTimeKind.Utc), 2, 49.80m, "Paid"),
    new("BS-1002", "Alan", new DateTime(2026, 11, 14, 11, 5, 0, DateTimeKind.Utc), 1, 24.90m, "Shipped"),
    new("BS-1003", "Grace", new DateTime(2026, 11, 15, 8, 0, 0, DateTimeKind.Utc), 25, 1120.50m, "AwaitingApproval"),
];

// 1) a single JSON array: the whole file is one value
JsonSerializerOptions indented = new(JsonSerializerOptions.Web) { WriteIndented = true };
File.WriteAllText("orders.json", JsonSerializer.Serialize(orders, indented));

// 2) JSON Lines: one compact value per line, each line ends with \n
await using (StreamWriter writer = new("orders.jsonl"))
{
    foreach (Order order in orders)
    {
        await writer.WriteAsync(JsonSerializer.Serialize(order, JsonSerializerOptions.Web));
        await writer.WriteAsync('\n');
    }
}

// read back line by line without loading the whole file
await using FileStream stream = File.OpenRead("orders.jsonl");
await foreach (Order? order in JsonSerializer.DeserializeAsyncEnumerable<Order>(
    stream, topLevelValues: true, JsonSerializerOptions.Web))
{
    Console.WriteLine($"{order!.OrderNumber} {order.Status} {order.Total}");
}

record Order(string OrderNumber, string CustomerName, DateTime CreatedAt, int Quantity, decimal Total, string Status);

$ dotnet run export-orders.cs
BS-1001 Paid 49.80
BS-1002 Shipped 24.90
BS-1003 AwaitingApproval 1120.50
$ cat orders.jsonl
{"orderNumber":"BS-1001","customerName":"Ada","createdAt":"2026-11-14T09:30:00Z","quantity":2,"total":49.80,"status":"Paid"}
{"orderNumber":"BS-1002","customerName":"Alan","createdAt":"2026-11-14T11:05:00Z","quantity":1,"total":24.90,"status":"Shipped"}
{"orderNumber":"BS-1003","customerName":"Grace","createdAt":"2026-11-15T08:00:00Z","quantity":25,"total":1120.50,"status":"AwaitingApproval"}

The topLevelValues: true parameter of DeserializeAsyncEnumerable has existed since .NET 9 and reads the file value by value; the writing side in .NET 11 is covered in the .NET 11 post of the I/O in .NET series. The easiest way to see the difference is to assume the process crashed while the export was being written. We cut both files at byte 330:

$ head -c 330 orders.json > broken.json
$ head -c 330 orders.jsonl > broken.jsonl
$ jq -c '.[] | .orderNumber' broken.json
jq: parse error: Unfinished string at EOF at line 16, column 19
$ jq -c '.orderNumber' broken.jsonl
jq: parse error: Unfinished string at EOF at line 3, column 76
"BS-1001"
"BS-1002"

Because a JSON array is a single value, not even one order came out of the half file. In JSON Lines only the last line is broken; jq printed the first two orders and reported the error (the stderr line appears before stdout because stdout is buffered when it writes to a pipe). A reader that processes line by line can skip the broken line and carry on:

import json
from decimal import Decimal

# each line is parsed on its own; a broken line does not stop the rest
with open("broken.jsonl", encoding="utf-8") as f:
    for number, line in enumerate(f, start=1):
        try:
            order = json.loads(line, parse_float=Decimal)
            print(number, order["orderNumber"], order["total"])
        except json.JSONDecodeError as e:
            print(number, "skipped:", e.msg)

$ python read_jsonl.py
1 BS-1001 49.80
2 BS-1002 24.90
3 skipped: Unterminated string starting at

For the same reason JSON Lines is the natural format for logs: appending a line does not require parsing the existing content, you can follow the file with tail -f, and split -l is enough to split a big file. Converting between the two formats is one line of jq:

$ jq -c '.[]' orders.json | diff - orders.jsonl && echo "same content"
same content
$ jq -s 'length' orders.jsonl
3
$ jq -s 'map(.total) | add' orders.jsonl
1195.2
$ echo '{"orderNumber":"BS-1004","customerName":"Linus","total":24.90}' >> orders.jsonl
$ wc -l orders.jsonl
       4 orders.jsonl

-s (slurp) collects all lines into a single array; that means loading the whole file into memory, so be careful with large exports. The total coming out as 1195.2 follows from the rule in the previous section: jq does arithmetic with doubles. For the .NET side of processing a big file line by line, see the streaming posts of the I/O in .NET series.

In Production

• Write the contract down: the date format (ISO 8601, UTC), money (string or decimal), large IDs (string), empty values (null or a missing field). JSON does not say any of this; your API documentation must.
• Treat duplicate keys in incoming JSON as a security matter. If a gateway or validation layer and the actual service use different parsers, they may see different values; in .NET 10, AllowDuplicateProperties = false closes that gap.
• Produce UTF-8 files without a BOM. Tolerate a BOM in files from outside, but don't produce one in your own pipeline.
• Use JSON Lines for long-running exports and logs: an interrupted write breaks one line, not the whole file.
• Not every file jq can read is valid JSON. If CI needs validation, use a strict parser or a schema.

Common Pitfalls

The first is carrying an ID as a number and not noticing that it gets rounded in a JavaScript client. The bug only shows up with the first ID above 2^53 and usually never appears in test data. The second is reading a money amount into a double and writing it back: the text 24.90 becomes 24.9. Even that much causes trouble in a payload compared against a signature or a hash.

The third is dismissing duplicate keys as something that never happens. In a large, hand-edited appsettings.json it is easy to write the same section twice; .NET catches it at startup, but a Node.js-based tool silently takes the last one. The fourth is turning values such as "0012" into numbers, which happens a lot when moving data from Excel into JSON. The fifth is Python's json.dumps producing invalid JSON for NaN; pass allow_nan=False and it raises an error instead.

Finally, mistaking a JSONC file for JSON. tsconfig.json and appsettings.json fail at the first comment when you try to read them with general-purpose JSON tools. Knowing which files are really JSON saves time when you write scripts.

A Small Experiment

★☆☆ Delete the second status line in order.json and turn orderId into a string. Run the four tools again; which of them now give the same result?

★★☆ Add a step to export-orders.cs that reads orders.jsonl with AllowDuplicateProperties = false. Write the same key twice by hand on one line and print which line failed. You will see that in JSON Lines you have to keep track of the line number yourself.

★★★ Produce an export of 100,000 orders as a JSON array and as JSON Lines. Compute the total once with jq -s and once line by line with jq -c and awk; compare the memory use of the two approaches with /usr/bin/time -v on Linux or /usr/bin/time -l on macOS.

Because JSON's specification is short, its surprises come from what the spec leaves open. YAML is the opposite: the spec is long and the surprises come from the rules themselves. In the next post we look at YAML's syntax, implicit typing and why Norway's country code turns into false, with real parsers.