Sensitive Data in Logs: Redaction, Data Classification and Privacy-Compliant Logging

Version note: The code in this post was built and run with .NET 10, Microsoft.Extensions.Telemetry 10.11, Microsoft.Extensions.Compliance.Redaction 10.11, Microsoft.Extensions.Diagnostics.Testing 10.11, Serilog 4.4 and xUnit 2.9.

Guess First

You marked a log parameter as [PersonalData], but forgot to turn on redaction on the logging side; there's no EnableRedaction() call. Does the customer's email address land in the log as plain text, come out empty, or does the app throw? And one more: with redaction on, the same email address is logged in two different events. Are the two masked values identical?

What Must Never Enter the Log

Some data must never enter the log under any circumstances: passwords, access and refresh tokens, API (Application Programming Interface) keys, card numbers and CVV (Card Verification Value), national ID numbers such as the Turkish TC (Türkiye Cumhuriyeti) identity number. If they show up in the log, everyone with access to the log system, every system that backs up the logs and the SaaS (Software as a Service) tool you pay for can reach that data. Logs are usually open to a much wider audience than the database and much less protected.

The second group is personal data: full name, email, phone, address, IP (Internet Protocol) address. Sometimes you really need these; a support team told to "look into this customer's complaint" must be able to find the customer. But the moment you write personal data into a log, that log becomes a personal data processing activity too. KVKK (Kişisel Verilerin Korunması Kanunu, the Turkish personal data protection law) and its European counterpart GDPR (General Data Protection Regulation) broadly require personal data to be limited to its purpose, proportionate and kept only as long as necessary. The retention period of logs, who can access them and what happens on a deletion request are part of that framework. This isn't a legal assessment; talk to your legal team about your own situation. On the technical side, the best you can do is to keep personal data out of the log entirely, and mask it when you must log it.

What should be logged from a security perspective was covered in the Web Security series, built on the OWASP (Open Worldwide Application Security Project) Top 10, in the post "OWASP A09: Security Logging and Alerting Failures — What to Log, What Never To". This post focuses on the mechanism: classifying data with Microsoft.Extensions.Compliance.Redaction and masking it automatically in the logging pipeline.

Data Classification: Taxonomy and Attributes

The basis of redaction is classifying data. First you define your own taxonomy, then an attribute for each class:

using Microsoft.Extensions.Compliance.Classification;

namespace Bookstore.Customers;

public static class BookstoreTaxonomy
{
    public static string Name => "Bookstore";

    public static DataClassification PersonalData => new(Name, nameof(PersonalData));
    public static DataClassification SensitiveData => new(Name, nameof(SensitiveData));
}

public sealed class PersonalDataAttribute() : DataClassificationAttribute(BookstoreTaxonomy.PersonalData);

public sealed class SensitiveDataAttribute() : DataClassificationAttribute(BookstoreTaxonomy.SensitiveData);

The redaction sample on Microsoft Learn marks the parameter directly as [MyTaxonomyClassifications.Private]. That doesn't compile: in C# only a class deriving from Attribute can be used as an attribute, not a static property. Small classes deriving from DataClassificationAttribute fill that gap; with a primary constructor each one is a single line.

I found two classes enough. PersonalData identifies a person but is sometimes needed for correlation (email, full name); SensitiveData must never be visible (national ID). Keep the number of classes small; every class is a rule everyone on the team has to remember.

Classified Parameters with LoggerMessage and [LogProperties]

We use the classification in two places: on a log method's parameter and on the properties of an object expanded into the log:

namespace Bookstore.Customers;

public sealed class CustomerSnapshot
{
    public int CustomerId { get; init; }

    [PersonalData]
    public string Email { get; init; } = "";

    [PersonalData]
    public string FullName { get; init; } = "";

    [SensitiveData]
    public string NationalId { get; init; } = "";

    public string City { get; init; } = "";
}

using Microsoft.Extensions.Logging;

namespace Bookstore.Customers;

public static partial class Log
{
    [LoggerMessage(2001, LogLevel.Information, "Customer {CustomerId} registered with email {Email}")]
    public static partial void CustomerRegistered(ILogger logger, int customerId, [PersonalData] string email);

    [LoggerMessage(2002, LogLevel.Information, "Customer signed in")]
    public static partial void CustomerSignedIn(ILogger logger, [LogProperties] CustomerSnapshot customer);

    [LoggerMessage(2003, LogLevel.Error, "Registration failed for customer {CustomerId}")]
    public static partial void RegistrationFailed(ILogger logger, Exception exception, int customerId);
}

[LogProperties] expands the object's public properties into separate log properties such as customer.Email and customer.City. We saw it in this series' "High-Performance Logging" post; the "watch out for sensitive fields" note we left there is exactly this topic. [LogProperties] and classified parameters are features not of the standard source generator but of the extended generator in the Microsoft.Extensions.Telemetry.Abstractions package; once the package is in the project, that generator takes over on its own.

Now the answer to the first question. These are the first two lines when I run without EnableRedaction(), with only AddJsonConsole():

{"EventId":2001,"LogLevel":"Information","Category":"Bookstore.Customers","Message":"Customer 42 registered with email (null)","State":{"Message":"CustomerId=42,{OriginalFormat}=Customer {CustomerId} registered with email {Email},Email=\u003Comitted\u003E ([Bookstore:PersonalData])","CustomerId":42,"{OriginalFormat}":"Customer {CustomerId} registered with email {Email}"}}
{"EventId":2002,"LogLevel":"Information","Category":"Bookstore.Customers","Message":"Customer signed in","State":{"Message":"customer.City=Izmir,customer.CustomerId=42,{OriginalFormat}=Customer signed in,customer.NationalId=\u003Comitted\u003E ([Bookstore:SensitiveData]),customer.FullName=\u003Comitted\u003E ([Bookstore:PersonalData]),customer.Email=\u003Comitted\u003E ([Bookstore:PersonalData])","customer.City":"Izmir","customer.CustomerId":42,"{OriginalFormat}":"Customer signed in"}}

The email didn't leak. The classified values weren't emitted as properties at all; the message has (null), and the text form has <omitted> ([Bookstore:PersonalData]). The behavior stays on the safe side: if you forget to turn redaction on, the data is lost but not exposed. The unclassified City and CustomerId come through normally.

EnableRedaction and Redactors

Redaction has two parts: turning it on in the logging pipeline and saying which redactor applies to which class:

bool redact = args.Contains("--redact");

ServiceCollection services = new();
services.AddLogging(logging =>
{
    logging.AddJsonConsole();
    if (redact)
    {
        logging.EnableRedaction();
    }
});
services.AddRedaction(redaction =>
{
    redaction.SetRedactor<ErasingRedactor>(BookstoreTaxonomy.SensitiveData);
    redaction.SetHmacRedactor(options =>
    {
        options.KeyId = 101;
        options.Key = Environment.GetEnvironmentVariable("BOOKSTORE_HMAC_KEY")!;
    }, BookstoreTaxonomy.PersonalData);
});

ErasingRedactor turns the value into an empty string. The HMAC (Hash-based Message Authentication Code) redactor hashes the value with a secret key. If you don't define a redactor for a class, the default fallback is ErasingRedactor as well. Running with --redact:

{"EventId":2001,"LogLevel":"Information","Category":"Bookstore.Customers","Message":"Customer 42 registered with email 101:nxi4FBX2XMdo97T/Cc1dKA==","State":{"Message":"Microsoft.Extensions.Logging.ExtendedLogger\u002BModernTagJoiner","Email":"101:nxi4FBX2XMdo97T/Cc1dKA==","CustomerId":42,"{OriginalFormat}":"Customer {CustomerId} registered with email {Email}"}}
{"EventId":2002,"LogLevel":"Information","Category":"Bookstore.Customers","Message":"Customer signed in","State":{"Message":"Microsoft.Extensions.Logging.ExtendedLogger\u002BModernTagJoiner","customer.NationalId":"","customer.FullName":"101:OzRdd1hZqtxqhLX9KgrWXw==","customer.Email":"101:gii2QzGZ2TvnZSDfmTSBKw==","customer.City":"Izmir","customer.CustomerId":42,"{OriginalFormat}":"Customer signed in"}}
{"EventId":2001,"LogLevel":"Information","Category":"Bookstore.Customers","Message":"Customer 43 registered with email 101:nxi4FBX2XMdo97T/Cc1dKA==","State":{"Message":"Microsoft.Extensions.Logging.ExtendedLogger\u002BModernTagJoiner","Email":"101:nxi4FBX2XMdo97T/Cc1dKA==","CustomerId":43,"{OriginalFormat}":"Customer {CustomerId} registered with email {Email}"}}

NationalId is empty, email and full name are hashed with a 101: prefix; 101 is our KeyId. Don't be confused by the class name in the State.Message field of the JSON (JavaScript Object Notation) console: EnableRedaction replaces the logger with the extended logger, and the JSON formatter writes that state's ToString(). The real values are in the properties.

An empty string can be misleading: was the field really empty, or was it erased? Writing your own redactor takes a few lines:

using Microsoft.Extensions.Compliance.Redaction;

namespace Bookstore.Customers;

public sealed class StarRedactor : Redactor
{
    private const string Stars = "****";

    public override int GetRedactedLength(ReadOnlySpan<char> input) => Stars.Length;

    public override int Redact(ReadOnlySpan<char> source, Span<char> destination)
    {
        Stars.CopyTo(destination);
        return Stars.Length;
    }
}

Writing SetRedactor<StarRedactor> instead of SetRedactor<ErasingRedactor> produces "customer.NationalId":"****". The fixed length is a deliberate choice: a mask like new string('*', input.Length) leaks the length of the value.

The HMAC Redactor: Mask but Keep Correlation

The real value of the HMAC redactor is this: the same input with the same key always produces the same output. Above, the Email value of customers 42 and 43 is the same (nxi4FBX2...), because I used the same address for both. When the support team asks "how many registration attempts were there with this email address?", they can hash the address with the same key and search for it in the log; someone reading the log can't recover the address.

But watch the second question: the Email parameter is nxi4FBX2..., while the customer.Email property of the same address is gii2QzGZ.... Same value, different hash. The reason is LoggerRedactionOptions.ApplyDiscriminator: it defaults to true and mixes the property name into the hash. The goal is to deliberately make correlation between fields with different names harder. Choosing consistent property names is enough for the same name to produce the same hash across events. If you really need correlation across fields:

logging.EnableRedaction(o => o.ApplyDiscriminator = false);

With this setting all three values matched under the same key: Email, customer.Email and the value returned when I called the redactor directly all became 101:6n1N4E1fLR2NVXihvTI/FA== (the key was different in that run, so the value is different too).

The key rules on Microsoft Learn are clear: base64 format and at least 44 characters, a distinct key for each major deployment, kept secret and rotated regularly. Don't write the key into code or appsettings; I read it from an environment variable, and in production you should use a secret store. Since KeyId is written in front of every value, after a rotation you can see which value was produced with which key; values with different KeyIds can't be compared with each other.

One note: the documentation says SetHmacRedactor is experimental and produces the EXTEXP0002 warning. When I built with the 10.11.0 package, the warning didn't appear; the build finished with "0 Warning(s)".

Know the limit of hashing too: predictable values such as email addresses and national ID numbers can be recovered by brute force if the key is compromised. Consider HMAC safe only as long as the key stays secret.

Masking with Destructuring in Serilog

If you use Serilog, Microsoft's redaction pipeline doesn't apply to Serilog's own {@Customer} destructuring. There, masking is done with a destructuring policy:

using Serilog;

namespace Bookstore.Customers;

public static class SerilogDemo
{
    public static void Run(CustomerSnapshot customer)
    {
        using Serilog.Core.Logger unsafeLogger = new LoggerConfiguration()
            .WriteTo.Console()
            .CreateLogger();
        unsafeLogger.Information("Customer signed in {@Customer}", customer);

        using Serilog.Core.Logger safeLogger = new LoggerConfiguration()
            .Destructure.ByTransforming<CustomerSnapshot>(c => new { c.CustomerId, c.City, Email = MaskEmail(c.Email) })
            .WriteTo.Console()
            .CreateLogger();
        safeLogger.Information("Customer signed in {@Customer}", customer);
    }

    private static string MaskEmail(string email)
    {
        int at = email.IndexOf('@');
        return at <= 1 ? "***" : $"{email[0]}***{email[at..]}";
    }
}

[05:12:02 INF] Customer signed in {"CustomerId": 42, "Email": "ada@example.com", "FullName": "Ada Lovelace", "NationalId": "10000000146", "City": "Izmir", "$type": "CustomerSnapshot"}
[05:12:02 INF] Customer signed in {"CustomerId": 42, "City": "Izmir", "Email": "a***@example.com"}

The first line shows how dangerous the @ operator can be: whatever is in the object ends up in the log. The second line contains only the fields we allowed. ByTransforming works as an allow list; if a new field is added to the class tomorrow, it doesn't enter the log by default. That's safer than a "remove the sensitive fields" approach. The full Serilog configuration is in this series' "Serilog in Depth" post.

Personal Data in the Exception Message

Redaction applies only to the parameters and properties you classified. The most commonly missed leak is the exception message:

try
{
    throw new InvalidOperationException($"Customer with email {ada.Email} already exists");
}
catch (InvalidOperationException ex)
{
    Log.RegistrationFailed(logger, ex, ada.CustomerId);
}

{"EventId":2003,"LogLevel":"Error","Category":"Bookstore.Customers","Message":"Registration failed for customer 42","Exception":"System.InvalidOperationException: Customer with email ada@example.com already exists\n   at Program.\u003CMain\u003E$(String[] args) in /src/Bookstore.Customers/Program.cs:line 47","State":{"Message":"Microsoft.Extensions.Logging.ExtendedLogger\u002BModernTagJoiner","exception.type":"System.InvalidOperationException","CustomerId":42,"{OriginalFormat}":"Registration failed for customer {CustomerId}"}}

Redaction is on, the log method is clean, but the email address is in plain text in the Exception field. The same problem shows up in Entity Framework Core (EF Core) constraint violation messages, or in HTTP (HyperText Transfer Protocol) clients that put an external service's error body into the exception. The fix belongs where the exception is created: put an identifier in the message, not personal data (Customer 42 is already registered).

Verifying with a Test: FakeLogger

Rules last only when they're tested. FakeLogger in the Microsoft.Extensions.Diagnostics.Testing package collects in memory the records that went through the real logging pipeline (redaction included):

public sealed class RedactionTests
{
    private const string Email = "ada@example.com";

    private static (ILogger Logger, FakeLogCollector Collector, ServiceProvider Provider) CreateLogger()
    {
        ServiceCollection services = new();
        services.AddLogging(logging => logging.EnableRedaction());
        services.AddFakeLogging();
        services.AddRedaction(redaction =>
            redaction.SetRedactor<StarRedactor>(BookstoreTaxonomy.PersonalData, BookstoreTaxonomy.SensitiveData));
        ServiceProvider provider = services.BuildServiceProvider();
        return (provider.GetRequiredService<ILogger<RedactionTests>>(), provider.GetFakeLogCollector(), provider);
    }

    private static void AssertNoEmail(FakeLogCollector collector)
    {
        foreach (FakeLogRecord record in collector.GetSnapshot())
        {
            Assert.DoesNotContain(Email, record.Message);
            Assert.DoesNotContain(Email, record.Exception?.ToString() ?? "");
            Assert.All(record.StructuredState ?? [], pair => Assert.DoesNotContain(Email, pair.Value ?? ""));
        }
    }

    [Fact]
    public void Customer_events_do_not_contain_the_email()
    {
        (ILogger logger, FakeLogCollector collector, ServiceProvider provider) = CreateLogger();
        using ServiceProvider _ = provider;

        Log.CustomerRegistered(logger, 42, Email);
        Log.CustomerSignedIn(logger, new CustomerSnapshot { CustomerId = 42, Email = Email, FullName = "Ada Lovelace" });

        Assert.Equal(2, collector.Count);
        Assert.Equal("Customer 42 registered with email ****", collector.GetSnapshot()[0].Message);
        AssertNoEmail(collector);
    }

    [Fact]
    public void Registration_failure_does_not_contain_the_email()
    {
        (ILogger logger, FakeLogCollector collector, ServiceProvider provider) = CreateLogger();
        using ServiceProvider _ = provider;

        try
        {
            CustomerRegistry.Register(42, Email);
        }
        catch (InvalidOperationException ex)
        {
            Log.RegistrationFailed(logger, ex, 42);
        }

        AssertNoEmail(collector);
    }
}

The second test calls the real code that creates the exception:

public static class CustomerRegistry
{
    public static void Register(int customerId, string email)
    {
        // we pretend every email is already registered
        throw new InvalidOperationException($"Customer with email {email} already exists");
    }
}

The dotnet test output:

[xUnit.net 00:00:00.08]     Bookstore.Customers.Tests.RedactionTests.Registration_failure_does_not_contain_the_email [FAIL]
  Failed Bookstore.Customers.Tests.RedactionTests.Registration_failure_does_not_contain_the_email [3 ms]
  Error Message:
   Assert.DoesNotContain() Failure: Sub-string found
                                ↓ (pos 54)
String: ···"Customer with email ada@example.com alrea"···
Found:  "ada@example.com"
Failed!  - Failed:     1, Passed:     1, Skipped:     0, Total:     2, Duration: 31 ms - Bookstore.Customers.Tests.dll (net10.0)

The test caught the leak. After changing the message to $"Customer {customerId} is already registered":

Passed!  - Failed:     0, Passed:     2, Skipped:     0, Total:     2, Duration: 28 ms - Bookstore.Customers.Tests.dll (net10.0)

Don't write this test once and forget it; when a new log event is added, cover it with the same helper method. A regression test is a far more reliable guard than a code review.

In Production

HTTP logging is another source of leaks: the Authorization header, cookies and the password in a request body. ASP.NET Core's HTTP logging middleware writes values only for headers on its allow list and shows [Redacted] for the rest; AddHttpLoggingRedaction in the Microsoft.AspNetCore.Diagnostics.Middleware package lets you classify headers and route parameters with the taxonomy from this post. Details are in the next post of the series.

Redaction has a processing cost; HMAC computes a hash for every value. If you log dozens of classified fields on every request, measure it. In most cases the cheaper solution is not logging that field at all.

Don't forget retention. Keeping logs that can be linked to personal data longer than necessary, even masked, is both a cost and a compliance risk. If your log tool lets you separate retention policies by class (for example longer for audit logs, short for debug logs), use it; we do that in Seq in this series' "Reading Structured Logs with Seq" post.

When you rotate the key, old and new logs no longer match. If you assume the support team will run a "last 90 days" query with the new key, plan the rotation schedule together with the retention period.

Common Pitfalls

First, assuming EnableRedaction also protects logs written with the extension methods. In a logger.LogInformation("Email {Email}", email) call the parameter has no classification; even with redaction on, the email goes out in plain text. Classification works only in source-generated log methods and in types expanded with [LogProperties].

Second, logging an object with an overridden ToString(). Today Customer.ToString() returns only the id; tomorrow someone adds the email there for debugging, and it leaks into every log line.

Third, masking only in the message text and forgetting the property. Next to a hand-written "a***@example.com" in the message, the full address can sit in a property of the same event. In a structured log, look for leaks in the properties, not the message.

Fourth, using a different redactor in tests and assuming production behavior. In the test I use StarRedactor, in production HMAC; the test verifies the "the value doesn't leak" rule, not the hash format. If you have an HMAC-specific expectation, verify it in a separate test.

A Small Experiment

★☆☆ Add a property called Phone to CustomerSnapshot without a classification. Run with --redact and see the phone number appear in plain text in the log. Then add [PersonalData] and run again.

★★☆ Add a third test to RedactionTests: call logger.LogInformation("Email {Email}", Email) and run AssertNoEmail. Explain why the test fails using the first pitfall above.

★★★ Rotate the HMAC key with two different KeyIds: in the same process, log the same email first with 101, then with 102. Write down which information has to be kept so that the support team can find both periods.

We brought sensitive data under control in log methods and objects. In the next post we look at the middleware that logs an entire HTTP request and response: AddHttpLogging, HttpLoggingFields, logging the body only for 4xx/5xx responses with IHttpLoggingInterceptor, the W3C (World Wide Web Consortium) logger and the measured cost of logging request and response bodies.