LINQ Set Operations: Distinct, Union, Intersect, Except and DistinctBy

Version note: The code in this post uses C# 10 and the .NET 6 APIs; it was built and run with the .NET 10 SDK.

Guess First

In the sign-ups coming from a newsletter form, the same person shows up twice. We clean up the duplicates with Distinct. Then we combine a wishlist with a cart; once with Concat, once with Union. What does this code print?

List<Customer> signups = new()
{
    new Customer("Ada", "ada@example.com"),
    new Customer("Alan", "alan@example.com"),
    new Customer("Ada", "ada@example.com"),
};

Console.WriteLine($"Distinct: {signups.Distinct().Count()}");

string[] wishlist = { "Dune", "1984", "Dune" };
string[] cart = { "1984", "The Hobbit" };
Console.WriteLine($"Concat: {wishlist.Concat(cart).Count()}, Union: {wishlist.Union(cart).Count()}");

class Customer
{
    public Customer(string name, string email) => (Name, Email) = (name, email);

    public string Name { get; }
    public string Email { get; }
}

What Distinct Counts as Equal

Output:

Distinct: 3
Concat: 5, Union: 3

Distinct removed nothing; Ada is in the list twice. Union, on the other hand, brought the five elements of Concat down to three. Both ask the same question: "are these two elements equal?" The answer comes from the type's default equality comparer (EqualityComparer<T>.Default). string works with value equality, so the two "Dune"s counted as the same. Customer, however, is a plain class that doesn't override Equals and GetHashCode; two separate new Customer(...) calls are two separate objects in memory even if their contents match, and LINQ sees them as different.

All the set operations that this post is about rest on this definition of equality: Distinct, Union, Intersect, Except, and DistinctBy, UnionBy, IntersectBy, ExceptBy from .NET 6. A note: these aren't the same thing as a join. A join matches the elements of two sequences on a key and produces new rows, while set operations compare the elements of two sequences of the same type with each other. We'll cover joins separately in the next post.

There are three ways to fix the equality:

List<Customer> signups = new()
{
    new Customer("Ada", "ada@example.com"),
    new Customer("Alan", "alan@example.com"),
    new Customer("Ada", "ada@example.com"),
};

Console.WriteLine($"Distinct on record: {signups.Distinct().Count()}");
Console.WriteLine($"Distinct emails: {string.Join(" | ", signups.Select(c => c.Email).Distinct())}");
Console.WriteLine($"Anonymous type: {signups.Select(c => new { c.Name, c.Email }).Distinct().Count()}");

record Customer(string Name, string Email); // Equals and GetHashCode come from the compiler

Distinct on record: 2
Distinct emails: ada@example.com | alan@example.com
Anonymous type: 2

A record makes the compiler generate an Equals and GetHashCode that look at all the properties, and Distinct just works. If you can't change the type, you can project to the field you want to compare first and then call Distinct; but what you're left with is emails, not customers. Since an anonymous type carries value equality too, it's a short way to compare several fields together.

Union, Intersect, Except: Set Logic

Remember the Venn diagrams from school: union, intersection and difference. Let's look at them through a wishlist and the purchased books:

string[] wishlist = { "Dune", "1984", "Dune", "Foundation", "Neuromancer" };
string[] purchased = { "1984", "The Hobbit", "1984" };

Console.WriteLine($"Union:     {string.Join(" | ", wishlist.Union(purchased))}");
Console.WriteLine($"Intersect: {string.Join(" | ", wishlist.Intersect(purchased))}");
Console.WriteLine($"Except:    {string.Join(" | ", wishlist.Except(purchased))}");
Console.WriteLine($"Where:     {string.Join(" | ", wishlist.Where(t => !purchased.Contains(t)))}");

Union:     Dune | 1984 | Foundation | Neuromancer | The Hobbit
Intersect: 1984
Except:    Dune | Foundation | Neuromancer
Where:     Dune | Dune | Foundation | Neuromancer

Union walked the first sequence, then the second, and yielded every element it hadn't yielded before. Intersect kept the ones in both, Except the ones in the first but not in the second. The point to watch is that all three deduplicate the result. "Dune" was in the wishlist twice; in the result of Except it shows up once. Writing the same thing with Where and Contains keeps the duplicate. Choose based on what you want: the answer to "which kinds of books haven't been bought" is Except, the answer to "which wishlist rows haven't been bought" is Where.

As for order: the documentation describes the order of the first sequence followed by the second for Union, and the order of the first sequence for Intersect. For Distinct, it says the result is unordered. Today's implementation keeps the order of first appearance, but if order matters to you, don't rely on it like a contract; add an OrderBy.

Comparing with a Comparer

Email addresses are case-insensitive, but the default equality of string isn't. The difference shows up as soon as we try to find the customers who aren't on the newsletter list:

string[] newsletter = { "ada@example.com", "Alan@Example.com" };
string[] customers = { "ADA@example.com", "grace@example.com", "alan@example.com" };

Console.WriteLine(string.Join(" | ", customers.Except(newsletter)));
Console.WriteLine(string.Join(" | ", customers.Except(newsletter, StringComparer.OrdinalIgnoreCase)));
Console.WriteLine(string.Join(" | ", newsletter.Union(customers, StringComparer.OrdinalIgnoreCase)));

ADA@example.com | grace@example.com | alan@example.com
grace@example.com
ada@example.com | Alan@Example.com | grace@example.com

Without a comparer, Except counted "ADA@example.com" and "ada@example.com" as different and filtered out none of the three customers. With StringComparer.OrdinalIgnoreCase, only Grace was left. Union with the same comparer both combined the two lists and collapsed the differently written copies of the same address into a single entry; the spelling that's kept is the first one encountered. Every set operator has an overload that takes an IEqualityComparer<T>. If you write your own comparer, remember the rule from the Dictionary post of the Collections and Generics series: two values you consider equal must produce the same hash.

DistinctBy and UnionBy (.NET 6)

Before .NET 6, saying "one book per author" meant either writing GroupBy(b => b.Author).Select(g => g.First()) or a comparer class just for this job. The *By variants let you define equality with a key selector. The examples from here on use the bookstore data from the earlier posts in the series; put the file below next to Program.cs as Data.cs:

record Book(string Isbn, string Title, string Author, decimal Price, int Stock, int Year);

static class Store
{
    public static List<Book> Books { get; } = new()
    {
        new("978-0441172719", "Dune", "Frank Herbert", 12.99m, 4, 1965),
        new("978-0553293357", "Foundation", "Isaac Asimov", 9.99m, 0, 1951),
        new("978-0451524935", "1984", "George Orwell", 8.99m, 7, 1949),
        new("978-0261102217", "The Hobbit", "J.R.R. Tolkien", 10.99m, 2, 1937),
        new("978-0547928210", "The Fellowship of the Ring", "J.R.R. Tolkien", 15.99m, 3, 1954),
        new("978-0553382563", "I, Robot", "Isaac Asimov", 7.99m, 5, 1950),
        new("978-0060850524", "Brave New World", "Aldous Huxley", 11.49m, 1, 1932),
        new("978-0451526342", "Animal Farm", "George Orwell", 6.99m, 9, 1945),
        new("978-0441569595", "Neuromancer", "William Gibson", 10.49m, 0, 1984),
        new("978-0441478125", "The Left Hand of Darkness", "Ursula K. Le Guin", 9.49m, 6, 1969),
    };
}

using System.Globalization;

CultureInfo.CurrentCulture = CultureInfo.InvariantCulture;

IEnumerable<string> onePerAuthor = Store.Books
    .DistinctBy(b => b.Author) // the first book of each author
    .Select(b => b.Title);
Console.WriteLine(string.Join(" | ", onePerAuthor));

List<Book> featured = new() { Store.Books[0], Store.Books[2] };
List<Book> newArrivals = new() { Store.Books[2] with { Price = 7.99m }, Store.Books[9] };

IEnumerable<Book> homepage = featured.UnionBy(newArrivals, b => b.Isbn);
Console.WriteLine(string.Join(" | ", homepage.Select(b => $"{b.Title} {b.Price}")));
Console.WriteLine($"Union on records: {featured.Union(newArrivals).Count()} books");

Dune | Foundation | 1984 | The Hobbit | Brave New World | Neuromancer | The Left Hand of Darkness
Dune 12.99 | 1984 8.99 | The Left Hand of Darkness 9.49
Union on records: 4 books

DistinctBy(b => b.Author) kept the book of each author that comes first in the list: Foundation from Asimov, 1984 from Orwell, The Hobbit from Tolkien. The rest of the books were skipped because they produced the same key again. You get the elements themselves back, not the keys; that's why it's different from Select(b => b.Author).Distinct().

The UnionBy example combines a homepage showcase. The "new arrivals" list contains a discounted copy of 1984. Combining by ISBN, the record from the first list won and the price stayed 8.99; the second record with the same key was skipped. A plain Union, on the other hand, looked at all the properties of the record, counted the two 1984s with different prices as separate books and returned four results. If some fields of your records can change, stating the definition of "the same book" explicitly with *By is safer.

IntersectBy and ExceptBy: The Second Parameter Is a List of Keys

The signature of these two operators surprises people at first. The second parameter isn't a sequence of the same type as the first, it's a sequence of the key type:

string[] soldOutIsbns = { "978-0553293357", "978-0441569595" };

IEnumerable<string> soldOut = Store.Books.IntersectBy(soldOutIsbns, b => b.Isbn).Select(b => b.Title);
int available = Store.Books.ExceptBy(soldOutIsbns, b => b.Isbn).Count();
Console.WriteLine($"Sold out: {string.Join(" | ", soldOut)}; available: {available}");

List<Book> inCart = Store.Books.Take(3).ToList();
IEnumerable<string> notInCart = Store.Books
    .ExceptBy(inCart.Select(b => b.Isbn), b => b.Isbn) // the second parameter is a list of keys
    .Select(b => b.Title);
Console.WriteLine($"Not in cart: {notInCart.Count()}, first: {notInCart.First()}");

Sold out: Foundation | Neuromancer; available: 8
Not in cart: 7, first: The Hobbit

The design comes from a real need. Most of the time what you have isn't a second list of Books to compare against, but a list of ISBNs: the codes of sold-out products, ids coming from an API, the rows a user ticked. ExceptBy(soldOutIsbns, b => b.Isbn) expresses that directly. If you do have a second Book list, you project it to keys first: inCart.Select(b => b.Isbn). Forget that and pass the list directly, and the compiler stops you:

List<Book> inCart = Store.Books.Take(3).ToList();

IEnumerable<Book> notInCart = Store.Books.ExceptBy(inCart, b => b.Isbn);

Program.cs(3,43): error CS0411: The type arguments for method 'Enumerable.ExceptBy<TSource, TKey>(IEnumerable<TSource>, IEnumerable<TKey>, Func<TSource, TKey>)' cannot be inferred from the usage. Try specifying the type arguments explicitly.

The error message isn't helpful on first reading. The compiler is trying to infer TKey from two places: Book from the second argument, string from the key selector. When the two contradict each other, type inference fails. If you see this message with the *By operators, the first place to look is the type of the second argument.

What Happens Inside: HashSet and Deferred Execution

Set operators use a hash set internally; with .NET 6, LINQ's own small set implementation gave way to HashSet<T>. That's why Except fills a set with the second sequence once and does a single hash lookup for each element of the first: roughly n + m work in total. Where(x => !list.Contains(x)), by contrast, scans the list from the start for each element: n × m. With ten books you won't see the difference; as the number of records grows, the cost of the second approach grows quadratically. We'll measure this in the series' performance post. Since these operators use deferred execution, when the second sequence is read matters too:

List<string> blockedIsbns = new() { "978-0441172719" };

IEnumerable<Book> visible = Store.Books.ExceptBy(blockedIsbns, b => b.Isbn);
Console.WriteLine($"First count: {visible.Count()}");

blockedIsbns.Add("978-0451524935"); // after the query was defined
Console.WriteLine($"Second count: {visible.Count()}");

First count: 9
Second count: 8

When the query was defined, the blocked list had a single ISBN. After a second ISBN was added to the list, the same visible query returned one book fewer. ExceptBy reads the second sequence again every time it's enumerated. Sometimes that's exactly what you want (a live block list), sometimes it's the source of a surprising bug. If you want a fixed result, materialize the query with ToList() where you define it.

Common Pitfalls

Calling Distinct on a class without equality. Nothing gets removed, and you don't get an error either. Use a record, DistinctBy or a comparer.

Mixing up Union and Concat. Concat appends two sequences end to end as they are, Union drops the duplicates. If duplicates are meaningful (like order lines), use Concat.

Expecting Except and Intersect to keep the duplicates of the first sequence. The result is a deduplicated set; if the row count matters, use Where with a HashSet.

Comparing user input (email, username) without a comparer. Because of case differences the same person is counted twice; pass StringComparer.OrdinalIgnoreCase.

Passing a list of elements as the second parameter to ExceptBy/IntersectBy. You need to pass a list of keys; if you see CS0411, check this first.

Relying on the order of Distinct. The documentation says unordered; if order matters, sort explicitly.

A Small Experiment

★☆☆ Take the wishlists of two customers (string arrays) and print, separately, the books both want, the books only one of them wants, and all the books without any duplicates.

★★☆ Find "the cheapest book of each author" without using DistinctBy. Then try to solve the same question with DistinctBy; explain why you have to sort first.

★★★ Write an IEqualityComparer<string> that compares email addresses case-insensitively and also ignores leading and trailing whitespace. Try it with Except, then deliberately write GetHashCode wrong (only calling Trim) and observe how the result breaks.

Set operations compared two sequences of the same type. In the next post we move on to Join and GroupJoin, which match two sequences of different types on a key, and to the left join pattern, which has no ready-made operator in .NET 6.