IIS in Production: Deploying, app_offline.htm, Locked Files, Application Initialization and Angular

Version note: The publish outputs and web.config files in this post were produced with the .NET 10 SDK (Software Development Kit) 10.0.402 and Angular CLI (Command Line Interface) 22.2.6, and the Windows Service sample was built and run with .NET 10 and Microsoft.Extensions.Hosting.WindowsServices 10.0.14. Since IIS (Internet Information Services) doesn't run on this Mac, IIS's own behaviour (file locks, recycling, URL (Uniform Resource Locator) Rewrite) wasn't tried; it's based on the documentation on Microsoft Learn.

Guess First

Bookstore.Api runs in-process on IIS on Windows Server. To deploy the second version you copy the new publish output over the site folder with robocopy. appsettings.json and web.config get copied, but at Bookstore.Api.dll the copy stops with a "file is being used by another process" error.

Guess: which version is running on the server right now, and which files are new? And did the change to web.config have any effect on the running app?

Deploying: Web Deploy, File Copy and app_offline.htm

On Windows, DLL (Dynamic Link Library) files loaded by a running process are locked; they can't be overwritten. In the in-process model the app is inside w3wp.exe, so the IIS worker process holds the lock. The answer to the guess is a mixed state: appsettings.json is new, the DLL is old. The change to web.config, on the other hand, triggers IIS to recycle the site's app pool (according to Microsoft's documentation a site is recycled when its web.config changes); so halfway through the copy a process combining old code with new configuration may have come up. The module we call ANCM (ASP.NET Core Module) in this post is the bridge from the ninth post. A half-finished deploy is more dangerous than one that never happened.

The mechanism on Microsoft's "App Offline file" page is for this problem: when you put a file called app_offline.htm in the root of the site folder, ANCM shuts the app down gracefully, stops processing new requests and answers every request with the content of this file. If the app doesn't shut down within shutdownTimeLimit (10 seconds by default), it stops the process. Once the file is deleted, the next request starts the app again. A deploy script (PowerShell; not run on this Mac):

$site = "C:\sites\bookstore-api"

# 1. take the app down; ANCM answers requests with app_offline.htm
Copy-Item .\app_offline.htm "$site\app_offline.htm"
Start-Sleep -Seconds 5

# 2. copy the new files (the locks are gone)
robocopy .\publish $site /MIR /XF app_offline.htm /XD logs

# 3. bring the app back
Remove-Item "$site\app_offline.htm"

For the duration of these steps users see the content of app_offline.htm instead of the app; write the file as a short "we're under maintenance" page. The documentation has two notes: the file must be smaller than 4 GB, and in the out-of-process model an open connection (for example a WebSocket) can delay the shutdown. This is exactly what Visual Studio and Web Deploy (msdeploy) do: when you publish with Web Deploy the tool puts app_offline.htm in place and removes it itself. Web Deploy's upside is diff-based synchronisation and remote publishing; its downside is a separate service on the server and credential management. In CI (Continuous Integration) and CD (Continuous Delivery) pipelines most teams now copy the artefact to the server and run a script like the one above; the pipeline side is in the upcoming CI/CD Fundamentals and GitHub Actions series.

Locked Files and Shadow Copy

app_offline.htm solves the lock problem by shutting the app down; the price is an outage for the duration of the copy. ANCM's shadow copy feature shortens it: the app's assemblies are loaded not from the site folder but from copies in another folder, the files in the site folder aren't locked and can be overwritten while the app runs. The feature arrived experimentally in .NET 6 as experimentalEnableShadowCopy and has been supported as enableShadowCopy since .NET 7. According to the "Advanced configuration" page on Microsoft Learn it's given as handler settings in web.config.

Writing these settings by hand into the web.config on the server means losing them on the next publish, because publish produces web.config. The right place is the project folder: if the project has a web.config, the SDK takes it as a template and fills in the missing parts. I put this into Bookstore.Api's project folder:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <location path="." inheritInChildApplications="false">
    <system.webServer>
      <handlers>
        <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModuleV2" resourceType="Unspecified" />
      </handlers>
      <aspNetCore processPath="%LAUNCHER_PATH%" arguments="%LAUNCHER_ARGS%" stdoutLogEnabled="false" stdoutLogFile=".\logs\stdout">
        <handlerSettings>
          <handlerSetting name="enableShadowCopy" value="true" />
          <handlerSetting name="shadowCopyDirectory" value="../ShadowCopy/bookstore-api/" />
          <handlerSetting name="cleanShadowCopyDirectory" value="true" />
        </handlerSettings>
      </aspNetCore>
    </system.webServer>
  </location>
</configuration>

The file in the publish output:

$ dotnet publish -c Release -o out/pub-iis
$ cat out/pub-iis/web.config
<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <location path="." inheritInChildApplications="false">
    <system.webServer>
      <handlers>
        <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModuleV2" resourceType="Unspecified" />
      </handlers>
      <aspNetCore processPath="dotnet" arguments=".\Bookstore.Api.dll" stdoutLogEnabled="false" stdoutLogFile=".\logs\stdout" hostingModel="inprocess">
        <handlerSettings>
          <handlerSetting name="enableShadowCopy" value="true" />
          <handlerSetting name="shadowCopyDirectory" value="../ShadowCopy/bookstore-api/" />
          <handlerSetting name="cleanShadowCopyDirectory" value="true" />
        </handlerSettings>
      </aspNetCore>
    </system.webServer>
  </location>
</configuration>

The SDK replaced the %LAUNCHER_PATH% and %LAUNCHER_ARGS% placeholders with real values, added hostingModel and left the handler settings alone. The app pool identity needs read, write and delete permission on the shadow copy folder, and each app should have its own folder; that's why the path above ends with the app's name. The documentation's warning is clear: shadow copy isn't for zero-downtime deploys. IIS still restarts the app and some requests can get a 503; it reduces the outage, it doesn't eliminate it. For zero-downtime deploys the documentation recommends blue-green deployments or Azure deployment slots; the topic of the upcoming Deployment Strategies series.

Recycling and the First Request: Overlapped Recycle, Application Initialization, Idle Timeout

IIS recycles app pools under certain conditions (by default when a periodic interval elapses, when config changes, when a memory limit is exceeded): it starts a new worker process and shuts down the old one. With overlapped recycle the old one isn't shut down until the new process starts taking requests. Microsoft's documentation is cautious here too: overlapped recycle helps but doesn't guarantee a zero-downtime deploy. After IIS signals a recycle, ANCM by default waits one second before shutting the app down; to reduce the likelihood of 503s on slow machines this delay can be raised with the shutdownDelay handler setting (milliseconds):

<aspNetCore processPath="dotnet" arguments=".\Bookstore.Api.dll" hostingModel="inprocess">
  <handlerSettings>
    <handlerSetting name="shutdownDelay" value="5000" />
  </handlerSettings>
</aspNetCore>

For the app, a recycle means the process restarts. Everything in IMemoryCache, in-memory sessions and counters kept in the process are gone; if the Data Protection keys aren't somewhere persistent, users lose their sessions too. Move state that must survive out of the process: a distributed cache, the database, a persistent key ring.

After a recycle or a server restart the app doesn't start until the first request arrives; the first user waits for JIT (Just-In-Time) compilation, the DI (Dependency Injection) container to be built and the database connection pool to warm up. IIS's Application Initialization feature prevents this by sending a request to the app as soon as the app pool starts. According to the documentation there are three steps: enable the Application Initialization role service in IIS, set the app pool's Start Mode to AlwaysRunning and the site's Preload Enabled setting to True, or add this block to web.config:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <location path="." inheritInChildApplications="false">
    <system.webServer>
      <applicationInitialization doAppInitAfterRestart="true" />
    </system.webServer>
  </location>
</configuration>

By default IIS sends the first request to the site root (/). Bookstore.Api has no endpoint at the root; pointing the request at the health endpoint with an add initializationPage="/health" inside applicationInitialization also opens the database connection as part of the warm-up.

The second source of waiting is the idle timeout: by default an app pool shuts down after 20 minutes without requests. In an internal app with no traffic at night, the first user in the morning goes through the cold start every day. In-process, the fix is setting the app pool's "Idle Time-out (minutes)" to 0. Out-of-process, the documentation recommends calling the app from outside at regular intervals or, if the app only hosts background services, running it as a Windows Service instead of on IIS.

Data Protection on IIS

The Data Protection problem from the third post exists on IIS too and shows up more often because of recycling: if the key ring is kept in memory, users lose their sessions and antiforgery tokens become invalid on every recycle. Microsoft's documentation lists four ways: creating registry keys for the app pool (on a single server there's a Provision-AutoGenKeys.ps1 script for that), setting the app pool's "Load User Profile" to True (keys are stored in %LOCALAPPDATA%\ASP.NET\DataProtection-Keys, protected with DPAPI (Windows' Data Protection application programming interface)), giving the file system as the key ring store in code and protecting it with an X.509 certificate, or defining a machine-wide policy.

In a web farm, that is with several IIS servers, the key ring must be on a share all servers can reach (a UNC (Universal Naming Convention) path, Redis, a database) and protected with a certificate. The code side is the same as PersistKeysToFileSystem and SetApplicationName from the third post; only the identity and permissions are IIS-specific.

Sub-Applications: Putting the API under /api

You may want to publish Bookstore under a single site on IIS: Angular at the root, ASP.NET Core at /api. According to the documentation IIS virtual directories aren't supported with ASP.NET Core; the right tool is a sub-application. You put the API (Application Programming Interface) folder under the site folder, right-click it in IIS Manager, choose "Convert to Application" and give it its own app pool. In the in-process model a separate app pool for the sub-app is mandatory.

Here a line from the ninth post becomes meaningful: the web.config publish produces puts the settings inside <location path="." inheritInChildApplications="false">. If the root app is ASP.NET Core too, this keeps its aspNetCore handler from being inherited by the sub-app. If it is inherited, two handlers definitions clash and the sub-app returns a 500.19; Microsoft's error reference lists this case separately. The sub-app's path becomes the app's PathBase; ANCM tells the app this itself, so routes don't see the /api prefix while links are generated with it. This is the IIS counterpart of what we built by hand in Nginx with X-Forwarded-Prefix in the third post.

Serving Angular on IIS: SPA Fallback with URL Rewrite

IIS can serve Angular's static files directly, but the SPA (Single Page Application) fallback from the sixth post, returning index.html for unknown paths, isn't in IIS itself. It needs the separately installed IIS URL Rewrite module. The rule lives as a web.config in the Angular project's public folder:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <system.webServer>
    <rewrite>
      <rules>
        <rule name="Angular SPA fallback" stopProcessing="true">
          <match url=".*" />
          <conditions logicalGrouping="MatchAll">
            <add input="{REQUEST_FILENAME}" matchType="IsFile" negate="true" />
            <add input="{REQUEST_FILENAME}" matchType="IsDirectory" negate="true" />
            <add input="{REQUEST_URI}" pattern="^/api/" negate="true" />
          </conditions>
          <action type="Rewrite" url="/index.html" />
        </rule>
      </rules>
    </rewrite>
  </system.webServer>
  <location path="index.html">
    <system.webServer>
      <staticContent>
        <clientCache cacheControlMode="DisableCache" />
      </staticContent>
    </system.webServer>
  </location>
</configuration>

The rule is the counterpart of try_files $uri $uri/ /index.html in Nginx: if the request doesn't match a file or folder and doesn't start with /api/, rewrite it to index.html. The /api/ condition keeps requests for the API sitting as a sub-application from falling into the SPA fallback. The location path="index.html" block carries the "don't cache index.html" rule from the sixth post over to IIS. Files in the public folder are copied into the build output as they are:

$ npx ng build
Output location: /work/bookstore-web/dist/bookstore-web
$ ls dist/bookstore-web/browser | grep -E "web.config|index.html"
index.html
web.config

I didn't try this rule on IIS for this post. The trap from the sixth post applies here too: the rule returns index.html for a missing hashed JavaScript file as well. You need a separate rule for hashed files, or to leave the old build's files on the server for one release.

Without IIS: Running Kestrel as a Windows Service

If you don't want IIS on a Windows server (there's already a load balancer in front, the app is internal only, managing IIS isn't the team's job), you can run Kestrel as a Windows Service. The Windows counterpart of AddSystemd() from the eighth post:

WebApplicationBuilder builder = WebApplication.CreateBuilder(args);

// no-op unless the process runs as a Windows Service
builder.Services.AddWindowsService(options => options.ServiceName = "BookstoreApi");

WebApplication app = builder.Build();
app.MapGet("/api/books/1", () => new { Id = 1, Title = "Dune" });
app.Run();

$ dotnet build
    0 Warning(s)
    0 Error(s)
$ dotnet bin/Debug/net10.0/WinSvc.dll --urls http://127.0.0.1:18590 > run.log 2>&1 &
$ grep -E "listening|started" run.log
      Now listening on: http://127.0.0.1:18590
      Application started. Press Ctrl+C to shut down.
$ curl -s http://127.0.0.1:18590/api/books/1
{"id":1,"title":"Dune"}

The code ran unchanged on the Mac; AddWindowsService doesn't kick in when the process isn't started as a Windows Service. When it runs as a Windows Service it sets the content root to AppContext.BaseDirectory. That matters because, according to the documentation, a Windows Service's working directory is C:\WINDOWS\system32; code that looks for files with Directory.GetCurrentDirectory() looks there. The PowerShell command from the documentation to register the service (not run on this Mac):

New-Service -Name BookstoreApi -BinaryPathName "C:\services\bookstore-api\Bookstore.Api.exe --contentRoot C:\services\bookstore-api" -Credential "BOOKSTORE\svc-bookstore" -Description "Bookstore API" -DisplayName "Bookstore API" -StartupType Automatic

In this model TLS (Transport Layer Security) ends at Kestrel or at the load balancer in front; the certificate and endpoint settings from the second post apply as they are. Using the server built on the HTTP (Hypertext Transfer Protocol) driver in the Windows kernel, HTTP.sys, instead of Kestrel on Windows is also an option (for port sharing and Windows authentication), but that's a separate topic. Running Worker Services that host background jobs as Windows Services is in the "Worker Service and In-Process Job Queue" post of the Background Jobs and Scheduled Jobs series.

In Production

Keep the deploy script in one place and run the same one on every deploy: app_offline.htm, copy, health check, remove app_offline.htm. Manual deploys are the most common cause of half-finished deploys.

Pick recycle settings deliberately. The default periodic recycle can drop sessions in the middle of the working day; turn it off if you don't need it or pin it to a time at night. Recycling on a memory limit hides a memory leak; find the leak and keep the limit as a safety net.

Don't touch the web.config on the server by hand. Every setting should live in the project's web.config or the project file (AspNetCoreHostingModel, EnvironmentName); the file publish produces should be the single source of truth on the server.

Common Pitfalls

Copying files over a running app. The DLLs are locked, the copy stops halfway, and if web.config changed the app restarts with mixed files.

Editing the web.config on the server by hand. A shadow copy, stdout log or environment variable setting silently disappears on the next publish.

Forgetting that an app pool recycle wipes in-memory cache and sessions. A pool that recycles not at 03:00 but on Tuesday at 14:17 empties users' baskets.

Trying to publish ASP.NET Core with a virtual directory. It isn't supported; use a sub-application and give it its own app pool.

Not installing the URL Rewrite module for Angular. Even if the rule sits in web.config, IIS doesn't know it without the module; the result is 404s on deep links or a 500.19 because of an unknown config section.

A Small Experiment

★☆☆ Add a web.config with the shadow copy settings to your own project, publish and compare the output file with the one in the project. What did the SDK change, and what did it leave alone? Are the handler settings kept when you set AspNetCoreHostingModel to OutOfProcess?

★★☆ Add a health check step to the deploy script above: after app_offline.htm is removed, try /health for 30 seconds, and if it fails copy back the backup of the previous release and exit with an error. How would you apply the releases/ and current idea from the eighth post on IIS by changing the site's physical path?

★★★ On a Windows machine set up Bookstore in a single site: Angular at the root (with the URL Rewrite rule), and a sub-application with its own app pool at /api. Look at the pathBase and link values /api/whoami returns. Then what happens when you remove inheritInChildApplications="false" from the API's web.config and make the root app ASP.NET Core?

We've seen all the pieces of bringing the app up on Linux and Windows. In the next post we step back and choose: how to decide between a Linux VM (virtual machine) and Nginx, Windows and IIS, Caddy, Traefik, YARP (Yet Another Reverse Proxy), containers and PaaS (Platform as a Service), and the checklist to go through before production whichever you pick.