More app_offline.htm woes

I recently posted about some app_offline.htm gotchas with ASP.NET MVC and in the process ended up stumbling across another long running (since ASP.NET 2.0) issue where the app_offline.htm doesn’t work completely as advertised.

Perhaps you have uploaded an app_offline.htm file, began uploading some dlls, only to see:

Could not load file or assembly '[assembly name]' or one of its dependencies. The process cannot access the file because it is being used by another process.

Maybe you even asked why this could be happening on the asp.net forums or consulted with Stack Overflow only to get the canned, I’m-gonna-ignore-your-question-and-regurgitate-what-Scott-Gu’s-blog-told-me response:

Make sure the app_offline.htm file is at least 512 bytes in size, that it’s named correctly, and in the root of your website.

What gives?

App_offline.htm is a sham

The idea that putting an app_offline.htm file in your website root will completely “turn off” your ASP.NET web application in every regard, for every request, no matter what file changes occur to it, is simply wrong. If you are uploading a particularly large dll and access the website before it is fully copied you’ll get the “could not load file or assembly” server error just like you would if you never used an app_offline.htm file at all.

I’ve scoured the net for a proper example and confirmation about this “feature” only to hit dead ends. I’m not the only one who has discovered this problem, but this post might be the first to actually address it directly. There is indeed a way to work around this behavior.

Getting around this problem

The httpRuntime element within the <system.web> section of the web.config has the following two attributes, in addition to others:

• waitChangeNotification: Specifies the time, in seconds, to wait for another file change notification before restarting the AppDomain.
• maxWaitChangeNotification: Specifies the maximum number of seconds to wait from the first file change notification before restarting the AppDomain for a new request.

This isn’t particularly revelatory, but it’s mentioned in Microsoft’s Walkthrough: Deploying a Web Site Project by Using XCOPY:

Add a maxWaitChangeNotification attribute to your httpRuntime element and set it to the maximum number of seconds to wait from the first file change notification before restarting the application domain. Set it to a number that exceeds the length of time to complete any file copy processes. File-change notifications are combined based on the value of this attribute and the waitChangeNotification attribute.

You may already have these set and if so, you may not ever experience this problem.

Take the app_offline.htm out of the picture for a moment here, and we’ll get to the heart of the issue. If you don’t use these attributes, ASP.NET will immediately restart the app domain as soon as any bin changes occur. Access the website while a dll is still being copied: yellow screen of death.

I did some further investigation and tested with an application that logged its ASP.NET application shutdown events, and the results are unsatisfying. On the first request after dropping in an app_offline.htm file, the application was shut down due to “Chainge in App_Offline.htm” as advertised. Subsequent bin directory changes that would typically register a “Change Notification for critical directories” event message, did not occur while the app_offline.htm file was present. Again, this is how the app_offline.htm feature is advertised.

Despite this, the application still throws a “could not load file or assembly” exception if a request is made while a dll is in the process of being copied over.

A complete solution to these app_offline.htm woes

In my previous blog post I discussed how ASP.NET MVC adds its own gotchas to the process of taking an application offline. Resources that don’t map to an obvious ASP.NET resource (a url without an .aspx extension) may bypass ASP.NET all together and cause IIS to throw a 404 or 403 error. The solution was to produce a temporary web.config file at the same time the app_offline.htm file was applied. The web.config file had one setting, runAllManagedModulesForAllRequests=”true” which forced ASP.NET to process all requests (and thus, render the app_offline.htm file).

And so the complete solution simply adds to this, by utilizing a web.config that also specifies the two waitChangeNotification attributes discussed above:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
    <system.web>
        <httpRuntime waitChangeNotification="300" maxWaitChangeNotification="300"/>
    </system.web>
    <system.webServer>
        <modules runAllManagedModulesForAllRequests="true" />
    </system.webServer>
</configuration>

But there’s one more catch.

ASP.NET will unload the application as soon as the web.config is changed, but it won’t actually reload it and apply the “waitChange…” settings until a request is made. So you could still drop the app_offline.htm and web.config in, and if the first requst occurs while a dll is only half copied, the exception will occur. To add insult to injury, the exception will persist until you replace the temporary web.config or the total duration of your “waitChange…” time expires!

To get around this final hurdle, you’d need to make a request to the application after uploading the temporary web.config, but before you start deploying the application files. The complete process is as follows:

1. Add app_offline.htm
2. Replace application’s web.config with a temporary web.config (as above)
3. Request any ASP.NET resource on the site so the new waitChangeNotification gets “applied”*
4. Make any file system changes necessary (deploy bin dlls, other site files)
5. Replace temporary web.config with the original application web.config
6. Delete app_offline.htm 

*You may dispute what happens in step 2-3, since ASP.NET shuts down the application after step 1. Indeed, it does not appear from my event viewer that the web.config change is registered or “applied” as a result of either step 2 or 3, but nevertheless the updated waitChangeNotification settings do get applied after step 3.

Conclusion

The app_offline.htm feature has its flaws. Like a lot of “out of the box” features, it’ll solve most of your problems until, well, it doesn’t. And when it doesn’t, you’ll have to move heaven and earth to get it working as intended.

I suppose the only consolation is that if you really need to update an application and exceptions of any kind are intolerable, you should be under some sort of load-balancing scenario anyway, updating your sites by bringing down a node at a time.