Over the past week, I’ve been on a fascinating “journey of discovery” related to self-hosted Web APIs and SignalR. This exploration helped me connect some dots and gain a deeper understanding of concepts I’ve encountered over the past decade.

Back in 2006, while exploring Python’s web development landscape (Django, Turbogears, etc.), I was thrilled to discover basic HTTP server modules. These modules allowed for application testing without the need for a full-fledged web server and even enabled embedding web servers directly into applications. I later found this concept in other platforms like node.js and .Net WCF, which has supported self-hosting from its inception.

Currently, many .Net libraries and frameworks offer self-hosting capabilities (Web API, SignalR via Owin, Service Stack, etc.). This led me to believe that the .Net framework itself likely provides the core functionality for building custom web servers. After some research, I came across this StackOverflow question, revealing that System.Net.HttpListener seems to be the source of this magic. While SignalR appears to use HttpListener directly, WCF and Web API seem to utilize an additional layer on top of it. It’s important to note that this differs from Hosted Web Core Applications, which rely on IIS.

Although HttpListener enables embeddable web servers within applications, it doesn’t handle the core HTTP logic itself. That responsibility falls on the http.sys driver. My familiarity with http.sys dates back to the Windows 2003 era. IIS6 marked a significant evolution of the beloved web server, primarily due to the introduction of http.sys. In simple terms, http.sys is a kernel driver that manages the HTTP (and HTTPS) protocol. When an HttpListener is created, it registers itself with http.sys, enabling http.sys to route traffic based on the registered URL prefix. This is well-explained in in this article:

“Basically, http.sys is a kernel-mode listener that has intrinsic knowledge of HTTP. Different parties can register with this listener to have requests forwarded to them. Typical examples nowadays include IIS 6 and SQL Server 2005”

and further elaborated in this other excellent one. Through this resource, I learned that HttpListener acts as a managed wrapper around the HTTP Server API.

As a core component of the Windows networking subsystem, http.sys is present in all Windows PCs from Windows XP SP2 onwards. More information can be found at here. To check if http.sys is installed on your system, simply type driverquery in the command line.

Opening listeners on http.sys is governed by ACLs. Unless running as an administrator, applications attempting to create listeners will encounter an AddressAccessDeniedException. To grant permissions for a user to register a listener on a specific URL prefix, use the netsh tool (running as administrator). For example:
netsh http add urlacl url=http://+:80/MyUri user=DOMAIN\user

One remarkable feature of http.sys is its support for port sharing between HTTP applications. This means multiple applications can be registered for the same port using different URL prefixes (e.g., app1 registers for localhost:80/app1 and app2 for localhost:80/app2). This is possible because the applications aren’t directly listening on the port; instead, http.sys handles the listening and forwards traffic to the appropriate application. This port sharing capability has been extended to other protocols through Net.TCP Port Sharing.

At the end of this excellent reading, you’ll discover that http.sys plays a central role in a perplexing and frustrating issue within modern web development: the lack of WebSocket support in .Net 4.5 (and consequently WCF, SignalR, etc.) on operating systems other than Windows 8 and Windows Server 2012.

“The problem is with the way that HTTP.sys works in Windows 7/Server 2008/R2 and earlier OS versions. Rather than issue a patch that could result in large regressions for existing customers that rely on the current behavior of HTTP.sys, the newest versions of Windows include a modification to HTTP.sys that enables WebSockets.”

This other post provides further insight into the workings of http.sys in Windows 7:

“The current technical issue for our stack is that the low-level Windows HTTP driver that handles incoming HTTP request (http.sys) does not recognize the current websockets format as having a valid entity body. As you have noted, the lack of a content length header means that http.sys does not make the nonce bytes in the entity available to any upstack callers. That’s part of the work we will need to do to build websockets support into http.sys. Basically we need to tweak http.sys to recognize what is really a non-HTTP request, as an HTTP request.”

I’ve read that WebSocket support in Windows 8/2012 is implemented in websocket.dll. This leads me to believe that the “WebSocket-enabled” http.sys in these OS versions handles the “almost HTTP-like” websocket handshake, while the standard communication is managed by websocket.dll.