Under the hood: What can you learn by building an HTTP Server from scratch?

Introduction: I've always been curious about how most of the technologies I use on a day-to-day basis work. One of the things we as programmers have to interact with daily is HTTP servers. Most of the time, we don't think much about it since HTTP has been around for a long time, and there are thousands of HTTP servers and clients, so most people don't give much thought to how these technologies work! As someone who worked on many web-based applications and built many restful APIs, I would like to think that I have a pretty good understanding of how the HTTP protocol works and how HTTP servers generally function, however, since I've always been using ready-made HTTP servers (Mostly Kesterl, IIS and Nginx) and HTTP clients (curl, postman, httpie) I felt that there's a lot that I can learn by writing an HTTP server from scratch, and so in this article, I will take you through my small HTTP server project that I've written and hopefully I will be able to share with you all the new things I got to learn along the way. It's just text! As you might already know, HTTP stands for HyperText Transfer Protocol, and it's simply one of the core protocols powering the internet and transferring data between your devices and servers all over the globe. Just by reading the name "HyperText Transfer Protocol" we can get a few clues as to what HTTP is and how it works. To put it simply, it's a protocol that allows us to transfer text! But could it be that simple? Per estimates, there are currently around 75 billion devices connected to the internet and over 1.1 billion websites accessible. Is it all running by simply sending text back and forth between servers and clients? Well, basically yes (for the most part). For us to understand more why HTTP is so popular, we need to understand a few things about HTTP, and hopefully by the end of the article, we will have a clear picture of why HTTP to this day remains the most dominant protocol for web apps and server-to-client communication. To avoid confusion about how HTTP works, let's first try to understand what HTTP is and what HTTP is not. HTTP is not a transport protocol, it is an application layer protocol. This means that HTTP depends on a transport layer protocol (such as TCP/IP) to send and receive the data packets. HTTP is designed to work with a client and server model and expects the client to initiate a request and a server to send a response back to the client, so HTTP is not optimized for real-time communication (Although it's capable of it, there are better alternatives) HTTP is not a secure protocol by default, since we are sending plain text over the wire, anyone with access to the transported data will be able to read the contents, so the HTTPS (HTTP + SSL/TLS) protocol was later introduced to provide a security through encryption on top of HTTP So, how does it all work? As we have already established, HTTP works by transferring text with a request-response model, and for it to be supported on billions of devices and websites, HTTP needs to behave the same across all servers and clients and so there are specific rules that these servers and clients need to follow. We've already established that the request and response messages are transferred as plain text, so how can the servers and clients parse these messages regardless of which HTTP server or client you are using? Well, that's where the "Protocol" part comes into the picture. HTTP is a standard that's maintained and documented by the IETF HTTP Working Group, and full documentation of the standard can be found here if you're curious to see more details. As long as the client and server are following the rules of the HTTP protocol, they should be able to communicate. The Anatomy of an HTTP Message Let's dig deeper into the HTTP message and the content it holds Both the request and response have a pretty similar structure 1- Description line: A single line of text that specifies the version of the protocol, then the route and HTTP method in the request, and the status code in the response 2- Headers: (Optional) A set of headers, each single line is a single header, and each header consists of a key and value pair separated by a colon. These headers are supposed to add more details about the request/response, such as the format of data being transferred. 3- Empty line: The purpose of this line is to indicate the end of the message metadata and the start of the message content. To be more accurate, this line is represented as \r\n\r\n (section 4.1 of RFC 2616), which will be important later. 4- Message body: (Optional) If the message contains any content, it will be included in this part. Usually, the headers will help us identify whether the message also includes a body. Now that we understand what HTTP messages look like and how they're supposed to be formatted, our HTTP server should just be able to read messages with this format and respond in

May 7, 2025 - 19:28

Under the hood: What can you learn by building an HTTP Server from scratch?

Introduction:

I've always been curious about how most of the technologies I use on a day-to-day basis work. One of the things we as programmers have to interact with daily is HTTP servers. Most of the time, we don't think much about it since HTTP has been around for a long time, and there are thousands of HTTP servers and clients, so most people don't give much thought to how these technologies work!

As someone who worked on many web-based applications and built many restful APIs, I would like to think that I have a pretty good understanding of how the HTTP protocol works and how HTTP servers generally function, however, since I've always been using ready-made HTTP servers (Mostly Kesterl, IIS and Nginx) and HTTP clients (curl, postman, httpie) I felt that there's a lot that I can learn by writing an HTTP server from scratch, and so in this article, I will take you through my small HTTP server project that I've written and hopefully I will be able to share with you all the new things I got to learn along the way.

It's just text!

As you might already know, HTTP stands for HyperText Transfer Protocol, and it's simply one of the core protocols powering the internet and transferring data between your devices and servers all over the globe.

Just by reading the name "HyperText Transfer Protocol" we can get a few clues as to what HTTP is and how it works. To put it simply, it's a protocol that allows us to transfer text!

But could it be that simple? Per estimates, there are currently around 75 billion devices connected to the internet and over 1.1 billion websites accessible. Is it all running by simply sending text back and forth between servers and clients? Well, basically yes (for the most part).

For us to understand more why HTTP is so popular, we need to understand a few things about HTTP, and hopefully by the end of the article, we will have a clear picture of why HTTP to this day remains the most dominant protocol for web apps and server-to-client communication.

To avoid confusion about how HTTP works, let's first try to understand what HTTP is and what HTTP is not.

HTTP is not a transport protocol, it is an application layer protocol. This means that HTTP depends on a transport layer protocol (such as TCP/IP) to send and receive the data packets.
HTTP is designed to work with a client and server model and expects the client to initiate a request and a server to send a response back to the client, so HTTP is not optimized for real-time communication (Although it's capable of it, there are better alternatives)
HTTP is not a secure protocol by default, since we are sending plain text over the wire, anyone with access to the transported data will be able to read the contents, so the HTTPS (HTTP + SSL/TLS) protocol was later introduced to provide a security through encryption on top of HTTP

So, how does it all work?

As we have already established, HTTP works by transferring text with a request-response model, and for it to be supported on billions of devices and websites, HTTP needs to behave the same across all servers and clients and so there are specific rules that these servers and clients need to follow.

We've already established that the request and response messages are transferred as plain text, so how can the servers and clients parse these messages regardless of which HTTP server or client you are using?
Well, that's where the "Protocol" part comes into the picture. HTTP is a standard that's maintained and documented by the IETF HTTP Working Group, and full documentation of the standard can be found here if you're curious to see more details.
As long as the client and server are following the rules of the HTTP protocol, they should be able to communicate.

The Anatomy of an HTTP Message

Let's dig deeper into the HTTP message and the content it holds

Both the request and response have a pretty similar structure

1- Description line: A single line of text that specifies the version of the protocol, then the route and HTTP method in the request, and the status code in the response
2- Headers: (Optional) A set of headers, each single line is a single header, and each header consists of a key and value pair separated by a colon. These headers are supposed to add more details about the request/response, such as the format of data being transferred.
3- Empty line: The purpose of this line is to indicate the end of the message metadata and the start of the message content. To be more accurate, this line is represented as \r\n\r\n (section 4.1 of RFC 2616), which will be important later.
4- Message body: (Optional) If the message contains any content, it will be included in this part. Usually, the headers will help us identify whether the message also includes a body.

Now that we understand what HTTP messages look like and how they're supposed to be formatted, our HTTP server should just be able to read messages with this format and respond in the same expected format to the client.

Let's start coding!

For this project, I will be using .NET 9 to write the server, although there are many libraries already available in .NET to provide such functionality, my goal here is learning, and not necessarily taking the easy path.

I will be mainly utilizing a TcpListenter class to handle the TCP communication part, as I'm not planning to rewrite the whole TCP stack from scratch (not yet).

And this is what the first draft of my project looks like Github repo
Don't worry, I will go over the code next and explain every function!

using System.Net;
using System.Net.Sockets;
using System.Text;
using Serilog;
using Serilog.Core;

public class SiriusServer
{
  public SiriusServer()
  {
    _logger = new LoggerConfiguration().MinimumLevel.Debug().WriteTo.Console().CreateLogger();
  }

  readonly Logger _logger;

  private async Task HandleRequest(TcpClient tcpClient)
  {
    try
    {
      // Get the network stream
      await using var networkStream = tcpClient.GetStream();


      // Parse the headers
      var buffer = new byte[1024];
      byte[] bodyBuffer = null;
      var bytesRead = 0;
      var headerCompleted = false;
      var headerSb = new StringBuilder();

      while (!headerCompleted && (bytesRead = await networkStream.ReadAsync(buffer.AsMemory(0, buffer.Length))) > 0)
      {
        // Fine the end of the headers indicated by the \r\n\r\n sequence 
        var headerTerminatorIndex = GetHeaderTerminaorIndex(buffer, bytesRead);

        // Headers not finished
        if (headerTerminatorIndex < 0)
        {
          // Add header chunk - as per the standards, the header should be encoded using ASCII. which is why we can't have UTF characters in the headers
          headerSb.Append(Encoding.ASCII.GetString(buffer, 0, bytesRead));
          continue;
        }

        // Headers end detected
        headerCompleted = true;

        // Add header chunk
        headerSb.Append(Encoding.ASCII.GetString(buffer, 0, headerTerminatorIndex));

        // Check if any body data was also read in the buffer and store it in a buffer
        var bodyBytesLength = bytesRead - (headerTerminatorIndex + 4);
        if (bodyBytesLength > 0)
        {
          bodyBuffer = new byte[bodyBytesLength];
          Array.Copy(buffer, headerTerminatorIndex + 4, bodyBuffer, 0, bodyBytesLength);
        }
      }

      // Parse the headers into a dictionary
      var headerLines = headerSb.ToString().Split(["\r\n"], StringSplitOptions.None);

      var descriptionLine = headerLines[0];
      _logger.Debug("Request description line read: {line}", descriptionLine);

      var headers = new Dictionary<string, string>();
      foreach (var line in headerLines.Skip(1))
      {
        if (string.IsNullOrEmpty(line))
          break;
        var headerData = line.Split(':');
        headers[headerData[0]] = headerData[1];
        _logger.Debug("Request header read: {line}", line);
      }

      // Check if the message has body and read it
      if (headers.TryGetValue("Content-Length", out var value))
      {
        // Copy any data that was read while parsing the headers
        var contentLength = int.Parse(value);
        buffer = new byte[contentLength];
        var totalRead = bodyBuffer?.Length ?? 0;
        if (bodyBuffer != null)
        {
          Array.Copy(bodyBuffer, 0, buffer, 0, bodyBuffer.Length);
        }

        // Read the rest of the body content
        while (totalRead < contentLength)
        {
          totalRead += await networkStream.ReadAsync(buffer.AsMemory(totalRead, contentLength - totalRead));
        }

        // I'm assuming that the body will be encoded using utf8, this can be enhanced later by reading the charset from the headers
        var requestBody = Encoding.UTF8.GetString(buffer);
        _logger.Debug("Body content: {body}", requestBody);
      }

      // Send back a response
      var response = "Hello, World!";
      var responseSb = new StringBuilder();
      responseSb.AppendLine("HTTP/1.1 200 OK");
      responseSb.AppendLine("Content-Type: plain/text");
      responseSb.AppendLine($"Content-Length: {Encoding.UTF8.GetByteCount(response)}");
      responseSb.AppendLine("Connection: close");
      responseSb.AppendLine();
      var headerBytes = Encoding.ASCII.GetBytes(responseSb.ToString());
      var bodyBytes = Encoding.UTF8.GetBytes(response);
      await networkStream.WriteAsync(headerBytes);
      await networkStream.WriteAsync(bodyBytes);
    }
    catch (Exception ex)
    {
      _logger.Error(ex, "An error occured while processing the request");
    }
  }

  private int GetHeaderTerminaorIndex(byte[] buffer, int bytesRead)
  {
    for (int i = 0; i < bytesRead - 3; i++)
    {
      // Determine the location of the end of headers indicated by the sequence \r\n\r\n
      if (buffer[i] == '\r' && buffer[i + 1] == '\n' && buffer[i + 2] == '\r' && buffer[i + 3] == '\n')
      {
        return i;
      }
    }
    return -1;
  }

  public async Task StartServer(int portNumber, CancellationToken cancellationToken)
  {
    var tcpListener = new TcpListener(IPAddress.Any, portNumber);

    try
    {
      tcpListener.Start();
      _logger.Information("Server started at port {Port}", portNumber);
      while (!cancellationToken.IsCancellationRequested)
      {
        var tcpClient = await tcpListener.AcceptTcpClientAsync(cancellationToken);
        _ = HandleRequest(tcpClient);
      }
    }
    catch (Exception ex)
    {
      _logger.Error(ex, "An error occured while starting the server");
    }
    finally
    {
      tcpListener.Stop();
    }
  }
}

class Program
{
  public static async Task Main()
  {
    await new SiriusServer().StartServer(8000,CancellationToken.None);
  }
}

So, what is all this code doing? Let's get into it

public async Task StartServer(int portNumber, CancellationToken cancellationToken)
{
  var tcpListener = new TcpListener(IPAddress.Any, portNumber);

  try
  {
    tcpListener.Start();
    _logger.Information("Server started at port {Port}", portNumber);
    while (!cancellationToken.IsCancellationRequested)
    {
      var tcpClient = await tcpListener.AcceptTcpClientAsync(cancellationToken);
      _ = HandleRequest(tcpClient);
    }
  }
  catch (Exception ex)
  {
    _logger.Error(ex, "An error occured while starting the server");
  }
  finally
  {
    tcpListener.Stop();
  }
}

This is the first part of the server getting executed, it creates a new TCP listener provided by the .NET System.Net.Sockets library, and what it does is that it starts listening on the desired port we provided to the server. The AcceptTcpClientAsync function accepts a connection request asynchronously, and whenever a new connection request is received, it will execute the HandleRequest function. There's also some exception handling around the TCP listener start since there are some scenarios that it might throw an exception, if, for example, the desired port is already being used by another process or if the process is unable to listen to the port for some other reason.

Once we receive a connection request, the HandleRequest function will start executing, and it will do the following:

await using var networkStream = tcpClient.GetStream();

First things first, we will grab the underlying network stream managed by the TCP client. This network stream will allow us to read and write data using the TCP connection.

Now that we have a way to read the data, the next few lines will read the request message by reading and parsing the headers portion of the request

var buffer = new byte[1024];
byte[] bodyBuffer = null;
var bytesRead = 0;
var headerCompleted = false;
var headerSb = new StringBuilder();

while (!headerCompleted && (bytesRead = await networkStream.ReadAsync(buffer.AsMemory(0, buffer.Length))) > 0)
{
  // Fine the end of the headers indicated by the \r\n\r\n sequence 
  var headerTerminatorIndex = GetHeaderTerminaorIndex(buffer, bytesRead);

  // Headers not finished
  if (headerTerminatorIndex < 0)
  {
    // Add header chunk - as per the standards, the header should be encoded using ASCII. which is why we can't have UTF characters in the headers
    headerSb.Append(Encoding.ASCII.GetString(buffer, 0, bytesRead));
    continue;
  }

  // Headers end detected
  headerCompleted = true;

  // Add header chunk
  headerSb.Append(Encoding.ASCII.GetString(buffer, 0, headerTerminatorIndex));

  // Check if any body data was also read into the buffer and store it in a buffer
  var bodyBytesLength = bytesRead - (headerTerminatorIndex + 4);
  if (bodyBytesLength > 0)
  {
    bodyBuffer = new byte[bodyBytesLength];
    Array.Copy(buffer, headerTerminatorIndex + 4, bodyBuffer, 0, bodyBytesLength);
  }
}

Since we don't know the size or length of the headers beforehand, we will be reading the data from the client in chunks of 1024 bytes until we encounter the header terminator index, which, as explained earlier, is a sequence of "\r\n\r\n"

As that's happening, we will read the chunk of bytes, then convert it to a string using ASCII encoding, which, according to the HTTP standard, is the encoding that should be used for the HTTP message headers section.

This is why you cannot include Unicode characters directly in the URL or cookies without doing a URL encoding first.

if (headerTerminatorIndex < 0)
{
  // Add header chunk - as per the standards, the header should be encoded using ASCII. which is why we can't have UTF characters in the headers
  headerSb.Append(Encoding.ASCII.GetString(buffer, 0, bytesRead));
  continue;
}

As we are reading these chunks of data, we are using a string builder to append the chunk to form the full headers section once we reach the terminator index.

// Check if any body data was also read into the buffer and store it in a buffer
var bodyBytesLength = bytesRead - (headerTerminatorIndex + 4);
if (bodyBytesLength > 0)
{
  bodyBuffer = new byte[bodyBytesLength];
  Array.Copy(buffer, headerTerminatorIndex + 4, bodyBuffer, 0, bodyBytesLength);
}

One thing to take note of during this operation is that since we are reading the data in chunks and we have no idea how long the headers are, the last chunk of data we read might include some data of the message body as well. That is why we are keeping track of that in a separate bodyBuffer array that we will use later to combine with the message body.

      // Parse the headers into a dictionary
      var headerLines = headerSb.ToString().Split(["\r\n"], StringSplitOptions.None);

      var descriptionLine = headerLines[0];
      _logger.Debug("Request description line read: {line}", descriptionLine);

      var headers = new Dictionary<string, string>();
      foreach (var line in headerLines.Skip(1))
      {
        if (string.IsNullOrEmpty(line))
          break;
        var headerData = line.Split(':');
        headers[headerData[0]] = headerData[1];
        _logger.Debug("Request header read: {line}", line);
      }

Now that we are done reading the headers section, we can split it into individual header entries by reading each individual line, knowing that each line is terminated with a sequence of "\r\n" and then each key value entry is separated by a Colon. By the end of that snippet, we will have first a description line which contains the HTTP method, URL, and the protocol version, as well as a header dictionary that we can use to access the provided headers easily.

      // Check if the message has a body and read it
      if (headers.TryGetValue("Content-Length", out var value))
      {
        // Copy any data that was read while parsing the headers
        var contentLength = int.Parse(value);
        buffer = new byte[contentLength];
        var totalRead = bodyBuffer?.Length ?? 0;
        if (bodyBuffer != null)
        {
          Array.Copy(bodyBuffer, 0, buffer, 0, bodyBuffer.Length);
        }

        // Read the rest of the body content
        while (totalRead < contentLength)
        {
          totalRead += await networkStream.ReadAsync(buffer.AsMemory(totalRead, contentLength - totalRead));
        }

        // I'm assuming that the body will be encoded using utf8, this can be enhanced later by reading the charset from the headers
        var requestBody = Encoding.UTF8.GetString(buffer);
        _logger.Debug("Body content: {body}", requestBody);
      }

As mentioned earlier the request body is an option part of the HTTP request message and if the request is to include a body then it's responsibility of the HTTP client to specify that information in the header by providing a Content-Type, which indicates the MIME type of the content provided in the message body as well as the Content-Length wich specifies the length of that content in bytes.
That's why the server will first check if the Content-Type header is present, which in turn will let us know if the message has a body or not.
Once that's determined, we can easily read the request body by first reading the Content-Length header and then reading that number of bytes from the network stream.

while (totalRead < contentLength)
{
  totalRead += await networkStream.ReadAsync(buffer.AsMemory(totalRead, contentLength - totalRead));
}

Having the content length for the body helps us avoid reading the body in chunks in the way we did for the header. Since we know exactly how many bytes we need to read, we can do that in a more streamlined approach.

var requestBody = Encoding.UTF8.GetString(buffer);

Finally, we can read the content of the body by decoding the byte buffer.

Note that in this implementation, I'm assuming that the content is encoded using UTF-8 encoding. However, that might not be the case for all requests, and a better approach would be to check for the charset parameter that might be included in the Content-Type header, e.g. "Content-Type: text/html; charset=utf-8"

// Send back a response
var response = "Hello, World!";
var responseSb = new StringBuilder();
responseSb.AppendLine("HTTP/1.1 200 OK");
responseSb.AppendLine("Content-Type: plain/text; charset=utf-8");
responseSb.AppendLine($"Content-Length: {Encoding.UTF8.GetByteCount(response)}");
responseSb.AppendLine("Connection: close");
responseSb.AppendLine();
var headerBytes = Encoding.ASCII.GetBytes(responseSb.ToString());
var bodyBytes = Encoding.UTF8.GetBytes(response);
await networkStream.WriteAsync(headerBytes);
await networkStream.WriteAsync(bodyBytes);

Now that we are done reading the request, all that's left to be done is to send the response back to the client. To do so, we can use a StringBuilder to build the response content as per the expected format, and then we can encode it and send it back using the same network stream. Again, in this part, you can notice that the headers part is encoded using ASCII as per the standard.

An additional header I'm adding to the response is the Connection header. This header is used to communicate between the client and server whether the connection should stay open after the transaction is done. In this case, I'm specifying that the connection will be closed. Another option is to allow the same connection to be utilized again for other requests by providing the value of "Keep-Alive"; however, I won't be implementing that functionality for now.

Fire it up!

Once you run the project, you should be getting the message (Assuming you don't have other services listening on port 8000)
Server started at port 8000

Now we can go ahead and send a test request

We can see that our client is receiving the expected response message with the correct headers and body.

On the server side, we can see the logs getting generated and the server being able to read and parse the headers and body as expected