Azure App Service - Hidden Network Trap

Introduction A few weeks ago, I was asked to investigate intermittent database connection failures affecting one of two Azure App Service instances. Both instances shared the same App Service Plan for a client, and—crucially—had been running smoothly for five weeks before these issues appeared. Interestingly, all development ("lower") environments were unaffected; only the production environments were experiencing failures. Each app used its own Azure Storage account and Azure SQL database. The Investigation Begins Both App Services were in production. The first app was failing to connect to its storage account, while the second was failing to connect to its Azure SQL database. The logs didn’t reveal anything unusual—just connection failures. Since the apps were already down, I decided to restart them, hoping a simple reboot might resolve any transient issues. However, after the restart, the problem flipped: now the first app was failing to connect to its Azure SQL database, and the second was failing to connect to its storage account. Digging Deeper: Networking Configuration This strange "mirrored" behavior prompted me to examine the networking setup. Both apps had regional VNet integrations, each with its own network and private endpoints for Azure SQL and Storage. However, network flow logs were not enabled, making troubleshooting more challenging. DNS Checks I checked DNS resolution, and it was returning the correct (private endpoint) IPs. But then I noticed something odd. The First Clue The private endpoint IP for the first app’s Azure SQL connection was the same as the second app’s private endpoint IP for storage. Conversely, the private endpoint IP for the second app’s Azure SQL connection matched the first app’s private endpoint IP for storage. This symmetry looked suspiciously similar to the "mirrored" issue I was seeing. The Discovery It turned out that both VNets used for regional VNet integration had identical address spaces and were running on the same single-instance App Service Plan. From the perspective of the App Service Plan’s networking stack, it was as if the VNets were peered. However, Azure VNet peering does not allow peering between VNets with overlapping address spaces. Why Did It Work Before? For weeks, everything worked because DNS and route table caches—along with Azure’s Software Defined Networking (SDN) caching—allowed the App Service Plan to resolve the correct IPs, despite the overlapping address spaces. But a single restart (perhaps triggered by a platform update or a move to a different underlying host) cleared these caches. As a result, DNS and route tables had to be recalculated, leading to SDN confusion and the connection failures I observed. Conclusion This experience was a reminder of how subtle networking misconfigurations—like overlapping VNet address spaces—can lurk undetected until a cache is cleared or a restart occurs. Always double-check your VNet address spaces, especially when using regional VNet integration with Azure App Service Plans! Azure App Service Plan regional VNet integration allows VNets with overlapping addresses which cause failures

May 19, 2025 - 23:20

Introduction

A few weeks ago, I was asked to investigate intermittent database connection failures affecting one of two Azure App Service instances. Both instances shared the same App Service Plan for a client, and—crucially—had been running smoothly for five weeks before these issues appeared.

Interestingly, all development ("lower") environments were unaffected; only the production environments were experiencing failures. Each app used its own Azure Storage account and Azure SQL database.

The Investigation Begins

Both App Services were in production. The first app was failing to connect to its storage account, while the second was failing to connect to its Azure SQL database. The logs didn’t reveal anything unusual—just connection failures.

Since the apps were already down, I decided to restart them, hoping a simple reboot might resolve any transient issues. However, after the restart, the problem flipped: now the first app was failing to connect to its Azure SQL database, and the second was failing to connect to its storage account.

Digging Deeper: Networking Configuration

This strange "mirrored" behavior prompted me to examine the networking setup. Both apps had regional VNet integrations, each with its own network and private endpoints for Azure SQL and Storage. However, network flow logs were not enabled, making troubleshooting more challenging.

DNS Checks

I checked DNS resolution, and it was returning the correct (private endpoint) IPs. But then I noticed something odd.

The First Clue

The private endpoint IP for the first app’s Azure SQL connection was the same as the second app’s private endpoint IP for storage.
Conversely, the private endpoint IP for the second app’s Azure SQL connection matched the first app’s private endpoint IP for storage.

This symmetry looked suspiciously similar to the "mirrored" issue I was seeing.

The Discovery

It turned out that both VNets used for regional VNet integration had identical address spaces and were running on the same single-instance App Service Plan. From the perspective of the App Service Plan’s networking stack, it was as if the VNets were peered. However, Azure VNet peering does not allow peering between VNets with overlapping address spaces.

Why Did It Work Before?

For weeks, everything worked because DNS and route table caches—along with Azure’s Software Defined Networking (SDN) caching—allowed the App Service Plan to resolve the correct IPs, despite the overlapping address spaces. But a single restart (perhaps triggered by a platform update or a move to a different underlying host) cleared these caches. As a result, DNS and route tables had to be recalculated, leading to SDN confusion and the connection failures I observed.

Conclusion

This experience was a reminder of how subtle networking misconfigurations—like overlapping VNet address spaces—can lurk undetected until a cache is cleared or a restart occurs. Always double-check your VNet address spaces, especially when using regional VNet integration with Azure App Service Plans!

Azure App Service Plan regional VNet integration allows VNets with overlapping addresses which cause failures