Microservices blindly trusting anything due to network segmentation?

So I just fell in a project where microservices are inside private subnets and therefore aren't reacheable through the internet. There is a balancer that can reach this microservices and this balancer is restricted through ACLs and Security Groups (AWS) to be accessible to very limited set of IPs (the IPs from the API Gateway). At a very high level, this is the architecture: As you see, the only entrypoint to everything is the API Gateway. They have multiple front-end applications and automated backend systems that access the microservices through the API Gateway. Until here, extremely common. Currently, human users (using some websites, apps etc..) authenticate via a password flow (through the API Gatewat) that involves a backend service issuing a JWT. This JWT is wrapped in an opaque token by the API Gateway and passed to the front-end. The API Gateway validates the opaque token on each request, extracts the JWT, and forwards it to the microservices, which validate the JWT using a shared secret. In this authentication, the API Gateway routes the request to an authentication operation from the backend (this is only possible in their Gateway when grant_type is password). However, automated applications authenticate directly with the API Gateway using grant_type=client_credentials. The Gateway returns an opaque token but does not issue a JWT, so the microservices cannot validate or authorize these automated requests. When a request from these automated applications arrive, the API Gateway authenticates them based on the opaque token (just like it does with the front-end apps) but doesn't pass anything to the microservices. The microservices simply blindly allows everything without a JWT to enter and accept every request. Basically it assumes these requests are safe because it already passed through all the network restrictions and client authentication on the API Gateway. I once asked another person on the team about it and they told me that they must use the API Gateway to generate tokens, because all the consumer APPs for their APIs are registered within the API Gateway and then the API Gateway becomes a "fail-fast" mechanism for non-existing and non-authenticated entities. Also, they told me it is safe because of all the network segmentations. Okay, understood, but it it hurts the Zero-Trust principle by allowing everything from within the boundaries of the network to reach the microservice without any validation whatsoever. Their API Gateway restrictions are the following: Their API Gateway can't issue an JWT; Their API Gateway can't connect to an external authorization server (it has one buil-in); Their API Gateway doesn't even expose an introspection endpoint to allow verification of the opaque token. I'm thinking about some options on this, and the only one I'm seeing is using Client Certificates from the API Gateway to the microservices. But even so, the microservices are unable to authorize who is calling and therefore are unable to restrict which operations it can perform. If there was a JWT with scopes in it, the microservices could look and authorize based on that. Morewise, this customer is extremely hard to deal with and will not help us by just telling us what we ask for and nothing more. Given their API Gateway constraints and their current state, what would you suggest to enable security (not only network, as it is already secured) in machine-to-machine communication in this case?

May 27, 2025 - 21:40

Microservices blindly trusting anything due to network segmentation?

So I just fell in a project where microservices are inside private subnets and therefore aren't reacheable through the internet.

There is a balancer that can reach this microservices and this balancer is restricted through ACLs and Security Groups (AWS) to be accessible to very limited set of IPs (the IPs from the API Gateway).

At a very high level, this is the architecture:

As you see, the only entrypoint to everything is the API Gateway.

They have multiple front-end applications and automated backend systems that access the microservices through the API Gateway.

Until here, extremely common.

Currently, human users (using some websites, apps etc..) authenticate via a password flow (through the API Gatewat) that involves a backend service issuing a JWT. This JWT is wrapped in an opaque token by the API Gateway and passed to the front-end. The API Gateway validates the opaque token on each request, extracts the JWT, and forwards it to the microservices, which validate the JWT using a shared secret. In this authentication, the API Gateway routes the request to an authentication operation from the backend (this is only possible in their Gateway when grant_type is password).

However, automated applications authenticate directly with the API Gateway using grant_type=client_credentials. The Gateway returns an opaque token but does not issue a JWT, so the microservices cannot validate or authorize these automated requests.

When a request from these automated applications arrive, the API Gateway authenticates them based on the opaque token (just like it does with the front-end apps) but doesn't pass anything to the microservices.

The microservices simply blindly allows everything without a JWT to enter and accept every request. Basically it assumes these requests are safe because it already passed through all the network restrictions and client authentication on the API Gateway.

I once asked another person on the team about it and they told me that they must use the API Gateway to generate tokens, because all the consumer APPs for their APIs are registered within the API Gateway and then the API Gateway becomes a "fail-fast" mechanism for non-existing and non-authenticated entities. Also, they told me it is safe because of all the network segmentations.

Okay, understood, but it it hurts the Zero-Trust principle by allowing everything from within the boundaries of the network to reach the microservice without any validation whatsoever.

Their API Gateway restrictions are the following:

Their API Gateway can't issue an JWT;
Their API Gateway can't connect to an external authorization server (it has one buil-in);
Their API Gateway doesn't even expose an introspection endpoint to allow verification of the opaque token.

I'm thinking about some options on this, and the only one I'm seeing is using Client Certificates from the API Gateway to the microservices. But even so, the microservices are unable to authorize who is calling and therefore are unable to restrict which operations it can perform. If there was a JWT with scopes in it, the microservices could look and authorize based on that.

Morewise, this customer is extremely hard to deal with and will not help us by just telling us what we ask for and nothing more.

Given their API Gateway constraints and their current state, what would you suggest to enable security (not only network, as it is already secured) in machine-to-machine communication in this case?