Where to store application events used to monitor application usage patterns and behavior?

Suppose you have an application and you have an APM platform like Datadog or Application Insights that you use to monitor the performance and the inner workings of your application. Sometimes there are application events that you want to record and check later. By analyzing these events you can spot bugs in your application, detect malicious users or understand that some of your customers are using the application in the wrong way. Example: you have a message queue in your backend and sometimes there are items in the queue that cannot be processed because some business validation rules fail. My question is: where do you usually store these kind of application events ? Please, notice that I'm not referring to domain-level events that you want to raise and / or handle in the application code to model the business domain. I'm referring to application events to be used by operation team to monitor the application behavior and usage patterns. I have seen different approaches to solve this problem: use a structured logging system and store this information in the logs store events at the APM platform level. As an example, Application Insights as a built in concept of custom event with associated metadata (see here, for an example in the C# sdk) store these monitoring events in the application database, by using a custom approach Each one of these approaches has some pros and cons. I'm just looking for guidance and some advice from people having experience with this problem. My favorite approach is storing this information using APM-level events, the problem with that is that not all the APM systems have the same concept of event and the same underlying data model.

Mar 17, 2025 - 22:28

Suppose you have an application and you have an APM platform like Datadog or Application Insights that you use to monitor the performance and the inner workings of your application.

Sometimes there are application events that you want to record and check later. By analyzing these events you can spot bugs in your application, detect malicious users or understand that some of your customers are using the application in the wrong way.

Example: you have a message queue in your backend and sometimes there are items in the queue that cannot be processed because some business validation rules fail.

My question is: where do you usually store these kind of application events ? Please, notice that I'm not referring to domain-level events that you want to raise and / or handle in the application code to model the business domain. I'm referring to application events to be used by operation team to monitor the application behavior and usage patterns.

I have seen different approaches to solve this problem:

use a structured logging system and store this information in the logs
store events at the APM platform level. As an example, Application Insights as a built in concept of custom event with associated metadata (see here, for an example in the C# sdk)
store these monitoring events in the application database, by using a custom approach

Each one of these approaches has some pros and cons. I'm just looking for guidance and some advice from people having experience with this problem. My favorite approach is storing this information using APM-level events, the problem with that is that not all the APM systems have the same concept of event and the same underlying data model.