How to do live self monitoring inside the application

Question

We are applying unittests, integration tests and we are practicing test driven and behaviour driven development.

We are also monitoring our applications and servers from outside (with dedicated software in our network)

What is missing is some standard for a live monitoring inside the apllication.

I give an example:

There should be a cron-like process inside the application, that regularily checks some structural health inside our data structures

We need to monitor that users have done some regular stuff that does not endanger the health of the applications (there are some actions and input that we can not prevent them to do)

My question is, what is the correct name for this so I can further research in the literature. I did a lot of searching but I almosdt always find the xunit and bdd / integration test stuff that I already have.

So how is this called, what is the standard in professional application development, I would like to know if there is some standard structure like xunit, or could xunit libraries even bee used for it? I could not even find appropriate tagging for this question, so please if you read this and know some better tags, why not add them to this answer and remove the ones that don't fit.

I need this for applications written in python, erlang or javascript and those are mostly server side applications, web applications or daemons.

What we are already doing is that we created http gateway from inside the applications that report some stuff and this is monitored by the nagios infrastructure.

I have no problem rolling some cron-like controlled self health scheme inside the applications, but I am interested about knowing some professional standardized way of doing it.

I found this article, it already comes close: Link

Answer 1

It looks like you are asking about approaches how to monitor your application. In general, one can distinguish between active monitoring and passive monitoring.

In active monitoring, you create some artificial user load that would mimic real user behavior, and monitor your application based on these artificial responses from a non-existing user (active = you actively cause traffic to your application). Imagine that you have a web application which allows to get weather forecast for specific city. To have active monitoring, you will need to deploy another application that would call your web application with some predefined request ("get weather for Seattle") every N hours. If your application does not respond within the specified time interval, you will trigger alert based on that.

In passive monitoring, you observe real user behavior over time. You can use log parsing to get number of (un)successful requests/responses, or inject some code into your application that would update some values in database whenever successful or not successful response was returned (passive = you only check other users' traffic). Then, you can create graphs and check whether there is a significant deviation in user traffic. For example, if during the same time of the day one week ago your application served 1000 requests, and today you get only 200 requests, it may mean some problem with your software.