Understanding and optimising your application with APM
Introducing an Application Performance Management (APM) tool can be useful in helping understand the health and efficiency of your web application. Besides giving you insights regarding performance (via indicators such as requests per minute or transaction durations), an APM can also help you understand why individual requests take longer (e.g. by breaking down the related SQL queries for you) and where errors are occurring (by logging them and reporting which functions are involved).
In this post, we will explore adding an APM to an existing Divio application.
APM is a general term that can comprise different metrics - such application performance, code level profiling and network-based performance. In general, the goal of APM is to turn a black box product or service into something more opaque by providing insights into performance - something that different testing strategies might otherwise miss.
Depending on your application type, different APM solutions can provide more granular and specialised tools to help you make sense of your application.
There are many different APM providers, but for this example, we will use Elastic APM as a popular, open-source choice and where the Elastic stack can be combined to add features such as logging and proactive monitoring. If you already have a preferred APM provider, the same general process should apply to integrating with a Divio project.
To get started, head over to Elastic and create an account for the Elasticsearch service. For this post, the free trial account that Elastic provide will be sufficient.
Once you have created and activated your account, start an initial deployment to the cloud provider of your choice which is ideally in the same region as your Divio project to minimise the time differential.
While the deployment is in progress, be sure to note the one-time credentials (username, password, APM secret token etc.). Existing credentials cannot be revealed once deployed and will require resetting.
Once your Elastic deployment is complete, from the Divio Control Panel, install the free "Divio Telemetry" addon into the application you wish to observe.
In order to complete the setup of the telemetry addon, the DEFAULT_APM_DSN environmental variable needs to be set. This can be done through the Control Panel, setting the variable for your test and live environments respectively and through the .env-local file on your local development environment.
Add the folowing to your .env-local file:
DEFAULT_APM_DSN = https://DIVIO_APP_NAME:APM_SECRET_TOKEN@ELASTIC_HOSTNAME.co:443
Note that the APM_SECRET_TOKEN and ELASTIC_HOSTNAME should reflect your settings in the Elastic Cloud. Your ELASTIC_HOSTNAME can be found through the Elastic deployment dashboard by copying the link the "APM Server URL".
We can begin locally testing the connection with Elastic APM Server by running "divio project update" followed by "divio project up" to launch your local project.
Once your project is launched, login to the APM service through the Elastic Cloud control panel using your noted credentials. You may have to wait a few minutes for your local data to become available in the APM application.
If you don't see any results within a few minutes, you may have to add an index pattern. Click on the APM configuration link on the Kibana/APM dashboard and hit the "Load Kibana objects" button. This essentially prompts a refresh of your project data to allow it to be visualised. You should also double check the DEFAULT_APM_DSN is correctly set.
In this post, we wont explore Kibana rather it is best explained through the extensive Elastic documentation.
Once your data is available, we can proceed to repeat the steps for your test and live environments which each environment having its own view within Elastic APM.
As your application is being used and data gathered, you can explore the telemetry data to find bottlenecks or areas ripe for optimisation using Elastic APM.
If you're interested in further configuration options and the Elastic stack, refer to the Elastic documentation at https://www.elastic.co/guide/en/apm/agent/python/current/django-support.html
You can find the source code for the Divio telemetry addon at https://github.com/divio/divio-telemetry-apm/blob/master/aldryn_config.py
Feedback? Questions? Let us know what you think!