LEMP Stack Performance Monitoring With Nginx Amplify on Ubuntu 20.04/18.04

Nginx Amplify is a free, flexible, and powerful way to analyze the load and performance of Nginx and Nginx Plus web servers. It’s easy to set up and use. Nginx Amplify also comes with MySQL/MariaDB, PHP-FPM plugins so you can monitor a full LEMP stack application. (LEMP stands for Linux, Nginx, MySQL/MariaDB, PHP.)

Nginx Amplify is developed by Nginx, Inc, the company behind Nginx web server. It’s a SaaS-based solution. You install the Nginx Amplify agent on your server and it will collect and send metrics to the SaaS service. The Nginx Amplify agent is open-source and lightweight. It collects many metrics, including:

• System metrics (CPU, RAM usage, network traffic, disk usage, disk I/O, disk latency, etc)
• Nginx metrics (connections, requests, HTTP status, response time, traffic, and more)
• MySQL/MariaDB metrics (connections, select queries, insert queries, update queries, slow queries, etc)
• PHP-FPM metrics (connections, connections queue, slow requests, etc)

Nginx Amplify can also

• Use static analyzer to help you improve Nginx configuration.
• Alert you of abnormal behavior

Installing Nginx Amplify on Your Ubuntu 18.04/20.04 Linux Server

The Nginx Amplify agent is open-source and there’s an install script that you can use to install Nginx Amplify on the following Linux Distributions:

• Debian 10, Debian 11
• Ubuntu 20.04, Ubuntu 18.04
• RHEL/CentOS 8

First, register an account at https://amplify.nginx.com

Verify your email address. Next, you will need to install the Nginx Amplify Agent on your Linux server. SSH into your server and login as root. The Nginx Amplify website is very nice to give the commands you need to run to install Nginx Amplify, so just copy and run these commands on your server.

Note that you need to run these commands as root.

Download the install script.

Run the install script. (Each Amplify account has a unique API key.)

Once it’s installed, the agent will automatically start. You can check its status with:

systemctl status amplify-agent

Output:

● amplify-agent.service - LSB: Stop/start nginx-amplify-agent
   Loaded: loaded (/etc/init.d/amplify-agent; generated)
   Active: active (running) since Wed 2019-02-27 03:37:05 UTC; 1min 17s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 9306 ExecStop=/etc/init.d/amplify-agent stop (code=exited, status=0/SU
  Process: 9341 ExecStart=/etc/init.d/amplify-agent start (code=exited, status=0/
    Tasks: 3 (limit: 2361)
   CGroup: /system.slice/amplify-agent.service
           └─9407 amplify-agent

Hint: press Q to gain back control of the terminal after running the above command.

To enable auto start at boot time, run

sudo systemctl enable amplify-agent

Configuring Stub_status in Nginx

Amplify agent can collect system metrics out-of-the-box. In order to collect Nginx metrics, you need to configure Nginx stub_status. Open the nginx.conf file.

sudo nano /etc/nginx/nginx.conf

Add the following server block in the http {...} context.

server {
     listen 127.0.0.1:80;
     server_name 127.0.0.1;
     location /nginx_status {
        stub_status on;
	allow 127.0.0.1;
	deny all;
     }
}

Save and close the file. Then reload Nginx for the changes to take effect.

sudo systemctl reload nginx

Now Amplify agent can start collecting Nginx metrics.

Creating Custom Nginx Log Format

Nginx Amplify agent can also collect information from Nginx log files. However, the default log format gives us limited information. If you really care about application performance, you should create a custom log format to show the request time, upstream response time, cache hits, etc.

Open the nginx.conf file.

sudo nano /etc/nginx/nginx.conf

Add the following lines in the http {...} context, above the include directives. Here we are creating a custom Nginx log format called apm (application performance monitoring) that includes information about request_time, upstream_response_time, upstream_connect_time and upstream_header_time. All are measured in seconds with millisecond resolution.

log_format apm '"$time_local" client=$remote_addr '
               'method=$request_method request="$request" '
               'request_length=$request_length '
               'status=$status bytes_sent=$bytes_sent '
               'body_bytes_sent=$body_bytes_sent '
               'referer=$http_referer '
               'user_agent="$http_user_agent" '
               'upstream_addr=$upstream_addr '
               'upstream_status=$upstream_status '
               'request_time=$request_time '
               'upstream_cache_status="$upstream_cache_status" '
               'upstream_response_time=$upstream_response_time '
               'upstream_connect_time=$upstream_connect_time '
               'upstream_header_time=$upstream_header_time';

Save and close file. Then open your Nginx virtual host configuration file. For example,

sudo nano /etc/nginx/conf.d/linuxbabe.com.conf

Your Nginx virtual host configuation file might be under /etc/nginx/sites-enabled/ directory. I like to use the /etc/nginx/conf.d/ directory.

In the server {...} context, add the following two lines to enable access log and error log. The access log is using the apm format and error log uses the warn log level.

access_log /var/log/nginx/linuxbabe.com.access.log apm;
error_log /var/log/nginx/linuxbabe.com.error.log warn;

Save and close the file. Then reload Nginx.

sudo systemctl reload nginx

Now I can see the upstream response time and other performance related metrics in the Amplify graphs. A good upstream response time is below 0.5 seconds. Otherwise your visitors can feel your website is slow. If you can tune your server to get upstream response time below 0.2 seconds, your website will be very snappy.

How do you find out which HTTP request takes long time? You can search the Nginx access log for the word pattern “request_time=1“, which shows you the requests that take more than 1 second to complete. Hint: If you set up Nginx FastCGI cache, the request_time will be reduced to 0 seconds for cached pages.

Possible causes of long request time:

• MySQL/MariaDB database is slow.
• The server is under heavy load. (Use the htop command to check load averages.)

If Amplify agent isn’t reporting Nginx metrics, check that Amplify agent runs as the same user as the Nginx worker process. You may also need to check if there’s a new version of Amplify-agent available.

How to Exclude Your Own IP Address in Nginx Access Log

Sometimes, your activity can skew the Nginx metrics. You can exclude your own IP address in the access log to avoid that. First, add the following lines in the http context of Nginx configuration file.  Replace 12.34.56.78 with your own IP address. If the HTTP request comes from your own IP address, Nginx will set the value of $log_ip variable to 0.

map $remote_addr $log_ip {
    "12.34.56.78" 0;
    default 1;
}

Then change the access_log directive as follows.

access_log /var/log/nginx/linuxbabe.com.access.log apm if=$log_ip;

Save and close the file. Then text nginx configuration and reload.

sudo nginx -t
sudo systemctl reload nginx

Note: If you use the WordPress Popular Posts plugin on your WordPress site, then there will be lots of HTTP POST requests sent from visitors’ browsers to your web server, and this can also skew the request_time in Nginx Amplify. To minimize this impact, you can go to Settings -> WordPress Popular Posts -> Tools and set log views from to logged-in users only. This will also reduce your server load.

How to Monitor MariaDB/MySQL Database Server

First, we need to create a user in MariaDB/MySQL to collect metrics. Log into MariaDB/MySQL monitor.

sudo mysql -u root

Create a new user for Amplify agent. Replace your_password with your preferred password. (It’s recommended to choose a password that is different from the MariaDB/MySQL root password.)

create user 'amplify-agent'@'localhost' identified by 'your_password';

Exit MariaDB/MySQL server.

exit;

Next, we need to enable the Amplify MariaDB/MySQL plugin in the Amplify configuariton file.

sudo nano /etc/amplify-agent/agent.conf

In the [extensions] section, you can see that mysql plugin is disabled by default.

[extensions]
phpfpm = True
mysql = False

Change False to True to enable this plugin.

[extensions]
phpfpm = True
mysql = True

Then in [mysql] section, find the following line.

password = amplify-agent

Replace the default password with the password you set for the amplify-agent user. Save and close the file. Then restart Amplify agent for the changes to take effect.

sudo systemctl restart amplify-agent

Now Amplify agent starts collecting MariaDB/MySQL metrics.

How to Enable MariaDB Slow Query Log

The slow query log can show you which queries are taking long time to give a response. It’s an important tool for optimizing MariaDB performance. To enable the MariaDB slow query log, edit the server configuration file.

sudo nano /etc/mysql/mariadb.conf.d/50-server.cnf

Find the following 4 lines:

#slow_query_log_file = /var/log/mysql/mariadb-slow.log
#long_query_time = 10
#log_slow_verbosity = query_plan,explain
#log-queries-not-using-indexes

Remove the # symbol and change the value of long_query_time to a low number like 1 second. (You can specify a smaller value like 0.5.)

slow_query_log_file = /var/log/mysql/mariadb-slow.log
long_query_time = 1
log_slow_verbosity = query_plan,explain
log-queries-not-using-indexes

Also, add the following line to enable the slow query log.

slow_query_log = 1

Save and close the file. Restart MariaDB for the changes to take effect.

sudo systemctl restart mariadb

or

sudo systemctl restart mysql

Now Nginx Amplify can show you slow queries in the graphs.

If your MySQL/MariaDB is slow, the first thing you need to do is to check QPS (Queries per second), which is a metric that monitors the load on the database server. As you can see from the following screenshot, one of my MariaDB database servers is under heavy load (around 1000 queries per second).

You can also check QPS from command line.

sudo mysqladmin status

Sample output:

Uptime: 405276 Threads: 11 Questions: 389965946 Slow queries: 4536289 Opens: 103692 Flush tables: 3 Open tables: 3999 Queries per second avg: 962.223

Enable Slow Query Log in MySQL 8

If you use MySQL 8 instead of MariaDB, then edit the server config file.

sudo nano /etc/mysql/mysql.conf.d/mysqld.cnf

Add the following lines at the end of this file.

slow_query_log_file = /var/log/mysql/slow-query.log
long_query_time = 1
log-queries-not-using-indexes = ON
slow_query_log = 1

Save and close the file. Restart MySQL server for the changes to take efffect.

sudo systemctl restart mysql

How to Monitor PHP-FPM

The Amplify agent PHP-FPM plugin is enabled by default, but we need to enable PHP-FPM status in order to collect metrics. Edit the PHP-FPM pool configuration file.

Ubuntu PHP7.4

sudo nano /etc/php/7.4/fpm/pool.d/www.conf

Find the following line and remove the semicolon to enable PHP-FPM status.

;pm.status_path = /status

It’s also a good idea to enable the PHP slow log, which can show you which requests are taking too long. Find the following line:

;slowlog = log/$pool.log.slow

Remove the semicolon and set the log path.

slowlog = /var/log/php-fpm/slow.log

Then find the following line.

;request_slowlog_timeout = 0

Remove the semicolon and set the timeout seconds like 1s.

request_slowlog_timeout = 1s

Save and close the file. Create a directory for PHP-FPM logs.

sudo mkdir -p /var/log/php-fpm/

Restart PHP-FPM for the changes to take effect.

sudo systemctl restart php7.4-fpm

Now Amplify agent starts collecting PHP-FPM metrics including the slow requests log.

Note that if you install a new version of PHP-FPM, like PHP8.0-FPM, you need to configure this again in /etc/php/8.0/fpm/pool.d/www.conf file.

The Amplify Overview Page

On the overview page, you can see the overall application health score and 5 key metrics for your system.

• Total requests
• HTTP 5xx errors
• Request time (You need to create custom log format as described earlier to show this metric.)
• How much bandwidth Nginx uses
• CPU usage

The application health score is affected by the number of HTTP 5xx errors. If there’s no HTTP 5xx errors, then the score will be 100%.

If you click the server icon, you can see how much CPU and RAM has been used on your system and how much Nginx is using.

Graphs

In the Graphs page, you can see predefined system, Nginx, MariaDB/MySQL and PHP-FPM metrics.

Nginx Amplify Dashboards

Amplify dashboards allow you to create custom graphs and charts and group them on one page. For example, you can assemble all performance related graphs on one page, or maybe you want to display all metric for a particular URL.

If you have configured Nginx FastCGI cache, you can also create graphs for Nginx cache hit ratio and cache expiration in the dashboard.

Analyzer

One of the useful tools in the analyzer is static analysis. Amplify can parse your Nginx configuration file and make performance, security and reliability recommendations. You can use it to identify mistakes and improve configurations.

As you can see, there are 5 warnings for my Nginx server, so I click the Open link to see how to adjust my Nginx configurations.

After making the adjustments, wait a few minutes and the warning will be gone.

Alerts

By default, if the Amplify agent stops working, an alert will be sent to your email address. On the alerts page, you can set various alerts for your server. For example, I can tell Amplify to send me an email if the upstream response time is above 1 second for the past 2 minutes.

If your server runs of out RAM, then it could kill MySQL/MariaDB process. You can create an alert rule to monitor the MySQL/MariaDB uptime like below. If the uptime is below 120 seconds over the past 2 minutes, then send an email notification.

You may also want to receive alert email when there’s a 5xx error on your server.

Amplify Agent Stops Working

Often I found that Amplify agent stops sending performance metrics for unknown reason. I create a Cron job to restart Amplify-agent once a day to solve this problem.

sudo crontab -e

Add the following line.

@daily systemctl restart amplify-agent

MariaDB Monitoring with Galera Cluster

It seems the Amplify agent will stop sending MariaDB metrics if its in a Galera cluster.

Linux Server Performance Monitoring

Nginx Amplify is useful for monitoring Nginx, MySQL/MariaDB, and PHP performance, but how about the performance of the underlying hardware/operating system? How do you know if your disks are slow? You can use Netdata to peek into those details.

• Linux Server Performance Monitoring with Netdata

Conclusion

I hope this article helped you install and use Nginx Amplify to monitor performance of Nginx or LEMP stack applications on Ubuntu 18.04 or 20.04. You might also want to read:

• How to Install Multiple Versions of PHP on Ubuntu

As always, if you found this post useful, then subscribe to our free newsletter to get more tips and tricks. Take care 🙂