โ€ข

How I Fixed Intermittent Cloudflare 524 Errors on an AWS Lightsail WordPress Site

If you rebooted your AWS Lightsail WordPress site and it’s not coming back, then you need to check the Linux CPU steal time and Lightsail CPU burst capacity.

In my case the server could technically boot, but it no longer had enough available CPU capacity to bring the full WordPress stack back under load. The burst capacity was exhausted, so the instance was being heavily CPU-throttled just as Apache, MariaDB, PHP-FPM and WordPress all started competing for resources.

The symptom of this was after rebooting the Lightsail instance it was intermittently returning Cloudflare 524 or 521 errors, loading very slowly, and the Lightsail dashboard showed surprisingly low CPU usage. So I went into SSH and checked Linux CPU steal time and Lightsail CPU burst capacity.

In my case, the fix was to move the site from a 1 GB / 1 vCPU Lightsail instance to a larger 2 GB / 2 vCPU instance.

The key symptoms were:

  • Lightsail CPU utilisation appeared to sit at only around 12%
  • Linux top showed 87โ€“88% CPU steal time
  • System load was above 5 on a 1-vCPU server
  • All five PHP-FPM workers were continuously busy
  • Lightsail showed 0% remaining CPU burst capacity
  • Apache was timing out waiting for PHP-FPM
  • Cloudflare intermittently returned 524 and 521 errors

After moving the site to the larger instance, CPU steal time dropped to 0%, load average fell to almost zero, and the site immediately became responsive again.

The commands that exposed the problem were:

uptime
top
free -m
ps aux --sort=-%cpu | head -20

In top, the critical value was:

87.5 st

st means CPU steal time. It indicates that the virtual machine wants CPU time but the hypervisor is not giving it enough.

The next check was in:

AWS Lightsail โ†’ Instance โ†’ Metrics โ†’ CPU burst capacity

That showed:

Remaining CPU burst capacity: 0%

That combination explained the problem.

Short version

The troubleshooting path was:

Cloudflare 524 / 521
โ†“
WordPress very slow
โ†“
AWS CPU graph only ~12%
โ†“
Linux load average > 5
โ†“
top shows ~88% CPU steal
โ†“
PHP-FPM workers saturated
โ†“
Lightsail burst capacity = 0%
โ†“
Resize Lightsail instance
โ†“
CPU steal = 0%
โ†“
Site responsive again

If you’re seeing the same combination, resizing the instance is likely the immediate fix. You should still investigate WordPress traffic, caching, bots, cron jobs and PHP-FPM afterwards, but the server needs enough CPU headroom first.

How the CPU got pegged at 100%

Here’s how I figured out what was going on.

I recently took over management of a WordPress site hosted on AWS Lightsail. The previous company preferred running their client websites in self contained instances on AWS.

A lot of extra work, but fair enough.

Everything appeared normal at first. Lightsail showed the instance as running, the domain was correctly using Cloudflare, and there had been no obvious infrastructure changes.

Then I get a request from the owner of the website.

“Shut it down, we don’t need that site any more”

*click* I shut it down, and logged off for the day

At 7:24am the next morning.

“Wait, we need the website back, can you put it online again?”

*click* I started up the instance, gave it a couple of minutes and nothing happened.

Connection timeout.

I waited 10 minutes and tried again. This time it loaded on my computer so I took a screenshot and sent it to the client. “all good” and I went to get breakfast. Five minutes later I get a message.

“Pete, the website still isn’t working”. So I opened my laptop on the kitchen table and went to the site.

Sometimes it loaded.

Sometimes it returned:

HTTP ERROR 521

At other times Cloudflare displayed:

A timeout occurred
Error code 524

Even SSH access through the Lightsail browser console started failing.

What initially looked like a WordPress problem turned out to be a much more interesting infrastructure problem.

The server had effectively run out of CPU.

Not CPU in the usual “100% utilisation” sense. AWS was reporting CPU utilisation of only around 12%.

The key was discovering that the instance was spending nearly 88% of its time waiting for CPU that Lightsail was no longer giving it.

This is how I worked it out.

The initial symptoms

The site sat behind Cloudflare, with WordPress running on an AWS Lightsail Bitnami instance.

The instance itself was tiny:

1 GB RAM
1 vCPU
40 GB SSD

Cloudflare was intermittently showing a 524 error.

A 524 means Cloudflare successfully connected to the origin server but the origin didn’t return a response quickly enough.

That immediately pointed towards the Lightsail server rather than DNS.

Things became more suspicious when the AWS Lightsail browser-based SSH connection also started failing.

At that point, it wasn’t just WordPress behaving badly. The server itself was struggling.

This page isn't working

If the problem continues, contact the site owner.

HTTP ERROR 521

Then a second Cloudflare error:

A timeout occurred

Error code 524

Browser: Working
Cloudflare: Working
Host: Error

First assumption: the server was overloaded

I rebooted the Lightsail instance.

It came back.

For a few minutes the site worked again.

Then it became slow.

Then it started timing out again.

The Lightsail CPU graph didn’t look particularly alarming. CPU utilisation was hovering around 12%.

That didn’t seem consistent with a server that was barely responding.

So I SSH’d into the machine and started checking the usual things.

First, memory:

free -m

The server had less than 1 GB RAM available in total and had started using swap, but it wasn’t completely out of memory.

Then disk:

df -h

The root volume was only around 38% full.

Then inodes:

df -i

Again, nowhere near exhaustion.

That ruled out two very common WordPress server failures.

The first big clue: load average

Running:

uptime

produced something like:

load average: 5.82, 3.79, 1.71

On a machine with one CPU core, a load average approaching six is extremely high.

Roughly speaking, it meant several processes were competing to get CPU time.

That was strange because AWS still wasn’t showing high CPU utilisation.

So I ran:

top

That was where the real problem became visible.

The CPU line looked roughly like this:

%Cpu(s): 12.7 us, 0.7 sy, 0.0 id, 86.6 st

The important part was:

86.6 st

st means steal time.

That means the virtual machine wanted CPU time, but the hypervisor wasn’t giving it CPU time.

In other words, the operating system had work ready to execute but the underlying infrastructure was preventing it from running.

The terminal looked something like this:

top - 08:26:53 up 7 min

load average: 5.82, 3.79, 1.71

%Cpu(s):
12.7 us
0.7 sy
0.0 id
86.6 st

PID   USER     %CPU   %MEM   COMMAND
1192  daemon   20.3   16.3   php-fpm
1155  daemon   19.5   20.3   php-fpm
1184  daemon   18.0   16.4   php-fpm
1196  daemon   18.0   15.1   php-fpm
1197  daemon   18.0   15.0   php-fpm

PHP-FPM was completely saturated

The next clue was the process list.

All five PHP-FPM workers were active.

Running:

ps aux --sort=-%cpu | head -20

showed five php-fpm workers at the top.

The PHP-FPM configuration confirmed why:

grep -E 'pm.max_children|pm.start_servers|pm.min_spare_servers|pm.max_spare_servers' /opt/bitnami/php/etc/php-fpm.d/www.conf

returned:

pm.max_children = 5
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3

The server could process a maximum of five PHP requests concurrently.

All five workers were busy.

New requests therefore had to wait.

That explained the chain of failure:

Incoming request
      โ†“
All PHP workers busy
      โ†“
Request waits
      โ†“
PHP gets almost no CPU
      โ†“
Request takes tens of seconds
      โ†“
Apache waits
      โ†“
Cloudflare waits
      โ†“
524 timeout

The Apache logs confirmed it

The Apache error log contained repeated messages like:

The timeout specified has expired
AH01075: Error dispatching request

and:

Failed to read FastCGI header
Connection reset by peer

Apache itself wasn’t the underlying problem.

It was accepting the requests.

The problem was that Apache was passing those requests to PHP-FPM, and PHP-FPM couldn’t complete them quickly enough.

There was also a lot of bot traffic

Looking at the access logs revealed another contributor.

I ran:

sudo tail -n 5000 /opt/bitnami/apache/logs/access_log 
| awk '{print $7}' 
| sort 
| uniq -c 
| sort -nr 
| head -30

The result was surprising.

A very large proportion of requests were going to:

/robots.txt

There were also requests hitting WordPress cron, Wordfence scans and the usual collection of automated vulnerability probes.

One suspicious-looking request was:

/wp-content/themes/seotheme/db.php?u

At first glance that looked worrying.

But checking the filesystem showed that the file didn’t exist, and the requests had originally returned 404.

They were simply automated scanners probing WordPress sites.

Later 504 responses occurred because the whole PHP stack had become too slow to answer.

That’s an important distinction.

Seeing malicious-looking URLs in an access log does not automatically mean the site has been compromised.

I disabled Wordfence temporarily

Wordfence scans were appearing repeatedly in the logs while the server was already under pressure.

To eliminate that workload temporarily, I renamed the Wordfence plugin directory and restarted PHP-FPM.

That helped remove one variable, but the server remained extremely slow.

The next top result was even more useful:

load average: 5.14, 5.07, 4.09

%Cpu(s):
11.8 us
0.0 sy
0.0 id
88.2 st

Almost 88% steal time.

At that point it became clear that WordPress optimisation alone wasn’t going to fix the immediate problem.

The Lightsail graph that solved the mystery

AWS Lightsail instances use burstable CPU performance.

A small instance can temporarily use more CPU than its baseline by consuming accumulated burst capacity.

Once that reserve runs out, the instance can be constrained back towards its sustainable CPU level.

I opened:

Lightsail โ†’ Instance โ†’ Metrics โ†’ CPU burst capacity

The graph showed:

Remaining CPU burst capacity: 0%

That was the missing piece.

The instance wasn’t reporting 100% CPU usage because it wasn’t being allowed to consume 100% CPU.

Linux was ready to use it.

PHP needed it.

But the Lightsail instance had exhausted its burst reserve.

That explained why AWS showed roughly 12% CPU while Linux showed almost 88% steal time.

The real fix: move to a larger Lightsail instance

The existing instance was:

1 GB RAM
1 vCPU

I decided not to keep trying to squeeze more performance out of it.

Instead I created a Lightsail snapshot and launched a larger instance from it:

2 GB RAM
2 vCPU

The migration process was straightforward.

I created a snapshot of the existing server, created a new Lightsail instance from that snapshot, verified WordPress and the Bitnami services, tested the site on the temporary public IP, then moved the existing static IP to the new instance.

Because the domain was already pointing to that static IP through Cloudflare, there was no DNS migration required.

Verifying the new WordPress server

Before moving the static IP, I checked the Bitnami services:

sudo /opt/bitnami/ctlscript.sh status

The result:

apache already running
mariadb already running
php-fpm already running

I then checked WordPress locally:

curl -I http://127.0.0.1 -H "Host: example.com"

WordPress returned a valid redirect to HTTPS, confirming that Apache, PHP, WordPress and the database were all functioning together.

The site was also tested directly using the new instance’s temporary IP.

Once everything looked correct, the existing Lightsail static IP was moved from the old instance to the new one.

The difference was immediate

After starting the replacement instance, I ran:

uptime

The load average had dropped to approximately:

0.04, 0.11, 0.09

Then:

top

showed:

100.0 id
0.0 st

The server was essentially idle.

CPU steal time had dropped from almost 88% to 0%.

The WordPress site loaded normally again.

The terminal showed something like this:

load average: 0.04, 0.11, 0.09

%Cpu(s):
0.0 us
0.0 sy
100.0 id
0.0 wa
0.0 st

MiB Mem:
1947 total
951 free
518 used
477 buff/cache

The lesson: low CPU utilisation can be misleading

The part that made this incident interesting was the AWS CPU graph.

If I’d only looked at it, I might have concluded:

CPU utilisation is only 12%. CPU clearly isn’t the problem.

That would have been wrong.

The machine wanted far more CPU than AWS was giving it.

The Linux steal metric exposed that.

If you’re troubleshooting a slow virtual machine, particularly a burstable VPS or cloud instance, top can tell you something that the hosting provider’s dashboard doesn’t make obvious.

Look at:

st

If steal time is consistently high, your operating system is waiting for CPU resources controlled by the hypervisor.

Then check your hosting provider’s throttling, burst or CPU-credit metrics.

Things I still wouldn’t ignore

Moving to a larger instance fixed the immediate availability problem, but it doesn’t mean the original site was perfectly optimised.

There was still excessive automated traffic hitting the origin.

There were frequent WordPress cron requests.

Wordfence was doing scanning work.

The site had only five PHP-FPM workers.

And /robots.txt was receiving an unusually large number of requests.

The next stage is therefore reducing the amount of work that ever reaches PHP in the first place.

For this site that means making sure static files are genuinely static, improving Cloudflare caching, rate-limiting abusive bots, reviewing WordPress cron, tuning Wordfence and then reviewing PHP-FPM limits now that more RAM is available.

A larger server gives you headroom.

It shouldn’t become an excuse to ignore unnecessary workload.

Commands that helped diagnose the problem

These were the commands that proved most useful during the incident:

uptime

free -m

df -h

df -i

top

ps aux --sort=-%cpu | head -20

ps aux --sort=-%mem | head -20

ps aux | grep php-fpm | grep -v grep

ps aux | grep apache | grep -v grep

ps aux | grep -E 'mysql|maria' | grep -v grep

sudo /opt/bitnami/ctlscript.sh status

grep -E 'pm.max_children|pm.start_servers|pm.min_spare_servers|pm.max_spare_servers' 
/opt/bitnami/php/etc/php-fpm.d/www.conf

sudo tail -n 200 /opt/bitnami/apache/logs/error_log

The single most important value in my case was hiding inside top:

88.2 st

That one number eventually explained why a server apparently using only around 12% CPU was unable to serve a WordPress page reliably.

Final takeaway

A Cloudflare 524 error doesn’t necessarily mean Cloudflare is the problem.

A running Lightsail instance doesn’t necessarily mean the server is healthy.

And low CPU utilisation in a cloud dashboard doesn’t necessarily mean CPU isn’t the bottleneck.

In this case the useful sequence was:

Cloudflare 524
โ†’ check origin server
โ†’ inspect Linux load
โ†’ inspect PHP-FPM
โ†’ notice high CPU steal
โ†’ check Lightsail burst capacity
โ†’ discover 0% remaining
โ†’ resize instance
โ†’ steal time falls to 0%
โ†’ site becomes responsive

The diagnosis only became obvious when the application metrics, Linux metrics and AWS metrics were looked at together.

That is the part I’d keep at the centre of the article.


I would also make the post visually distinctive by using reconstructed screenshots rather than your actual screenshots. Keep the same visual language as AWS, Cloudflare and a terminal, but replace the domain, IP addresses, timestamps and process IDs. You don’t actually need photorealistic screenshots. Clean diagrammatic recreations in Canva will probably make the article easier to understand.

I’d also create one hero diagram showing:

Visitor
   โ†“
Cloudflare
   โ†“
Apache
   โ†“
PHP-FPM
   โ†“
WordPress / MariaDB
   โ†“
AWS Lightsail CPU

                โ†‘
        0% burst capacity
        88% CPU steal