•

My WordPress site goes down when bots crawl it

A lot of your site’s traffic isn’t people. Search engines, AI crawlers, SEO tools, price checkers and vulnerability scanners all visit, some politely and some not. Most of the time the server copes. But when a burst arrives and the requests are for pages that WordPress has to build from scratch, a server that’s fine for real visitors can fall over in minutes, and it’s often back up by the time anyone checks.

Is this what you’re seeing?

  • Your WordPress website keeps crashing, or goes down when bots crawl it, then recovers by itself.
  • Your hosting server keeps running out of memory, or your host says you’ve hit your resource limits.
  • The site crashes under traffic spikes, though the spikes in your analytics are small.
  • The logs show 500, 502, 503 or 504 errors, timeouts, and lots of requests from a few addresses or unfamiliar bots.
  • On your own server, you see PHP-FPM errors, cache connection errors or “out of memory” messages.
  • Restarting PHP or the server fixes it, until the next time.

What usually causes it

Bots reaching pages that aren’t cached

A cached page costs the server almost nothing. An uncached one makes WordPress run PHP and query the database. Bots are very good at finding the uncached ones: search results, filtered shop pages, “add to basket” links, and event calendars, which can offer an endless run of next-month and next-day pages. A crawler following all of them can generate thousands of uncached requests. How to tell: the access log shows a bot requesting addresses with ? in them, search pages, or calendar dates far into the future.

More PHP workers allowed than the memory can hold

Each request that WordPress handles runs in a PHP process, often managed by a service called PHP-FPM. The server’s settings say how many can run at once. Each one uses memory, and a busy WordPress site’s processes can be large. If the limit is set so high that all of them together need more memory than the server has, a burst of traffic doesn’t queue politely: it uses up the memory, and the system starts killing processes, sometimes the database or the cache, to stay alive. Set too low, the opposite happens: requests queue and time out. How to tell: on your own server, the logs show memory warnings or “out of memory” events around the same time as the errors.

Scanners and badly behaved crawlers

Automated scanners try hundreds of addresses in seconds looking for weaknesses, such as .env files or old plugin files. Some AI and SEO crawlers ignore polite limits and fetch pages as fast as the server will answer. Slow or unfinished requests can tie up workers far longer than a normal visit. How to tell: lots of requests for files that don’t exist on your site, or one user agent making a large share of all requests. I describe a case like this in the site that looked down even though the server was fine.

Shared hosting limits

On shared hosting you can’t change PHP-FPM yourself, but the same thing happens inside your account’s limits. When bots push you past your share of processes or memory, the host slows or pauses your site, usually with a 503 or a “resource limit reached” page. How to tell: your control panel’s resource-usage graphs show limits being hit at the times the site went down.

What you can safely try

Take a backup first, at a quiet time. These steps don’t change your content, and each one can be undone.

  1. Note the times the site went down. Ask your host for the access and error logs for those minutes, or look at your control panel’s access logs or visitor statistics. Look for which bots, and which addresses, were busiest.
  2. Check that page caching is switched on and working for logged-out visitors. Open a page in a private window twice; the second load should be noticeably quicker. Your caching plugin or host can confirm it.
  3. Check the Crawl stats report in Search Console’s settings. If Google’s own crawling jumped at the same time, that’s useful to know. If it didn’t, the traffic was from other bots.
  4. If you use Cloudflare, look at its security settings. There are options to challenge bad bots and block known AI crawlers. Turn on one at a time and check that real customers can still buy and log in.
  5. Use your robots.txt to ask well-behaved crawlers to stay out of pages with no value to them, such as search results or far-future calendar views. Most SEO plugins have a robots.txt editor. Copy the current version first so you can put it back, and never block the whole site. Scanners ignore robots.txt, so this only helps with polite bots.
  6. Don’t raise PHP worker or memory limits yourself to “give it more room”. Without knowing how much memory each process uses, that can make the crash worse.

When to stop and get help

  • You run your own server or cloud instance, and the logs show out-of-memory events, PHP-FPM errors or cache connection failures.
  • The site stopped working after someone changed the PHP-FPM settings.
  • Your host is warning that your account will be suspended for using too many resources.
  • Blocking bots has started blocking genuine customers, payment notifications or Google.
  • The logs show so many different errors that you can’t tell which came first.

A real example

A membership and events site had a serious outage on its own Linux server. It wasn’t a bargain-basement setup: Nginx as the web server, PHP-FPM to run WordPress, and two layers of caching in Redis and Varnish.

The server logs from the incident showed a burst of crawler activity, followed by several different kinds of failure: Redis connection errors, HTTP 500 errors, aborted requests, network timeouts, memory pressure and finally an out-of-memory event. Any one of those, read on its own, could have sent the investigation off in the wrong direction. The Redis errors, for instance, made it look like a cache problem.

Rather than assume the first error in the log was the cause, I looked at how the application, the server’s resources and the incoming traffic related to each other as the failures happened. The server had about 4GB of memory, and PHP-FPM was allowed a very high number of processes for that amount. Under a burst of simultaneous requests, that raised a clear concern about the server exhausting its memory, which could explain so many different parts failing at once. The likely cause was resource starvation under concurrent load, rather than one faulty plugin.

The broader point applies to much smaller sites too. Downtime often comes from the way several systems interact under load: traffic, caching, PHP limits and memory. That’s why I start with the logs and the server’s configuration together, not with the most alarming error message. If your site goes up and down without a clear pattern, see my WordPress site keeps going offline as well.

Getting it fixed

Send me the times of the outages and what hosting you’re on. I read the access and error logs for those times, find what the bots were requesting and why it wasn’t cached, and look at whether the server’s PHP and memory settings fit what it has. A WordPress repair is a fixed £119, which covers finding the problem and up to 90 minutes of fixing it. Most repairs fit in that, and you’re never charged more without agreeing first. Server tuning that needs longer is £75 an hour, agreed with you first.

If your site is down and it’s costing you money right now, see emergency WordPress support for a faster start.

Questions

Can someone troubleshoot PHP-FPM problems on my server?

Yes. I need SSH access, or a hosting panel that shows the PHP-FPM settings and logs. I look at how much memory each PHP process really uses on your site, how many can safely run at once, and what’s sending the requests, then change the settings to match. If someone recently changed them and the site broke, the first step is putting back the last settings that worked.

My WordPress website is running out of server memory. Should I raise the PHP memory limit?

Those are two different limits. The PHP memory limit is how much one request may use, and raising it helps with one heavy page or import. Running out of server memory is about how many requests run at once. Raising the per-request limit on a server that’s already short can make things worse.

My site has too many PHP workers running. Is that bad?

It depends on the memory. Many busy workers on a server with plenty of memory is fine. Many on a small server, especially if they’re all serving bots, is a warning sign. The answer is usually fewer uncached requests, not just a different number.

Can you set up Cloudflare to reduce bot traffic?

Yes, including caching rules, bot challenges and rate limits. The care is in not blocking what you need: Google, payment providers’ notifications, your booking or shop integrations, and real customers on mobile networks. I test each rule against those before leaving it on.

Will blocking bots hurt my Google rankings?

Not if you leave the search engines alone. Blocking scanners and aggressive crawlers has no effect on Google. Blocking Googlebot, or the whole site in robots.txt, would. If you’re unsure whether a rule catches Google, check the Crawl stats report after you change it.

Do I just need faster hosting?

Sometimes, but a bigger server serving the same uncached bot traffic only lasts longer before it falls over. Fix what’s reaching PHP first, then decide whether you need more. If the site is slow for real visitors even on quiet days, see fast when logged in, slow for visitors.

Written by Peter Claridge, who fixes WordPress sites for businesses across the UK from Reading. Last checked on WordPress 7.1, November 2026.