Your Magento store is running along nicely. Customers are browsing, orders are flowing in. Everything’s humming.
Then—without warning—checkout starts lagging. Page loads spike. Error messages appear. Your support team gets messages like “Why is your site so slow?” or “I can’t add to cart.”
But traffic hasn’t gone up. There’s no promo running. What’s happening?
Bots.
Not bad bots, necessarily. Just busy ones. Crawlers, scrapers, AI agents… sometimes helpful, sometimes annoying, always resource-hungry.
And here’s the thing: if your store is falling over because a few bots are crawling it, the real issue might not be the bots at all.
It might be that your hosting stack just isn’t ready for real-world load.
Let’s talk about that—and what to do about it.
🏎️ Start With the Stack: Why Magento Crumbles Under Load
Many Magento stores are built on shaky foundations.- Old or underpowered hardware
- Generic LAMP stacks
- Inefficient PHP configurations
- Heavy reliance on Varnish, without proper tuning
- No cache warming
- Redis setups that collapse under concurrent load
- Questionable coding decisions or extensions
At Kualo, we see it all the time. Magento sites suffering because the platform hasn’t been engineered to scale.
That’s why optimisation has to come first.
Want the deep dive? Read our post:
👉 Supercharging Magento: Mastering Speed, Security and Scalability for E-Commerce Success
Here’s the short version:
⚙️ LSAPI: PHP at the Speed of “Now”
Magento is built on PHP. And PHP is… not known for being snappy under pressure.Unless you’re using LSAPI, the LiteSpeed API handler that outperforms traditional methods like FastCGI and PHP-FPM.
Unlike those older handlers, LSAPI doesn’t spin up brand-new PHP workers for every request. Instead, it keeps a pool of master processes in memory, and when a request comes in, it forks a lightweight child process from one of those masters.
That’s important, because:
- Forking is faster than spawning a new process from scratch
- Memory is shared, reducing overhead and duplication
- Startup time is near-instant, so pages render faster
- System resources are used more efficiently, which means better performance under load
- Faster page generation
- Lower CPU usage
- Greater resilience when traffic spikes
🚀 LiteSpeed + LiteMage: Supercharged Caching
Caching is everything in Magento.With LiteMage, our stack caches more pages (even for logged-in users) using intelligent Edge Side Includes (ESI) and selective hole-punching. That means:
- Logged-out and logged-in pages are served fast
- Checkout and cart updates remain dynamic
- Most of your store is delivered from cache, instantly
🧠 KeyDB (Not Redis): Concurrency Without Choking
Redis is single-threaded, which means it can only process one request at a time per core—regardless of how many are coming in. On busy stores, where hundreds or even thousands of simultaneous requests are hitting your backend—especially from bots—Redis can become a bottleneck.That’s because it’s handling:
- Session storage
- Backend cache reads/writes
- Locking for shared data structures
That's why we use KeyDB—a multi-threaded drop-in Redis replacement that handles concurrency far better. It’s faster, lighter, and doesn’t cause your CPU to burst into flames when traffic spikes—especially the kind you didn’t ask for.
🔥 Cache Warming: Beat the Bots to the Punch
Even the fastest Magento stack can be brought to its knees if bots or users are constantly hitting uncached pages. And that’s not unusual—especially for stores with frequent product changes, layered navigation, or multi-store setups.This is where cache warming becomes critical.
Without cache warming, purged or lesser-visited pages must be rebuilt on the fly, which means triggering full Magento execution: PHP, database queries, sessions, object caching, and more. Multiply that by hundreds of simultaneous bot requests and you’ve got a performance crisis.
This risk can be mitigated by using a Cache Warmer, such as Amasty’s Full Page Cache Warmer, which continuously regenerates your site’s cache in the background. Here’s how it helps:
- Controlled crawl behaviour: The cache warmer sets the pace, so Magento pages are recached at a manageable rate—unlike bots, which can crawl erratically and cause spikes.
- Rapid rehydration after purges: When pricing or stock changes invalidate cache, the system quickly regenerates affected pages without waiting for organic traffic to trigger it.
- Focus on what matters: The warmer can prioritise high-traffic and high-conversion pages, ensuring they’re always lightning fast.
- Comprehensive coverage: It supports multiple currencies, customer groups, and device types, so each visitor type gets the right cached experience.
Contrast this with stores that don’t use cache warming. They may have full-page caching enabled, but in practice, large swathes of their catalogue remain uncached for long periods. When a bot starts crawling thousands of these pages in quick succession, the server is forced to dynamically generate each one—resulting in CPU saturation, slowdowns, and even downtime.
In short, cache warming shifts you from reactive to proactive. You’re not waiting for trouble—you’re preparing your store so that when it comes, you’re ready.
🤖 Enter the Bots: Helpful but Hungry
Okay, now your stack is battle-ready.But what about the bots?
Here’s the thing: most bots aren’t malicious. They’re just doing their jobs—indexing, checking prices, compiling data for AI summaries.
But some bots can overdo it. Especially on Magento stores with:
- 50,000+ products
- Multiple store views
- Deep category nesting
- Dynamic layered navigation
- Multiple currencies
- Or a hosting platform that doesn’t scale well
🚫 Don’t Block. Rate Limit.
Some hosts hit the panic button. They try to block bots with robots.txt, server firewalls, or Cloudflare’s Bot Fight Mode.Now of course many bots don’t even respect robots.txt, but even if you did manage to block them, it means:
- You might impact SEO crawling
- Limited AI visibility
- The possibility of broken price feeds
- Lost partnerships
At Kualo, we take a smarter approach.
We rate limit.
🎛️ What Is Rate Limiting?
Instead of saying “No,” we say:“Whoa there, buddy. Try again in a sec.”
Bots get HTTP 429 “Too Many Requests” responses. The polite ones back off and come back later. Many AI bots—like those used by Arc or Perplexity—understand crawl budgets and adjust their speed accordingly.
This means:
- Bots get their data (eventually)
- Humans don’t get slowed down
- Your site stays fast and online
🌐 Rate Limiting at the Edge with Cloudflare
If you’re using Cloudflare, things get even better.We can apply rate limits before the traffic ever hits your server. That’s like putting a bouncer outside the nightclub, not inside the bar—filtering out trouble at the door, rather than once it’s already causing a scene.
To do this, we add a set of predefined custom WAF rules to your Cloudflare account. When needed, we can interact with these rules instantly—toggling between blocking, challenging, or allowing traffic based on what we’re seeing.
So when aggressive bots show up—AI scrapers, marketplace crawlers, or anything blatantly ignoring robots.txt—we can:
- Switch on edge-level rate limiting to slow things down
- Trigger challenges or blocks for specific user agents, IP ranges, or patterns
- Shield sensitive endpoints from excessive load—without touching your core application
- PHP workers don’t get tied up handling junk
- Cache layers like KeyDB remain responsive
- ElasticSearch doesn’t get overwhelmed by a flood of bot-originated one-off search queries
📈 What We’re Seeing, and How We Help
More and more Magento stores are being targeted by bots that go far beyond the usual Google crawl. These aren’t the old-school indexers of the past—they’re faster, less respectful, and far more disruptive.Here’s a snapshot of what we’re seeing:
- AI scrapers pulling entire catalogues at speed, often for use on third-party marketplaces—ignoring crawl-delay rules or disguising their identity with generic user agents
- Competitor price bots, hitting product pages every few seconds, especially painful for stores with dynamic pricing or layered navigation
- Proxy and residential IP abuse, simulating human users and triggering search/filter-heavy routes at scale
- Bots loading dozens of filter combinations, not to browse—but to stress test or scrape variations, often from badly built tools or malicious scripts
- Making thousands of requests per minute
- Appending unique query strings to each URL, rendering full-page caching ineffective
- Spoofing the Referer header as https://www.google.com/—to appear as though users were clicking in from search results
The impact?
- Magento tries to dynamically render every request
- PHP processes queue and multiply
- ElasticSearch gets overloaded by rapid-fire queries
- And site performance degrades for real customers at exactly the wrong moment
- Triggered edge-level blocking, cutting the issue off before it reached the server
- Returned 406 status codes, neutralising the attack while minimising SEO disruption
- Once things stabilised, we applied ongoing rate limits to keep future activity in check
🧠 Think Bigger: Optimisation + Control = Resilience
Bot traffic is still traffic—some of it useful, some of it not.The answer isn’t to shut it all out.
But when bots go from curious to chaotic, you need the power to slow them down—or shut them out—with precision.
It’s about building a platform that can handle the load, and the tools to control what gets through.
With:
- LiteSpeed and LiteMage for powerful, Magento-native caching
- LSAPI for efficient, scalable PHP performance
- KeyDB for high-concurrency session and object caching
- Smart cache warming to keep pages fast—even for bots
- Granular rate limiting at both the server and the Cloudflare edge
- Real-time insights and expert guidance to take action—fast
💬 Struggling with bot traffic, slowdowns, or Magento instability?
You're not alone—and our Magento hosting solutions are built to handle exactly that.Book a free consultation with our team. We’ll ask a few quick questions about your current setup, identify what’s causing performance issues, and walk you through how we can solve them.
Every Magento store is unique, and so is our approach. Whether you’re on shared hosting, running your own VPS, or need serious firepower, we’ll match the solution to your needs.
From fully optimised shared environments to powerful bare metal servers with AMD EPYC CPUs clocked at up to 5.7GHz, we build solutions that are engineered for Magento’s quirks—and built to handle whatever bots, customers, or campaigns throw your way.
Talk to us at Kualo—and take the first step towards a store that performs the way it should.
