What is Semrush Bot and Why Is It on Your Site?

What SemrushBot Actually Is

A bot — also called a web robot, web spider, or web crawler — is software designed to automatically browse the internet and perform repetitive tasks far faster than any human could. The most common job for bots is web crawling: visiting pages, following links, and collecting data.

SemrushBot is Semrush’s version of this. It’s the crawler Semrush sends out continuously to discover new web data and keep its index fresh.

Here’s what that means in practice: every time a Semrush user checks backlinks for a competitor’s domain, runs a Site Audit, or pulls keyword data, a lot of that information ultimately traces back to what SemrushBot collected while crawling the web.

What It’s Actually Collecting From Your Site

This is where it gets specific. SemrushBot doesn’t crawl your site for one purpose — it feeds data into several different Semrush tools simultaneously. Based on Semrush’s official documentation, here’s what the collected data powers:

• Backlink Analytics — the public backlink index (also called the webgraph of links)
• Site Audit — on-page SEO, technical issues, and usability analysis
• Backlink Audit — identifying potentially harmful backlinks in your profile
• Link Building Tool — finding link prospects and monitoring acquired backlinks
• SEO Writing Assistant — checking URL accessibility
• On Page SEO Checker & Content Template — content analysis and recommendations
• Topic Research — surfacing relevant content ideas
• SplitSignal — supporting SEO A/B testing
• Content Toolkit — content-level reporting
• Plagiarism Checker — verifying if content appears on your site

There’s also RyteBot — a separate crawler that Semrush uses via Ryte.com for additional on-page and technical analysis — and SemrushBot-ESI for Semrush’s Enterprise Site Intelligence product.

One thing most people miss: SemrushBot isn’t a single crawler. It’s a family of specialized bots, each with its own user-agent string targeting a specific tool.

The Full List of SemrushBot User-Agents

If you’re checking your server logs or planning a robots.txt block, knowing which user-agent belongs to which tool matters. Here’s the complete breakdown:

In my experience reviewing server logs across client sites, SiteAuditBot and SemrushBot tend to be the most frequent visitors — especially on sites that other SEOs are actively auditing or tracking through Semrush.

How the Crawl Process Works

SemrushBot’s process is straightforward but worth understanding.

It starts with a list of URLs. When it visits a page, it saves every hyperlink it finds and adds those to what’s called the “crawl frontier” — essentially a queue of URLs to visit next. This list gets revisited repeatedly based on Semrush’s internal policies, which means SemrushBot comes back to your site periodically to catch content changes, new pages, and dead links.

It’s not a one-time visit. It’s an ongoing cycle designed to keep Semrush’s data current.

Should You Block SemrushBot?

Honestly, most sites shouldn’t. SemrushBot isn’t harvesting your content to compete with you — it’s building an index that SEO professionals use to analyze the web. If your site shows up in Semrush’s backlink data or Site Audit results, that’s SemrushBot doing its job.

That said, there are legitimate reasons to block it:

• You’re running a private or staging environment
• You want to conserve crawl budget on a large site
• You don’t want competitors to audit your site through Semrush tools

The method is simple: robots.txt.

Personally, I’ve never blocked SemrushBot on any of the sites I manage — and there’s a practical reason for that. SemrushBot’s crawl is what keeps Semrush’s data about your site accurate. If you’re running regular Site Audits, the crawler needs fresh access to your pages to surface real issues. Block it, and you’re essentially auditing stale data. Beyond that, backlink data, indexability signals, and on-page analysis all depend on SemrushBot being able to reach your pages freely. For SEO work, that visibility matters more than the minor crawl overhead.

How to Block SemrushBot Using robots.txt

Blocking via IP address won’t work — Semrush explicitly states they don’t use consecutive IP blocks. The only reliable method is robots.txt.

To block the primary backlink crawler:

User-agent: SemrushBot
Disallow: /

To block the Site Audit bot:

User-agent: SiteAuditBot
Disallow: /

You can stack multiple user-agent blocks to cover all SemrushBot variants. Refer to the full user-agent list in the table above for tool-specific blocks.

A few technical rules to keep in mind:

• The robots.txt file must be placed in the top directory of your web host, not a subdirectory. Otherwise, SemrushBot will ignore it.
• If you have subdomains, each one needs its own robots.txt file. A robots.txt on the root domain doesn’t automatically cover subdomains.
• Your robots.txt must return an HTTP 200 status code. If it returns a 4xx error, SemrushBot assumes no file exists and will crawl freely. A 5xx response will block crawling of the entire site. A 3xx redirect is handled fine.
• After updating robots.txt, it can take up to one hour or 100 requests for SemrushBot to discover and apply the changes.

SemrushBot for Backlink Analytics also respects two non-standard extensions:

• Crawl-delay — it honors delays up to 10 seconds. Anything higher gets reduced to the 10-second limit automatically.
• Wildcard (*) rules — it supports wildcard patterns in robots.txt.

Semrush Bots Block List Table

The table below lists each tool, the name of its robot, and the method for blocking it in robots.txt:

Code to Block All Bots at Once

If you want to add all these bots to your robots.txt file simultaneously, you can copy and paste this code:

User-agent: SiteAuditBot
Disallow: /

User-agent: SemrushBot-BA
Disallow: /

User-agent: SemrushBot-SI
Disallow: /

User-agent: SemrushBot-SWA
Disallow: /

User-agent: SplitSignalBot
Disallow: /

User-agent: SemrushBot-OCOB
Disallow: /

User-agent: SemrushBot-FT
Disallow: /

User-agent: RyteBot
Disallow: /

User-agent: SemrushBot-ESI
Disallow: /

A Few Quirks Worth Knowing

“Why is SemrushBot trying to crawl a URL that doesn’t exist?”

If SemrushBot detects a non-existent page on your site, it stops. But if other sites still link to that dead URL across the web, the crawler may continue checking it — because the link exists externally even if the page doesn’t. This is expected behavior.

“Why is SemrushBot attempting to log in or submit forms?”

If your login or survey forms use the GET method, the input data becomes part of the URL, which makes it accessible to any crawler following links. Semrush recommends switching those forms to the POST method to prevent this.

“SemrushBot is ignoring my robots.txt after a site migration.”

If you migrated from HTTP to HTTPS, make sure your robots.txt file was also migrated and is correctly served at the new URL. The crawler looks for it at the current domain location.

Where to Go If You Need Support

If SemrushBot continues crawling pages, it shouldn’t after you’ve updated robots.txt. You can contact Semrush directly at bot@semrush.com. Include your website URL and the relevant server log entries showing the unexpected crawl behavior — they’ll investigate.

For a broader understanding of how web crawlers and robots.txt interact, the official reference is robotstxt.org.