We swept 10,066 live AI tools in October 2026: 54% serve llms.txt, 4% have a Wikidata item, 535 block AI training crawlers and 75 block AI answers.


Mikhail Savchenko12 min
Contents10 sections
  1. Key numbers
  2. How do AI tools describe themselves to machines?
  3. Do AI tools block AI crawlers?
  4. What do AI tools cost to start?
  5. How ready are AI tools' own sites for AI search?
  6. What does this mean if you sell an AI tool?
  7. How we measured
  8. Download the data
  9. How to cite
  10. Changelog

Between 30 September and 7 October 2026, the INITE Atlas crawled the websites of 10,066 live AI tools and recorded what each one publishes for machines: llms.txt, structured data, sitemaps, robots.txt rules for AI crawlers, public registry entries and the entry price on its pricing page. The most surprising result is about crawlers. 535 AI tools block at least one crawler that collects training data, but only 75 block an agent that fetches pages for live AI answers. AI companies are seven times more willing to stay out of model training than out of ChatGPT's or Perplexity's replies.

Key numbers

#154%Of 10,066 AI tools swept between 30 September and 7 October 2026, 5,473 (54%) served an llms.txt file.

#279%Of 9,980 AI tools whose homepage the October 2026 sweep could parse, 7,887 (79%) carried Organization schema.

#392%Of 10,066 AI tools swept in October 2026, 9,257 (92%) published a sitemap.

#44%Of 10,066 AI tools swept in October 2026, only 408 (4%) had a Wikidata item.

#55.3%In the October 2026 sweep, 535 of 10,066 AI tools (5.3%) blocked at least one AI training crawler in robots.txt, while 75 (0.7%) blocked at least one AI retrieval agent.

#6375Bytespider (375 tools) and CCBot (369 tools) were the AI crawlers most often blocked by the 10,066 AI tools swept in October 2026; GPTBot followed with 258.

#71,004The median homepage of 9,980 AI tools swept in October 2026 carried 1,004 words in its server HTML, and 382 homepages carried fewer than 100.

#8$20Across 3,868 AI tools with a USD monthly entry price read in September and October 2026, the median entry plan cost $20 a month, with quartiles at $10 and $49.

#971%Of 7,930 AI tools whose pricing the Atlas could read in September and October 2026, 5,630 (71%) offered a free tier or free plan, and 444 (5.6%) sold only through sales.

#1058/100The median Atlas Readiness score across 10,066 AI tools in the October 2026 sweep was 58 out of 100 (quartiles 54 and 62); 251 tools scored 80 or above and 1,440 scored below 50.

AI tools that block each AI crawler in robots.txtOf 10,066 AI tools swept 30 September to 7 October 2026, the most-blocked training crawlers are Bytespider (375) and CCBot (369); the most-blocked retrieval agent is ChatGPT-User (62).AI training crawlers, blocked by 535 toolsBytespider: blocked by 375 of 10,066 toolsBytespider375CCBot: blocked by 369 of 10,066 toolsCCBot369GPTBot: blocked by 258 of 10,066 toolsGPTBot258Google-Extended: blocked by 194 of 10,066 toolsGoogle-Extended194ClaudeBot: blocked by 188 of 10,066 toolsClaudeBot188Applebot-Extended: blocked by 155 of 10,066 toolsApplebot-Extended155Meta-ExternalAgent: blocked by 154 of 10,066 toolsMeta-ExternalAgent154Amazonbot: blocked by 154 of 10,066 toolsAmazonbot154AI retrieval agents, blocked by 75 toolsChatGPT-User: blocked by 62 of 10,066 toolsChatGPT-User62PerplexityBot: blocked by 48 of 10,066 toolsPerplexityBot48OAI-SearchBot: blocked by 30 of 10,066 toolsOAI-SearchBot30Perplexity-User: blocked by 29 of 10,066 toolsPerplexity-User29Claude-User: blocked by 25 of 10,066 toolsClaude-User25Claude-SearchBot: blocked by 23 of 10,066 toolsClaude-SearchBot23Bingbot: blocked by 19 of 10,066 toolsBingbot19

Tools that block each crawler in robots.txt, out of 10,066 live AI tools, INITE Atlas sweep of 30 September to 7 October 2026. One tool can block several crawlers. Data: atlas-ai-tools-2026-10.csv.

This chart is free to republish with a link to this page.

How do AI tools describe themselves to machines?

What we found. The basics are nearly universal: 92% publish a sitemap and 79% carry Organization schema. llms.txt, a file that most of the web still ignores, sits on 54% of AI tools' sites. The step that takes outside work is the rare one: only 408 tools (4%) have a Wikidata item, a public record that knowledge graphs draw on to confirm that a company exists and which site is its own. The median homepage carries 1,004 words in the HTML the server sends, so most of these sites can be read without running JavaScript. 382 send fewer than 100 words, which leaves a crawler that does not run scripts little to read.

So what. Files a team can add in an afternoon have spread fast. Identity, the part an AI engine uses to decide that "Acme AI" in a question is the same company as acme.ai, has not. When every competitor in a category has an llms.txt and a sitemap, those files stop separating anyone.

Now what. If you run an AI tool's site, check the server HTML of your homepage before anything else, then the entity record. Whether llms.txt deserves your time is a separate question; we weighed the evidence in is llms.txt dead.

Do AI tools block AI crawlers?

What we found. Rarely, and selectively. 535 tools block at least one training crawler (Bytespider, CCBot, GPTBot, Google-Extended, ClaudeBot, Applebot-Extended, Meta-ExternalAgent, Amazonbot). Only 75 block at least one retrieval agent (ChatGPT-User, PerplexityBot, OAI-SearchBot, Perplexity-User, Claude-User, Claude-SearchBot, Bingbot). Retrieval access is fully open on 5,456 sites and fully closed on 16. The remaining 4,594 are "partial": their robots.txt fences some paths from retrieval agents, which can be nothing more than an admin area closed to every bot, so it is not a sign of hostility to AI.

So what. AI companies separate the two jobs. Keeping your content out of the next model's training set costs you little; keeping it out of the answer a buyer reads today costs you the sale. The robots.txt files show vendors acting on exactly that trade.

Now what. Write rules per agent, not one blanket rule for "AI bots". Our AI crawler allowlist for 2026 lists which user agent does which job.

What do AI tools cost to start?

What we found. The Atlas could read pricing for 7,930 tools. 71% offer a free tier or free plan, and 4,292 are freemium. 702 bill by usage and 1,693 by plain subscription. Among the 3,868 tools that print a USD price per month, the median entry plan is $20: 908 start under $10 and 396 start above $100. The 462 tools that price per user have a median of $19 per user per month.

Median entry price by category, for the twelve largest categories in the Atlas:

CategoryTools in the AtlasWith a USD monthly priceMedian entry price
Developer tools2,9781,065$19.99
Marketing2,1601,084$25
Automation1,601644$20
Analytics696259$30
Finance650245$19.99
Sales505228$49
Security460119$19
HR423121$29
Customer support301128$29
Agents294134$24
Legal19129$49
Health18239$14.99

The legal and health medians rest on 29 and 39 prices, so read them as rough.

So what. The common entry price is $20 to $25, and tools that sell to sales or legal teams start at twice that. AI visibility trackers are a pricier corner: across the 176 in the Atlas, the median USD entry plan is $49 a month (quartiles $29 and $99), and 124 of the 166 with readable pricing have a free tier.

Now what. If you are pricing a new tool, the category median is the number a buyer has already seen. Pricing above it needs a reason written on the pricing page itself.

What we found. Atlas Readiness scores each site out of 100 on four measured parts: entity (25 points), machine-readable content (35), retrieval access (20) and footprint (20). A part we could not measure is left out and the score is rescaled, not zeroed. The median is 58, half of all tools fall between 54 and 62, 251 score 80 or above and 1,440 score below 50.

So what. The distribution is narrow because most sites do the same easy things and skip the same hard one. Machine-readable signals are common across the catalogue; the entity part, with Wikidata at 4%, is where the gap sits.

Now what. A score in the 50s is ordinary for an AI tool, not a failure. Moving into the 80s means earning a public identity and a footprint beyond your own domain.

What does this mean if you sell an AI tool?

Your competitors already have the files. Assume the llms.txt, the sitemap and the schema are table stakes, and spend effort on what most of them lack: a homepage readable without JavaScript, a resolvable public identity, and robots.txt rules that let retrieval agents in.

Then measure whether assistants actually name you. Two of our guides cover the tooling: AI visibility tools that track mentions and citations, and generative engine optimization tools that change what engines can read. To see your own site scored on the same parts as this dataset, run the free check.

How we measured

A weekly crawler reads each tool's own website: the homepage as the server sends it, robots.txt, sitemap, llms.txt and pricing page. It also queries a few public registries, Wikidata among them. A tool counts as live when its own site answered during the sweep window, and every tool here was swept between 30 September and 7 October 2026. Denominators differ where a page could not be parsed: Organization schema and homepage words are counted over the 9,980 homepages we could read.

Prices are what each vendor's pricing page showed when the Atlas read it in September or October 2026. The USD figures cover only tools whose entry price is printed in US dollars per month (3,868); per-seat, usage-based and other currencies are counted separately or left out.

The limits matter. A robots.txt rule states intent; it does not prove that a crawler obeys it. A missing price means the page did not print one, not that the tool is free. And the Atlas measures AI tools, not the web at large. For context only, SE Ranking found llms.txt on 10.13% of the domains it checked (SE Ranking), a different population from ours.

Download the data

Both files are licensed under CC BY 4.0. Reuse them with credit to the INITE Atlas and a link to this page.

How to cite

INITE Atlas, "What 10,066 AI tools publish for machines", October 2026, https://inite.ai/en/blog/ai-tools-statistics.

Changelog

2026-10-07: first edition. Refreshed at least every 13 weeks.

Frequently Asked Questions

  • How many AI tools are there in 2026?

    Nobody has a complete count. The INITE Atlas tracked 10,066 live AI tools in its sweep of 30 September to 7 October 2026, counting only products whose own site answered the crawler. Directories that list more usually include dead sites, duplicates and wrappers that no longer respond, so treat any single number as the size of one catalogue, not of the market.

  • What percentage of AI tools have an llms.txt file?

    In the October 2026 Atlas sweep, 5,473 of 10,066 AI tools (54%) served an llms.txt file at the site root. That is far above the general web: SE Ranking found llms.txt on 10.13% of the domains it checked. AI companies adopt the file because their buyers ask AI assistants for recommendations, whether or not any engine reads it.

  • Do AI companies block GPTBot and other AI crawlers?

    A few do. In the October 2026 sweep, 258 of 10,066 AI tools blocked GPTBot in robots.txt, and 535 blocked at least one AI training crawler. Bytespider (375) and CCBot (369) were blocked most often. Blocking the agents that fetch pages for live AI answers was much rarer: 75 tools blocked any of them, 62 of those blocked ChatGPT-User.

  • How much does an AI tool cost per month?

    Among 3,868 AI tools whose pricing page showed a USD monthly entry price in September and October 2026, the median was $20 a month. A quarter started at $10 or less, a quarter at $49 or more, 908 cost under $10 and 396 cost over $100. Sales and legal tools had the highest category medians at $49; 71% of tools with readable pricing had a free tier.

  • Can I reuse these AI tools statistics?

    Yes. Both CSV files are published under CC BY 4.0, and the chart on this page is free to republish. Credit the INITE Atlas and link to this page; each number has its own anchor, such as #stat-3, so you can link the exact figure you quote. We refresh the dataset at least every 13 weeks and log each edition in the changelog.

AEOAI ToolsStatisticsllms.txt