Flo Luccioni

If you've published your PWA, it's serving a robots.txt right now — the first thing any crawler reads, and the file that decides what happens next: which parts of your site get fetched, and by whom. Yours has been saying yes to everyone.Everything below concerns that web version of your app. Your iOS and Android apps aren't crawled by anyone, and the tab only appears once your PWA has been published at least once — before that, there's no file to configure.That was a defensible answer. It just wasn't one you made — and the population it answers has changed. A few years ago, a crawler on your content was Google, Bing, or a scraper you didn't care about. Today a growing share of that traffic belongs to AI companies, and they don't all want the same thing from your pages. Treating them as one block means choosing between all of them and none of them.The new tab splits them into four families, in plain language: search engines (Google, Bing and the others that list your site in their results), AI answers (ChatGPT, Claude or Perplexity citing your site when they answer), AI training (robots collecting your content to train their models), and on-demand reading (someone asks an AI to open one specific page of your site). Four families, because those are four different deals.