User-agent: * Allow: / Disallow: /users/ Disallow: /search$ Disallow: /search/*/shortlist Disallow: /search/shortlist Disallow: /search/result Disallow: /search/fulltext Disallow: /out/ # Google-Extended is Gemini's TRAINING opt-out. It is a robots.txt-only control # token — there is no such user agent, so it cannot be blocked in Apache. It is # read by Googlebot and is independent of Search: this does not affect crawling, # indexing or ranking on Google Search. # # Narrowed from "Disallow: /" on 2026-11-01 (owner, after Zuan's SEO mail): the same # token also gates GROUNDING in the Gemini app, so the blanket opt-out kept NetMaid # and its agencies out of Gemini answers. Training and grounding cannot be split, so # the line is drawn by content instead: directory, agency, browse and FAQ pages are # open; the helpers' own data stays out, because a trained model cannot forget it the # way a 410 removes an index entry. # /maids/ profiles (/maids/:id.html and every other shape) # /1 … /9 //… — helper photos on mm., agency files on files. (this file is # served on every host), and the legacy /N-NAME.html maid URLs on www User-agent: Google-Extended Disallow: /maids/ Disallow: /1 Disallow: /2 Disallow: /3 Disallow: /4 Disallow: /5 Disallow: /6 Disallow: /7 Disallow: /8 Disallow: /9 # Google's non-search crawler (internal R&D/product), separate from Googlebot. # 167K requests in week 32, 78.6% of them wasted on redirects and 404s, and it # sends no traffic. Also 403'd in production_common.conf; this is the polite # request that saves us the 403. User-agent: GoogleOther Disallow: / # Common Crawl — the feedstock for most AI training corpora. 23K requests in # week 32, 96.2% waste (21,755 404s). User-agent: CCBot Disallow: / # Sogou serves mainland-China search. Netmaid's market is Singapore employers, # who do not use it: 147K requests in week 32, zero referrals. User-agent: Sogou web spider Disallow: / User-agent: Sogou Pic Spider Disallow: / # Apple's TRAINING opt-out, the counterpart of Google-Extended above: a # robots.txt-only token, not a crawler. Applebot (Apple search, Siri, # Spotlight) is unaffected and stays allowed. User-agent: Applebot-Extended Disallow: / # Everything below is also 403'd in config/apache/bad_bots.conf. Until # 2026-10-19 those bots got a 403 for this file too, which tells a crawler it # may crawl everything, so they never stopped trying. Now they can read this. # Keep the two lists in step. # AI training crawlers. User-agent: GPTBot User-agent: ClaudeBot User-agent: meta-externalagent User-agent: Bytespider User-agent: Amazonbot User-agent: Reflectionbot Disallow: / # Search engines for markets we don't serve. User-agent: Baiduspider User-agent: Yandex User-agent: PetalBot User-agent: 360Spider Disallow: / # SEO and backlink tools. User-agent: AhrefsBot User-agent: SemrushBot User-agent: DotBot User-agent: MJ12bot User-agent: DataForSeoBot User-agent: BacklinksExtendedBot User-agent: SERankingBacklinksBot Disallow: / # Search APIs sold to other people's AI agents, and listing scrapers. User-agent: ExaSearchBot User-agent: LinkupBot User-agent: KeenableBot User-agent: DesearchBot User-agent: FirmCredBot User-agent: AionBot User-agent: VelenPublicWebCrawler User-agent: BarrigazBot User-agent: SEBot-WA User-agent: MarketingDataBot User-agent: KIMISUITE-SalesCenterBot Disallow: / Sitemap: https://www.netmaid.com.sg/sitemap.xml