Operator Intelligence from Evolette Locin: Agentic SEO and Technical Dominance, Singapore
1. The Operational Flaw: Nobody Decided Who May Read the Site
Open the robots.txt file of a typical Singapore business site and you will find one of two things. Either it is the default file the website platform generated years ago, or it is a block of rules pasted in by a developer or plugin that nobody on the commercial side has read. In both cases, the question that now matters was never asked: which automated systems are allowed to read, quote and learn from this business, and on what terms?
That question used to have one answer, because there was effectively one crawler that mattered. It no longer does. Search engines, AI search products, AI training pipelines and link-preview bots all request pages from your site, and they do not all behave the same way or serve the same purpose. The operational flaw is treating “the crawlers” as a single group. Blocking everything protects content you may want found. Allowing everything hands the same content to systems whose use of it you never weighed. Either outcome is a decision made by default, and a decision made by default is a decision made by whoever last touched the file.
There is a second and quieter failure. Robots.txt controls crawling, not whether a page appears in results. A page blocked from crawling can still be listed if other sites link to it, and a page you want removed needs a different mechanism, such as a noindex directive that the crawler must be allowed to see. Founders who believe a disallow line is a privacy control are often wrong, and the error tends to surface at the worst moment, for example when a draft price list or a staff-only page shows up in a search result.
2. The Live Proof-of-Concept: Four Properties, Four Different Answers
We run several sites for businesses that customers find by name and by need, and no two of them have the same right answer. Miyu Omakase wants to be found and quoted accurately, because a guest asking an assistant where to book a counter dinner should receive the correct outlet, the correct booking rules and the correct contact route. Its public pages are short, factual and meant to be read by anyone, human or machine, so the access policy is open for search and AI answers.
Winchester Tennis Arena, TAG International Tennis Academy and PickleChoo are more varied. Their public programme and venue pages should be readable. Their booking flows, member areas, parent portals and any internal tooling should not be surfaced at all, and the right control for those is authentication and noindex, with robots.txt as a courtesy layer on top. We learned to separate these two jobs by seeing how often a rule written for one purpose was quietly doing the other badly.
We do not publish crawl statistics, citation counts or traffic figures for these properties in a public essay, and we would be cautious of any adviser who quotes “AI crawler share” as proof of success. What we can describe is the method and who is accountable for it. The identity work that sits underneath it is covered in our piece on why disambiguation beats keyword volume, and the consistency work in building a fact record that agrees with itself.
3. The Execution Architecture: A Six-Step Access Policy
Step 1: Inventory what is actually on the site. List every section by purpose: public marketing pages, booking and checkout paths, member or parent areas, staging copies, internal documents, search-result and filter pages that create thin duplicates. Most owners discover at least one section they did not know was public.
Step 2: Classify each section into one of three tiers. Tier one is content you want found and quoted. Tier two is content that may be crawled for search but that you do not want an AI product to reuse. Tier three is content that should not be reachable by anyone without a login. Write the classification down in plain language. It is a commercial policy, and the person who owns revenue should sign it, not only the person who owns the website.
Step 3: Map each tier to the right control. Tier three is solved with authentication, not with a robots line. Content that must stay out of results is handled with noindex or removal, and the crawler must be able to fetch the page to see that instruction. Robots.txt is reserved for managing crawl behaviour: keeping well-behaved bots away from low-value URL patterns and pointing them to your sitemap. Using the right tool for each tier avoids the false comfort of a disallow line.
Step 4: Treat each named crawler as a separate decision. The operators of the major AI products publish documentation naming their crawlers and what each one is for, and some separate the bot used to fetch pages for live answers from the bot used to gather material for model training. Google, for example, documents a control token that affects use of content by its generative AI products without changing how pages are crawled for Search. Read each operator’s current documentation before writing a rule, because names and behaviours change and a rule copied from a blog post may target a bot that no longer exists or may not do what you assume. Then decide per crawler against your tiers. An agent can draft the rules and check them for syntax, and a person approves the policy.
Step 5: Test the file before it goes live. A single misplaced slash in a robots.txt can block an entire site from search. Use the testing tools in Google Search Console and a plain request to the file to confirm that your tier-one pages remain crawlable and your key templates are not caught by a wildcard. Run the check again after any site migration, theme change or plugin update, since these are the usual moments a good file gets overwritten.
Step 6: Review quarterly and keep a change log. Date every edit, record who approved it and why, and re-read the operators’ documentation each quarter. Check your server logs, or the crawl statistics your platform exposes, for requests from crawlers you did not plan for. Where the platform does not let you edit robots.txt directly, as is common on hosted site builders, record that limitation in the log and use the controls it does offer.
4. The Asymmetric Advantage: Why Agencies and Consultants Do Not Do This
An hours-billing agency has little reason to care about a ten-line text file. It produces no report worth showing a client, and getting it wrong is rarely traced back to the person who edited it. An academic consultant can describe the landscape of AI crawlers accurately and still never have had to explain to a venue owner why a staging copy of the site outranked the real one for a week.
An operator treats the file as what it is: an access policy over a commercial asset. We decide what is public because we live with the consequences on our own booking pages and our own enquiries. The same instinct comes from our contract and risk background. Deciding who may copy, reuse or train on your content touches intellectual property and terms-of-use questions, and the position that suits a restaurant is not the one that suits an academy publishing coaching material. Robots.txt is also not a legally binding instrument on its own. It is a convention that well-behaved crawlers follow, so where the content matters commercially, the website’s terms and your actual access controls need to say the same thing.
5. The WhatsApp Conversion Bridge: Request a Private Portfolio Audit
A quick test: open your robots.txt in a browser, then ask who wrote each line and what business decision it expresses. If nobody can answer, you are running an access policy by accident. In a Private Portfolio Audit we inventory your public surfaces, classify them into the three tiers, and rank the fixes by how much booking and enquiry volume depends on them.
Request a Private Portfolio Audit via WhatsApp.
This article is general commercial commentary, not legal or financial advice. Crawler names and platform behaviour change; confirm current operator documentation and take advice on intellectual property and terms-of-use questions before changing access rules.
