September 1st, 2016


Robots.txt Disallows All User-Agents

Robots.txt Disallows All User-Agents

Short, concise description of the idea
Robots.txt disallows all user-agents, which means that my livejournal cannot be indexed by search engines.

Full description of the idea

Despite setting my journal to public, and unchecking "minimize inclusion to search engines," my robots.txt remains the same, it blocks all search engines. What's the use of having a blog if not crawled by search engines anyway? How would one get new visitors?

My LiveJournal URL is:


An ordered list of benefits
  • It's user-friendly.
An ordered list of problems/issues involved
  • Robots.txt is not changed despite setting privacy setting to PUBLIC.