# All robots will spider the domain User-agent: * Disallow:/recip.cgi Disallow:/links3.html Disallow:/enter.html Disallow:/world.htm Disallow:/hub.html