Author Topic: Robots.txt?  (Read 1138 times)

y00tz

  • Autococker
  • Posts: 2742
Robots.txt?
« on: December 01, 2007, 03:12:26 PM »
If we added a robots.txt to the top level domain with:

Code: [Select]
Disallow: /testwiki1/
It should cut down on *some* wiki spam, yes?

bctrainers

  • PGP
  • Posts: 16
Re: Robots.txt?
« Reply #1 on: December 02, 2007, 04:12:25 AM »
Sadly, most indexers do not follow the robots.txt rules any more. :(

Best is to set certain IP ranges to being able to view it via .ht file. :)

sk89q

  • Global Moderator
  • Autococker
  • Posts: 1049
Re: Robots.txt?
« Reply #2 on: December 02, 2007, 11:12:51 AM »
What incentive do spam bots have to follow robots.txt?

User-agent: Spambot*
Disallow: /

Search engines, however, do follow robots.txt.

However, the easiest and most effective way is just to require editors to type in a string (i.e. "jitspoer0x") into a text field.

y00tz

  • Autococker
  • Posts: 2742
Re: Robots.txt?
« Reply #3 on: December 02, 2007, 12:32:50 PM »
Spam bots don't use Google anymore?  Wow I'm getting old..

I just get tired of clearing the revisions of the DP wiki pages (the community guide just broke 1,000 spam revisions), eventually with the new wiki I'd like to see CAPTCHA.

I'm surprised one of our community members hasn't stepped up and made a wiki for Digital Paint, after all, it's not something that needs to be done specifically by Jitspoe.

Zorchenhimer

  • Autococker
  • Posts: 2614
Re: Robots.txt?
« Reply #4 on: December 02, 2007, 02:18:37 PM »
I'm surprised one of our community members hasn't stepped up and made a wiki for Digital Paint, after all, it's not something that needs to be done specifically by Jitspoe.

http://dplogin.com/forums/index.php?topic=9795.0

lekky

  • Autococker
  • Posts: 2449
Re: Robots.txt?
« Reply #5 on: December 02, 2007, 03:12:33 PM »
I don't think he meant that Z :P

sk89q

  • Global Moderator
  • Autococker
  • Posts: 1049
Re: Robots.txt?
« Reply #6 on: December 02, 2007, 04:10:20 PM »
Spam bots *can* spider on their own, so blocking a search engine is not 100% effective.

Like I said... just entering static text into a field works.

jitspoe

  • Administrator
  • Autococker
  • Posts: 18802
Re: Robots.txt?
« Reply #7 on: December 04, 2007, 01:26:17 PM »
I just need to get mediawiki set up.  It has anti-spam provisions built in.