Digital Paint Discussion Board

Digital Paint Community => Other Stuff => Topic started by: y00tz on December 01, 2007, 04:12:26 PM

Title: Robots.txt?
Post by: y00tz on December 01, 2007, 04:12:26 PM

If we added a robots.txt to the top level domain with:

Code: [Select]

Disallow: /testwiki1/
It should cut down on *some* wiki spam, yes?

Title: Re: Robots.txt?
Post by: bctrainers on December 02, 2007, 05:12:25 AM

Sadly, most indexers do not follow the robots.txt rules any more. :(

Best is to set certain IP ranges to being able to view it via .ht file. :)

Title: Re: Robots.txt?
Post by: sk89q on December 02, 2007, 12:12:51 PM

What incentive do spam bots have to follow robots.txt?

User-agent: Spambot*
Disallow: /

Search engines, however, do follow robots.txt.

However, the easiest and most effective way is just to require editors to type in a string (i.e. "jitspoer0x") into a text field.

Title: Re: Robots.txt?
Post by: y00tz on December 02, 2007, 01:32:50 PM

Spam bots don't use Google anymore?  Wow I'm getting old..

I just get tired of clearing the revisions of the DP wiki pages (the community guide just broke 1,000 spam revisions), eventually with the new wiki I'd like to see CAPTCHA.

I'm surprised one of our community members hasn't stepped up and made a wiki for Digital Paint, after all, it's not something that needs to be done specifically by Jitspoe.

Title: Re: Robots.txt?
Post by: Zorchenhimer on December 02, 2007, 03:18:37 PM

Quote from: y00tz on December 02, 2007, 01:32:50 PM

I'm surprised one of our community members hasn't stepped up and made a wiki for Digital Paint, after all, it's not something that needs to be done specifically by Jitspoe.

http://dplogin.com/forums/index.php?topic=9795.0

Title: Re: Robots.txt?
Post by: lekky on December 02, 2007, 04:12:33 PM

I don't think he meant that Z :P

Title: Re: Robots.txt?
Post by: sk89q on December 02, 2007, 05:10:20 PM

Spam bots *can* spider on their own, so blocking a search engine is not 100% effective.

Like I said... just entering static text into a field works.

Title: Re: Robots.txt?
Post by: jitspoe on December 04, 2007, 02:26:17 PM

I just need to get mediawiki set up.  It has anti-spam provisions built in.