Robots.txt: Too Complicated for Dave

October 24th, 2007 by EyeOnWiner

Dave wishes that TechMeme had an item-level opt out.

Of course people would like to have an item-level opt-in, to guarantee a post would earn a top position on TechMeme, but obviously that’s impractical, so why not give us the power to say please don’t include this story in TechMeme, it’s too meta.

Gabe Rivera has said before that TechMeme will follow the Robots.txt spec, and Robots.txt has all the tools you need for item-level exclusion. Unfortunately, Dave can’t be bothered to understand the Robots.txt spec. Then again, there’s this:

[I]nstead of making me turn off the TM spider for my whole site, why not give me a way to say “Stop TechMeme spider, this item is off-limits.” We could come up with a TechMeme namespace for RSS 2.0. I’d be happy to help.

When the only tool you know how to use is a hammer, everything looks like a nail.

4 Responses to “Robots.txt: Too Complicated for Dave”

  1. You missed his weird little smiley: ;-> or whatever it is.

    OTOH, I’ve never grokked what he means by that.

    I move that the TechMeme exclusion tag should be part of the Atom namespace, that would mix it up a bit.

  2. McD says:

    Gabe should cut his losses and just exclude scripting.com pointers all together and let Dave send him traffic when he doesn’t see any links coming back. ;->

  3. Tom says:

    Semi-related to this topic, Dave’s having a really interesting twitter day today. My favorite quote so far has been…

    ” I want a BDG, not a spec. I just want to know what goes on the wire. I don’t have time to go to college”

    Now, is it just me, or does Dave have nothing but time. Seems to me the only thing he’s actually created in at least a year is a NYTimes outline that would take any developer 5 minutes and an updates OPML spec which since he created it entirely by himself shouldn’t have taken more than an hour either

    One more bonus quote though…

    “I think tinyurl.com may be starting to suck “

    Is there anything he won’t turn against after a couple months?

  4. Bullshit Mancuso says:

    Let’s reinvent robots.txt as robots.opml.