I always have thought web directories should have backfill from a search engine when you search them.  Especially, general directories.

Yahoo (dubbed “Mighty Yahoo”) was the first directory to add a backfill provider on it’s search results. First Alta Vista, then Inktomi, then Google, then Inktomi again provided backfill results for Y!.  Snap/NBCi directory had backfill and so did Looksmart.

How backfill worked was somebody would search the directory, when the directory ran out of listings that matched the query, you would see search results from the backfill provider.  (Being Mighty Yahoo’s backfill provider cemented Google’s reputation and proved to be one of the biggest mistakes Yahoo ever made.)  What was important is the search portal needed to provide something in a search that would satisfy the person searching, otherwise they would start going elsewhere. And they could go elsewhere because there was genuine competition in web search back then.

It gets harder for niche directories.  For example: if you go to a Star Trek niche directory and search for “uniforms” it is assumed you are only going to get uniform sites that are related to Star Trek.  In this instance you don’t want a general backfill giving results for police uniforms or nurses uniforms.

For backfill to work on a niche subject directory you almost need a place to put in an extra search “slug” to bring it into context.  So in our example, you need to have a place in your admin panel to add “star trek” as a term to the backfill.  So that somebody searching for uniforms on your Star Trek directory are automatically searching for “star trek uniforms” on the backfill.  The “star trek” is always added to the backfill.

It gets even harder for a local or geographic directory.  Frankly I think backfill should not be used with a local directory.

I did some looking around, Google and Bing are way too expensive to get a search feed from.  Smaller engines like Mojeek.com (or Mojeek.co.uk) or Gigablast.com provide search results API’s.  Both are affordable, but Gigablast’s is really really affordable.

Of course, I don’t know how to actually code this.  That’s what coders are for.  😀

But the trick here is this:

  • The more listings you get in your directory the more users will use the search function.
  • You have to have a search function.
  • No matter how sophisticated your directory search is, you have a finite amount of listings.
  • Mobile devices make the search function even more important.
  • This is why most will need backfill.

I’m doing a soft launch of my new blog directory.  It is ready for submissions by bloggers.  If you have one blog or several you can add your URL’s.  Listings are free, I don’t want or need the money, I want you to find more readers. Mind, we do have guidelines. We take all types of blogs: regular blogs, photo blogs, podcast and microcast blogs, micro blogs.

I have seeded just a few sites in the directory, just enough to give you an indication of what types of sites go where.  Some lists have no listings because they should be self explanatory.

Even if you do not want to add your URL, I would appreciate it if you would spread the word to your blogger friends.  They might want a listing.

This is a continuation of my previous pipe dream.  I decided to do something more modest, a micro directory if you will.  It is not quite as personal as a Blogroll but it is smaller than a stand alone web directory.  It falls somewhere in between.

Some things I’m excited about:

  • The directory is responsive to mobile.  So phone users can use it. I have this vision of a bored train commuter, takes out his phone, selects a listing in the directory and she’s off reading somebody’s blog.
  • The directory search feature seems to work better than most.  This is important both in general and for mobile.
  • The directory is part of the blog here.  Which makes it easier to find users.

This is part of my decentralized search proposal.

That’s enough for now.  More thoughts later.

This was also posted to
/en/linking.

Also on:

Yesterday, like Snoopy, I almost got the Sopwith Camel, out of the barn to get ready for one last dogfight with the Red Baron.  Meaning, I almost bought a domain, directory script and hosting for it to start a directory of blogs.  I had it all researched, script picked out, ideas as to how to make it good, everything, then I started writing the blog post below (italics). It’s not complete but it was my reality check.

Directories and Search Engines Do No Good if Unused

Here is the cold hard reality of search engines and directories:

  1. They have to have a starter set of listings. I call this “seeding” the directory.  If a searcher comes to use the directory you need to provide them with something or they will never come back.  This means the Editor(s) of the directory have got to add a starter set of sites to the database manually.  It’s a lot of work.
  2. You have two sets of customers: A. searchers, the public, B. webmasters adding their URLs.  The expectations are different for each set.  You have to market to both.
  3. You have to provide traffic to sites listed.  With a new search engine or directory you have to build a user base.  This means you have to gain exposure, either by hard advertising or by word spreading within whatever niche you serve.  Also links back, articles and search boxes on websites help.  This is why it is better to start in a niche, because members of the niche are more likely to support you.
  4. You are not going to make any money at it for years, if ever.

Time!

I scrabbled back from the edge of the cliff.

I’m not willing to put the time and money forward to make such a directory at this moment.  That and the time commitment to promote it, that is, gain traffic to it would be huge.   So I scrapped the Big Plan.

As I said before, there is a need.  Blog directories are a stopgap measure for a few years until the next generation can cook up better solutions to the blog discovery problem.  The need will grow in the near future as Twitter and FB either decline or restrict our being able to syndicate our posts to them or both.

I do have other “clever Plans,” in the works. I’m not giving up.  This is not retreat, I’m just attacking in a different direction.  More when those plans are closer to launch.

(Plus I salvaged that post I started yesterday.)  😄

Also on:

In a previous conversation, I made a rough list of types of blog search directories and search engines.

Blog Discovery:  I’m sure directories are not the best solution for blog discovery, but like blogrolls they have a place at the table because they are low tech and cheap. Here’s a rough hierarchy:

  • Do In A Pinch: Blog acting as a directory.

  • Minimum: A proper directory script (ie. phplinkdirectory or similar) This allows for blog owner to submit their blog, write a description etc.

  • Better:  A directory script that not only lists blog URL but also lists blogs RSS feed.

  • Better x 2: Directory described above which also generates it’s own RSS feed for each category and subcategories.

  • Better x 3:  Some sort of fusion of the x2 directory above, and Indieweb stuff to some degree.  Maybe a fusion of a standard directory with Kicks Condor’s Indieweb.xyz.  This is just brainstorming.

  • Better x 4: Probably an RSS search engine like the old defunct Daypop or Icerocket. Because this leads the searcher to individual posts about a topic in close to real time.  Such an engine could use Post Kinds as filters for the searcher to refine their search.  There used to be a lot of RSS blog search engines I could find only RSSMicro today.

  • Best: Some sort of hybrid directory/RSS/crawler engine listing only blogs.  The search crawler digs deep into a blog for those posts from 2015 or before that are buried and won’t appear in feeds.  The RSS search engine for the newest posts.

I think I need to expand on this.

The more advanced the solution, the greater the technology bar to entry.  Just about anyone can start a human edited directory, but creating an RSS search engine requires more programming skill.  Moreover, the more advanced solutions require more money to run which again raises the bar to entry.

We see this now with web search, the cost to build a search engine that will rival Google is more than many countries can afford.

But if we only follow the technology route we create another set of silo’s like Google and Bing.

So in the blog world we need lots of different ways to discover blogs and blog posts.  We need ways for new blogs to be found by readers before their bloggers lose heart in talking to themselves.  We need many high tech search engines and many low tech directories so that the blogosphere does not become consolidated once again into a few silos.  No one source is going to index the entire blogosphere, and that’s okay. because each index is a collection.

The Indieweb plays an important role in this.  Webmentions will become the new backbone of making the Web the social network.  And they will also help blogs get discovered.  But right now Indieweb adoption is relatively small.  We also need to find audiences that are just readers who do not blog.  Because even siloed social networks have their own search function.

  • If there are 100,000 bloggers, then we need 100,000 blogrolls.
  • We need site searches, like phinde, that can index multiple domains.
  • We need many dozens of blog directories of all sorts: human reviewed and, somehow, Indieweb automated.
  • We need a dozen blog and RSS feed search engines.

All at the same time.  Let people choose their trusted sources.

Other reading:

Search Engine History

Bomis Webrings had Important Differences.

Indieweb, Discovery and Search.

Also on:

Replied to

In reply to: Chris Aldrich replying to the Blog Snoop Memo.

Chris,

Cow Paths:  Yes we have this concept in both New Urbanism and planning municipal bike/hike trails.  It’s a good analogy for what I’m doing: “What’s available off the shelf and free as a stopgap until somebody designs something better?”

Blog Discovery:  I’m sure directories are not the best solution for blog discovery, but like blogrolls they have a place at the table because they are low tech and cheap. Here’s a rough hierarchy:

  • Do In A Pinch: Blog acting as a directory.
  • Minimum: A proper directory script (ie. phplinkdirectory or similar) This allows for blog owner to submit their blog, write a description etc.
  • Better:  A directory script that not only lists blog URL but also lists blogs RSS feed.
  • Better x 2: Directory described above which also generates it’s own RSS feed for each category and subcategories.
  • Better x 3:  Some sort of fusion of the x2 directory above, and Indieweb stuff to some degree.  Maybe a fusion of a standard directory with Kicks Condor’s Indieweb.xyz.  This is just brainstorming.
  • Better x 4: Probably an RSS search engine like the old defunct Daypop.  Because this leads the searcher to individual posts about a topic in close to real time.  Such an engine could use Post Kinds as filters for the searcher to refine their search.  There used to be a lot of RSS blog search engines I could find only RSSMicro today.
  • Best: Some sort of hybrid directory/RSS/crawler engine listing only blogs.  The search crawler digs deep into a blog for those posts from 2015 or before that are buried and won’t appear in feeds.  The RSS search engine for the newest posts.

Something like that.

I don’t think we have tapped RSS fully.

OPML: I use Inoreader and I subscribed to your Indieweb list from your Following Page.  Brilliant list and thanks.  I think there is a future for this for deep discovery.

rel=directory: My instincts tell me this is important and they are on the right track.  The directory also is a low tech way to sort by topic.  Because it’s not enough to just have all the Indieweb plugins, but people need to know what the blog’s topics are.

Vouch: Yes. Needed in some form.  I used to move at the periphery of the spam crowd although I was never part of it and I can tell you, right now Indieweb stuff, (webmentions etc.) are in a honeymoon.  The spammers will figure out how to exploit these in ways you never dreamed of.

Thanks to you and to Kicks Condor for taking the time to reply so extensively.  The first step needed for Blog Snoop is to have an extensive index of blogs.  Only then will it be of any use.  And this was a useful discussion because it gave me some ideas that might work.  Again thanks.

Also on:

I know I do go on about old web directories.  Eventually I’ll get it out of my system.

For a younger audience, it is helpful to see what I’ve been talking about, in the flesh, er, pixels.  Back in the late 90’s and early to mid 2000’s there were quite a few niche science fiction and fantasy directories.  I found two of these directories still functioning on the Web.  At one time these were the prominent examples of their kind. I’m glad they are still around to see.

First, is Speculative Vision Network.  It’s always been part of a portal, art gallery, forums, interactive stories, directory.

Second is Scifisource. Which has always been a directory.  It looks pretty much the same as I remember it.

I do not know to what extent these are being maintained but they still function.  These are the real deal, sorta sealed in amber, and at least serve as examples of the niche directories of that era.

Also on:

I’m hitting a fork in the road with this site and the experiment of using a blog as a directory of blogs.  The problem here is me: I’m running out of time.  I’m duplicating a lot …

Source: Announcement: The Future of Blog Snoop – Blog Snoop Weblog Directory

We’ll see what happens.  It was a guerrilla blog discovery idea I thought I would experiment with.  Details at the link.

Also on:

Replied to

In reply to: The Awesome Directories.

Kicks,

I think I have an answer to your linkrot concern. Not for Awesome but a hypothetical directory – “someday”.

The idea really isn’t mine, I got it thinking about the Microcast.club directory, which is really a new Indie style webring that also has a List of Sites page aka a flat directory page.  When you join you get a webring code to place on your site, plus listing in the directory.  Cool.

Linkrot.

On some hypothetical future niche directory it’s that webring code that prevents the linkrot.  As long as the webring/directories robot keeps finding that validation code you stay in the directory, no code and eventually you will be dropped. Not perfect but it automates the process a bit.

It wouldn’t have to be a webring code, it could just be a validation code, I suppose.  I like the idea of indiestyle webring/directory combined.  My worry would be scale on the ring.  Old style webrings had a sweet spot of 50-200 sites.  Over 200 and they become difficult to manage and it was found that bigger destination sites in the ring siphoned off too much ring traffic and gave very little back.

Still adding that directory page was a good evolutionary move for indie-style rings.

Massaging this idea a bit:  one could let ring members self define subject categories when they join very much like you did on Indieweb.xyz (tags sorta) which would help when it gets larger.

This was also posted to
/en/linking.

Also on: