Despite my earlier protestations, I am working on a web directory project on a different domain.  It’s not a big deal but there is work to do, like: seed the directory with a starter set of links.  I mean you go to a directory to find web pages or websites and it better have something for you to find or you will never come back.

Yeah, so I’ve got this used web crawler/indexer, that would be me, who is an old, slow and cranky, old git, but  works cheap. The slacker likes taking a nap during working hours. This could take awhile.

Plus I’ve got to edit CSS font sizes which I’ve never done, and write help pages that explain what the heck I’m trying to do.

All of which is to say, I’ve got to cut back on blogging here for awhile and just get this directory ready for launch.

Thanks.

Signed, Me.

I’ve built a lot of different niche web directories over the years.  Frankly there are some that I have forgotten about.  But here are some highlights.

Planet Doom – 2001 – 2005 this was my first directory/portal.  I learned a lot operating it and it had quite a lot of traffic.  At the time, Google was pretty poor at delivering results for niche searches so they would often put niche directory categories in their SERPs.  In effect they would “hand off” the search to the specialists.  As Google got better at it they moved away from featuring niche directories in the results.  Template – custom. Logo – custom both by Lynne Scott.

I went with Scifi/Fantasy/Horror in order to make myself a little different from most of my competition who were doing just Scifi and Fantasy.  Doing all three never was a good fit.

This is the directory that got carpet bombed by Google after my host, Searchking.com, sued Google.

Scifimatter.com – 2003 – 2012  Once it became clear that Google would never send traffic to Planet Doom (above) ever again, I took the backup database from Planet Doom, discarded the Horror listings and created Scifimatter.  The earliest version used a free version of the Gossamer Threads script which had a flat file database.  Later versions (pictured) used WSNLinks, if I remember right.  At various times this directory offered webring hosting and banner exchanges for webmasters.  By about 2008 it became clear that directories had had their day.  I stubbornly hung on to this neglected site until 2012 when I finally pulled the plug.  This was my favorite.  Template – off the shelf.

Shadowdark.org – This was a catchall domain.  From it I ran lots of different perl and php scripts on subdomains to save some hosting fees.  The sites were of all sorts of genres.  Three directories stand out and are listed below.

Planet Doom II –  experimenting, I revived the Planet Doom name as a Horror, Dark Fantasy and Scifi Monster Directory.  I used the free version of Fluid Dynamics Search Engine which worked pretty good for search but was hard to administer. The categories are fake, if you hover over a category you see it triggers a search engine search.  It did okay but was never super popular.  Ironically, today, using a site search engine script, something like this might make a better niche directory than a directory script.  PD II might have been ahead of it’s time.  Template – off the shelf.

The Ring Codex – I rushed this out to take advantage of all the hype about Peter Jackson’s Lord of the Rings movies which were forthcoming with fevered anticipation.  I used a free version of the Gossamer Threads perl script.  Codex had a lot of people using it.  I should have spent more effort on promotion and building the index.  The script had no captcha protection on the Add URL form so it eventually attracted automated spam submissions.  I think at one point I was manually deleting, one at a time, 200 spam submissions a day, with very few legit submissions.  Rather than spend money on upgrades, I closed it down.  Template – off the shelf.

Spy Fiction Guide – I was getting burned out on SF/F/H,  but espionage fiction had always been a favorite genre of mine.  With the fall of the Berlin Wall and eventual collapse of the USSR the spy genre seemed to be dying out. So this was sort of a labor of love on my part to keep the genre alive.  It was never wildly popular but I had fun.

The index of most of these were built by me using a dial-up internet connection which was a very time consuming process.  Niche directories were at their best when people were still building serious websites on Geocities, Tripod, Angelfire and the other free web hosts.

If you have read this far, I thank you.  This post was an itch I had to get out of my system.

 

This was also posted to
/en/linking.

This is a follow on of: Let Us Build a New Web, so you might want to start with that.

Here I want to talk about expanding beyond a static site or just a blog.  For most of these I think I would probably also have a blog just because it’s easy to post updates and announcements on one.

Website Ideas:

Wiki – if you have used Wikipedia then you already have used a wiki.  Wiki’s are very good for collaborative websites.  You can build a knowledge base  with a wiki. You might not need a blog if you have a wiki.  Wiki’s are best for making vast globs of sprawling information able to be found through the wiki’s site search and hyperlinks.

You can use one for group journal type role play.  I have always wanted to use a wiki for world building for table-top RPG’s like D&D, CoC, Metamorphosis Alpha and/or Traveler.

Knowledge Base – a KB is great for making a detailed manual.  You see a lot of knowledge bases used in software support.  Here is an example for WSNLinks.

So if you have some detailed, step-by-step knowledge you want to share, a knowledge base might be perfect.

Some KB ideas: how to paint RPG miniatures, naval miniatures wargame rules,  table-top RPG rules manual, any kind of howto guide.

Directory – I’ll talk about two variants, there are more but I’ll stick with two for now.  1. Links Directories and 2. Business Directories.  Links Directories are collections of hyperlinks to websites (ie. Yahoo started out as a links directory.) Business Directories, may or may not have hyperlinks, but they generally list the name, address, phone number and hours of operation of a business.  Most include a map showing the business location (ie. Yelp, and the online Yellow Pages.)

  1. Links Directory – this could be something as simple as using a directory for hosting your own bookmarks.  (ie. back in the day I knew someone who had a “Cool Directory” which was anything he thought was cool.  Basically his bookmarks.)  Make a topical directory, links to websites about a topic you are passionate about. (A cause, hobby, science fiction, anime, comics, etc.)  I do think that a blog compliments a directory well.  It gives the directory owner a voice.
  2. Business Directory – these make perfect local directories, their strength is they can list bricks and mortar businesses that do not have a website.  These can also be used as restaurant review sites.  I always thought a directory of weird old tourist attractions would be cool.

Forums – forums are an old school social network.  For niche sites they can be perfect for like minded people to have in depth discussions. One advantage a forum has over social networks with moving timelines, a post or a reply, much like email, will sit there waiting for you until you return.  So maybe you only visit once a week, all activity will be there waiting for you.

Most hosting accounts have a couple of free forums scripts ready to deploy at the touch of a button.  I like SimpleMachines forum the best. YMMV.  If your community thrives and becomes big you can move up to something like Invision Community.

The down side of forums are they are very hard to get started.  They work best when you and a few friends decide beforehand you need one.  Otherwise, start a blog on the topic first, attract a following and then ask your followers if they would be interested in a forum for more chat.

For almost all the above I think you should have a blog.  It is always a good way to reach out by syndicating to social networks.  You can mix and match all the different scripts described above whatever works for you.  Again if all you need is X number of pages and then your topic is exhausted just make a static site.  Do it for you. Do what pleases you.

If you have ever had the yen to build a website the above can give you some ideas.  Feel free to comment if you have ideas of your own or questions.

This was also posted to
/en/web.

Here is an interesting artifact from the Web 1.0 past.  A list of known search engines and directories.

See the List Search Engine List

  1. This is only a partial list.
  2. This does not list the thousands of niche, national, regional and local directories of that time.

Almost all these are gone.  If you want a glimpse at what web search was like in 2001 click the link.  A list from 1999 would be even better and more vibrant.  A lot of these search engines had no hope of surviving in 2001, but at least most of them had their own index.  Now we are down to 2: Google and poor second Bing, plus a couple of smaller engines with their own indexes.  And you don’t call that a monopoly?  You don’t call that a silo, and a dangerous one at that?

Look at that list – at least they were trying.

If you have serious bookmarks scattered about on various browsers and bookmarking services and want to consolidate them into one index you might consider a web directory script.

This solution is not for everyone, but if you already have a C-panel type web hosting account for your blog, the kind of hosting where you can create either more subdomains or host more domains you can do this with ease.

As an example: 1. I don’t have the hosting account but I need one for other projects anyway, 2. I’ve got about 30 parked domains I could be using, 3 or I could put the bookmark directory on a subdomain.

Pros:

  1. Web based so you have access anywhere across all devices.
  2. Searchable and can be categorized using any structure you devise.
  3. Most directory scripts will let you have hidden categories that you can keep private.
  4. Nobody is tracking what you bookmark and selling the info.
  5. Dead link checker.
  6. No ads.
  7. It’s on your own domain so you have control.  You don’t have to worry about third party bookmark site getting bought up or going pay or changing business models.
  8. Many directory scripts have a free version with fewer features.  This might be enough for a bookmarks directory.

Cons

  1. Not quite as easy to add bookmarks to.
  2. More set up and maintenance.

 

I’d never really thought about this until Amit Gawande asked about bookmarking.

Yahoo!, Dmoz, Looksmart, Snap/NBCi, Go.com, at one time there were a lot of First Rate web directories and now they are all gone.  There were also quite a few Second and Third Rate directories plus regional, local and niche.

With the rise of link popularity and Google’s Page Rank, all too soon, junk directories were springing up all over.

Most of the top tier directories are gone, but I found 7 directories, still alive, that have been around a long time.

Illumirate –  I remember it’s predecessor Hotrate from back in the 1990’s.  Sounds like they have tried different business models and lost several databases over the years and have settled down to be a small non-commercial general directory.  Kinda decent.  Nothing to really separate them from the pack but is not filled with spam.  Looks like a legit navigational directory.

Skaffe – They started out as a general directory and now they appear to be pushing themselves as a regional directory.  Sometimes you have to try different things to survive.

Gimpsy – This is kinda unique, it is organized around what you want to do (buy, get, read). It was never a favorite but it is pretty decent.

SeekOn – The results do not looked spammed out which is good.  They are pay inclusion only which is not good, but somebody has to pay for hosting.  They accept both websites and articles.

JoeAnt –  If I remember right this was founded by former Go Guides (editors) who were let go when Disney pulled the plug on it’s own portal at Go.com.  The look has really not changed over the years except somebody scrunched the categories all into one column to make it more mobile friendly.  I always felt the SERPs were too cluttered.  And in my quick testing I found some dead wood in the index.  JoeAnt was always hard to get into.

GoGuides.org – another directory founded by former Go.com Guides.  Sites are listed by rating. Another one that was hard to get into.  Seems to be maintained.  Fairly clean look.

SoMuch.com –  A general directory slanted towards internet and tech.  It started out as one man’s bookmark collection and went from there.  I like this because it is unpretentious and I do like the Category hierarchy.

Most of these have an option to pay a fee for faster review.  Frankly, I would not pay any general directory for inclusion.  Use the free submit option.

My random thought for the day.  These can be dangerous.  Hold my beer.

What would happen if you combined a standard web directory script with Indieweb.org features like webmentions and such?  I think you could end up with a very powerful tool for a directory. I have not the slightest idea how one would actually do it.

Presuming that both parties have webmentions here is how I see it working:

  1. Editor adds a website to the directory.  The directory (equipped like an Indieweb blog, send out a Mention to the page and/or root URL that was added.
  2. The owner of the URL that was added to the directory, may keep or discard that Mention but at least they now know that they are listed.  They might want to go to the directory and “claim” that listing and edit it or not.
  3. For the directory there is a chance that the directory gains a link.  It may well be a nofollow link in a facepile somewhere but as long as it is a clickable link back there is a gain for the directory.

Potential for Abuse

There are two kinds of directories 1. rare legitimate directories that are trying to be a navigation aid on the web, and 2. link popularity directories that are basically there to sell a link to websites for SEO purposes.  A mention from the former should be welcome to any webmaster.  And a directory that is trying to be a legit web navigation aid needs to attract searchers to use it. Win-win.  On the later, I could see spam directories trying to abuse this. The moderating factor is unlike a decade ago, very few new directories of any type are being started these days.

Why?

The web has changed in 20 years.  Webmasters no longer run out and submit their sites to directories, they are used to social media and the search engines just find them eventually.  So a very high percentage of listings in a new directory are going to have to be added by an Editor rather than by a submission to the  Add URL form.  Because of this it is even more important for the directory to let webmasters that have not submitted to the directory know they have been listed.  It’s a point of contact.

 

This was also posted to
/en/linking.

 

I’ll be the first to admit, I’d like to run either a search engine or a directory of my own.  It’s one of those itches that isn’t going away.

I almost started a real directory a couple of months ago but better judgment prevailed.

Then today I discovered this:  WSN Links WordPress plugin.  Suddenly many of my objections from my previous post were solved.  Attaching a full directory to a blog allows you to build traffic on a budget.  While a stand alone directory would be hard to promote without spending lots of cash.

See, the other problem is I’m on managed WordPress hosting so I can’t just install other stand alone scripts on the server.  But a plugin, yes that I could install.

I had a ton of other questions: General vs niche, if niche what niche, how to seed it with listings and more?  The only way to answer many of these questions would be to install the plugin and see exactly what the plugin’s capabilities are.

So I tried and the installation failed.  WordPress plugin uploader wants .zip files NOT .php and without FTP access to the server uploading that file just is not going to happen.

Saved, I think.

Maybe someday, maybe soon, maybe not.

Searchking.com had two divisions: 1. A search engine, 2. remotely hosted web directories (think WordPress.com only for directories,) run by different individuals. This article will deal with the search engine alone.  The Searchking search engine I am going to talk about is not the Searchking directory that exists today even though it is using a similar concept in presentation.

This is all from memory having used it quite a lot.  I have no insider knowledge of how it worked or was administrated.

What was Searchking?

Searchking was a minor search engine born at a time when there were 7 – 8 major search engines, many dozens of minor search engines, and many hundreds of directories.  It would seem very crude by today’s standards, but back in 1997 it was on par with many of the majors and almost all the minors for search quality.  It was designed by Searchking CEO Bob Massa and coded by Sargeant Hatch.

The reality was SearchKing search engine was not really a search engine.  It had no crawler other than, maybe a meta tag grabber, my memory is a little cloudy on that.  It was, really a flat directory.  That is it had no hierarchy of categories.  All it searched was the Title, Description and Keywords submitted for each page.  I’m sure there was a rudimentary algo, for example, giving more weight to a keyword in the tile than in a description, but that was it.

How Did it Work?

For the person searching, it appeared to be a search engine.  There was just a searchbox.  When you searched, you got a SERP with 10 results on page one, ten more on page two.  When you clicked through to a site it was shown to you with a frame at either the top or bottom.  On the frame you could vote for the quality of the site.  You could also report spam or dismiss the frame.  The quality votes also effected ranking.  The reality was that very few searchers ever voted.  This means a key feature of the algo was rarely used.  The spam report was used a bit.  Mostly, people dismissed the frame as quickly as they could.

There was some human review.  I think Admins running the search engine kept a weather eye on the submissions and did some random checks.  They did act on reports.  But remember, the Web was wide open, wild frontier when this was built. Nobody knew what worked and what didn’t.  Even some of the major search engines like Infoseek mainly indexed only meta tags and maybe a little on page text for that one page.  Hardly a deep crawl.

Submitting your website to Searchking really meant submitting each page by hand, manually.  Again keep in mind, there were no CMS’s yet nor blog platforms in general use, so most websites were hand coded in HTML. Plus everyone was on dialup which was slow.  So websites tended to be 5 – 25 static pages.  Because of the slowness of dialup internet, we all tended to keep Titles, Descriptions and keywords short.  Search too was rudimentary everywhere.  Keyword searches were one or two words on all search engines.  In 1997 when Searchking search engine was built, it was built to work within the limitations of the day.

By year 2000, Searchking was starting to show it’s age, but it remain viable.  One feature the Searchking search engine had was instant listing.  When you added a page, it went live instantly.  This would play a key role in 2001.

By 2001, “Mighty” Yahoo was still the king of search, Google was rapidly gaining popularity for it’s deep spidering and better search results.  The other major search engines were falling behind rapidly or had disappeared.  In those days, it took Google about a month to start listing a new site that had been submitted too it.  Getting a new site to rank in Google was yet another matter, you might be listed, but you might be on page 20 of Google’s SERPs.  Also, Google was still just a web search engine, it was not a news search engine. News stories maybe got in and ranked faster but it still took a week or two before it would appear on Google.

9-11 and the need for Instant News and Information

Then the 9-11 terrorist attacks on the Twin Towers in New Your City occurred  on September 11, 2001.  Everyone was stunned.  Over on the Searchking directory hosting side, we had a forum community of directory owners.  I think we all spent most of the 11th glued to our televisions and radios, but we were also hitting all the news websites for updates.  We started sharing the URL’s to news stories on the forums.  Google had been caught flat footed, for days there was little useful or current information on it, which they acknowledged and tried to correct, but that was slow.

Over at Searchking, CEO Bob Massa told us in the forums that they were monitoring people desperately searching Searchking for any kind of information about the attacks, survivors and relief efforts.  Because Searchking could list new pages instantly, he asked us to help: to dig out and add the URL’s to news articles, new web pages of survivors, relief news, defense news, background news, to Searchking to help those looking for information.  And the SK community responded, we were searching for any news we could find ourselves anyway, TV and radio announcers were reading our makeshift URL’s of survivor lists, and we could jot down and share those too.  This was most important in those first couple of days after the attack, but people from the UK, Canada, the US, even a person in Greenland, cranked out listings of information for about 2 weeks straight. Eventually Google reprogrammed their crawlers and started catching up as did the mainstream media.  All the fancy high tech crawlers failed, but little low tech Searchking actually delivered.

It was probably Searchking search engine’s last best moment.

Why is this Important Today?

There are lessons to be learned from this example that would help make a new directory viable today.

  1. The idea of a flat directory, without drilling down through a hierarchy of categories fits in with the way people use search in 2018.  Make a directory, look and act like a search engine to the searcher.
  2. Even if you have a hierarchy you can hide it on another page so you don’t scare people off.  Just present the searchbox front and center to the user.  (Personally, I would still have a hierarchy of categories as “spider food” for Google, Bing and Yandex, but just make the searchbox so prominent users won’t ignore it.)
  3. One might combine a directory with one of the many available open source web search engine scripts  in some way and combine human reviewed results with crawler results.
  4. Instant listing after a human review is still fast.
  5. Don’t rely on user voting for rankings. People want their information in as few clicks as possible.
  6. Allow for longer Titles, Descriptions and Keywords.  You can better capture what a site is about that way.  We kept these short because of dialup slowness, but that isn’t a problem anymore.

 

Interested in the directory hosting side of the old SK?  I will have more on that in a later post.

 

This was also posted to
/en/linking.

 

 

A project of Web Directory Reviews Org, the topics under discussion include directory reviews, editing, administrative issues, SEO, hosting, & related subjects.

Bookmark: Web Directory Forum

 

Update:  The place is abandoned, a ghost ship.  With some spammers running around as the most recent posts.  To bad they once had some decent discussions.  Never mind.

This was also posted to
/en/linking.

Also on: