In Reply to Kicks Condor Cataloging Horror Fiction.
Track Expired Links: Many directory scripts had a bad link checker but that really didn’t work well. Basically you had to go through the directory categories and subcategories manually and get rid of dead wood. Now I was in a “hobby and fan” niche so a lot of sites listed were static sites on Geocities, Tripod or other free host. (Blogs, on Blogger and Wordpress.com, were new and didn’t have Pages yet) Surprisingly the free hosted sites were more stable than sites on their own domains. Webmasters on free hosts might abandon their site but they left them up and Geocities didn’t delete so even without updates their existing essays and info were still good. Sites on their own domains: would either go dark or get snapped up by the p*rn industry and go p*rn. We did have a way for users to report bad links but only a few went to the trouble. So clearing out dead wood was a time consuming chore and part of what I mean when I talk about difficulties of getting directories to “scale”.
Updates: No way to know really. On the Horror search engine I had the spider reindex the Index page every few months so changes made there would eventually show up.
Wikis as replacement for Directory: No and sorta. A directory is about navigating the Web. The job of a directory is to get rid of the searcher by helping them find what they want as quickly as possible. You are sending people off to a primary source. A wiki is about providing information and if it links out, that is a secondary function. Take Wikipedia as an example: Wikipedia (which I use daily) is a silo. It’s just not a commercial silo. Wikipedia has filled the vacuum left by the closure of free hosts like Geocities. Before Wikipedia the experts were all building pages about their areas of expertise either on free tilde pages from their ISP or university or on GeoCities and the like, now they are editing Wikipedia pages. Then Google intentionally filtered out free hosted sites, so it fell to niche directories and webrings to help free sites get found, then the plug got pulled and Geocites went dark. But that is how we found information before Wikipedia. Wikipedia is a mixed blessing. It provides a needed service - free, but it also smothers out many small informational sites from being created (unless they can contain more useful detail than Wikipedia.) It’s more efficient, but lacks funkiness and fun.
Blogs can act as curated directories. I have a small experimental one. But they really can’t do it as well as a directory even though they might aid in discovery they are improvisations.
Webrings are like taking a subcategory and adding navigation between like sites in that subcategory.
Crawler: Having a real crawler that you control is like having the secret decoder ring! Even if you can’t see all the uses right now, you will quickly find all sorts of new uses and utility. You can detect bad links, changed content, who they are linking too, etc.
Moderation: yeah I hate to say it but you will eventually need it. I closed down many of my directories partly because they were obsolete but partly because those old directory scripts did not have spam protection. They got spammed out by submit bots. It got to be too much work deleting out hundreds of spam Add URL’s each week from the review cue. Right now you are flying under the radar and the Indieweb movement is small, but once both your directory and the Indieweb blips up on the screen the spammers will come.
As you have pointed out, directories are still around us: Yelp, yellow pages, Craig’s List, eBay, Amazon etc are all directories.
Directories can exist without a hierarchy. The now defunct Searchking “search engine” was really just a linear directory. You submitted each page of a site (no crawler) you wrote a title, description and 10 keywords (Tags?) and your page was instantly findable in the index. That was okay in the early days of the Web when sites were static and small.
Your experiments are going to be a lot of fun. The technology has changed from my day and that will work in your favor. I can’t help with coding, that’s above my pay grade, but if you have any questions keep asking, I be happy to tell you what I can.
