Google started out ridiculously smart, but it’s been stagnant for too long. They clearly need competition.

If I were building a search engine…

1. You need to start building your own index of the web.  That means you need a crawler (robot) and it needs to be good.  It takes time to build an index and it is not cheap.
2. I would gather in info from other providers: Wikipedia, Wolfram Alpha whatever I could cobble together.
3. Until your own index is ready, you need to have a search results from a big search engine.  There are only two left: Google and Bing in English.
4. What happens if Google or Bing refuse to sell you a search feed or if they refuse to renew because you are getting too big?  You could probably hammer together a good blended backfill feed by combining Mojeek, Gigablast and Yandex using your own algorithm.  You either lead with your own results and use the other three as backfill or you blend your own results in with the three others in a sort of seamless metasearch.  (You do need to plan for “what if” scenarios.
5. All the above is to buy you time while you learn and refine how to crawl the web on your own.
6. Eventually you roll out your own search index as the backbone of the search results with the others maybe on standby for queries you are weak on.

Qwant seems to be the other player.  Their strategy is more open:  they are actively building an index by crawling the web.  Bing is providing backfill.  It is not obvious where Qwant’s index ends and Bing’s begins.  The results are good and seamless.  With a little luck, marketing and money Qwant will eventually need to use Bing less and less.  This too is a good plan.

Fortunately, Bing is willing to sell it’s search feed to just about anyone that can pay the fee.  For now.  I do not think Microsoft really knows what to do with Bing, except to somehow milk it for all the dollars they can get until they decide what to do with it and how it fits in with the company strategy.

Search engines are important but they are not as important as they were before the social media silos of Facebook and Twitter.  You don’t need Google to find a company’s Facebook page.

If I were trying to build a search engine today those are some of the things I would try. All this could change in a year.

Just follow these simple instructions to create the perfect shakespearean insult: [Via]

The finding of the Coroner’s Court is that 1990’s style webrings are officially dead.

Evidence of Demise

• Two of the three remaining Ring Hosts are broken.  Both Webring.org and Ringsurf.com are broken in such a way that nobody can sign up as new members and it has been that way for a time.
• The third remaining ring host, Webringo.com, is functioning.  It just appears that they have no traffic.  But points to them for keeping things in working order.
• New 1990’s style rings created have had zero take up.  This is too small a sample to really tell but it is a small indicator.
• A newer Indie-tech style webring has little useful traffic despite a user base.

What Killed the Webring?

1. Generation shift.  Web 1.0 users “surfed the Web” so they liked the idea of a curated grouping of websites they could surf to.  Modern web users are used to helicoptering into a single web page via a search engine.  They only care about that page and it’s information, not websites or surfing.
2. Rings are passive.  They sit there and wait to be discovered.  They are passive in recruiting members and they are passive in finding users.  Passive cannot break through the noise of the modern web.
3. Search engines used to suck.  That was one reason for webrings you couldn’t find anything.
4. Geocities, Tripod and Homestead.  Webmasters on these free hosts wanted to be found, joining a webring got you traffic.  Those free host webmasters were also familiar with HTML so they were not intimidated by having to put a ring code on their sites.  Modern webmasters use CMS’s and are more intimidated by messing with HTML code.
5. Young webmasters may have heard of webrings in passing but have never seen one in the wild.  They don’t know what they are. Ditto the public visitor, they don’t know what they are.
6. Commerce.  The web in the 1990’s was little used for commerce.  It was a place to explore, have fun, find neat things, exchange information and ideas.  Rings were good for explorers but not daily commuters.  Today commerce has taken over the web, efficiency rules so we can maximize sales, revenue and consumption. Webrings were never good for that.
7. Lack of traffic.  Webring hosts had hubs.  These were a directory of webrings organized by subject. Example.  Many tens of thousands of visitors went to these ring hosts to find rings to surf, because search engines sucked.  So a webring gained traffic from both the ring host and the ring codes on individual sites.  The biggest reason you joined a ring or started your own ring, was to tap into the hundreds of thousands of eyeballs at those hubs, the code on other websites was icing on the cake.

There may still be some life in old style webrings: it seems to me neocities.org is a perfect match for webrings.  But it would take some promotion.  A ring host would need to get listed in Neocities webmaster resources pages and it might catch on. They would be a good match just as they were for Geocities et al.  But it would take effort.

The demise of the webring does not make me sad.  It’s time has passed and there are better ways to find websites.  It would have been nice to have it as a tool in the fight against the Google search monopoly silo but it’s a bit like fighting Delta Force with a sword.

This is part of a series: See Part II Here.

This was also posted to /en/linking.

To Do’s

Internal ads:  I need to look into ad or banner system plugins for Wordpress.  I don’t need it right now but will need.  I am not talking about Adsense or outside ads, but I need to drive some traffic to my own content.  The big example would be the Blog Directory.  I need to let people know it is there.

Graphical banner ads are a pain.  Mainly because I’m terrible at making graphics.  The other thing is they can screw up the view on mobile.  The good thing is you set up the campaigns and forget it.

Text ads? Maybe.  Worth trying.

Placement?  I hate ads between paragraphs in articles but it is effective.

I need to think this through and see what free plugins are available.

More directory listings:  I need to add more.  I could easily add the famous blogs, but that really isn’t the focus of the blog directory, which is hopefully for good new or unfound blogs - the underappreciated.  Will work on this.

UK based Mojeek.com is the next privacy search engine I am going to make my default and use for a few weeks.  (There is a UK specialized version at Mojeek.co.uk, plus German and French versions.)

I did a quick look over before, but this test will be daily use as my default browser.  I will try my best to use it exclusively, without resorting to a backup search engine.

I really don’t know what to expect.  Mojeek has it’s own unique database that it is building by crawling the web with a robot.  But, unlike Duckduckgo or Qwant, it has no backfill from a massively larger search engine to back up the Mojeek search index.  They out there with no training wheels.  That takes guts.

So we will just have to see how I get on.

I don’t have to do this alone.  You can join me!  Just make Mojeek your default search engine for a few weeks and use it as your daily driver.  I would love to hear how you do and what you think.

My experiment with Qwant search engine is ending early.  On July 20th, 2018 I started using it as my default search engine for a month long test.  Today, August 7th, I am terminating further testing.

The test terminated when I got yet another captcha because they detected unusual activity from me and thought I was a robot. That never happens to me.  I just don’t type that fast.  When I’m trying to find something I do keep refining my search query.  I guess that is too much for Qwant.  They apparently only want dilettantes doing an occasional search. This is not a robust search engine for somebody who seriously works and searches.

The truth is, I was already getting irritated with Qwant.  Not for the quality of it’s search results, but because the search results page was taking too long to load.  The page is pretty but slow.  I’d want lean and fast results.  The actual search results were pretty darn good.

But I can’t use a search engine that craps out on me right in the middle of a project.

Pros:

Trying to build their own unique index.

Decent search results.

Privacy

Cons:

Slow

Not robust. Can’t handle multiple searches typed by a slow human typist.

Like: Best Browsers of 2018 | Expert Reviews.

For my own amusement and because the Wordpress plugin was free, I made a little bookshelf of free public domain ebooks available for the Kindle.  It’s a bit like having a Little Free Library on this blog.  I enjoyed them so I thought I would point them out.  They remind me of a gentler era.  Folks from the UK should be able to find the same free titles on Amazon.co.uk.

Bookshelf here.

Disclaimer: All the books are free and in the public domain.  I am not an Amazon Affiliate so I derive no benefit from this.  The button “Purchase from Amazon” just takes you to the Amazon page where it is free.  Just encouraging reading.

Enjoy.

I just installed OStatus plugin for Wordpress.  It’s a bit short on details as to how it all works.  Scant few controls too, one check box on the plugin panel about activating feed summary or some such and nothing on the Post screen.

For me the biggest reason to try it is better two way communication with Mastodon.

I always have thought web directories should have backfill from a search engine when you search them.  Especially, general directories.

Yahoo (dubbed “Mighty Yahoo”) was the first directory to add a backfill provider on it’s search results. First Alta Vista, then Inktomi, then Google, then Inktomi again provided backfill results for Y!.  Snap/NBCi directory had backfill and so did Looksmart.

How backfill worked was somebody would search the directory, when the directory ran out of listings that matched the query, you would see search results from the backfill provider.  (Being Mighty Yahoo’s backfill provider cemented Google’s reputation and proved to be one of the biggest mistakes Yahoo ever made.)  What was important is the search portal needed to provide something in a search that would satisfy the person searching, otherwise they would start going elsewhere. And they could go elsewhere because there was genuine competition in web search back then.

It gets harder for niche directories.  For example: if you go to a Star Trek niche directory and search for “uniforms” it is assumed you are only going to get uniform sites that are related to Star Trek.  In this instance you don’t want a general backfill giving results for police uniforms or nurses uniforms.

For backfill to work on a niche subject directory you almost need a place to put in an extra search “slug” to bring it into context.  So in our example, you need to have a place in your admin panel to add “star trek” as a term to the backfill.  So that somebody searching for uniforms on your Star Trek directory are automatically searching for “star trek uniforms” on the backfill.  The “star trek” is always added to the backfill.

It gets even harder for a local or geographic directory.  Frankly I think backfill should not be used with a local directory.

I did some looking around, Google and Bing are way too expensive to get a search feed from.  Smaller engines like Mojeek.com (or Mojeek.co.uk) or Gigablast.com provide search results API’s.  Both are affordable, but Gigablast’s is really really affordable.

Of course, I don’t know how to actually code this.  That’s what coders are for.  😀

But the trick here is this:

• The more listings you get in your directory the more users will use the search function.
• You have to have a search function.
• No matter how sophisticated your directory search is, you have a finite amount of listings.
• Mobile devices make the search function even more important.
• This is why most will need backfill.

Yesterday I noticed that there is an Add URL form at Gigablast search engine.  So I took a few seconds and added both of my current domains.  Why? Because it’s free, it’s there and it puts my domiains, both newish, on Gigablast’s eventual crawl list.  At least they know about them.

Back in the day, all the engines and directories had an add url form.  When a new directory or search engine launched all the webmasters would race over to add their sites.  Especially on the directories, it was easier to get listed in a new directory for free when they were new and needed to build an index.  The longer you waited the harder it became to get listed.

The whole idea behind having a website on the Web is to be found.  To do that you want to be listed in as many legit places as possible.  You never know where a faithful reader is going to come from.  We’re living in a unique time, where social networks like Facebook and Twitter have loosened Google’s throttling, monopoly, grip on web discovery.  We could go back to that overnight, so get yourself listed in as many search engines large and small as you can.

Gigablast has been around for years.  It used to supply almost all the meta-search engines.  That brought traffic. I think most meta’s have died because there are so few general search engines to get search feeds from that it is not worth running a meta-search.  But Gigablast still has it’s own index, it still crawls the web and if adding my URL’s aids them and me then it’s worth doing.

This is so simple, I don’t know why I didn’t think of it before.  You can make a decent blog search engine, for free, in minutes with Duckduckgo.

1. Go to the Duckduckgo Search Box page.
2. Do whatever customization you like.
3. In the field “Site Search” type in: blogspot.com,wordpress.com,medium.com

Medium.com is optional.  Copy the searchbox code and Paste it on an HTML page and you are done.

Pros:

• Because it’s DDG all searches are private.
• Blogspot and Wordpress are the two largest blog hosts on the web, at least in the Western World.
• Fast results with very few ads.
• DDG does not insist on having their branding all over the searchbox.
• You can do multi word, complex search queries.

Cons:

You are not searching for all blogs. Only blogs hosted on a subdomain “blogname.blogspot.com” are going to be searched.  Blogs on these same hosts that pay to use their own domain will be overlooked.  Also, blogs that are self hosted on their own servers will be overlooked.

Have fun. Adapt this to your own needs.  If you find this useful on your own site, come back and let me know how you use it.  Thanks.

Blogspot search allows to to look for blogspot blogs in this Blogspot blogs search engine . To help you search blogs on blogspot

This is actually a good find.  Very current, it searches all the blogs hosted on Blogger.com (aka Blogspot)  Very good tool for discovery.

Is Wordpress.com quietly building a blog based social network with Reader?

I have no inside information, but the answer is, They are half way or more there.

Aside

If I were a good (slick) blogger I would fill this post up with neat screenshots of Reader pages that illustrate each of my key points.  You could then squint at these unreadable pics to your hearts content.  Instead, just go to Wordpress.com and register.  You can see Reader for yourself.

/Aside

WP Reader is an aggregater of all the blogs hosted on Wordpress.com PLUS remotely hosted installs of Wordpress that use Jetpack.  Everytime you login to Wordpress.com the first thing you see is the Reader.  The Reader is also on all the WP mobile apps.  You can search for posts and blogs by keywords and subscribe to (follow) any blog in the index.  The Reader can also serve up the newest posts on the system whether you have subscribed or not.

1. The Reader is fast and attractive, well laid out and easy to use. It’s like a timeline.
2. It is everywhere and has a huge user base.
3. It can send you notifications if you want.
4. You can “Like” a post from Reader.
5. I think you can even comment on a post from Reader if the owner of the blog has comments turned on.
6. Reader is a very powerful discovery tool.
7. Reader is a powerful traffic tool. Not just one time traffic but repeat traffic. It is so easy to subscribe to a blog you will find yourself with followers soon after making a few posts.
8. You can see who follows you in Reader.  They have a profile page.

What if’s…

But what happens if they give you the ability to add RSS feeds from any blog hosted anywhere?

What would happen if they adopt Indieweb webmentions from the Reader and all WP blogs? So you could comment on another blog from your blog but all in Reader.  And get replies. These types of Indieweb feed reader (see webmention link above) are already available, so it would be just a matter of WP coding these capabilities into Reader.  Don’t ask me how, I’m not a coder.

With just a few more features a de facto social network would emerge.  I don’t know if it would succeed.  I don’t know if it would be a good fun community or be filled with trolls.

If I were WP, I would be at least thinking about it in my off time.

Micro.blog is also a blog based social network and seeing how well it’s features work tells me you can have a successful community built around blogs.

All this is just speculation.  A social network may be the furthest thing on WP’s mind.  All I am pointing out is that the foundations are there at least by accident.

The inspiration for this post came from Greg McVerry.  He very astutely asked the same question in Indieweb chat room a week or so ago.  But, other pressing matters were being discussed so no more was said.  But Greg’s observation got me thinking so I thought I would start a discussion.

Could it be done? Do you think they are doing it?