Quote from: phoenix_nl on May 31, 2013, 03:28:29 PM

Interesting idea, but I think it would be hard to cover all bitcoin related sites. You'd still need google for complete coverage I'm afraid.

you don't seem to understand how this works, so ill explain it for you. google works as a spider. every searchengine does. it searches domains for links and tags. it indexes these tags to and matches them to a link list. when you search google, you pull up a list of links that closely match your search term. basically if i set my search engine to index to a depth of six, and all bitcoins sites had some random hash tag like "hashBTC1321ljwljaslsdjaflasorewralesfjlsadfjasdl99999999999" i could set the spider(search engine) to leave the domain, and it would find every website on the internet with that tag in its meta tag list.

This would take months. the spider would just be cranking band with crawling the internet. the solution? what google does. "web master tools" is a really fancy multi featured way of letting web site owners do the part of the heavy lifting of the search engine process. this is the next step in my development. right now, i've got a form that gets your email address a description of your website, the url of your homepage and makes you fill out a captcha. when you hit submit if it verifies, it adds your site to qeue to be indexed by the spider. for my idea to be possible, on a global scale, i would need a virtual private server with very liberal mysql connection allowances. the trick is to run multiple instances of the spider at once to crawl pages and speed up the time it takes index. then you have temp tables which serve to hold site information until the spider crawls it. you have to clear these manually. So in essence, you can compete with google, but you have to have a damn nice hosting setup with a very flexible memory and mysql database access abilities. my ideal setup would be 100 simultaneous database connections, unlimited bandwidth, 3 databases, 10gb diskspace.