风之谷: Web Search

Google如何评价Blog的重要性

Tue, 06 May 2008 09:06:50 GMT

读了一下Google关于Blog Search的Patent。总结一下Google是如何评价Blog的重要性的。

正面的指标：

订阅数：查看blog在各个RSS定制服务中的被订制的数量。被订制的越多，blog越重要。但同时要尽量避免"订制Spam"（Spamers通过订制自己的博客来提升其质量）。避免的方法包括验证订制人的ID和IP。
搜索点击数：查看blog作为搜索结果时被点击的次数。当一个blog同时和其他一些blog作为搜索结果而又被用户点击时，Google会认为这个blog比较引人关注也因此比较重要。
Blogroll链接数：很多blog都有自己的blogroll。blogroll包含一些对外的链接（很多都是blog)。包含指向一个blog的链接的blogroll越多，该blog就越重要。
来自高质量的blogroll的链接：高质量的blogroll的链接大多都指向著名的或值得信任的blog。
来自高质量的blog的blogroll的链接：这里的假定是著名的或值得信任的blogger很少会放链接连向spam blog。
有Tag：有Tag表明作者对文章比较认真负责，呵呵。
来自邮件和聊天记录的链接：看来Google也会通过Gmail和GTalk来修正blog的重要度。
PageRank：PageRank越高对应的blog也就越重要。考虑到blog的更新比较频繁，最新的blog post可能还没有PR。这时可以用对应的blog的PR来代替。

负面的指标：

更新频率：更新过于频繁或非常有规律的博客会被认为是在spam。因此会被降低重要性。
feed内容和blog内容的不一致：考虑到spammer可能会为了提升自己的排序而把有价值的内容放到feed里面，同时在博客内容里面放一些指向不相关内容的广告链接。为了惩罚这种情况，对于feed内容和blog内容不一致的情况，要降低blog的重要性。
重复内容：有些spammer为了让某些内容能够多次长时间的出现在feed里面，会重复发布同样的内容。这样的情况会被惩罚。
用词不当：Google会收集一些spam的文章，做一些词频的统计分析（比如提取bi-gram或tri-gram）。一旦发现某个博客含有过多的spam的词汇，就会降低其重要性。
博文长度分布：一些spammer会用机器自动产生和发布一些内容长度相同或类似的文章。Google会统计文章的长度分布。对文章长度分布过于均匀的，会降低其重要性。
链接：有些spammer会在一篇文章或一个博客中插入多个指向同一网页的链接以增加该网页的PageRank。为了惩罚这种情况，Google会降低该文章或博客的重要性。
广告：如果一个博客含有过多的广告，它的重要性会降低。另外，一般博客会包括三方面的内容：最近发表的文章；blogroll和metadata（比如博主的信息）。通常广告会放在blogroll或metadata部分。对于放在文章部分的广告，Google会予以惩罚。

FAROO---P2P Web Search

Mon, 07 Apr 2008 07:56:51 GMT

目前绝大多数的搜索引擎都是server-client结构的。考虑到搜索本身的实时性，高并发性和海量的存储，这种架构对服务器的要求很高。如果换个思路，采用P2P的方式来做搜索，那么很多问题都有了截然不同的解决方案。FAROO就是这样一个P2P搜索引擎。FAROO有如下几个特点：

分布式的crawling：当某个用户打开了一个网页以后，这个网页就会被加入到P2P网络的分布式存储中。这种crawling机制和已有的根据链接的方式有很大的不同。假设一个理想的情况，当FAROO的用户非常多以后，这种用户触发的crawling机制能做到几乎实时的覆盖全网！
分布式的indexing：FAROO没有一个中央索引服务器。每个FAROO的用户都是它的索引的一部分。每个用户负责一些词。如果某个用户访问了一个包含某个词的网页，那么这个网页的URL将会被存储在对应该词的用户的索引列表中。考虑到用户不会一直在线，会有多个用户对应同一个词。一旦一个用户离开了，其他用户可以补充上去。
分布式的Ranking：与常用的根据链接分析的方式来决定网页的重要性的方式不同，FAROO采用用户浏览次数来决定网页的重要性。这可能可以克服比较新的高质量网页在链接分析中不利的问题。
个性化的Ranking：FAROO的客户端可以搜集到用户访问了哪些网页，这样它有可能可以提供一些个性化的查询。

我能想到的FAROO的几个问题：

如何保证效率：因为index是分布存储在很多peer的机器上的，如何快速的搜集和集成这些index data是个很严重的问题。
如何保证覆盖率：FAROO在早期的时候用户不会很多，这个时候他的index会很有限，而且同时在线的用户也不多（我试了一把，目前大概也就几十人同时在线），这样很多查询都会找不到结果。当然，一旦他的用户数突破了一个瓶颈，情况会大不一样，但如何达到这个Tipping Point看起来是很有挑战的。
如何评价网页质量：链接分析已经被证明了是网页质量分析的一个很有效的工具。FAROO的浏览次数指标更偏向于热门的东西，对挖掘长尾信息不利。

Searchme

Tue, 01 Apr 2008 09:28:49 GMT

从Google创新性的推出异常精简的搜索界面，以达到让用户在最短的时间内找到自己感兴趣的网站并离开以来，主流的搜索引擎的界面一直都延续了这种风格。近年来，也有过一些尝试，比如Clusty的搜索结果聚类，Ask.com的分类查询提示，A9.com的多搜索引擎mash-up，Answers.com的搜索结果聚合，甚至是Microsoft的“美女”搜索Ms.Dewey。但这些尝试都没有取得突破性的成功。最近，随着Searchme开放自己的private beta search engine，搜索领域又出现了一个颇有生机的参与者。

Searchme是05年6月成立的公司。创始人是硅谷成功的企业家Randy Adams。他成功的创立了一系列的公司，包括最早的Internet Shopping网站Internet Shopping Network，著名的喜剧视频网站funnyordie.com等。他还是Sequoia Capital和Yahoo!的媒人和Yahoo!早期的director。在Sequoia的倾力支持下Searchme已经融了四轮资，共计31million$的资金。搜索引擎是一个竞争异常激烈的市场，有Google, Yahoo, Microsoft这样无比强大的玩家。但同时，搜索引擎市场也是一个有无限潜力的市场，任何有实力，敢于创新的参与者都有机会从中获利。正如Randy所言，Google目前的模式已经让它相对稳定的拥有了60%的市场，不太可能会做本质上的创新。而没有任何包袱的像Searchme这样的小公司可以尽情的发挥自己的创造力，从不断的创新中探索能够提高用户搜索体验的方式。

Searchme的搜索体验有两大特色。

1）对查询和网页的分类：有了这些分类用户就可以快速的定位自己感兴趣的网页。目前阶段，分类还不是多级的，如果能提供多级的分类就更方便了。因为是自动的分类，难免会有分不准确的时候。如果能根据用户的浏览行为来不断的优化分类结果，那么分类就会更准确了。用户的浏览行为还可以用来调整查询的分类顺序，比如根据分类的点击数来排序分类。一些很少有人点的分类甚至可以过滤掉。

2）网页的图片快照：Searchme除了支持用传统的文字方式来显示结果外，还采用了一种类似于苹果的 iTunes copycat的方式来显示搜索结果的图片快照。这种图文并茂的相信会更容易让用户定位自己感兴趣的网页。图片显示也为整合adsense和adwords提供了可能。具体来说，就是可以同时考虑查询和当前的网页的上下文信息在网页的图片快照中嵌入广告。

下面是Youtube上的一段介绍Searchme的视频：

Google Universal Search

Wed, 23 May 2007 09:44:56 GMT

Google released its universal search recently. The key idea is simply as search and display all kinds of information in a unified framework. As the application of universal search framework, youtube is integrated into the general search results. As Pruitt said on SearchEngineWatch, "For example, YouTube is immensely popular, but there are huge segments of the online population (probably older) who search via Google regularly, but have never visited, and are unlikely to directly visit YouTube.com. Universal Search could help them reach that market,"

More and more vertical searches of Google will be seamlessly integrated into its general search. The key issue here is query categorization that is to understand the search intent of users and categorize it into several verticals. For example, a query like "Tiger woods photo" should trigger Image Search and "Content-based image retrieval survey" should trigger Gogole Scholar. I guess Google China will apply the framework for Chinese search soon and integrate its Chinese specific vertical searches, e.g. its latest 生活搜索 (daily life search including real estate and ticket search). This will be a huge threat to those vertical search engines like Kooxoo. It's a kind of monopolization just like the binding of IE with Windows. I guess Google will face some lawsuits soon:-)

Baidu likes my blog too:-)

Mon, 14 May 2007 00:28:41 GMT

See this:-)

Google really likes my blog:-)

Sat, 12 May 2007 04:13:22 GMT

Search with social search advertisement using Google. There are more than 0.1 billion results. My post ranks FIRST:-)

垂直IM和社会化搜索 (Vertical IM and Social Search)

Thu, 22 Mar 2007 17:44:07 GMT

受Keso最新一篇关于百度和腾讯的文章的触发，我也来谈谈我对搜索和社区的一些想法。

和垂直搜索类似，IM也应该是垂直的。比如针对C2C（淘宝）的旺旺。针对搜索也应该有相应的IM。这样的IM比general的IM如 (腾讯的QQ) 有个优势是搜索服务可以更主动的帮助用户找到可能感兴趣的朋友，进而建立更多的连接并增强社区的关联度。而不是像目前这样只是把真实世界的朋友搬到了网上。结识新朋友完全是碰运气的行为，呵呵。

另一方面，对搜索来说，针对一些搜索意图和搜索结果的交流和讨论可以用来提高搜索的准确性。具体的模型和算法我正在目前的一些关于娱乐和学术的项目中尝试。相信最终搜索，社区，IM，Q&A这些东西都会无缝的整合在一起。这天的到来也不会很远了，呵呵。那个时候的搜索才是真正意义上的social search。

解密Chacha (Understanding Chacha)

Tue, 20 Mar 2007 16:13:55 GMT

To figure out the underlying process of Chacha, I did another experiment with the query Chacha. Following is the communication results. Actually, I just discussed with the guy about their scheme and algorithm. Now they have thousands of guides working for Chacha now. They are organized into four levels: Apprentice, Pro, Master, and Elite. Masters have the hiring authority. The earning per session is related to the level. The higher the level, the more pay you will earn. For the new comers, they have some training process too. Currently, they have too many applicants so that they have to suspend the applying process for they don’t have enough resources for the training. When it accepts new guide, I’ll apply for it to figure out the details of this interesting engine.

For the underlying algorithm, according to the conversion and my guesses, there is a TWO layer taxonomy with hundreds of classes. Each guide is allowed to select up to 50 classes (keywords) from the taxonomy. The guide who has served me selected 18 classes. For the query, they should have some classification algorithm to automatically classify it into the classes of the taxonomy. Based on the classification results, Chacha will select a guide for the query (the searcher). During the search session, the guide will change the class of the query according to the feedback of the query intention of the searcher. In this way, Chacha could refine it’s classification results of the queries and accumulate more training data for future classification tasks (a smart way). I have not figured out how it will handle the ambiguity issue of queries. Maybe, it should consider about using multiply guides for a queryJ. For more details about Chacha, please refer to the following links provided by the guide:

ChaCha Search - Connect and search with a live ChaCha Guide | | Sign up to become a ChaCha Guide
Interested in learning how to become a ChaCha Guide
http://www.chacha.com/info/guidesignup

Guideshq.com - Making Better Guides, One Search at a Time
Chacha.com is set to revolutionize finding information on the internet. It succesfully augments the the simple search function with the power
http://www.guideshq.com/

ChaCha Guides Network :: Index
Do You Want to Become a Guide? Are you interested in working from home any hours you would like on your computer as a guide for ChaCha? Look here for more information!
http://searching-4-cash.com/forum/index.php

Wired News: Get Rich (or at Least Paid) Quick
The ChaCha guides earn between $5 and $10 an hour as a starting wage
http://www.wired.com/news/technology/0,71850-0.html

The communication between Janine (a nice guide) and me (I rated her excellent):

Status: Connecting ...

Status: Looking for a guide ...

Ring: ...

Status: Connected to guide: Janine(84173)

Janine(84173): Welcome to ChaCha!

You: hi

Janine(84173): Hello

Janine(84173): Can you please be more specific on your search?

You: I'm looking for a guy working at Chacha

Janine(84173): A Master guide?

You: What is master guide?

Janine(84173): A Master guide is the guides that are allowed to invite new guides

You: Yes, he is

Janine(84173): Ok. Is this a specific person?

You: Actually, I'm very interested in the role of search guide

You: I wish to be one

Janine(84173): Ok! Would you like me to give you information on becoming a guide?

You: Sure, that's great

You: I'm interested in the relationship between a guide and Chacha

Janine(84173): Please give me a moment and I will you the information you are looking for.

Janine(84173): What do you mean?

You: For example, I want to know how much will Chacha pay for a session

You: I think I'm a good searcher and I hope to help other people on search tasks.

You: At the same time, I hope I could earn something too.

Janine(84173): Pay depends on the level you are at.

You: I think Chacha is a great service that could fulfill my wishes

Janine(84173): It is a great service.

You: How many levels currently?

Janine(84173): I just sent you an article on Wired News that talks about the pay for ChaCha guides.

Janine(84173): There are 4 levels: Apprentice, Pro, Master, and Elite

Janine(84173): Right now, ChaCha has temporarily suspended invites but the websites I gave you (in exception to Wired News and ChaCha) will let you know when it is back up.

You: Which level are you at?

You: Why suspend?

You: not enough money to support more guides?

You: What a pity:-(

Janine(84173): It is suspended because there have been a lot of new guides that have applied and they need to be trained.

Janine(84173): But, it is only temporary.

You: Is there any test process to judge the qualification of an applicant?

Janine(84173): The GuideSHQ website I sent you does but it is very simple.

Janine(84173): The ChaCha Guides Network will keep you in touch with Master Guides and most of them will just ask for basic information.

Janine(84173): For example, how long you are on the computer per week, if you are a United States citizen over the age of 18, etc.

You: ok. got it

You: I have a question here

Janine(84173): Ok. Feel free to ask.

You: How dose Chacha select a guide?

You: Does each guide have some profile to describe his or her expertise

Janine(84173): ChaCha does not necessarily "select" a guide. You are just invited by a master guide.

Janine(84173): No, not necessarily. Just as long as you are good at searching, you are qualified. As a ChaCha guide you will be given the option to select certain keywords that you know more about.

You: Sorry. I mean for a user. If he or she input a query, how will chacha select a guide to serve him

You: How many keywords will a guide select generally?

You: For example, how many keywords have you selected?

Janine(84173): I do specifically know. I do know that it has to do with your (a Chacha guide) keywords. It seems pretty random at least.

Janine(84173): Right now, I have about 18 keywords but I can have as many as 50

You: Will there be any guide on the selection of the keywords?

You: I mean is there a taxonamy

Janine(84173): There is what we call a "Keyword Tree" which just categorizes your keywords

You: How many levels of the tree?

Janine(84173): For example: your ChaCha search was categorized under Computers>Internet

You: ok. I see

Janine(84173): There are about 15 Levels but hundreds of SubLevels

You: Sorry, what do you mean by sublevel?

You: I mean how many layers does the tree have?

Janine(84173): For example: Computers is a main level and "Internets" is one of the Sublevels

Janine(84173): There are three levels of the tree.

You: ok. I see

You: So totally how many nodes does the tree have?

You: Just an approximation is needed

You: I want to know the difficulty of selecting keywords.

Janine(84173): I cannot give you the exact number but I suppose that there are hundreds of keywords

Janine(84173): It is not difficult at all.

You: ok. i see

You: But how to decide the keyword or class of a query?

You: will that be decided by a person or machine?

Janine(84173): You select based on your knowledge of the keyword.

Janine(84173): You decide your own keywords.

You: I notice that for some query, the keyword will change according to the feedback information

You: for example, if I told you that I want to find a car named as Chacha

You: will you change the class of Chacha to mobile?

Janine(84173): Yes, then we (the chacha guide) would change the keywords

Janine(84173): Yes exactly.

You: But, for some word, for example, Jaguar, there are different meanings?

Janine(84173): If you need any more information, the websites I sent you will answer all your questions.

You: got it

Janine(84173): Yes, then the ChaCha guide will have to clarify the search

You: But will that information be stored for future uses?

Janine(84173): Yes it does.

You: I mean if another guy come later and search for Chacha again

You: and now he wants this site

You: but the new guide for him will give him the car

You: Anyway, thank you so much for your time

Janine(84173): No, the guide should not unless the InfoSeeker asked for it

Janine(84173): You are very welcome. If you have any more questions just visit the sites I sent you. They will help you

You: I'll take a good look at the sites.

You: Thank you

Janine(84173): Have a wonderful day and please come back again!

Janine(84173): You are very welcome.

You: yes. I will

You: Hope I could be your peer later:-)

Janine(84173): Thank you for using ChaCha.

Janine(84173): Maybe so.

Janine(84173): Goodbye.

You: Bye

Janine(84173): Please RATE ME. Thanks for using ChaCha.

Status: Session ended.

感受Chacha (Experiencing Chacha)

Tue, 20 Mar 2007 14:28:21 GMT

很多人都说下一代搜索是社会搜索(Social Search)。一种对Social Search的理解是如Eurekster这样的Social Powered Vertical Search Platform。Wikipedia的创始人Wales也想用类似的方式来打败Google。而更直接的一种理解就是雇人（暂且叫这些人search guide）来搜索。这种模式和已有的搜索模式并不矛盾。已有的方式是提供了一些基本的工具。而帮助他人来搜索的search guide们也需要依赖传统的如Google这样的工具来完成分配给他们的搜索任务。在这种模式下，一个核心的技术问题是如何合理有效的分配查询。一方面，那些search guide们需要比较准确的描述自己的专长。另一方面，对用户提交的查询也要做比较准确的自动分类以找到对应的search guide(s)。Chacha是这个领域的先锋。他是最早采用这种模式的搜索引擎。也是目前这个领域最引人注目的。我用两个query感受了一下Chacha这个特色服务。

第一个查询是比较popular的查询Tom Cruise。如下是整个交流过程:

Status: Connecting ...
Status: Looking for a guide ...
Status: Connected to guide: Roxanna(17533)
Roxanna(17533): Welcome to ChaCha!
Roxanna(17533): Hi
You: hi
Roxanna(17533): what can I find for you on tom cruise
You: I want to find tom cruise's girl friend in his high school
Roxanna(17533): your search is who tom cruise had for a girl friend in high school
You: yes
Roxanna(17533): well, I am not sure if we can find that , but I will certainly do my best to see if we can, please give me a few moments while I search
You: the name, birthday, hair style, height and weight of her
You: and also what she has been married to
You: you can send me the answer to scenery_jf@hotmail.com
You: I have some urgent things to do now. Thanks
You: Bye
Transfer: You are being transfered to another guide who can help you search even better!
Status: Looking for a guide ...
Status: Connected to guide: mark(52951)
mark(52951): Welcome to ChaCha!
mark(52951): Hi
mark(52951): Please be more specific as to what you're looking for on this topic.
You: I want to find tom cruise's girl friend in his high school \
You: the name, birthday, hair style, height and weight of her
You: and also what she has been married to
You: Specific enough?
You: ???
You: Can you answer that in 1 minute?
mark(52951): Thank you for using ChaCha!
Status: Session ended.

对于两个guide我给的评价都是bad。不是因为他们没有帮我找到答案。这么八卦的问题我压根也没指望有什么好的答案。而是因为他们的服务不够专业。比如第一个人，他完全可以答应我的请求，回头做些调研然后发信给我的。而第二人一旦觉得问题有难度，马上就闪人，sigh......。其实我觉得这种人力搜索相对于机器搜索来说，一个最有优势的地方就是解决一些比较困难的查询。如果是很简单的查询，Google完全可以在他们connecting的时间内就给用户满意的答案了，呵呵。Chacha和目前的Q&A也很类似。不过Chacha更强调实时性，而Q&A更强调答案的准确性。相对来说Q&A的方向是更有道理和前途的。

另一个查询是Feng Jing。第一个人找了两个关于那个体操运动员的网页，在我的一再提示下（比如做image analysis方面的研究人员），还是不能给我答案。最后抛给我Wikipedia里面的一个条目，然后就闪了，呵呵。我还是给了个Bad的评价。接下来来了个稍微专业一些的女士sherri。如下是我和她的对话：

Status: Connecting ...
Status: Looking for a guide ...
Status: Connected to guide: Sherri(17326)
Sherri(17326): Welcome to ChaCha!
You: hi
Sherri(17326): hello
You: Can you help me to find some info about Feng Jing
Sherri(17326): just a moment
Sherri(17326): is this first link who you are looking for?
You: Yes, this is the one.
Sherri(17326): ok, good, now I know whwat I am looking for
You: I want to find his wife's blog site
Sherri(17326): what is her name
You: I don't know
Sherri(17326): ok
You: Otherwise, I won't start with his name
You: How long will you take?
Sherri(17326): Thanks for being patient! Rest assured I'm finding the most relevant results for your search.
Sherri(17326): well I revised the search to seek his wife
Sherri(17326): just a momnet
You: ok
You: any progress?
Sherri(17326): if he has a wife, there is nothing posted about her yet
You: I'm sure that she has a blog site
You: I just can't remember the site address
You: Anyway, could you do more research on this and send me the results later to me via email
Sherri(17326): giv eme you eamil
You: scenery_jf@hotmail.com
You: Thanks in advance
Sherri(17326): ok got it
Sherri(17326): Are these results sufficient?
You: no
You: Thank you for your time
You: CU
Sherri(17326): no problem
Sherri(17326): bye
Sherri(17326): Please RATE ME. Thanks for using ChaCha.
Status: Session ended.

她先是很快找到了我的主页。我肯定了她的成果后，要她帮我找我老婆的blog的链接。她在一番努力后说没有这个东西，呵呵。在我的坚持下，她又努力了一番，最后也是抛了个Wikipedia的条目给我。后来我把email给她，让她找到后发信给我。24小时后没有任何回复。当然她相对于之前的几个还是要更专业些的，因此我给了个ok。

我不清楚Chacha是如何和这些Guide合作的。是按session的个数给钱，还是按质量给钱，或者是用质量加权。

总之这种利用guide的模式是值得借鉴的。也是很有勇气的一种尝试。但是目前Chacha的服务质量和方式离用户的期望还是有相当大的距离的。

This is really COOL.

Wed, 07 Mar 2007 21:40:03 GMT

Search with 最近的股票 in Google, my post is ranked first!

BTW: There are a lot of visits from Google and Baidu with the query 新婚感言 for lots of people got married recently:-).

Update on Kanweb--Query Substitution of Google

Wed, 21 Feb 2007 07:43:15 GMT

I did some experiments with Google and had some interesting findings on the query analysis and understanding of Google. More details could be found at: Query Substitution of Google of KanWeb.

My Swicki on Web Search and Mining

Sat, 10 Feb 2007 17:00:51 GMT

This is my swicki on Web search and mining. Welcome to it:-)

Trick of Baidu and Google.cn---Leverage UCC in core search

Sat, 13 Jan 2007 16:24:05 GMT

l Why UCC?

UCC is generally defined as the user (not creator) created content, such as Blogs, forums, comments, reviews, Q&A and so on. With the flourish of Web 2.0 that emphasizes the data and users, tremendous UCC are generated everyday. Comparing to the traditional Web content, UCC has the following three unique characteristics:

First, the quality of some UCC is really high. They are the foundations of many great Web companies, such as Amazon (the rating and comments), Ebay (the credits of the sellers) and Myspace (the profiles). Even with the same purpose, UCC will be more important than traditional contents. For example, in the travel domain, the information in the travel journals of real travelers is much more valuable than the information from the official websites. There is a very strong reason behind the high value of UCC, which is the power of users and communities. However, how to assess the quality of UCC is a nontrivial problem. It might be more challenging than the assessment of common Web pages because there is little link information. Therefore, the link analysis techniques such as PageRank will be unsuitable. I’ll discuss this issue latter.

Second, for most of the UCC, there is temporal information that could be used to gauge the importance and freshness of the information. For example, the temporal information for Blog posts could be the time the posts were posted. There are at least two benefits. On the one hand, the most appropriate information could be indexed in our index. Imaging that in the near future, there will be too many contents to be indexed with the existing servers and infrastructure, how to select the optimal subset is a very crucial issue. On the other hand, during the online search process, the most relevant and valuable information could be returned to the users.

Third, there will be less spam in UCC. Anti-spam is treated as the most critical thing of a commercial search engine. It is a game between spammers and anti-spammers. I believe, the game will not over as long as the search engines exist. Currently, the advantage of spammers is the number of people and the nature of behind the scene, while the advantage of Anti-spammers is the deeper understanding of their own search engines. The game is similar to that between criminals and police. There will always be criminals. However, the police will keep the weather in most cases. Why? I think the main reason is that common people don’t like bad guy and would like to help the police to catch them. Without leveraging the huge resource of good citizens, I think the police can’t be such successful. Back to the anti-spam issue, leveraging common Web users is the key. Since most of the UCC are shared, judged and supervised by community users, the life of spammers will be very tough there. For example, in a forum of car, the advertisers could be easily identified by many other experienced users and will be kicked out forever.

l Current Status of UCC in Core Search

Given the aforementioned three advantages of UCC, they have been leveraged in both general core searches and many vertical searches. For the vertical search, one good example is the “酷讯” of Tony Chen, former employee of MSRA. 酷讯 indexes and extracts daily life related information from many authoritative UCC-based web sites, e.g. forums. One successful application is buying train tickets.

In this report, I’ll focus on the UCC in core search. Actually, most of the existing leading search engines have leveraged a lot from UCC. For example, Yahoo! has acquired many Web 2.0 companies, such as, http://del.icio.us (the most successful Web page Tagging site), Flickr (the most successful image Tagging site), Webjay (a playlist sharing site) and the recent Bix (an online contest site). Besides these acquired techniques and data, Yahoo! has also innovated several own services, e.g. My Web, 360 degree and Yahoo Answers. Yahoo!’s next big bet is very clear: the social search (leveraging UCC). Google has also acquired some social search companies, e.g. Blogger (one of the earliest BSP), Orkut (the most famous Web site in Brazil) and the recent big deal Youtube. However, I think Google’s investments on social search is not as huge as Yahoo!. Different from the acquiring strategy, Baidu built its own social services to generate UCC, such as “知道” (a Q&A service), “贴吧” (a forum with the topics generated and maintained by users), “空间” (a BSP). Similar to Yahoo!, Baidu highlights the social services very much by putting them at distinct places. Yanhong Li (the CEO of Baidu) has officially claimed that the next strategy of Baidu is to “search human brains”.

Besides some technique forums, Microsoft has Live Spaces, a very important and unique service that continually producing valuable UCC WW. Live Spaces is the BSP with largest number of registered users. I’ll show how our competitors have leveraged from Spaces in the following experimental results.

To show how and how much existing search engines have leveraged from UCC, I design the following experiments with my Blog: 风之谷. Since most of my Blog posts were written in Chinese, I considered three Chinese search engines: Baidu (the dominant Chinese search engine), Google.cn, and Live search with the preference set to search Chinese page only.

(1) To ensure that my space is in the index of all three search engines, I searched them with the query “风之谷 spaces”. With Baidu, my space ranked 4^th. There are many spaces named as “风之谷” (14 in the first 20 results). Surprisingly, my space ranked 1^st with Google.cn. More surprisingly, I can’t find my space with the query in the top 1000 results of Live. I have to expanded the query to “风之谷 spaces Michael dell” (Michael Dell is one of the titles of my posts). This time, I found my space. This results show that both Baidu and Google deem Live Spaces as very important sources while MS the owner of Live Spaces doesn’t think so.

(2) I searched the three engines with each title (only title without other information) of my Blog posts and collected the following information: the rank of my post, the number of Blog posts from any Blog sites, and the number of Blog posts from Live Spaces in the first 10 results of each search engine.

• Position of my posts:

– 新婚感言 (long, 18 comments, 8/6) : 9 (Baidu, 104,000); 1 (Google, 72,600); >100 (Live, 3,232)

– 李彦宏马云 (very long, 2 comments, 8/13) : 14 (Baidu, 199,000); 11 (Google, 141,000); >100 (Live, 10,907)

– 按摩椅 (very short, 4 comments, 8/23): >500 (Baidu, 511,000); 159 (Google, 541,000); >500 (Live, 67,381)

– 朱雅琼 (short, 0 comments, 8/25): 323 (Baidu, 290,000); 70 (Google, 104,000); >500 (Live, 5,620)

– Michael Dell (long, 3 comments, 8/30): 3 (Baidu, 174,000); 12 (Google, 311,000); >200 (Live, 3,851)

– 一句有道理的话 (short, 1 comments, 8/31): 3 (Baidu, 4,410,000); 1 (Google, 2,290,000); >200 (Live, 678,082)

– 照片更新 (short, 1 comments, 11/6): >300 (Baidu, 898,000); 28 (Google, 2,750,000); >500 (Live, 4,506,404)

• Number of Blogs (Spaces) in top 10

– Why Blog?: 10/3 (B, 10,700,000), 9/3 (G, 1,260,000), 7/0 (L, 244,633)

– 运动与伤病: 1/0 (B, 12,300), 0/0 (G, 3,270,000), 0/0 (L, 16,378)

– My idols: 9/6 (B, 132,000), 7/1 (G, 145,000), 4/2 (L, 3,037)

– Harry Potter 4: 5/4 (B, 1,310,000), 1/0 (G, 963,000), 0/0 (L, 16,279)

– 与glass聊天心得: 9/1 (B, 10,900), 2/1 (G, 13,400), 4/1 (L, 162)

– Search MSN space: 9/9 (B, 215,000), 3/3 (G, 1,250,000), 4/1 (L, 38,693)

– 打台球: 4/0 (B, 1,200,000), 0/0 (G, 218,000), 0/0 (L, 93,638)

– Patent Meeting: 4/3 (B, 26,400), 0/0 (G, 281,000), 0/0 (L, 1,442)

– 唱歌: 2/1 (B, 31,200,000), 1/0 (G, 4,640,000), 0/0 (L, 1,118,860)

l How to leverage UCC in a better way?

To maximumly leverage the UCC, following three key issues should be resolved:

1) Quality assessment: many kinds of information could be used to assessment the quality of UCC, such as, the number of replies, comments, clicks, and ratings. Within a community, the link information could also be considered. For example, the latest released feature of friend links of Live Spaces could be leveraged. The temporal information should be considered too. The potential usefulness of temporal information has been discussed earlier.

2) Anti-spam: although there might be less spam in UCC, we should realize that spammers are very smart guys. They will figure out ways to spam us once they know our search strategies. For example, spammers could also build a fake forum or write some fake Blogs. Anti-spam in UCC is another interesting and challenging issue.

3) Query understanding: for some very general queries, the common contents, e.g. the authoritative sites should be ranked very high. For some specific queries, the UCC should be ranked higher. Since the quality measurements of common pages and UCC are very different, a proper learning algorithm should be used to learn the optimal fusion strategy. For example, for the queries with very high quality general search results, the UCC should be used less. I found that searching with a very popular and general query with Baidu, the UCC will also be mixed into the top results.

How Google will rank this post (search results of Google for the query "黄金甲")

Thu, 11 Jan 2007 16:01:06 GMT

风之谷: Web Search

Google如何评价Blog的重要性

FAROO---P2P Web Search

Searchme

Google Universal Search

Baidu likes my blog too:-)

Google really likes my blog:-)

垂直IM和社会化搜索 (Vertical IM and Social Search)

解密Chacha (Understanding Chacha)

感受Chacha (Experiencing Chacha)

This is really COOL.

Update on Kanweb--Query Substitution of Google

My Swicki on Web Search and Mining

Trick of Baidu and Google.cn---Leverage UCC in core search

How Google will rank this post (search results of Google for the query "黄金甲")

张艺谋《满城尽带黄金甲》_影音娱乐_新浪网

《满城尽带黄金甲》电影官方网站

张艺谋《满城尽带黄金甲》-搜狐娱乐

满城尽带黄金甲_腾讯娱乐

张艺谋06年年度巨献——《满城尽带黄金甲》_TOM娱乐

千里之外

名称最特别的传情达意最动人的特快专递送一首歌，胜过千言万语…… [汤潮]

黄金甲- 标签- 搜狐博客

标签：黄金甲-新浪播客

:: 满城尽带黄金甲::