Comments on: Google: How do you do it? http://kubasik.net/blog/2007/11/02/google-how-do-you-do-it/?utm_source=rss&utm_medium=rss&utm_campaign=google-how-do-you-do-it Kevin Kubasik's Personal Blog Sat, 17 Jul 2010 15:28:22 +0000 hourly 1 http://wordpress.org/?v=abc By: RyanTheRobot http://kubasik.net/blog/2007/11/02/google-how-do-you-do-it/comment-page-1/#comment-285 RyanTheRobot Sun, 11 Nov 2007 08:21:47 +0000 http://kubasik.net/blog/2007/11/02/google-how-do-you-do-it/#comment-285 I seriously wonder how google does half the things that it does... and how their product quality is constantly one of the best in the industry. They are such an awesome company... I seriously wonder how google does half the things that it does… and how their product quality is constantly one of the best in the industry. They are such an awesome company…

]]> By: Tobu http://kubasik.net/blog/2007/11/02/google-how-do-you-do-it/comment-page-1/#comment-284 Tobu Fri, 09 Nov 2007 14:55:25 +0000 http://kubasik.net/blog/2007/11/02/google-how-do-you-do-it/#comment-284 I've been reading these lectures: http://www.ee.technion.ac.il/courses/049011/spring05/index_files/Page337.html The second one touches on indexes. No huge insights there (delta compression maybe?), but it did clarify some concepts for me. I’ve been reading these lectures:
http://www.ee.technion.ac.il/courses/049011/spring05/index_files/Page337.html

The second one touches on indexes. No huge insights there (delta compression maybe?), but it did clarify some concepts for me.

]]> By: Kevin Kubasik http://kubasik.net/blog/2007/11/02/google-how-do-you-do-it/comment-page-1/#comment-283 Kevin Kubasik Mon, 05 Nov 2007 06:07:31 +0000 http://kubasik.net/blog/2007/11/02/google-how-do-you-do-it/#comment-283 @Tobu: We don't do this at the moment, I've tought about doing it, but it seems like it would cost far too much at retrieval time.. However I might look into a proof of concept implementation to test this, however its a lot of work if it ends up being too expensive =/ @noname: yeah, I checked out some dbm implementations, I couldn't figure out if we were gonna get compressed text content, it seems like most of them just dump the content on disk, much like our old system which took up far too much space... anyone know anything specific wrt this? @Tobu: We don’t do this at the moment, I’ve tought about doing it, but it seems like it would cost far too much at retrieval time.. However I might look into a proof of concept implementation to test this, however its a lot of work if it ends up being too expensive =/

@noname: yeah, I checked out some dbm implementations, I couldn’t figure out if we were gonna get compressed text content, it seems like most of them just dump the content on disk, much like our old system which took up far too much space… anyone know anything specific wrt this?

]]> By: noname http://kubasik.net/blog/2007/11/02/google-how-do-you-do-it/comment-page-1/#comment-282 noname Sat, 03 Nov 2007 02:55:32 +0000 http://kubasik.net/blog/2007/11/02/google-how-do-you-do-it/#comment-282 I'm not sure but maybe you find this useful: http://tokyocabinet.sourceforge.net/ I’m not sure but maybe you find this useful: http://tokyocabinet.sourceforge.net/

]]> By: Tobu http://kubasik.net/blog/2007/11/02/google-how-do-you-do-it/comment-page-1/#comment-281 Tobu Fri, 02 Nov 2007 21:01:08 +0000 http://kubasik.net/blog/2007/11/02/google-how-do-you-do-it/#comment-281 Re index size. In beagle, do you compress words by replacing them by a reference to an index entry (assuming a minimum length and a minimum number of occurences)? I suspect it would cut down size. You would need index entries to be permanent then (or possibly refcounted). Re index size. In beagle, do you compress words by replacing them by a reference to an index entry (assuming a minimum length and a minimum number of occurences)? I suspect it would cut down size. You would need index entries to be permanent then (or possibly refcounted).

]]> By: Kevin Kubasik http://kubasik.net/blog/2007/11/02/google-how-do-you-do-it/comment-page-1/#comment-280 Kevin Kubasik Fri, 02 Nov 2007 20:37:12 +0000 http://kubasik.net/blog/2007/11/02/google-how-do-you-do-it/#comment-280 If we were just querying Google, then that would be an easy solution, however, many people like having search results locally, and its a marketed feature of GDS, unless I'm misunderstanding what people want/are looking for. If we were just querying Google, then that would be an easy solution, however, many people like having search results locally, and its a marketed feature of GDS, unless I’m misunderstanding what people want/are looking for.

]]> By: grakic http://kubasik.net/blog/2007/11/02/google-how-do-you-do-it/comment-page-1/#comment-279 grakic Fri, 02 Nov 2007 20:33:27 +0000 http://kubasik.net/blog/2007/11/02/google-how-do-you-do-it/#comment-279 Can't you just do HTTP to get search results from Gmail. IMHO, search results is useless if user is offline so i don't see major drawback here. Can’t you just do HTTP to get search results from Gmail. IMHO, search results is useless if user is offline so i don’t see major drawback here.

]]> By: Walther http://kubasik.net/blog/2007/11/02/google-how-do-you-do-it/comment-page-1/#comment-278 Walther Fri, 02 Nov 2007 19:32:26 +0000 http://kubasik.net/blog/2007/11/02/google-how-do-you-do-it/#comment-278 Take a look at http://libgmail.sourceforge.net It is a python library for gmail. I think they are something http-based, but I didn't look how it works exactly. It seems to be pretty efficient for searches. Take a look at http://libgmail.sourceforge.net
It is a python library for gmail. I think they are something http-based, but I didn’t look how it works exactly. It seems to be pretty efficient for searches.

]]> By: glandium http://kubasik.net/blog/2007/11/02/google-how-do-you-do-it/comment-page-1/#comment-273 glandium Fri, 02 Nov 2007 15:50:51 +0000 http://kubasik.net/blog/2007/11/02/google-how-do-you-do-it/#comment-273 Surely, GDS uses some open source library for SSL encryption, so you could replace it with your own that would log the data sent to be encrypted. Anyways, my guess is that GDS directly gets indexes from GMail, via a proprietary Google POP extension. Why download a lot of data and index it yourself when powerful servers already did it for you ? All GDS has to do is use the same indexing algorithms as Google, which is not impossible. At least, that's how I'd do it. Surely, GDS uses some open source library for SSL encryption, so you could replace it with your own that would log the data sent to be encrypted.

Anyways, my guess is that GDS directly gets indexes from GMail, via a proprietary Google POP extension. Why download a lot of data and index it yourself when powerful servers already did it for you ? All GDS has to do is use the same indexing algorithms as Google, which is not impossible. At least, that’s how I’d do it.

]]> By: Kevin Kubasik http://kubasik.net/blog/2007/11/02/google-how-do-you-do-it/comment-page-1/#comment-275 Kevin Kubasik Fri, 02 Nov 2007 12:47:55 +0000 http://kubasik.net/blog/2007/11/02/google-how-do-you-do-it/#comment-275 @nick: There is some support, however, the problem is that IMAP tends to be a much more complex protocol, the Gmail implementation isn't 100% and right now there are some performance problems on that front. Also, since the indexing is really a one time sequential skim, pop really meets most of our needs. Really, I would think that most users (especially now with IMAP) could just register their Gmail accounts in Thunderbird, Evolution or Kmail to get them indexed. @nick: There is some support, however, the problem is that IMAP tends to be a much more complex protocol, the Gmail implementation isn’t 100% and right now there are some performance problems on that front.

Also, since the indexing is really a one time sequential skim, pop really meets most of our needs. Really, I would think that most users (especially now with IMAP) could just register their Gmail accounts in Thunderbird, Evolution or Kmail to get them indexed.

]]>