Mono GSOC Projects: Linq to SQLite

So I noticed that one of the accepted proposals for the Mono project is to create a LINQ provider for SQLite. Major props to this (its something I totally want to see!) and I’m glad to see that LINQ in Mono is going to be its own beast, I love it when the FOSS community just takes a technology and runs with it! Anyways, I wanted to try and get in touch with the mentor/student of this project and share my experience (as the author of the current LINQ to SQLite component ). But contact info seemed hard to come by, so I thought I would post what I had learned.

First, people really want this, and there are several half-complete implementations floating around, including mine (read only, no commit/update/delete support) and this one.

Second, support for just queries is quite easy. Support for complete CRUD, tedious but not to difficult (lots of examples already exist). Support for the generation/mapping/reflection of a database to real Linq objects, this is the tricky part (specifically the UI elements when unable to just piggyback the Visual Studio work).

Anyways, all the luck in the world to this GSOC project, I would really like to see a working implementation come from this!

Tomboy Hackfest: Part 2

Alright! Some cool news! The Mono Hackfest at the Novell OSTC in Provo, Utah was a success ( I would say, people showed up and we talked about features )  And while I took several photos in the hope that I might have another photo-riffic blog post, but alas, my flash wasn’t on, and they are all pretty much worthless.  That aside, it was pretty cool to root around more of the Novell OSTC campus.

As a by-product of the ‘Hack’ing portion of the hackfest, I am happy to report the enabling of tagging in Tomboy, while we are still working out the specifics of the tagging interface, a super-experimental version of our newest iteration (based somewhat strongly on the Blogger.com tagging interface which we all agreed was somewhat well designed). While this is mostly implemented, there are a few issues (mostly based on my lack of Gtk knowledge/experience) with some of the autocomplete logic. I basically created a new Gtk Window which is composed of a ListView and hovers over the entry area (actually a GtkTextView, as a GtkEntry wouldn’t handle text markup). There are 2 real problems at the moment.

1. I need to handle keyboard input intelligently enough to allow selection of an autocomplete option, I just need someone more familiar with how keypress events are handled to take a look at my code and figure out what widgets I should listen to the keypress events on etc.
2. I need to get the autocomplete popup widget to show in the right place (should be easy enough to get, again I just need someone a little more familiar with the Gtk API so I don’t spend another hour looking for the window positioning information.) and the widget needs to close when a note is closed. (Right now the autocomplete box hangs around, should again just be subscribing to a window destroy event, but my previous attempts have resulted in some messes.)

Anyways, if anyone has the time to offer a hand/check any of this out, just drop by #tomboy (I’ll be in and out due to exams, but I’ll do my best to answer any questions) or feel free to just fix it right off the bat Here’s a quick and dirty screenshot of the problem as it exists (you can see the autocomplete dropdown isn’t quite right).

In addition to traditional tags, we have added a new little tidbit for Addin Developers, the concept of System Tags. In short, any tag added to a Tomboy note with the system: prefix will not be displayed. While this seems a little stupid at first glance, this allows us to easily implement things like Tasks, and allow Addins to associate their own data with tags while not implementing their own data store, and still maintaining backwards compatibility. For example if I wanted to implement ‘Contacts’ in Tomboy (NOT A FEATURE THAT SHOULD BE IMPLEMENTED IMHO)  I could simply add the following tags to store all the information I needed for my Addin:

• system:Contact
• system:FirstName:Kevin
• system:LastName:Kubasik
• system:EMail:KevinAtKubasikDotNet

And so on, anything with the ’system:’ prefix will be hidden from the user, but still stored with each note.

Mono 1.2.6 Memory Usage

So, I’ve heard a lot of hype about the upcoming 1.2.6 release of Mono being faster, leaner, and more stable then ever before (due largely to Novell’s acquisition of a QA team dedicated to Mono). Beagle has always gotten flack over memory use, and as a result, we are relentless in our hunt for abused memory. And while it is wonderfully satisfying to reduce memory usage, its really hard to beat dropping megabytes of resident memory for free . I’m running Ubuntu Gutsy and its 1.2.4 release of Mono, but in my quest for some real numbers to back up all this talk I built the current SVN trunk of Mono.

Even my most optimistic expectations put our potential benefit around maybe 2 or 3 MB resident less than beagle running under Mono 1.2.4. On my test setup, Beagle 0.3pre consumed (after my recent Opera backend fix) around 110 MB of VM and 36 MB of RSS (averaged over a 2 hour run).After building and installing Mono 1.2.6, the same 2 hour run was averaging 72 MB of VM and 27 MB of RSS! Its still far from perfect, but free memory reduction is just plain cool .

Some observations about the general pattern of allocation and collection under 1.2.6, it ‘idles’ much lower than 1.2.4. While some actions always push the memory usage up, 1.2.6 *appeared* to return to its lower memory point much faster, and more regularly.

Anyways, I just wanted to say, props to everyone on the Mono team for rocking my socks.

Google: How do you do it?

So its not a big surprise that an oft-requested feature for Beagle is the ability to index a users Gmail messages (like Google Desktop Search). Today we (the Beagle developers) started to investigate just how this is done. While POP3 (and now IMAP) are available, downloading all of a users mail, indexing it, and then caching the text so we can display it. Now, my initial investigation into GDS for Linux revealed that it was calling home via POP3s and downloading lots of data. I have assumed that it was simply iterating over all messages (via POP3), downloading them, indexing them, and caching the compressed content somewhere in Google’s custom indexes.

Now, I had originally planned on this post being an open plea to any and everyone at Google asking them to open up the Gmail access API, but seeing as its just the plain old ugly POP3 (maybe a cool extension), were stuck biting the bullet and implementing a remote mail access layer.

Anyways, given how incredible Google has been in a million other situations, I thought I would throw out 2 wildly out-of-this-world questions, I wouldn’t expect to get a response, but before I spend the time figuring it all out, I felt like I should at least ask.

• Are there some special POP Extensions available in Gmail? Is there some helper web api? Or does GDS really just have a POP3 crawler?
• Is your compression/text storage library open source? (or documented in some research paper at all?) Beagle has always struggled with how to best handle storing copies of a documents text so that it might be made available in interfaces. While we do have a new hybrid text cache (text over 4k on the filesystem, under in a sqlite db, all compressed) we were still no where near as small as the GDS indexes. A cursory examination reveals that the GDS indexes are some form of b-tree on disk, but how are you compressing all that text so small? Is there some substitution/reconstruction algorithm? (It seems like that would be wildly expensive, but who knows).

Anyways, its a long shot, and its pretty far out there, but for the sake of not passing up answers that I can’t seem to find elsewhere on the net, I have asked.

How Much I Rely on Class Libraries

Ok, so I was stamping out a recent (quite simple) assignment for a class. The assignment required the separation of each place value in an integer. Since the information came into the program as a string, I stamped out a simple solution using some easy loops and handy methods available in the String class (of the .Net 2.0 class libraries). However, a quick skim over the rubric included 5% for correctly using div and mod to parse out the integer values. I immediately flipped back into my program, then froze for a second..

then another…

then 2 more…

I was completely blanking. I know its a simple task, its just I have become so accustomed to the incredible abundance and availability of a dozen methods for every little task that I blanked for a good minute, just stonewalled. I knew I had written this code before, everyone has done it at least once in some ‘Intro to Programming Theory’ class, and the concept was easy enough, but it just wouldn’t come. Cursing the professor for such a trivial demand, I went and got a cup of coffee.

Upon returning I realized how stupid and petty I was being. While I do often rely on cool class libraries and the methods they provide, I really just have to stop being so self-righteous and realize how likely it was that much of the class would be completely stuck on such a task. We spend so much time today learning about existing technologies and API’s that we forget the core of programming: Problem Solving.

I quickly slapped together the following C# method:

static int[] toIntArray(string input)
{
int i = 0;
int digit;
List<int> ints = new List<int>();
for (int numdigits = input.Length; numdigits > 0; numdigits–)
{

digit = Convert.ToInt32(input) / (int)Math.Pow(10.0, (double)(numdigits – 1));

digit = digit % 10;

ints.Add(digit);

i++;

}
return ints.ToArray();
}

In retrospect, its quite simple, I just hope that this was my ‘moment of realization’ and I don’t get so inundated again as to the point where I can’t do the simple stuff on my own anymore.

SqlLite Linq Provider

Ok, so as many of you may have noticed in my last post I’ve taken a real interest in C# 3.0 and its new nifty features. Now, I’m mostly just excited for the simple collection manipulations, but the whole Linq to SQL thing is nagging in the back of my mind. Now while complete CRUD will no doubt take some serious code, simple query support is not that difficult to implement. After following mattwar’s blog series on a generic DB provider for Linq, I decided that I wanted that awesome glory against a lean and mean sqlite db.

So, about an hour or so of messing and meddling I have a few samples working. (The attached zip) Its a Visual Studio 2008 Beta 2 project (I’ll be writing autotools magic for mono later this week) so sorry for that, the code can still be imported into Monodevelop, just the solution/project files won’t work. Anyways, I have tested a smattering of JOIN’s and all sorts of simple selects without issue. However, the elements of sqlite that behave differently tend to do so silently and without complaint, making it harder to be certain that everything is working. However, I’m planning on fleshing out a set of test queries, however, for now I could really just use the help with testing/checking the SQL (as I’m no sqlite Guru).

Known Issues

• DataType sloppyness – I hope to handle this better (storing a DateTime string in TEXT would extract to DateTime successfully) right now you need to pretty much just use strings or numeric values.
• Inefficient Queries – Not being a Sql master, I can’t say that much of whats generated is the best way to do things, please, if you know then share!
• OrderBy issues – Its just hard as heck to get working, it seems to work fine sometimes, but no promises.

Anyways, play around, have fun, and note that you need the Sqlite provider for ADO.Net (duh).

Linq To Sqlite Download

I’m looking at db_linq, which is a full (bi-directional, change tracking, general awesome crazyness) solution, this is really just a way to query sqlite db. I might try to add a sqlite provider to db_linq at some point, its just that their system is very different from my implementation, so there wouldn’t be too much shared code.

UPDATE: This post is obsolete, please see this post on Sqlite Linq Providers.

The Changing Face of High-Level Programming

Ok, so I’m sure most MS .Net dev’s have already seen these posts far too many times, for the Mono users out there, I have a little treat. While Moonlight and WPF get tons of hype, I think the biggest and most exciting change coming soon to a C# compiler near you is support for lambda expressions, anonymous types, and extension methods.

Now on the whole this doesn’t sound all that exciting, I mean, before a few months ago, I had never really used lambda expressions to accomplish much beyond pass that unit in an intro to CS class. Individually, there’s nothing to jump for joy about, but when used in conjunction, we can produce startlingly clean and readable code.

To demonstrate this I’ve whipped up two examples that I was fiddling with as I read a million tutorials. They aren’t fancy XML or Database providers, just some simple (and quite common in my experience) text parsing tasks that have disproportionately complex code. We will use some of the new C# 3.0 features to make far cleaner and more readable code.

The first example is an exclusion string, or a set of characters that are not allowed in another.

var illegalchars = "abcdefg"; 

string testString1 = "Kevin"; 

string testString2 = "hijkmlppp";

The ‘old’ way of checking both strings for one of the illegal chars:

 foreach (char c in illegalchars) { 

if (testString1.Contains(c) || testString2.Contains(c)) 

 Console.WriteLine("illegal char!"); 

}

Using awesome new stuff:

 if (testString1.Intersect(illegalchars).Any() 

|| testString2.Intersect(illegalchars).Any()) 

Console.WriteLine("Linq found it too");

Our next example is ‘exploding’ or splitting a series of values out of a string (CSV and PSV are common examples of this) into an array:

 string pipeDelined = "Kevin | McCool | Kubasik";

An old solution might have been (I know we could optimize this, or clean it up, just making a point ):

 List<string> names = new List<string>(); 

foreach (string s in pipeDelined.Split('|')) { 

var ts = s.Trim(); 

if (ts == "") continue; 

names.Add(ts); 

} 

var allNames = names.ToArray();

Using our cool new C# 3.0 tools, we can change this to the super-sexy:

 var allLinqNames = pipeDelined.Split('|') 

.Select(s => s.Trim())

.Where(s => s != "")

.ToArray();

While a hardened child of OOP (via C# and Java) might baulk at the new syntax, I think that it can quickly start to grow on a developer. Moreover, it has the distinct advantage of being unambiguous, and makes reading someone else’s dense code much more fluid. 

I really can’t wait for C# 3.0, and not for those flashy API’s, just the simple syntactical sugar that is already making me lazier by the minute.