A Slightly More Usable IRC Client

I’ve made a number of changes to snowflake in the last few days. Been trying to figure out why my network stack is fail, and I think it might be to do with sloppy interrupt handling or a poorly written rtl8139 driver. And then there is the upgrade to an svn snapshot of ocaml-3.12.0+dev, followed by a bunch of fixes to that change. And some cosmetic work like an updated IRC client.

I have a lame workaround for the network issues, which makes IRC work a bit better at the very least: a thread that constantly sends out ARP packets. Cheesy, but it suffices for the short-term. And means I can spend some time enjoying working on the IRC client.

Like all things I do, it is riddled with bugs! But it makes it slightly more tolerable to use. I also tried to see if my network hack would let DAAP work, but I’m afraid it’s too buggy and damaged for me to get that far.

Tracking Down Bottlenecks

My friend Taras does some really intriguing work on Firefox, and of late has been working hard to reduce cold startup time, which involves a lot of tracing. And anyone who reads this blog every now and then will realise I have serious issues with my network stack, which got me thinking.

One of Taras’ posts, “When in Trouble, Draw a Picture” could equally apply to my current problem. It’s pretty easy to use the rdtsc instruction to get relative timing information, and a pretty graph should be fairly easy to produce from timestamped debug messages.

As an indication of how disastrous my network stack is, I added a little bit of code in my IRC module to reply to CTCP pings. The result? Between 30-60s for a CTCP ping reply. Ridiculous!

Unfortunately, I can’t work on it right now as I managed to inadvertently forkbomb my unbuntu install and need to physically restart the machine. Easier said than done, as it requires a trip in to work to do. The fork() system call can be dangerous in the hands of the uninitiated, but lesson learned.

Turning Ideas Into Code

I know what I want to do. Kind of. But not how to turn the idea in my head into real-life, working code. It’s frustrating. Very frustrating. Some days I wish computers could simply suck the ideas out of our heads and turn them into something that works.

I’m trying to figure out how to address the severe I/O problems in Snowflake, and have a picture of how I imagine it should work. Alas, trying to get it out on (virtual) paper is ridiculous. Why does it have to be so hard?

The basic premise is to reduce all the copying, and have I/O classes (or whatever) that do translations between layers in terms of reading/writing the actual data. For example, to go from a bunch of wave samples in the audio driver to the raw bytes on the disk via the used filesystem. At some point, offsets and lengths change due to other offsets and changes to unit sizes. A sector is 512 bytes long, and the position in a file relative to the partition gets offset by the starting position of the partition relative to the disk.

If the I/O code could do all those sorts of translations so that the IDE driver could put the bytes the AC’97 driver requests directly into the buffer provided by the AC’97 driver, it would supposedly become a whole lot more efficient.

Just need to get it all out of my head and into the text editor. And then change all my existing code to try and make it work with it. And hope I don’t make things worse!

Listing Directories

This was a weird way to run snowflake today. My IDE code sucks under Virtual Box; really no idea what’s wrong with it. So I decided to test it in QEMU, without an X Server in sight.

To make it even more fun, I’m running it inside another VM, my Ubuntu installation in Virtual Box. I have one ssh session for building snowflake and killing QEMU (I don’t know how else to quit) and another for running my QEMU session.

Anyways, despite the weird ways in which I do things, I booted snowflake in QEMU, with the primary hard disk pointing to my Ubuntu installation’s hard disk. And with my ext2 file system code, managed to list some directories.

The above is a directory listing of the following folder in Ubuntu: ~/google-code/snowflake-os/kernel. I had to run QEMU as: sudo qemu -boot d -hda /dev/sda -cdrom snowflake.iso -curses, for those curious about what I was doing. Also, appears it’s not doing the full directory list…

A Fractured Collarbone

Whilst not directly related to Snowflake, my fractured collarbone has given me a lot of free time with very little to occupy it.  So after downloading a whole bunch of tools and deleting some music to make way for the installation of a Linux virtual machine, I now have a small development environment for Snowflake.

And now that I have everything I need to hack away on Snowflake again, I have to wonder what I should work on, what may interest me enough to get some fresh code cooked and committed.

I got Snowflake built with Ubuntu and gcc 4.x, booted the ISO image in VirtualBox, and mucked about in the extremely limited command interpreter.  Then I got to thinking that I need to start investigating what I will do about making processes, or at least something akin to processes.

One thought I’ve always had floating around is typed processes—like a function is typed in O’Caml.  Of course, this could get complicated if a program/process were to have complex types as inputs and outputs, such as records, sums, objects, and phantom types.

Then there is also the need to build a proper device/driver interface, with a file system becoming increasingly necessary.  Lastly, a hard think about how I/O will work is needed—everything is just so haphazard and likely inefficient.

The other big problem is the heavy use of locking; and these locks aren’t light-weight either.  I’m fairly certain most of my performance woes, if not all, are being caused by my excessive use of the Mutex and Condition interfaces.  Instead, extremely lightweight locks are needed, with the addition of atomic semantics within O’Caml itself.

So much to think about.  So much to plan.  So much to rewrite.  So much to throw out.  So much to create.

Network Frustrations

My useless network stack is driving me insane.  I need to fix it, but I also need to know where the actual problem lies.  So I need to profile it.  And that is where the problem lies.  So, I’ve decided to try and tackle this from an alternate angle.

Earlier today, I was using trusty time command to see how long it takes to parse about 100 million ARP packets, which aren’t all that large; and I’ve come to the realisation that I could build a test framework that simulates sending the tonne of packets that my daap server was sending.

I don’t think the dataset would be that large to test against.  After all, it did die after like less than half a MB of data was sent to it.  A simple capture with Wireshark should suffice to generate a nice little data set to test against.

And with that, I just might be able to profile the network stack without too much effort.

However, I’m not at my development machine, so it’ll have to be a task to start on tomorrow.

One of the things that this highlights, however, is the advantage C has over the likes of O’Caml.  One can just cast a bunch of bytes to a struct, and manipulate it with some semblance of typing available without much runtime overhead.

Conversely, in O’Caml, I need to actually parse the bytes into a type (if I want to use strong types, which I like to do), which requires a lot more effort than C does.  And that kind of sucks.

Might be interesting trying to extend the O’Caml compiler to have a notion of structs (as opposed to records), so could get some of the benefits and maybe even speed of C shoe-horned into O’Caml.  The only problem would be the GC…

Almost Everything I Need

Well, it’s not quite everything I need, but it’s certainly becoming more fun.  I now have an IRC client—barely usable, but tolerable—which I can chat away on whilst listening to some music.  All in Snowflake!

There are even more problems to add to the growing list of problems, like it dying with not much RAM available.  I ran out with 64MB after two songs finished playing.  And did I mention the barely usable IRC client?

Seems I just add stuff without fixing the old stuff, but I promise I’ll get around to fixing the old stuff.

Next on my plan is to try out Taras’ old approach to the network stack and parsing integer lists.  It seems inefficient at first glance, but I don’t remember having the performance problems I do at present.  Anyways, all such sweeping changes can always be undone, so it worth pursuing :)