Turning Ideas Into Code

I know what I want to do. Kind of. But not how to turn the idea in my head into real-life, working code. It’s frustrating. Very frustrating. Some days I wish computers could simply suck the ideas out of our heads and turn them into something that works.

I’m trying to figure out how to address the severe I/O problems in Snowflake, and have a picture of how I imagine it should work. Alas, trying to get it out on (virtual) paper is ridiculous. Why does it have to be so hard?

The basic premise is to reduce all the copying, and have I/O classes (or whatever) that do translations between layers in terms of reading/writing the actual data. For example, to go from a bunch of wave samples in the audio driver to the raw bytes on the disk via the used filesystem. At some point, offsets and lengths change due to other offsets and changes to unit sizes. A sector is 512 bytes long, and the position in a file relative to the partition gets offset by the starting position of the partition relative to the disk.

If the I/O code could do all those sorts of translations so that the IDE driver could put the bytes the AC’97 driver requests directly into the buffer provided by the AC’97 driver, it would supposedly become a whole lot more efficient.

Just need to get it all out of my head and into the text editor. And then change all my existing code to try and make it work with it. And hope I don’t make things worse!

Listing Directories

This was a weird way to run snowflake today. My IDE code sucks under Virtual Box; really no idea what’s wrong with it. So I decided to test it in QEMU, without an X Server in sight.

To make it even more fun, I’m running it inside another VM, my Ubuntu installation in Virtual Box. I have one ssh session for building snowflake and killing QEMU (I don’t know how else to quit) and another for running my QEMU session.

Anyways, despite the weird ways in which I do things, I booted snowflake in QEMU, with the primary hard disk pointing to my Ubuntu installation’s hard disk. And with my ext2 file system code, managed to list some directories.

The above is a directory listing of the following folder in Ubuntu: ~/google-code/snowflake-os/kernel. I had to run QEMU as: sudo qemu -boot d -hda /dev/sda -cdrom snowflake.iso -curses, for those curious about what I was doing. Also, appears it’s not doing the full directory list…

A Fractured Collarbone

Whilst not directly related to Snowflake, my fractured collarbone has given me a lot of free time with very little to occupy it.  So after downloading a whole bunch of tools and deleting some music to make way for the installation of a Linux virtual machine, I now have a small development environment for Snowflake.

And now that I have everything I need to hack away on Snowflake again, I have to wonder what I should work on, what may interest me enough to get some fresh code cooked and committed.

I got Snowflake built with Ubuntu and gcc 4.x, booted the ISO image in VirtualBox, and mucked about in the extremely limited command interpreter.  Then I got to thinking that I need to start investigating what I will do about making processes, or at least something akin to processes.

One thought I’ve always had floating around is typed processes—like a function is typed in O’Caml.  Of course, this could get complicated if a program/process were to have complex types as inputs and outputs, such as records, sums, objects, and phantom types.

Then there is also the need to build a proper device/driver interface, with a file system becoming increasingly necessary.  Lastly, a hard think about how I/O will work is needed—everything is just so haphazard and likely inefficient.

The other big problem is the heavy use of locking; and these locks aren’t light-weight either.  I’m fairly certain most of my performance woes, if not all, are being caused by my excessive use of the Mutex and Condition interfaces.  Instead, extremely lightweight locks are needed, with the addition of atomic semantics within O’Caml itself.

So much to think about.  So much to plan.  So much to rewrite.  So much to throw out.  So much to create.

Network Frustrations

My useless network stack is driving me insane.  I need to fix it, but I also need to know where the actual problem lies.  So I need to profile it.  And that is where the problem lies.  So, I’ve decided to try and tackle this from an alternate angle.

Earlier today, I was using trusty time command to see how long it takes to parse about 100 million ARP packets, which aren’t all that large; and I’ve come to the realisation that I could build a test framework that simulates sending the tonne of packets that my daap server was sending.

I don’t think the dataset would be that large to test against.  After all, it did die after like less than half a MB of data was sent to it.  A simple capture with Wireshark should suffice to generate a nice little data set to test against.

And with that, I just might be able to profile the network stack without too much effort.

However, I’m not at my development machine, so it’ll have to be a task to start on tomorrow.

One of the things that this highlights, however, is the advantage C has over the likes of O’Caml.  One can just cast a bunch of bytes to a struct, and manipulate it with some semblance of typing available without much runtime overhead.

Conversely, in O’Caml, I need to actually parse the bytes into a type (if I want to use strong types, which I like to do), which requires a lot more effort than C does.  And that kind of sucks.

Might be interesting trying to extend the O’Caml compiler to have a notion of structs (as opposed to records), so could get some of the benefits and maybe even speed of C shoe-horned into O’Caml.  The only problem would be the GC…

Almost Everything I Need

Well, it’s not quite everything I need, but it’s certainly becoming more fun.  I now have an IRC client—barely usable, but tolerable—which I can chat away on whilst listening to some music.  All in Snowflake!

There are even more problems to add to the growing list of problems, like it dying with not much RAM available.  I ran out with 64MB after two songs finished playing.  And did I mention the barely usable IRC client?

Seems I just add stuff without fixing the old stuff, but I promise I’ll get around to fixing the old stuff.

Next on my plan is to try out Taras’ old approach to the network stack and parsing integer lists.  It seems inefficient at first glance, but I don’t remember having the performance problems I do at present.  Anyways, all such sweeping changes can always be undone, so it worth pursuing :)

Playing Music

Whilst I've previously been able to play a wave file loaded as a multiboot module, and also given up on streaming a wave file over the network, I've decided to find a new approach…

With the new addition of a very primitive IDE driver, I can now pass a tar file full of wave files as a hard disk image to QEMU, and play wave files from 'disk' using my shell. It's all a bit of a hack from start to finish, but still quite cool.

Now some more additions are needed. Like a real file system layer, and a way to implement optional buffering for block io. And a lexer/parser for the shell so I don't need this nasty escaping hack to use names with spaces.

A 'Lil Shell

I've gotten sick to death of the performance problems with the TCP stack, and my buggy RealTek driver. Instead, I put together a lil shell type thing. It's not really a shell, since a shell does things mine doesn't. And it doesn't do quoted strings either. But it does do stuff.

This is configuring my network stack with a static IP:

Performance Woes

It had been a few months since I had last worked on Snowflake.  Now that I’ve gotten back into, I’ve ported across the old AC’97 driver, and attempted to build a functional network stack.

And now I’ve been trying to get Snowflake to stream a wave file across the network and play it.  Unfortunately, I’ve run into a rather large problem: after a few hundred packets in less than a second seems to put a very heavy load on the garbage collector.

I’m fairly certain it is the garbage collector, but I haven’t yet figured out how to make a Linux app such that I can profile my network stack.  Given I also have customised O’Caml libraries, I’m even more reluctant to try to do this.

The other problem with trying to profile the network stack in Linux is that without the RealTek 8139 driver, the performance characteristics will likely change as well.  One would have to write the fake driver to mimic the behaviour of my RealTek driver fairly accurately in order to get somewhat reliable information.

I’m at a loss what to do next.  I was really excited about the thought of streaming wave files over the network and playing them with Snowflake.  Now I feel dejected with no hope of fixing this stupid problem.

Sure it is related to memory allocation, but every time I feel like I’ve eliminated potentially excessive or large memory allocations, and fix the bugs that crop up as a result, it seems to be no better.

There’s still one place I haven’t pruned memory allocations from yet, and that’s building the packets to be sent out on the wire.  And I’m not sure how I’m going to fix that either.  Ideally, I’d be like the parsing, and just write bytes to a pre-allocated Bigarray – preferably the actual buffer given to the driver for DMA.

However, if I do that, and it still doesn’t fix the problem, then I’ll be really screwed.  I want this to work so bad!

I’d also hate to see how much CPU Snowflake actually uses to stream the file across the network – I can imagine the CPU pegged at 100% – which wouldn’t be very efficient at all.