Network Frustrations

My useless network stack is driving me insane.  I need to fix it, but I also need to know where the actual problem lies.  So I need to profile it.  And that is where the problem lies.  So, I’ve decided to try and tackle this from an alternate angle.

Earlier today, I was using trusty time command to see how long it takes to parse about 100 million ARP packets, which aren’t all that large; and I’ve come to the realisation that I could build a test framework that simulates sending the tonne of packets that my daap server was sending.

I don’t think the dataset would be that large to test against.  After all, it did die after like less than half a MB of data was sent to it.  A simple capture with Wireshark should suffice to generate a nice little data set to test against.

And with that, I just might be able to profile the network stack without too much effort.

However, I’m not at my development machine, so it’ll have to be a task to start on tomorrow.

One of the things that this highlights, however, is the advantage C has over the likes of O’Caml.  One can just cast a bunch of bytes to a struct, and manipulate it with some semblance of typing available without much runtime overhead.

Conversely, in O’Caml, I need to actually parse the bytes into a type (if I want to use strong types, which I like to do), which requires a lot more effort than C does.  And that kind of sucks.

Might be interesting trying to extend the O’Caml compiler to have a notion of structs (as opposed to records), so could get some of the benefits and maybe even speed of C shoe-horned into O’Caml.  The only problem would be the GC…

Almost Everything I Need

Well, it’s not quite everything I need, but it’s certainly becoming more fun.  I now have an IRC client—barely usable, but tolerable—which I can chat away on whilst listening to some music.  All in Snowflake!

There are even more problems to add to the growing list of problems, like it dying with not much RAM available.  I ran out with 64MB after two songs finished playing.  And did I mention the barely usable IRC client?

Seems I just add stuff without fixing the old stuff, but I promise I’ll get around to fixing the old stuff.

Next on my plan is to try out Taras’ old approach to the network stack and parsing integer lists.  It seems inefficient at first glance, but I don’t remember having the performance problems I do at present.  Anyways, all such sweeping changes can always be undone, so it worth pursuing :)

Playing Music

Whilst I've previously been able to play a wave file loaded as a multiboot module, and also given up on streaming a wave file over the network, I've decided to find a new approach…

With the new addition of a very primitive IDE driver, I can now pass a tar file full of wave files as a hard disk image to QEMU, and play wave files from 'disk' using my shell. It's all a bit of a hack from start to finish, but still quite cool.

Now some more additions are needed. Like a real file system layer, and a way to implement optional buffering for block io. And a lexer/parser for the shell so I don't need this nasty escaping hack to use names with spaces.

A 'Lil Shell

I've gotten sick to death of the performance problems with the TCP stack, and my buggy RealTek driver. Instead, I put together a lil shell type thing. It's not really a shell, since a shell does things mine doesn't. And it doesn't do quoted strings either. But it does do stuff.

This is configuring my network stack with a static IP:

Performance Woes

It had been a few months since I had last worked on Snowflake.  Now that I’ve gotten back into, I’ve ported across the old AC’97 driver, and attempted to build a functional network stack.

And now I’ve been trying to get Snowflake to stream a wave file across the network and play it.  Unfortunately, I’ve run into a rather large problem: after a few hundred packets in less than a second seems to put a very heavy load on the garbage collector.

I’m fairly certain it is the garbage collector, but I haven’t yet figured out how to make a Linux app such that I can profile my network stack.  Given I also have customised O’Caml libraries, I’m even more reluctant to try to do this.

The other problem with trying to profile the network stack in Linux is that without the RealTek 8139 driver, the performance characteristics will likely change as well.  One would have to write the fake driver to mimic the behaviour of my RealTek driver fairly accurately in order to get somewhat reliable information.

I’m at a loss what to do next.  I was really excited about the thought of streaming wave files over the network and playing them with Snowflake.  Now I feel dejected with no hope of fixing this stupid problem.

Sure it is related to memory allocation, but every time I feel like I’ve eliminated potentially excessive or large memory allocations, and fix the bugs that crop up as a result, it seems to be no better.

There’s still one place I haven’t pruned memory allocations from yet, and that’s building the packets to be sent out on the wire.  And I’m not sure how I’m going to fix that either.  Ideally, I’d be like the parsing, and just write bytes to a pre-allocated Bigarray – preferably the actual buffer given to the driver for DMA.

However, if I do that, and it still doesn’t fix the problem, then I’ll be really screwed.  I want this to work so bad!

I’d also hate to see how much CPU Snowflake actually uses to stream the file across the network – I can imagine the CPU pegged at 100% – which wouldn’t be very efficient at all.