low-level programming geekiness

[cjsmith]

I don't talk about work details much, 'cause either they're so minor they're meaningless, or they're something my company wants to talk about before I talk about it. But every so often there's a tidbit I can share for my geeky friends.

You know you're a real Embedded Systems Programmer when...

...you find and fix a bug that involves changing this code

var--;

to this:

disable_interrupts();
var--;
reenable_interrupts();

Extra bonus geek points if it doesn't happen reliably, doesn't happen in the debugger, and/or takes a long time to reproduce. More extra bonus geek points for not having been the one to put that bug in there in the first place.

That was late last week, and I was pretty proud of it, actually.

Comments

[lesliepear.livejournal.com]

That looks like a nasty bug to track down.

What language is that - C or C++ or something else?

[cjsmith.livejournal.com]

C. It was indeed nasty to track down. Adding breakpoints would, as you might imagine, change the timing of the entire thing, and the bug wouldn't occur. So no traditional debugging. I was glad when I found that one!

[wispfox.livejournal.com]

*shakes head* Damn interrupts!

I hated interrupts when I was in CS in school. Exactly because of how thoroughly it kills debugging!

(I note that I am not a programmer now)

[gdmusumeci.livejournal.com]

...man, that would be a total pain in the ass to track down.

I'm glad I'm not in kernel land much any more. Tracking down multithreaded bugs isn't so bad when the debugger actually functions as designed. :-)

[cjsmith.livejournal.com]

It was indeed annoying. Wasn't the debugger's fault, either, really -- it simply didn't have, was never designed to have, the capability to do what I needed.

I haven't been in kernel land long enough that a bug like this is a no-brainer for me. I know people who have (okay, not many of 'em), or at least who are far quicker with such things. If I'm going to stay in kernel land, I want to get that good.

[cjsmith.livejournal.com]

Amen! Either you've got a debugger carefully customized for your system, aware of interrupts and able to "backtrace" out of them by reading the right weird registers... nah, even that wouldn't have helped for this one. We simply didn't give it the capability to halt all involved hardware within one clock cycle. When your processor's talking to (or listening to) something else, debugging is challenging.

[gorillashaman.livejournal.com]

hehehehe...

Along the same lines, you know you're a real embedded prgrammer when you fix a bug by changing a line like this:

int foo;

to this:

volatile int foo;

[cjsmith.livejournal.com]

Ohhhhhhh yeah. Those I've done many a time! :-)

We should start a list. You're a real embedded programmer when you fix a bug by...

Heck, I'd probably learn something!

[oddhack.livejournal.com]

I shouldn't be too critical about that original section of code, but it does seem atomically bad. The responsible programmer ought to be locked up.

[cjsmith.livejournal.com]

*snicker*

[joedecker.livejournal.com]

Heh. var isn't a machine word, is it. ;)

[cjsmith.livejournal.com]

Nah, I just made up the name. That's not the troublesome part...

[joedecker.livejournal.com]

Sorry, I wasn't being clear. I was suggesting that the variable wasn't of a size that would allow it to be decremented with a single instruction, as it was of a "int" size likely larger than the size of integers decrementable on the CPU (if in fact there's any such concept in the CPU at all)

[allanh.livejournal.com]

It's either a pitiful comment on my state of humanity, or a most excellent compliment on my state of geekiness ... that I actually understood this joke.

OK. I'm scared now. Time for me to go into management...

[hitchhiker.livejournal.com]

It hurts us, precious!

[cjsmith.livejournal.com]

Ahh, got it. Come to think of it, I've had THOSE problems too! In this case it was 8 bits, so this was the more "usual" problem (load/decrement/store isn't one cycle).

[cjsmith.livejournal.com]

Umm, don't look now, but don't you already have at least one foot in management?

[cjsmith.livejournal.com]

Yeah, those are just annoying, aren't they? It's easier to design thread-safety into a chunk of code in the beginning than it is to retrofit it later.

[joedecker.livejournal.com]

Gotcha. :)

[cjsmith.livejournal.com]

I haven't been programming at this level (very low level) for long enough to have hundreds of these under my belt. Each bug still teaches me something, if only some new trick for how to find 'em.

[joedecker.livejournal.com]

I enjoyed the frustration of some of those things myself. :) I still tell the story of the prototype handheld software that worked in the little prototypes in California but wouldn't boot in the prototypes in New Jersey, a sordid tale of an interrupt handler using a byte of memory before it'd been intialized, and an ASIC design that put the LCD contrast control and a "completely reorganize the memory mapping of the device" bits into the same "convienent" write-only control byte. :)

[joedecker.livejournal.com]

You can hose that off pretty easily. :)

[allanh.livejournal.com]

Well, THAT explains the massive sucking sounds from the vicinity of my left foot.

[cjsmith.livejournal.com]

When your toe hairs start forming into little pointy cones on either side, then worry.

[cjsmith.livejournal.com]

*I* can. Farthest I ever got was team lead and I got right back outta that. But some people have a harder time shedding it!

[cjsmith.livejournal.com]

They're frustrating as anything when you're sleuthing, and then the whole thing always turns into a well-loved story the instant you actually find the bug!

[airporter.livejournal.com]

You're scared...

(I'm not a geek, I'm an accountant. I'm not a geek, I'm an accountant.I'm not a geek, I'm an accountant.I'm not a geek, I'm an accountant.I'm not a geek, I'm an accountant.I'm not a ....)

[simplykimberly.livejournal.com]

I'm giving a whole talk with that as the punchline in October ;)

[cjsmith.livejournal.com]

Ummmm... I'm curious, but do I really want to know? ;-)

[simplykimberly.livejournal.com]

lol, it's not something scary ... well, except for the fact that lots of programmers end up doing multi-threading without really understanding it - that's scary!

http://www.softwaresummit.com/2005/speakers/jennery_kimberly.htm (http://www.softwaresummit.com/2005/speakers/jennery_kimberly.htm)

But really, that's one of my punchlines - that designing concurrency into a program/system is a whole lot better than trying to retrofit it into an existing one!

[cjsmith.livejournal.com]

well, except for the fact that lots of programmers end up doing multi-threading without really understanding it - that's scary!

Amen! It's easy to say "create new thread" but a lot harder to make sure you as a programmer have upheld your half of the bargain.

And yeah, retrofitting it is time-consuming and error-prone. This is obvious to the most casual observer, even the folks in management (that last was said with a smile). Yet we still wind up doing it!

[apparentparadox]

Jane sent out email telling us to wear our bad dancing shirts on Sun morning. I replied with "Ack", but it turned out she didn't know what that meant. It's hard to remember that not all my friends are geeks (or radio people).

[cjsmith.livejournal.com]

Some non-geeks have also picked up the use of the word "ping" as a verb, eg "ping me again about that on Saturday". But when I say "ICMP_ECHOREPLY" back at them it doesn't work all that well. :-)

[zoratu.livejournal.com]

I'm facing a problem right now, where if I touch a short length of insulated wire to a lead in the path of where I think my software problem resides, there is just enough capacitance in the wire to make the problem go away. I can't use a voltmeter or a scope because of this; I feel your pain.

[cjsmith.livejournal.com]

Oh ouch. You attach a scope probe and the bug goes away? And it's a software problem? I'm impressed!

[zoratu.livejournal.com]

Well, the guys in Taiwan are disavowing manufacturing problems, so it has become a software problem. Software hacks for hardware problems are fun, 'though, right? ;-)