The Opinions Expressed Herein are My Own

Upgrading the Linux Kernel when Using a USB Boot Disk

2013-05-11T04:03:48Z

In order to protect any sensitive data on my work laptop, I use full-drive encryption. To aid recovery in case of theft, it normally boots to a honeypot installation of Windows with Prey installed. To boot into Linux, one must use a USB boot disk[1] which contains the contents of /boot.

Once in a while, it is necessary to upgrade the kernel. I don't do this very often, so it seems like the sort of process which should be documented.

Mount /boot read-write: `mount -o rw /boot`
I generally have /boot mounted read-only, in part because I don't usually have the thumbdrive plugged in (I don't boot very often, after all), and to ensure I don't accidentally make changes to it without being prepared to undergo this full process.
Install kernel updates: `sudo aptitude full-upgrade`
Reboot. Pray it works.
Insert backup boot disk. (Just in case the thumbdrive on my keychain goes bad or gets lost.)
Copy files from the updated boot disk to the backup: `rsync -av /boot /media/usb2`
Remove the original boot disk. This will help prevent unfortunate mistakes.
Alter grub.cfg of backup boot disk to refer to the proper UUID: `sudo sed -i s/$UUID_OF_BOOT/$UUID_OF_BACKUP/ grub/grub.cfg`
`blkid` comes in handy for finding the relevant UUIDs.
Reboot. Pray it works.

Assuming both boot disks successfully boot the machine with the updated kernel, congratulations! You have successfully updated the kernel!

[1] It's not quite as much a waste of a thumbdrive as you might think. The sticks have a FAT partition of ~90% or so of the advertised capacity, so they can still be used to transfer files between computers or for data storage.

New year, new job, new blog. I'd 301 redirect you if I could, but since I can't you'll have to click through to read comments or leave your own.

A Public Service Announcement About rm and tildes

2013-04-22T04:09:30Z

If you happen to be lax in upgrading software, you might have an older version of magit where if M-x magit-status is allowed to initialize a git repository where none was before, it has a tendency to create said repository in the directory '$HOME/repo/~/repo'[1].

If this happens, and one wishes to remove that annoying and non-useful '~' directory, do not, I repeat, do not run the command:

rm -r ~

That is bad. Remember shell expansion! You need to quote the tilde:

rm -r '~'

But also note that even the latter should not be done within eshell, as it seems to be intent on expanding the tilde anyway. Use a real shell.

Also, always do `ls $dir` before you do `rm $dir`, just to double-check what you're deleting. Even if you're sure, check anyway.

Because otherwise, you might accidentally delete your home directory.

On the bright side, I can now confirm my backups work.

[1] see bug 383

New year, new job, new blog. I'd 301 redirect you if I could, but since I can't you'll have to click through to read comments or leave your own.

Live from the DMV! (This Program was Recorded Earlier)

2013-01-26T20:32:15Z

I recently renewed my driver's license. There were a few changes from the last time I did it, and they aren't really for the better.

First, they now require removal of glasses, hats, and human expression from your picture, to aid the facial recognition software used. While I offered a mild objection to removing my glasses and having a blank expression, it regrettably did not occur to me to state my objection to the usage of facial recognition software at all until after I was done.

Wish I had, though. For several reason: first, because I do object to the government using facial recognition software on my pictures, or really storing any biometric data on me at all; and second because I want to know if there's a process to handle such objections. Obviously the DMV is not the place to effect change, but knowing if there are procedures in place for objections would at least give me a place to start.

Well, I had always planned on moving out of state eventually, anyway. I guess I'll add "DMV uses facial recognition software" to the list of reasons to leave, and the list of things to check for any place I might otherwise be inclined to move to.

Second, they no longer provide your actual license. Instead, they provide a temporary paper license and mail the real one. So now, instead of potentially being lifted by someone showing their face who would have to look similar enough to fool the person giving it to you, your license can be lifted anonymously by someone within the postal system, or anyone with access to your mailbox. This is apparently so they can take time to more carefully verify your identity to prevent identity theft, so the fact it makes licenses easier to steal is hilarious.

Third, unless you specifically object, signing up for a license now automatically enrolls you into the selective service database if you're eligible. While I object to the very existence of that database, auto-enrollment makes vastly more sense than the manual enrollment that was supposedly mandated when I was within the eligible age range.

It gets harder and harder to keep a low data profile. Le sigh.

New year, new job, new blog. I'd 301 redirect you if I could, but since I can't you'll have to click through to read comments or leave your own.

Getting a Job, a Retrospective

2013-01-13T01:37:40Z

Being largely burnt out on the skin cream business, I decided near the beginning of 2012 it was time for a change of pace and decided to look for a job.

Step one: Leave the house. Achievement unlocked!

What I heard from people in these groups, and from various places on the Internet, is that now is a great time to be a programmer. Jobs are easy to come by, they said! The pay is great, they said!

As it turns out, there are a few things that make it somewhat more difficult for me to find work: first, is location. This town, which I would love to leave but not quite yet, thanks to the banks and insurance industry which dominate it, is a Java and .NET shop through and through, and they demand experience in those languages. Also, banks and insurance companies are very stodgy, and maintain a considerably stricter dress code than I can bring myself to adhere to. So for multiple reasons, this town's two primary industries are almost entirely out.

So I'm largely left looking at the PHP, Python, and Ruby shops. PHP because however much I hate it I have experience in it, and Python and Ruby because places which use Python and Ruby are enlightened enough to realize programmers can learn new programming languages. That's a relatively small number of places in this town, but at least it's non-zero.

Attending meetings is fun, but I'm terrible at networking (it requires talking, which I am wont not to do), so wasn't particularly helpful in terms of finding work. But a conveniently-time presentation by one of the CIALUG members on how he got a $20k/year pay hike with his new job proved interesting.

First was the suggestion that, contrary to the opining on Hacker News, recruiters were not all terrible people, and a recommendation of the people at Palmer Group.

First sign of trouble: their baraccuda was doing deep packet inspection and rejected my e-mails on the basis of my home IP not being authorized to act as an SMTP server—that it isn't being used as one is, apparently, entirely irrelevant. Had to craft a special rule to remove the Received: header before I could relay mail to them. (Alas, postmaster@ was completely non-responsive.)

Once I got through, my initial contact was quite nice. Then I got handed off to a fellow who clearly was not interested. But, with some encouraging from my initial contact, I met with him anyway and ... it did not go well. We parted ways mutually unimpressed, and I never heard from him again.

So much for that recruiting agency.

Second was the presenter's resume (alas, I cannot now find it), which I thought was pretty cool and modeled mine after. Which is a pretty high compliment, actually: I've looked at resume examples online, and they all result in pretty much the same reaction: "This is boring and stupid. Why the hell would anyone read this?". That abundance of awful resumes kept me from even trying to make my own for years, and it wasn't until I saw the presenter's that I realized they could be not-terrible.

Unfortunately for my desire for privacy, my resume isn't terribly impressive if restricted solely to work-related items, so I had to pad it out with a little open-source work. I'd have vastly prefered to keep those lives separate (hence my not linking to a copy of my resume, though I've actually mentioned most of the things on it here), but felt I needed the extra oomph. I expect my resume post-new-job to better hold up without violating that separation.

There's not a lot of privacy in job hunting, actually. People want things like your name. Dice has the audacity to require a phone number and street address. I'm an introverted nerd: I don't take phone calls, and I sure as heck don't want mail. Fortunately, "000-000-0000" and "Undisclosed" are considered valid inputs for those fields.

Anyway, privacy issues aside, and resume in hand, I start looking for work. So I peruse Craiglist want ads, and post my resume on Dice (the trick, of course, is that unlike last time, I'd actually like to end up with a job), and browse job sites.

Great googly moogly are help wanted ads ever awful! I can't even read more than a few a week, they're just so terrible. They either leave me wondering what the job even is, or they leave me wondering why anybody would want it.

So I send my resume out to a few places. Nobody bites. One actually sent me a "hey, thanks for sending us your resume", but otherwise I get nothing. I periodically adjust my info on Dice, and see from my server logs that some of the links I helpfully included in my resume are being clicked on.

Dice nets me two contacts at this point. A recruiter and an interview request.

The company requesting an interview doesn't sound all that interesting—just basic PHP site development—but I figure I could use the practice and go. And I was right about needing the practice, because I bombed the interview, hard. Couldn't even accurately define MVC. Naturally, they passed.

The recruiter was a nice lady who tried to get me a job at WebFilings as a "Scrum Master", whatever that is. I wasn't terribly thrilled about the commute, but she made a convincing argument for giving them a try. But, after a month or two of her contacts being out of the office, they eventually decided to fill the position from within.

Visibility supposedly helps, so I gave my presentation at Pyowa. That netted me an interview—and their head guy from Chicago seemed to really like me—, but for whatever reason, they too passed. Which made me sad, because I really liked them and would likely have accepted in spite of the non-stellar pay because the work that was 3-6 months out (converting their monolithic architecture to a service-oriented one) sounded really fun.

Somewhere in here I also send my resume to a company which hosted one of the CIALUG meetings, because they said they were looking for people. They do a little googling, end up on my father's personal site, and peruse the pictures page, presumably trying to remember who I was (I'm not recognizably there, but they don't know that) and possibly confusing me for my dad. I get a polite brush-off.

Monster is mentioned at a Pyowa meeting. I sign up and post my resume. And, while it results in vastly more contacts than Dice, they're all offers to become an insurance salesman. And they have unsubscribe links, which should give you and idea of how personal they are.

Suddenly, another recruiter contacts me through Dice. She loves my resume, then proceeds to ask questions which can largely be answered by reading it or my profile. I grate my teeth and point this out as politely as I can while I answer the questions. During the phone conversation, "How much do you make now?" "I'd rather not say." "Well I understand and maybe you'll tell me later." (paraphrased), which I think is a little odd (the only thing about money anybody else has asked is how much I'd like to get) but don't worry too much about. I get asked to fill out a form online to get into their system. This makes me not happy, but people aren't exactly beating down my door with money, so I go for it. And their form wants information I can't give it (What month did I start? It was nine years ago, I don't remember!), and information I won't give it (current pay). Not to be deterred, I e-mail the recruiter asking for suggestions. That went something like this:

Recruiter: "You should tell us your current salary."
Me: "No. Stop asking or we're done here."
Recruiter: "Okay, bye."

The exchange was somewhat more politely verbose, of course, but that's the gist of it.

And then I pretty much forgot about Dice until I get an interview request from a little startup which enticed me with things like private offices and no dress code. So I do the interview, and I have no idea how to judge it. No real technical questions, no coding on the whiteboard, just "tell me about yourself / your work" (after a long pause: "I don't normally talk about myself, help me out here and ask me some questions!"), and chumming with a few of the existing coders (during which I did very little of the talking).

But they made an offer, and it wasn't half bad in terms of pay, either. They agreed to a significant narrowing of the scope of their non-compete—which was originally so vague you could construe it as prohibiting clipping coupons from the newspaper—, and I accepted.

So new job for Pixie! It's not perfect—the commute is longer than I'd like (about 20 minutes), the company-provided computers are laptops of all things, and they use Google apps¹—, but it should be interesting.

Here's some things I learned during this experience:

Recruiters "love my resume", but if I could just go ahead and make it completely boring that'd be great.
You can e-mail a link to your resume, using the e-mail as a cover letter. Links to resumes do, in fact, get clicked. (Handy for tracking!)
Links on resumes also get clicked. (Also handy for tracking.)
Visibility is a really good way to get companies interested. It's also the most work.
Monster may get you more contacts faster, but Dice will actually get relevant contacts.
Like dating, you get much better response rates from those who come to you.
Wearing a t-shirt and shorts to an interview will not prevent you from getting an offer.
Every interviewer will bring up Lisp if it's on the resume.
Questions about the technology stack and development process tend to elicit a response of "that's a good question".

And now, some months after accepting a position and working for a while (this entry was in my drafts folder so long I forgot about it!), I can add a few things to the list:

Somewhere along the line, my Dice-tagged e-mail address was acquired by spammers. Ugh.
While I was worried having coworkers would reveal I'm a terrible programmer, turns out I'm not half-bad. (Assuming my measurements are correct, my net effect on our primary repository has been roughly -74,000 lines, in spite of numerous feature expansions and improvements.)
Working on a laptop isn't so bad. My only real complaint is that they cheaped out on the screen, so it's got even fewer pixels than the already-pathetic "full HD" so many laptops tout as if it were decent.
Working 40-50 hours a week kills nearly all of my free time. I have yet to figure out how other people manage full-time work plus side projects.
Having an office is awesome. How anybody gets work done without the ability to make quiet something which was noisy is beyond me.
I really like the new job. Yay!

¹ Good T.B.Lee, people use this crap? None of it works with JavaScript disabled, and with JavaScript enabled it brings my computer to a griding halt. And then some of it still doesn't even work (go go Groups infinite redirect!). Fortunately, I've managed to almost entirely avoid it.

New year, new job, new blog. I'd 301 redirect you if I could, but since I can't you'll have to click through to read the comments or leave your own.

The Reunion

2012-09-04T03:59:04Z

I wouldn't have expected it ten years ago, but I've found myself curious lately¹ where my high school peers ended up and if I was the biggest loser among the class², so I made one of my rare trips outside the house to attend the ten-year high school reunion.

Regrettably, my camera is currently in Chicago undergoing repairs, so I was unable to document the event with pictures. From what I saw of picture-taking, there likely won't be many pictures for posterity. That's unfortunate, but we can at least document the event in words, so here goes:

Men of the 2002 graduating class, we have gotten fat! Some of the women have too, of course, but not nearly in the staggering proportions of us men: by rough estimates, I would say 90% of us men have grown in girth, to maybe 50% of the women.

Well, at least we're well-fed.

Facial hair is surprisingly popular for the men. I was additionally surprised at how many people remarked on my beard; apparently people do not recall that I was much too lazy to shave in high school and generally had a beard back then as well.

I was quite pleased to see many of the women wearing comfortable shoes, rather than high-heels. Still, there was no shortage of women wearing shoes which essentially forced them to spend the evening on their tippy-toes. 8+-inch heels are crazy, ladies!

I felt about as short as I did in high school, though a number of people I remember being taller than I was are now much closer to eye level. Either I've gotten taller, or some of you have gotten shorter.

Facebook is not only the most popular method of keeping in touch, as far as I can tell it's the only method. There was not one "Hey, what's your e-mail?", not one "Hey, what's your phone number?", not one "Hey, what's your blog?", not even an "Are you on Google Plus?"—just "Are you on Facebook?".

No. No I am not.

A special note to their husbands: You bastards! You decimated the reunion hookup market!

Much to my surprise, a number of the ladies who were in my social circle are married and have kids now. Bucking both the trends of highly intelligent women waiting until mid-thirties to have children and of women failing to lose the baby fat, because they are lookin' fiiiiiiine. *ahem*

More people than I expected are still in town. I thought I was the only one! You'd think this town would be small enough I'd have run in to one of them at some point, but apparently not. (Though that may also be explained by my disinclination to leave the house.)

Many people are still in the midwest, even if not in Iowa. Omaha, Kansas City, South Dakota, and Chicago were all represented.

IT was well-represented as a professional group, though there's undoubtedly some bias towards it given my high-school peers. Lawyers were also quite well-represented.

Nobody advertised what they were paid, myself included, so I do not know how we all rank in terms of wealth, but I think I can safely surmise that those of us who showed up are doing okay. Not spectacular, but well enough.

On a more personal level (and boy I really hope I'm getting the names right, because I'm trying dagnabbit):

Amy A. said hi to me! Okay, that sounds a little weird when I say it like that, but we never hung out back then, and I may have had a crush on her at one point. She's a sexy lawyer now, which would have been nice to know a few weeks ago when negotiating the non-compete for my new job, but is nonetheless something to keep in mind for the future.

Chris K. still tells the best stories, which will serve him well in the criminal justice system.

Tori L. is a teacher now, and mad props to her for it. As someone who sometimes thinks about trying his hand at teaching, but knows he'd be terrible, I've got a lot of respect for those who can actually do it.

Congrats to Josh C. on surviving so many years of retail. You're a better man than me, my friend. Also congrats on getting out of it! :)

Sri, me and Laura both miss you and hope you're doing well.

A big congratulations to Lauren H. for sobering up during the past ten years. That kind of successful life change is truly inspirational.

Andrew W., whom I did not get a chance to talk to, was looking quite dapper. Way to show up the rest of us men and get more handsome, rather than more rotund, Andrew!

A big ~~Ha ha, suckers!~~ Congratulations! to those of you who have chosen to join the ranks of parenthood, as well as congrats to those of you who are now permanently off the dating market. There are more of you in both categories than I really expected, but I'm happy for you all the same.

If you weren't there, you were missed, and you should come next time! I didn't even get an invite and still managed to show up, so the rest of you have no excuse. :P

And, of course, a big thanks to Pete L. for arranging the whole affair. He got a 10+% turnout for our class, which I thought was pretty darn impressive, and could not have picked a nicer night for it. Hopefully next time I'll be included in the list of invitees. ;)

P.S.: If you're wondering what I've been up to, it amounts to "not much" punctuated by a few interesting excursions. Also, if you'd like my e-mail address, just ask Pete; he's got it.

^curious I found out about the reunion on Friday, because I googled for it. Fortuitous timing!
^loser Yes. Yes I am.

Introducing clavatar

2012-08-26T03:02:41Z

Earlier this week I discovered Libravatar, a federated Gravatar clone. "Neat.", I said to myself in a very Bender-like tone.

Then I found myself in need of a GitHub account and repo to play with BountyOSS's progressing GitHub integration. A test repo would be boring, so I thought to myself "Hey, this federated avatar stuff is pretty neato, I'll do a repo involving that.". So I did.

And the result is clavatar.

If you're building some sort of site with users, and you want to add avatar support without having to host all those pesky images yourself, you can use clavatar to find an appropriate avatar URL for a given e-mail address or OpenID/URL. It supports libravatar-style federated avatars, Libravatar-as-a-service, Gravatar, and even Unicornify.

(clavatar:get-avatar-url "avatars@example.com" :size 120)
; => #<URI ...>
(clavatar:get-avatar-url "http://example.com/user/avatars")
; => #<URI ...>

Most of the dependencies are readily available through Quicklisp, but you will need a more recent copy of iolib. And I do mean recent—the patch required for clavatar to work landed earlier today.

Video of Burgled-Batteries Talk is Available

2012-07-12T05:03:15Z

(This will be my last post on this talk, I promise!)

You can now watch my talk on Youtube, if you are so inclined.

Sadly, the first 22 minutes have an echo which makes it...awkward to listen to. Due to being very nervous, I stuck more closely to the script than in my rehearsals, though, so if you've read the talk you can skip the video. Q&A starts at 30:30; unfortunately I was a bad presenter and failed to repeat the questions (even after being reminded that I should, gah!), so my answers may not make much sense.

Nobody laughed at my scripted jokes, but people did say afterwards they found the talk interesting, so hooray for that! It was a lot of work, and I'm kinda happy it's over and I can get back to actual programming now. :)

Thanks to the Pyowa group for having me and recording the presentation, and also thanks to Textura for sponsoring both the meeting and drinks afterwards!

burgled-batteries: The Talk (July 10th!)

2012-07-09T09:43:28Z

I will be giving my talk on embedding CPython into Lisp (burgled-batteries) at Pyowa, on Tuesday, July 10th. The meeting is scheduled to start at 6PM CDT.

If anybody wants to show up and heckle me in person, the address is 4445 Corporate Drive, West Des Moines, IA (Google maps). But I understand if nobody wants to drive hours on end for that. :)

Unfortunately, I have been unable to confirm whether there will be a Google+ hangout for lispers to join in, or any video recorded. (I remain hopeful, because Pyowa has done them before, but can make no promises.) I'll try to pester the appropriate parties to link to either of those things from the Pyowa twitter feed, if they're available. You might also try Pyowa's youtube or Google+ accounts, just in case.

Apologies that I cannot be more definitive about the possible methods of remote participation. :/

Slides, and my current script, are already available (last run-through was ~35 minutes, somewhat less than my original timings, though I've expanded the latter half of the talk). Thanks to those who have taken the time to review and provide feedback on my earlier drafts. There's still time to offer some if you have any!

burgled-batteries: The Talk (Coming Soon?)

2012-06-27T06:01:49Z

My local Python users group has expressed some interest in my giving a talk on burgled-batteries. I figured "hey, why not?".

They happen to be pretty good about recording presentations and doing Google+ hangouts and posting the videos to YouTube and so forth, so once I have a date for the talk and know where any of that stuff will be I'll mention where to find it and then the Lisp world can admire my awkwardness in front of a crowd, possibly maybe even join in and embarrass me with questions I should know the answers to but don't.

In the mean time, if anybody would like to help me with my talk, I'm all ears. I've got a draft here that currently takes 45-50 minutes to run through. Am I glossing over dealing with CPython too much? Is there anything fellow Lispers would like to see more or less of in the talk? Is it so terrible I should just toss it and start over? Etc.

It's a git repo, so you can do git clone http://talks.kepibu.org/2012/pyowa-bb/.git/ pyowa-bb if you feel the urge to send patches, but general advice would be great too. It'll be my very first talk, it'd be nice if it were at least mildly interesting to somebody somewhere. :)

Firefox has Gone Rogue

2012-06-12T00:50:44Z

Firefox just force-"upgraded" itself to v12, from 3.6. Since I was staying on 3.6 for some very specific reasons (one of which being so I could take screenshots for an entry noting everything they've screwed up in newer versions of FF), I am not happy. Of course, since I never gave it permission to update to 12, I'm also not happy for reasons of "installing without permission is what malware does, jerkwads". There's also anger due to the "I just wanted to do some work, not spend all day reconfiguring firefox" angle.

Seeing as I am now unable to take aforementioned screenshots, here's the entry, minus illustrating pictures:

The Many and Horrifying UI Sins of Firefox

Elimination of browser.link.open_external

For those of us with browser.link.open_newwindow set to the only sane value (1—or "If I want it in a new tab/window, I'll ****ing tell you!"), the merging of the completely different browser.link.open_external option was a sad day indeed, because it meant that links from other applications (say, your mail program) would destroy whatever page you were on, rather than opening in a new window (or tab) as would be proper.

see bug 469082, bug 324164, bug 509664

Note comment 20 in bug 509664, which states "links from external programs should never ... open new windows". Which is, ya know, the complete opposite of how many of the people configuring open_external had it set. Good job listening!</sarcasm>

Corrective actions:

Restore Open_External
might also be correctable by setting the undocumented preference browser.link.open_newwindow.override.external in very new versions of FF

Opening Add-Ons configuration in a tab

see comment #3 on the largely unrelated bug 597164 for what is pretty much my reasoning for thinking it bad, but written by somebody else.

Corrective action: Add-ons manager Dialog Returns (can not be uninstalled)

Combined back/forward history dropdowns

If I bring up the backwards history menu, it's because I want to go back, and showing the opposite direction does not make sense. Likewise for forwards. Further, disjoint menus nicely follow Fitts' Law, where the greater number of pages skipped requires the greater amount of effort, making it perfectly intuitive.

[img of combined dropdown] Quick, what do I click on to go back four pages?

Corrective action: NoUn Buttons (was also necessary in FF3)—but beware it is not yet up to snuff on newer firefoxen: it sorts the dropdowns properly, but fails to remove back items from forward list, and vice versa. (But does if you right click. The hell?)

Combined reload / stop buttons

"Reload" and "Stop" do two completely different things and on that point alone should not be combined.

Corrective actions:

Can be suppressed by adding a spacer between the two buttons. Unfortunately, this means you get an ugly extra space between them for your troubles.
Active Stop Button claims to include correction in addition to other, possibly unwanted, behavioral changes.

Moved reload / stop buttons

Since the dawn of browsers, the buttons have been Back, Forward, Reload, Stop. Thou shalt not alter this order nor insert items in between. You don't even need any extra buttons (am I the only one who finds the Home button completely pointless?).

Corrective action: Rearrange.

Tabs on top

This [img] is the browser UI. This [img] is the browser UI with tabs on top. Any questions?

NB: The illustration would have colored browser UI, document, and document selector different colors, thus highlighting that with "tabs on top" (really "tabs in the middle") the document selector is in the middle of the browser UI, rather than between the browser UI and the document, where it belongs as the part of the browser UI which transitions between documents.

Corrective action: May be deselected in UI.

URL Formatting

Why show the full URL, when you can confuse the hell out of people by adding obnoxious coloring in an attempt to make the full URL less readable?

Corrective action: set browser.urlbar.formatting.enabled to false in about:config

http:// trimmed from URL bar

Dear Firefox, please stop copying Chrome. If I wanted Chrome, I'd be using it. Kthx.

Corrective action: set browser.urlbar.trimURLs to false in about:config

Non-native widgets

While the non-native tabs in FF3 bugs me (and, by extension, the non-native tabs in FF4+), the non-native widget that bugs me the most in newer versions of Firefox is the URL bar. They took a beautiful, native-looking thing and shat all over it, turning this lovely thing [img] into this monstrosity [img].

Corrective action: None known.

Remember password and Share location dialogs

This is what it looked like in FF3: [img]
This is what you get in newer versions of FF: [img]

Notice the large number of problems immediately apparent:

FF3 offered each available option with a single click. FFn requires selecting the item from a drop-down menu.
In FF3, the default was obvious. In FFn it is not. In fact, notice how the share location dialog makes it look like FF will share your location if you simply dismiss it.
FFn's dialog is vastly more intrusive. Whereas FF3's could be ignored until one was ready to deal with it, FFn's demands attention by covering part of the page.

Corrective action: None known.

Elimination of the status bar

First off, the status bar is where all the little bits you sometimes use to interact with extensions go. Sticking that stuff in the toolbar would take up way too much space for anybody with more than one extension. Second, it provides a stable place to look for certain information.

Of course, it wouldn't bother me so much if I didn't have to install a gorram extension just to get the status text on the status bar. Look, I'm sorry you can only afford a netbook as a Firefox UI designer, but some of us have real computers and can afford the twelve pixels for the status bar. There's no need to be so hostile towards it.

Corrective action: Status-4-Evar

Status bar has a close button

Who the hell decided a status bar needed a close button? And on the opposite side of every other close button, to boot!

Corrective actions:

Right click -> View Link Info has gone away

I'm what, ten minutes into actually using FF12 and already I'm noticing missing features? Ugh.

Corrective action: None known.

URL in status bar is abbreviated in the middle, rather than the end

I very nearly missed that a thing was abbreviated because the marker I was used to no longer exists at the end. That's not necessarily bad (I was thrown off when "Find" no longer popped up a dialog box, for instance, but that was a clear improvement), but putting "..." in the middle means it's now impossible to tell the difference between "long thing with ... in it" and "long thing cut short" (and you can't exactly format "..." in a distinguishing manner, to differentiate).

Corrective action: None known.

Frikkin' Scavengers

2012-06-05T06:36:51Z

My city's Spring Cleanup thing, the once a year they come around and pick up large, loose trash, is upon us. With it come the scavengers—people who drive around picking up things they'd consider useful. It's sort of like a city-wide garage sale where every item is free.

So we threw a few items on the curb for the scavengers. What I expected would happen is if a scavenger wanted an item, they'd take it. Turns out, not so much: they'll destroy your item to get the part they want, then leave you holding the pieces of your now broken and useless item.

I put out some old CRT monitors—not knowing whether the city will actually take them—thinking "well, maybe a scavenger will want one, and if they don't, and the city doesn't take it, I can just bring it back inside". I'm no worse off either way, maybe somebody is a little better off. At least, that was the plan.

Some chap seemed interested in the monitors, turning them over. I figured he was looking for some bit of info on the backs. But then he left, failing to put them monitors upright. "Well, that's rather rude, I thought." and went to put the monitors upright again. Upon which I realized the monitors were now missing their built-in VGA cables, because the fellow had sliced them off; taken the copper (presumably), and left the monitors.

What the hell, man? Look, once you take the monitor, I don't care what you do with it. Extract the copper, trash the rest, whatever. But breaking my stuff (it's still in my yard, I'm still responsible for it, ergo still mine) and leaving me with a now-useless piece of equipment? That's willful destruction of property, and a dick move.

And I am pissed.

Can't do anything about it because by the time I realized what he had done he was gone, but I'm still pissed. So very, very pissed. Because now the monitors are completely useless. If the city doesn't take them I'll have to pay to recycle them because I can't even stick them back on the shelf as a spare.

Asshole.

Threw some printers on the curb, too. Didn't see the person who took them, but they took the printer and left the paper guides and trays. Look, they're a part of the thing you're taking! If you don't want those pieces, throw them out with your own trash, dammit.

Not nearly as dickish a move as bricking the CRTs, though.

The guy who took the box of assorted Jeep parts I left out looked sheepish, like he thought he was stealing something. That, at least, was hilarious.

Even so, I think I'll just rent a dumpster next time. Because Mr. Slice-n-Dash has left me bitter. You don't smash things for the piece you want and leave the pieces, you take the whole thing. Smashing for parts is completely disrespectful of the people who are leaving these nice things out for those who might find them useful.

Am I wrong here, internet? Am I wrong to expect that if someone is going to take something they should take the entire thing and tear it apart on their own turf?

Passwordless Drive Encryption for Virtual Machines on Linux

2012-05-18T01:10:49Z

Recently, I acquired a used Xserve RAID. Good storage capacity, and way cheaper than buying new drives. In the interests of data security I'd like easily removed drives, such as those in the RAID, to have their contents encrypted.

The master server is just a house for VMs. And, because those VMs are generally free to be rebooted with wild abandon, entering passwords into the VMs controlling the RAID arrays would be a real downer. Fortunately, the master computer to which the RAID is attached is already set up with encrypted hard drives—which means I can store keyfiles for the RAID arrays on it without compromising security.

In fact, thanks to proper layering, the VMs don't even have to know the drives they're accessing are encrypted.

Here's how.

Set up Drive Encryption

First, we'll need a key file. Random data will do nicely.

KEY_FILE=/etc/luks-keys/$VMNAME
head -c /dev/random 1024 > $KEY_FILE
chmod 400 $KEY_FILE

Next, set up encryption on the RAID array.

cryptsetup luksFormat $RAID_DEVICE $KEY_FILE

And then add it to crypttab, so encryption will be automatically set up and torn down at system start/shutdown.¹

UUID=`cryptsetup luksUUID $RAID_DEVICE`
echo $RAID_VOLUME UUID=$UUID $KEY_FILE luks >> /etc/crypttab
cryptdisks_start $RAID_VOLUME

The encryption set up and running, it's just a matter of creating the VM.

Configuring the VM

I like to create a partition on the regular drive for the VM's OS. While not strictly necessary, it gives a bit of flexibility (e.g., the ability to switch between two different VMs controlling the RAID while testing different NAS distributions), and leaves the maximum amount of space on the RAID array for files.

lvcreate -L $SIZE -n $VOLNAME $VOLGROUP

That done, actually make the VM.

virt-install --name $VMNAME \
  --disk /dev/$VOLGROUP/$VOLNAME \
  --disk $RAID_VOLUME \
  …

Or, if adding the encrypted disk to an existing VM:

virsh attach-disk $VMNAME $RAID_VOLUME $VMDEV --persistent

Once inside the VM, you can format the attached volume as normal, none the wiser to the encryption. It just works.

Of course, encryption means if your RAID array loses one too many disks your data is hosed irrecoverably, so be darn sure to have a good backup policy in place before you start storing critical data on it!

For the purpose of those backups, it might be desirable to ensure you can take snapshots of the filesystem. That's doable, too:

VGNAME=$RAID_VOLUME
vgcreate $VGNAME $RAID_DEVICE
lvcreate -L $SIZE -n volume $VGNAME
mkfs -t $FSTYPE /dev/$VGNAME/volume

where $SIZE is some size smaller than the extent of the device. Exactly how much smaller depends on how much data churn you're expecting while the volume is snapshotted, but I'd say leaving ~10% for the snapshot is probably sufficient.²

Then you can attach the logical volume to your VM

virsh attach-disk $VMNAME /dev/$VGNAME/volume $VMDEV --persistent

and use snapshots to back up the VM from outside with the VM none the wiser.

Note that the mkfs turns out to be important: if you don't format the logical volume before mounting it within the VM, the VM will see the LV rather than an unformatted block device and be very confused.

A Note about Fibre Channel

Using the mptfc driver with my fibre channel card, I was unable to get echo '- - -' > /sys/class/scsi_host/$HOST/scan to recognize a change in RAID size, and don't recall having much luck actually specifying the parameters, either.

Rescanning was, however, successful with echo 1 > /sys/class/scsi_device/$DEV/device/rescan

So if scan doesn't help, try rescan. It certainly beats rebooting!

Footnotes

¹ Alternatively, you could add a libvirt hook to set up and tear down the encryption before and after the VM is booted and shutdown. That is left as an exercise to the reader.

² As a point of reference, upgrading Debian from Stable to Testing requires ~7% of a 32GB volume.

DLNA and Netflix: A Review

2012-05-14T05:14:26Z

Some time ago, we got a TV that supports DLNA. I had no idea what that was, but a quick consultation with Wikipedia left me eager to give it a try.

I've tried a couple of DLNA servers since then, and found all of them wanting in some manner. I've also been using Netflix streaming of late, so I'll add that to the comparison seeing as it fills the same niche.

DLNA

Why?

I gave DLNA a shot because my TV supports it and it seemed like a good idea.

Pros

Works even when internet is down.
Doesn't leak information about watched shows.
Content stays available as long as you want it to.
No commercials!
No monthly fees.

Cons

No fast-forward, rewind, chapter skipping, or single-frame progression.
Can't handle interactive content (DVDs, etc.).
Client capabilities differ significantly (e.g., in whether they support switching between multiple audio streams).
Wildly varying client capabilities necessitates remuxing and/or transcoding, complicating servers.

Verdict

DLNA is a nice idea, poorly implemented atop a terrible protocol. Because none of the DLNA servers I've tried are perfect, I've ended up running two beside each other so I can use the fast one when it works, and the featureful one when the fast one can't do what I need.

PS3 Media Server

Why?

I tried this at the recommendation of the local Linux User Group.

Pros

Supports pausing, skip forward, and skip backward.
Ability to force audio to desired language (if available in file, of course).
Debian package available (third-party).
Can run headless.

Cons

Slow to browse. You can speed up browsing a bit by enabling the cache, but that broke audio language forcing for me.
Slow to play. Files require multiple seconds to start playing, skip forward, back, or unpause.
Unpausing often restarts slightly before or after the pause time.
Had to scrounge around forums for a profile that (mostly) works with my TV.
Documentation assumes you're running the GUI, leaving most of the config file options un- or poorly-documented.

Verdict

PMS is okay. The slowness is infuriating, and configuration involved more than starting it up, but it's been very stable. The ability to prefer some languages over others is an excellent feature.

MediaTomb

Why?

I tried this because it was in Debian.

Pros

Debian package.

Cons

XML config file.
Web interface.
Can't pause or skip when remuxing or transcoding.

Verdict

MediaTomb is crap. XML-based configuration files are terrible, and web interfaces are a blight upon mankind that should be abolished. But even if you can look past that, the inability to do something as basic as pause makes it completely worthless.

Serviio

Why?

I tried this because it was mentioned somewhere on the interwebs.

Pros

Browses fast.
Plays fast.
Resumes fast.

Cons

No Debian package.
Can't switch audio language.
Must be configured with included GUI client, no config file.

Verdict

Serviio is decent. The fast pausing and unpausing is a welcome relief from pms's multi-second unpause, and the fast browsing makes getting to the file I want to watch much less painful. The lack of a .deb is a bit of a downer, but installation is relatively straighforward (untar and run). While I'd much prefer a proper config file, the only thing preventing me from ditching pms entirely is the inability to prefer a particular audio language.

Edit (16.05.2012): I seem to have spoken too soon on Serviio. Further testing on a wider range of files has shown it to sometimes have infuriatingly-long start-to-play times and that it works with fewer files than pms. Alas, it had such promise!

Netflix Streaming

Why?

I tried this because hey, free trial.

Pros

Much wider array of content than my local media server.
Skipping around the video is much nicer than DLNA.
Selection UI is way better than DLNA.
Good for discovering things you didn't know about or old things you'd forgotten.
No commercials!

Cons

Stops working when the internet goes down mid-video.
No rewind, fast-forward, or single-frame movement.
The video progress meter never goes away when paused. (Especially annoying when pausing to try and read something, only for that thing to end up covered.)
Never more than 720p.
Leaks watched show information.
Many things I'd like to watch aren't available (e.g., no Jack Benny).
Content regularly disappears or threatens to disappear with minimal notice.
Many series are only partially available. E.g., you can watch ReBoot seasons 1 and 2, but not 3 or 4; and Sesame Street is missing the first forty years or so.

Verdict

Netflix is awesome, when they have something you'll watch. Searching for something specific is an exercise in futility (if you're looking for it, Netflix won't have it), but browsing can often result in something interesting. I very much enjoy when the "Comedies from the 1940s" category pops up, for instance; and the availability of the wealth of long-running series from the Star Trek and Stargate franchises will likely keep me subscribed and entertained for quite some time.

Summary

Netflix and DLNA are both awesome ideas with not-so-awesome implementations, but between the two Netflix is an incredible triumph. DLNA generally works, and is vastly superior to watching things on a computer monitor, but has little else in its favor (I suspect I'd ditch it entirely if I had an HTPC running XBMC). Netflix certainly has areas in need of improvement, but I was surprised at just how quickly and completely it supplanted any other method of watching video content—my biggest annoyance is actually the pause thing, believe it or not, because I pause to read things all the time.

Accept-Charset is Still Necessary

2012-03-01T20:38:13Z

Some people have apparently been arguing that the Accept-Charset HTTP request header is unnecessary. Obsolete. Not worth the extra bytes.¹ And they've been doing a pretty good job of arguing it: most newer browsers have stopped sending it.

This is a mistake.

The HTTP/1.1 spec has this to say:

If no Accept-Charset header is present, the default is that any character set is acceptable.

But practically speaking, sending any old character set won't work. For instance, a modern browser is likely to have no idea what to do if you send it EBCDIC. Which means, from a practical standpoint, you can only assume the available charsets are: what you're told the browser can handle via accept-charset, and what you're told by the standards the browser must support.

HTTP/1.1 requires only one charset: ISO-8859-1.

Section 14.2 implies this with:

If no "*" is present in an Accept-Charset field, then all character sets not explicitly mentioned get a quality value of 0, except for ISO-8859-1, which gets a quality value of 1 if not explicitly mentioned.

While section 3.7.1 makes it explicit:

The "charset" parameter is used with some media types to define the character set (section 3.4) of the data. When no explicit charset parameter is provided by the sender, media subtypes of the "text" type are defined to have a default charset value of "ISO-8859-1" when received via HTTP. Data in character sets other than "ISO-8859-1" or its subsets MUST be labeled with an appropriate charset value. See section 3.4.1 for compatibility problems.

ISO-8859-1 is a strict superset of ASCII, which means you can actually assume the availability of two charsets: ISO-8859-1 and ASCII. Note, however, the complete and utter lack of a Unicode requirement.

Now, if you've been around the web for a while, you'll know: way back before utf-8 became largely ubiquitous, not every browser sent an accept-charset header. Some of those browsers did not support unicode. (WebTV comes immediately to mind, I'm sure there were numerous others.)

Which means that, even if we ignored the theoretical concern, we still should not assume utf-8 support in the absence of a corresponding accept-charset because there are clients it would break.²

Now, certainly, feel free to require utf-8 support in the next version of HTTP (Google's SPDY does this, IIRC), and eliminate the header then. That would be appropriate and acceptable.

But not now.

Because if you do it now, you bifurcate the HTTP/1.1 protocol. You split it into the "early 2000s HTTP/1.1" where a missing accept-charset meant you needed to send ISO-8859-1 or ASCII, and the "2012 HTTP/1.1" where it means fuck all.

There are enough broken things on the web today—don't add to it. Put accept-charset back, and let's pretend this never happened.

You want to save bytes? You can save quite a bit more by changing browser UA strings to "<browser>/<version>", rather than the current "Mozilla/5.0 (<tons of crap>; <browser>/<version>; <even more crap>) <possibly extra crap>". Better yet, discourage any remaining remnants of browser sniffing (a terrible idea from the start) and eliminate the header entirely.
That virtually every website in existence wrongly assumes utf-8 support and sends it regardless does not justify modern browsers wrongly failing to advertise their support. Some of us actually care enough to do this right, and you're making our job harder than it needs to be.

Accidental Load Testing

2012-01-29T15:10:31Z

I run a low-traffic webserver. I'm also marginally obsessed with returning pages quickly (for our primary site, as of this writing, Google reports the average time Googlebot spends downloading a page as 159 milliseconds, and that it takes 0.4 seconds to load for users on average).

However, that's not terribly impressive when your busy season is significantly under 100,000 hits per month. So I've often wondered what would happen if we were to get a sudden influx of traffic. Would the server hold, or would it fall over?

Thanks to an unintentional inclusion in Planet Lisp, my previous picture-heavy entry provided a small taste of what would happen in the event of sudden popularity.

A sudden influx of traffic. It dies off quickly, probably at least in part due to Xach's quick removal of the entry from the Planet Lisp feed.

As you might expect, lots of hits to the webserver has a strong correlation to the amount of outgoing traffic.

Load and memory usage stayed essentially constant, thanks to the use of lighttpd for static content such as images. Had all of that traffic hit Apache, I imagine these two graphs would look somewhat different.

In words, what happens with a large influx of HTTP requests is: not much. Which is exactly what I want. A larger influx of traffic, or traffic that would involve PHP pages, is of course another matter entirely and I still don't know how things would hold under that case. But I'm quite pleased with the non-effect I saw in this case.

An interesting, and to me surprising, side effect is shown here: SSH probes had a significant jump in number. I wonder why?

Nashville, 2010

2012-01-05T11:34:14Z

Way back in 2010 I visited my friend sparky in Nashville. It was good times, but I never got around to doing a write-up.

Instead, I present my visit to Nashville in pictures.

Getting to Nashville

I took a plane to Nashville. This involved some waiting around in a terminal.

Not much to do in a terminal but take pictures of airporty things.

These chairs were reserved for people in wheelchairs. Kinda seems like a step down from being in a chair with wheels.

Don't forget to turn off fluid pumps when not in use.

It was an afternoon flight.

I'm not sure, but this might be a midget joke. (Clearly it's not meant for kids, there's nothing to color.)

Apparently elevator permits are actual things. Unlike my ability to hold a camera steady.

I blame these on feeling "artsy".

Hooray, the plane!

Vroom vroom.

During takeoff, the airfoils were set to 9 degrees.

A wild herd of school buses.

Guys, I think your boat missed the water!

For landing, 22 degrees.

A quick change of planes near Gate D28. So quick I didn't even go inside the airport.

Must be cold outside.

I realize the sign itself is asymmetrical, but surely I can't be the only one bothered by the asymmetricality of the arrow locations?

Fluffy.

Nashville! Clearly, I was ... moved by the music.

Threat level Orange means the guy in the orange shirt is a threat, right?

Ah, there's my host! Don't just stand there, I'm being mugged you fool!

Wandering around Nashville

Apparently water meters are often confused with traffic signals.

Vegetation!

Don't forget kids, it's important your hair color matches your car.

I wish you well, young pine, but I do not expect things will work out for you.

My host decided he should take a picture.

This wall clearly needed to be remembered. I wonder why?

More trees should wear nametags. Really takes the guesswork out of what species they are.

if it weren't for the Bank of Nashville, this could have been anywhere.

A real American Darling.

This is mostly what I remember of Nashville.

I'm not sure, but I think a nearby power plant might burn coal.

Who goes to a city known for its music and takes pictures of plants? This guy!

A Quick Stop in Union Station

Opulent!

They don't make 'em like that anymore.

"Lost Divas"?

Even the bathroom is swanky!

Harper's Weekly

Really? You guys couldn't spring for real logs?

Dammit Frank, you weren't supposed to stare at the sculptor!

The ironworker was feeling a little loopy that day.

Even the carpet was fancy!

Back Outside

Where unattended packages will be confiscated and destroyed. By lasers!

I wonder what this thing is for? Oh, handy, they tell me who to ask.

I've got hundreds of out-of-focus pictures of interesting things, but sparky's ass is one of the few in-focus pictures. Go figure.

Apparently the rebellion against good grammar is on Facebook.

"So how's the parking?" "Eh, it's allright."

Personally, I think commissioned art should look better than graffiti.

Cop! Act natural!

In the event of a rampaging giant, this building comes equipped with handles.

Wait, that's not how you put a building together.

This was the secret entrance until budget cuts forced them to close it up.

Should I be happy Nashville's citizens can read, or saddened they need this much instruction on how to cross a street?

Rockets!

During my trip to Nashville, we took a trip to Florida to watch a space shuttle take off.

NASA's Theme Park

Turns out NASA has their own theme park.

I think it may have been the shuttle Atlantis.

NASA prefers to get their flags from Patriots.

I think my lens is dirty.

Just hangin' out.

For your safety, please do not bring a change of clothes.

The ground of NASA!

The ashtray of NASA!

The admission price of NASA!

Space tourism is fraught with DANGER!

The trash can of NASA!

Oh good, I was really concerned about your environmental policy.

A map!

Finally, inside the NASA Theme Park.

Some sort of re-entry vehicle?

Naturally, I'm interested in the device holding it. That makes sense.

A rare Space Croc!

I wonder if the darts are included with that twelve-thousand-dollar rock.

Consumerism is in full swing at the NASA Gift Shop.

Warning: do not steal this shuttle and fly into space. You will burn up on re-entry.

The line to see the fake shuttle.

I had no idea what those people were standing around for, and so just walked past everybody. Nobody said anything.

No warning lights went off.

But there were some ladder problems.

So I took the elevator back down.

I attended a presentation by Buzz Lightyear.

I wandered into some building to get out of the heat. Air conditioning!

Listen for British Intelligence.

There's something strange about that cargo.

Turns out I'm in line for a movie!

3D doesn't work very well.

After the movie, we traveled back in time.

Unfortunately, they keep the doors locked. Sure looks like fun in there!

At NASA theme park? Check. Taking pictures of plants? Check.

Atlantis' Final Launch

Everybody's getting ready to watch the launch.

Quick, take a picture of a tree!

A flyby to, uh, make sure nobody is packing a bazooka?

Turns out this is about the closest we're gonna get to the shuttle.

Whoooooooosh!

The crowd looks on in awe.

While I take a moment to check out some guy's ass.

Sparky felt like a kid again; I still had my zoom lens on.

More NASA Theme Park

How do you top a shuttle launch? With LEGOs!

Some complete strangers pose while somebody else takes a picture.

I suspect this isn't a robot so much as a sculpture.

The answer is in the data!

We take revenge against HAL.

And journey to Mars!

Only to get trapped inside Tron.

To ensure astronaut happiness, NASA cultivates Space-Weed.

The barcode is a nice touch.

Oh, there you are, Sparky.

Thanks to Sparky's amazing planning, we took a cab back to where we parked.

And, without any sleep, drove back to Nashville.

A Sandy Detour

But first, let's check out this test hole.

Oh, and maybe spend some time at the beach.

It's a very welcoming beach.

I still need to clean that lens.

I took a lot of pictures of water.

Surfin' USA!

Sir, I don't care how handsome your smile is, that baby carriage still looks ridiculous.

Nashville at Night

Sparky enters a trance and catches up with the Internet.

Meanwhile, I explore the local cuisine.

In the Parks

We stop at a park for some sunset pictures.

It's a little creepy.

Berries!

Sparky looks for just the right angle.

And leaves a message for future generations.

Anyswing for the shot!

We head off to a different park, where we pick up a modeling client.

And I find some roses.

Sparky aims for a water shot.

I may have intruded upon a private moment between him and his camera.

Guys, I think you're supposed to land at sunset, not take off.

It was a pretty sunset.

What's this? We've been transported to Europe!

I try my hand at a water shot.

Too dark for pictures? Not with long exposures!

Oh screw it. Flash!

In the City

The Nashville night life!

Sparky sets up a shot.

I become enamoured.

Celebrate the music scene.

Don't forget to buy a t-shirt.

But no parking.

This building not well ventilated.

Whirrr.

Turns out they don't actually print things in Printers Alley.

Apparently the date did not go well.

Sparky takes notes.

Don't blink.

Ugly Mugs

While I have surprisingly few pictures of the place, we spent a large amount of time at Ugly Mugs, Sparky's coffeeshop of choice.

My laptop is bigger.

They kept the place nice and warm.

In case you wondered about the name.

A fine selection.

Sparky, get internet at home for that sort of thing!

The Countryside

We took a trip out to the countryside and took some pictures.

The grass was happy to be in focus.

A bug photobombs.

We pass some bioxide.

Heavy traffic.

Don't look at me, I'm naked!

That soccer field's a real dump.

I believe it safe to assume this fellow is monied.

House of Spark

All my pictures are blurry and out of focus. Boo! There's a lovely bamboo forest in the backyard, and the interior is actually decorated (a former housemate was a decorator, apparently). The house itself, however, could stand some TLC. And possibly a new roof.

A Trip to the Library

We went to the library to watch little girls in tutus.

At the end they took a bow.

The place cleared out quickly.

And we could explore the rest of the library.

Craig's Church

What does an atheist do at church?

I'm pretty sure it involves fancy equipment.

And sound checks.

Let's slip into something a little more comfortable for the Lord.

Going Home

Before I know it, it's time to say goodbye to Nashville. So I hopped on a plane and went home.

The in-flight entertainment for the first leg of the flight home was pretty good.

Due to a small problem, my connecting flight taxied from the runway back to the airport, where we all got off the plane and waited.

And waited.

And waited. Finally, we got back on the plane to go home.

Miscellanea

Though not pictured, we both really enjoyed "The Butler Dunnit" at the in-the-round Chaffin's Barn Dinner Theatre. If you're in Nashville, it's totally worth a visit.

I'm not really a music lover, so Nashville's many bars and live music venues had little to offer me. We did not attend any.

Unfortunately, Nashville was struck with floods before I arrived, and many of the places we would have visited were closed for repairs.

Because it's taken me so long to even do this much, I haven't done any post-processing on the pictures. That means some (mostly night pictures) are much too dark. It also means some of my comments may not make much sense because the thing I'm commenting on is only really visible at full size, and I haven't done the necessary cropping for the scaled-down versions presented. Apologies.

Sparky: I'd love to include some of the pictures you took while I was taking pictures of you taking pictures. If you happen to have any handy. :D

Strategies for Automatic Release of Foreign Resources

2011-11-30T09:18:19Z

Wanting to avoid forcing users of burgled-batteries to deal with cpython reference counts for Python types with unknown translation, I'm exploring not one, but two separate strategies for automatically handling the issue.

The second, upon which I based my previous ill-advised CFFI trick ¹, and for which I am grateful to Paul Khuong for correcting my error, is to let the GC decide when a resource is no longer needed. As I discovered attempting to test it, however, this does not necessarily free one from thinking about when a resource can be freed.

The first approach, which is considerably more deterministic, is also somewhat more limiting in extent. In that approach, pointers are also wrapped, but the wrapped-value has dynamic-extent within a barrier². Leave the barrier, and any wrappers are relieved of their wrapped value.

Both approaches have at least one downside in common: wrapped values cannot be used in place of a POINTER. This can be lessened somewhat for CFFI's TRANSLATE-*-FOREIGN functions (with some caveats involving the use of the EXPAND- variants):

(defmethod translate-to-foreign ((value wrapped) type)
  (translate-to-foreign (wrapped-value value) type))
(defmethod translate-from-foreign ((value wrapped) type)
  (translate-from-foreign (wrapped-value value) type))
(defmethod free-translated-object ((value wrapped) type param)
  (free-translated-object (wrapped-value value) type param))

but MEM-REF and friends (reasonably!) assume POINTERs, so there's no getting around callers noticing the difference. Whether that's a worthwhile tradeoff depends on your goals.

Release by GC

The Good

Foreign resource has indefinite extent

The Bad

Freeing of foreign resource is unpredictable

The Story

While I'm not always particularly good about testing, refcnts are incredibly easy to screw up and tracking down refcnt bugs is unpleasant, so I've been trying to make sure I test things involving them to catch when I screw up.

After a while flailing about with no idea what I was doing (seriously: at some point I decided adding threads into the mix would make it easier to test things involving the GC), I had an epiphany. "Gee, I wonder if trivial-garbage has finalizer-related tests?" It does. Well then, I'll just copy whatever they do.

;;; I don't really understand this, but it seems to work, and stems
;;; from the observation that typing the code in sequence at the REPL
;;; achieves the desired result. Superstition at its best.
(defmacro voodoo (string)
  `(funcall
    (compile nil `(lambda ()
                    (eval (let ((*package* (find-package :tg-tests)))
                            (read-from-string ,,string)))))))

I'm not particularly thrilled with the "call with strings of Lisp code" interface for VOODOO, so I mess with it a little but not too much because I understand it even less than the original author.

(defun voodoo (expr)
  (funcall (compile nil `(lambda () (eval (read-from-string (prin1-to-string ',expr))))))
  (values))

Having simplified my tests, and confident the remaining problems I was seeing were in fact problems with my tests and not with the thing being tested, I set about discovering what those problems might be.

(let* ((wrapped (run code))
       (unwrapped (wrapped-value wrapped)))
  (do-things-with unwrapped))

Spot the problem? If not, here's a hint: after what point is WRAPPED eligible for GC?

WRAPPED might get GCed at any point after (wrapped-value wrapped) is calculated. If that happens, the finalization on WRAPPER changes, and potentially destroys, what UNWRAPPED points to. And then you've got a problem with the test (or, potentially, user code).

Lesson learned: don't wander around with an unwrapped pointer in a &body. But it's worse than that. Don't even pass the unwrapped pointer to a function if you can avoid it:

(let ((wrapped (run code)))
  (do-things-with (wrapped-value wrapped)))

WRAPPED becomes eligible for GC as soon as WRAPPED-VALUE returns, which means the C object might go away before the pointer is passed to DO-THINGS-WITH, or any time during the execution of DO-THINGS-WITH.

If you're lucky, liveness will be ensured by wrapped pointers being part of some larger datastructure, but if not, it becomes necessary to make sure WRAPPED is required later on, to avoid it becoming eligible for GC too soon.

(let* ((wrapped (run code))
       (unwrapped (wrapped-value wrapped)))
  (do-things-with unwrapped)
  (calculate-something-using wrapped))

Unfortunately, if using wrapped pointers, one must keep in mind the liveness of the wrapper, lest foreign resources disappear too soon—making them not quite so effortless as I had hoped.

Release by Barrier

The Good

Foreign resource is freed predictably

The Bad

Ham-fisted approach

The Story

I started with refcnt barriers first because they're considerably less scary to write. Their predictability also makes them considerably easier to debug and reason about.

(defclass barrier-wrapper ()
  ((value :initarg :wrap :reader wrapped-value)))

Wrapper. The invalidation of pointers upon exiting the barrier necessitates an ability to modify the VALUE slot. Hopefully nobody does.

(defvar *barrier-objects*)
(defun initialize-instance :after ((object barrier-wrapper) &key)
  (push object *barrier-objects*))

Nothing fancy. Just quick-and-easy tracking of wrappers created within a barrier.

(defmacro with-barrier (&body body)
  (let ((*barrier-objects* (list)))
    (unwind-protect
        (progn ,@body)
      (mapcar #'invalidate-wrapper *barrier-objects*))))

(defun invalidate-wrapper (wrapper)
  (free-somehow (wrapped-value wrapper))
  (slot-makunbound wrapper 'value))

Clearly I'm taking liberties with #'FREE-SOMEHOW, but you get the point. Upon exiting the extent of the form, we free the resource and invalidate the wrapped value. I unbind the slot, but setting it to NIL might better suite your tastes.

This is certainly useful in some cases, such as where you're processing things in a loop and each iteration is unrelated to the next, so you don't need to carry any values over. And it's not as if WITH-THING is unheard-of. But the complete lack of subtlety bothers me.

While it'd certainly be possible to provide a little more nuance by adding in the ability to promote objects to the next barrier or GC-ability, I'm not convinced reintroducing parts of manual memory management is an appropriate method of dealing with the shortcomings of a system meant to automate it away.

Conclusions

While both are (to me) clear winners over forcing the issue on library users, I don't think either is a clear winner over the other. Letting the GC handle things is more conceptually elegant, but practically speaking it has some thorny issues that are difficult to overcome and even harder to spot.³ Barriers are easy to reason about, but they're lacking in useful nuance, and I suspect would wind up being too large to be practical, or too small to be useful.

So, for now at least, burgled-batteries admits both approaches, though I have not yet come up with a good way to enable the approaches to be interleaved.

A third approach might be a sort of hybrid, where an implementation similar to barriers is used not to ensure foreign objects are destroyed upon exiting the form's extent, but to ensure they remain available within it. A "corral", I guess you might call it?

At any rate, the search continues for the perfect means of automatically dealing with foreign resources.

Footnotes

One of the great things about being wrong on Planet Lisp instead of just my own little echo chamber: somebody is vastly more likely to notice and correct me. Trying to debug the problems that would have arisen from my original approach would have been Not Fun.
It is entirely possible I have incorrectly co-opted the term "barrier". If so, I'd be happy to switch to a more correct term, if one exists.
For instance: You unwrap and pass the value to foreign code, foreign code issues a callback, callback triggers a GC, wrapper goes away taking the wrapped foreign value with it.

CFFI Tricks: Automatic Freeing of C Resources

2011-11-13T02:50:49Z

Lisp finalizers, at least the ones provided by trivial-garbage, are not at first glance particularly useful.

You can't reference the object being finalized—on some lisps, the object is already gone by the time you get to the finalizer—so you can't inspect the constituent parts of an object to operate on them. Closing over the parts only works in the event an object cannot be destructively updated—since most objects in CL can be destructively updated, that would require resetting the finalizer on update. Of course, since objects can have multiple finalizers, and you can't selectively clear them, you might accidentally wipe away someone else's finalizer.

In short, finalizers are only good for unique, constant objects. ~~Like, say, your typical usage of a foreign-pointer.~~

Note (added 2011.11.13): Paul Khuong has pointed out the below is a bad idea akin to trying to stick a finalizer on a fixnum. Whoops!

(defvar *ptr* (gced-foreign-alloc :int))

For this example, we're going to define a wrapper for #'FOREIGN-ALLOC which sets up the finalization of our pointer object. In your code, you'll probably bake the finalization setup into a method on #'TRANSLATE-FROM-FOREIGN which returns a pointer.

(defun gced-foreign-alloc (type &rest rest)
  (let* ((ptr (apply #'foreign-alloc type rest))
         (addr (pointer-address ptr)))
    (tg:finalize ptr
                 (lambda ()
                   (foreign-free (make-pointer addr))))))

Because the object being finalized is not passed to the finalizer function, and because closing over it would result in keeping the pointer around indefinitely, we instead close over just enough information to recreate the pointer: the address.

That's right, we recreate the pointer in order to free it. This feels a little odd, but the restrictions on portable finalizers rather limit our options; so, we cross our fingers and hope for the best.¹

If you were expecting a long entry, I'm sorry to disappoint: that's pretty much all there is to it.

(setf *ptr* nil)
(tg:gc :full t)

There are, however, some caveats to this approach.

Because GC runs cannot be guaranteed to occur at any given time, the pointed-to object may be held far longer than you'd prefer.
Recreating the pointer with something like (make-pointer (pointer-address *ptr*)), or even just having a #'POINTER-EQ pointer returned by a C function call, may result in multiple non-EQ objects pointing to the same memory address and too-early freeing.
If the foreign side moves objects around, the address may no longer be correct once the finalizer runs.

One is only an annoyance, and three is liable to cause you problems elsewhere anyway. That second caveat is a biggie, however, and means this technique is unsafe for use when dealing with arbitrary C code—careful consideration of applicability is necessary. For example, it's useful in burgled-batteries to automatically handle decrementing reference counts on untranslatable objects, but I wouldn't use it to free() memory.

Footnotes

Because finalizers operate on a specific object, so long as (eq (make-pointer x) (make-pointer x)) is NIL, there should be no issues. If it's T, some GC implementations could conceivably be tripped up, but I think that's unlikely.

Aside: Greetings Planet Lisp!

This being my first new post since I was added to Planet Lisp, here are some other, more generally applicable, CFFI Tricks you may have missed:

CFFI Tricks: Instrumenting the Groveller

2011-10-26T16:17:57Z

When interfacing with C, sometimes you want to know something which isn't provided in the shared object or DLL you're using (perhaps because it's tucked away in a C header file instead) and your options are: write a little C to get at that bit of info, or embed that information into your program and hope it doesn't change. While the latter is certainly less complicated—and often good enough—, the former is the "proper" solution, for which CFFI-Grovel provides an excellent start.

But sometimes you want to do something CFFI-Grovel doesn't support. Writing C yourself is an option, of course, but then you lose out on the help CFFI-Grovel provides getting C code to produce Lisp forms. So instead, we'd like to tell CFFI-Grovel how to do things it doesn't currently do.

Enter CFFI-GROVEL::DEFINE-GROVEL-SYNTAX. It's an internal symbol, but given the terrible machinations we've done using CFFI internals already, this should hardly raise an eyebrow.

D-G-S is sort of like DEFUN, except it only applies to forms in grovel files. It takes three hopefully self-explanatory arguments: NAME, LAMBDA-LIST, and BODY. Within the scope of body, it provides all the variables within LAMBDA-LIST, plus an additional variable CFFI-GROVEL::OUT which is an output stream to the C file being created.

Let us suppose we would like to grovel the assorted values of a bitfield. CFFI-Grovel does not currently provide support for this, so we must add it ourselves. (A responsible Lisper would send a patch upstream for this particular example, but don't let that distract you from the ability to define other things which have no reason to go upstream.)

First, let's figure out a syntax. Something like:

(bitfield (name &keyword base-type)
  &body (lisp-name c-name)+)

suits my fancy, and has a nice parallel with the existing CENUM. (In fact, most of the code is stolen from cenum.)

In use, it might look something like this excerpt from burgled-batteries:

(bitfield method-convention-flags
  (:positional-arguments "METH_VARARGS")
  (:keyword-arguments    "METH_KEYWORDS")
  (:mixed-arguments      "METH_KEYWORDS | METH_VARARGS")
  (:no-arguments         "METH_NOARGS")
  ...)

This nicely parallels the CFFI:DEFBITFIELD form which it generates, so it should be relatively easy to pick up the usage from one to the other.

(cffi:defbitfield (method-convention-flags)
  (:positional-arguments #x00000001)
  (:keyword-arguments    #x00000002)
  (:mixed-arguments      #x00000003)
  (:no-arguments         #x00000004)
  ...)

So let's get cracking at making this thing work!

(in-package #:cffi-grovel)

We should probably define our own package, but this is example code and all the symbols are internal to the cffi-grovel package anyway, so we'll cheat.

(define-grovel-syntax bitfield (name &body masks)
  (destructuring-bind (name &key base-type)
      (ensure-list name)

This gets us started. The D-BIND gives us the option of passing (name :base-type some-type) as the name, for bitfields that are unusually large or small. It parallels the same option in CENUM, and DEFBITFIELD.

    (c-section-header out "bitfield" name)

C-SECTION-HEADER produces a comment in the C code. While not strictly necessary, it can help you find your place when trying to debug the generated C.

    (c-export out name)

C-EXPORT produces C code which produces Lisp code which exports the name of the bitfield. (Modulo CFFI-GROVEL::*AUTO-EXPORT*, which defaults to NIL.)

    (c-format out "(cffi:defbitfield (")

C-FORMAT is a wrapper around CL:FORMAT which generates a string from the control-string and format arguments provided, then uses that string to produce an fprintf form in C which writes the string produced by FORMAT to the lisp file being generated. In other words, this makes C code which writes the provided string to our generated lisp file.

    (c-print-symbol out name t)

C-PRINT-SYMBOL, as you might guess, produces C code which writes out a Lisp symbol. You may run into package issues if you're doing something particularly fancy in that department.

    (when base-type
      (c-printf out " ")
      (c-print-symbol out base-type t))
    (c-format out ")")

I think you can figure this out.

    (dolist (mask masks)
      (destructuring-bind (lisp-name c-name)
          mask
        (check-type lisp-name symbol)
        (check-type c-name string)

Nothing fancy here. Just some sanity checking.

        (c-format out "~%  (")
        (c-print-symbol out lisp-name)
        (c-format out " ")
        (c-printf out "#x%08x" c-name)
        (c-format out ")")))

As if functions in CFFI-GROVEL were named by some logical and consistent scheme, C-PRINTF produces C code which calls fprintf. Its arguments are a C printf string, and the names of C arguments to pass to fprintf. Though not strictly necessary, we produce a hexadecimal number and set the width of the output to emphasize the flaginess of the numbers in the resulting lisp file.

    (c-format out ")~%")))

Closing things off, and that's it. A few lines of code, and we can now grovel bitfields.

The functions and macros we've used here are defined, along with some others, in cffi/grovel/grovel.lisp. While there isn't much in the way of docstrings, the code is straightforward, and reading more of it may help you better understand how to define your own grovel forms.

Now go forth and grovel heretofore ungrovelled things!

CFFI Tricks: Mapping C Arguments to Lisp Return Values

2011-10-17T19:11:04Z

Unlike Lisp, C lacks the ability to return multiple values, and so some C functions take arguments which are used to return a value. When dealing with such a C function, we'd really rather convert such things into something more Lispy, like a secondary return value, and not pass an argument to the function at all.

In other words, given the C function:

void function return_arg(int num, void * ret)

We'd like to call it as:

(return-arg num) ; => ret

Clisp's native FFI can do this—they call it an :out argument. But of course, being conscientious of other implementations, we'd prefer a solution using a portability layer like CFFI. Unfortunately, CFFI doesn't have anything like Clisp's :out or :in-out arguments.

That leaves us with three possible options:

Write extra code for FFI functions which return values through the argument list
Make library users write extra code to deal with return values in the argument list
Write code to convince CFFI to handle it for us

Being lazy, and not wanting to write essentially the same boilerplate multiple times, we'd like to avoid 1. Being fans of clean APIs, we'd like to avoid 2. Which leaves us with 3.

Naturally, we'll need a name for our foreign-type wrappers. What Clisp calls :out, I think I'll call RETURN for what I hope is an obvious parallel. What Clisp calls :in-out, I shall call PLACE. (What we'll end up defining won't perform reassignment, so PLACE is not strictly accurate, but it'll work for now.)

(define-foreign-type output-arg (cffi::enhanced-foreign-type)
  ((real-type :initarg :real-type :reader real-type)))

This just defines a base-type, so we can share code between the RETURN and PLACE foreign-types. We give it a real-type slot, which differs from the actual-type slot from CFFI. CFFI's actual-type slot will be set to :pointer, because a :pointer is how you pass data around in C. real-type will cover the user-visible type translation.

(define-foreign-type return (output-arg) ()
  (:documentation "A RETURN foreign-type is like Clisp's :OUT."))
(define-parse-method return (real-type)
  (make-instance 'return :actual-type :pointer :real-type (parse-type real-type)))

(define-foreign-type place (output-arg) ()
  (:documentation "A PLACE foreign-type is like Clisp's :IN-OUT."))
(define-parse-method place (real-type)
  (make-instance 'place :actual-type :pointer :real-type (parse-type real-type)))

And these are our foreign-types.

To use them, we need to produce two things: a CFFI:DEFCFUN form with all the output-args as :pointers, and a DEFUN which wraps a call to that function to pass in appropriate pointers and do type translation on the values coming out.

burgled-batteries has this functionality baked into DEFPYFUN—however, that comes with a lot of other baggage, so here's a more single-minded macro.

(defmacro defcfun* (name return-type args)
  (multiple-value-bind (lisp-name foreign-name options)
      (cffi::parse-name-and-options name)
    (let* ((internal-lisp-name (symbolicate "%" lisp-name))
           (args (mapcar #'normalize-arg args))
           (*values-accumulator* (gensym "ACCUM"))
           (retval (gensym "RETVAL"))
           (lisp-args (mapcar #'first (remove-if-not #'include-in-argument-list-p args :key #'fourth))))
      `(progn
         (defcfun (,foreign-name ,internal-lisp-name ,@options)
             ,return-type
           ,@(mapcar (lambda (arg) (cl:list (first arg) (third arg))) args))
         (defun ,lisp-name ,lisp-args
           (let ((,*values-accumulator* (cl:list))
                 (,retval '#:you-should-never-see-this-value))
             ,(loop :for (value var actual-type parsed-type) :in (cons '(nil nil nil nil) (reverse args))
                    :for body = `(setf ,retval (,internal-lisp-name ,@(mapcar #'%choose-symbol args)))
                    :then (expand-to-foreign-dyn value var (cl:list body) parsed-type)
                    :finally (cl:return body))
             (push ,retval ,*values-accumulator*)
             (values-list ,*values-accumulator*)))))))

To produce an appropriate DEFCFUN, we start with the lisp-name and add a "%" as is the usual convention for internal helper bits. A gensym would work as well, but this provides something callable for debugging purposes or what-have-you. The foreign-type of any output-arg argument is simply :pointer. We'll deal with type translation of those arguments in our wrapper.

The DEFUN is where the real fun happens: we have an accumulator, which accumulates the return values from both the C function, and the output arguments. (Alas, no consideration is made for :void functions—for them, the first return value will always be useless.)

That loop you see calls #'EXPAND-TO-FOREIGN-DYN, which is the most general manner in which CFFI handles the conversion of arguments (as contrasted with #'EXPAND-TO-FOREIGN and #'TRANSLATE-TO-FOREIGN). We'll get to the implementation of E-T-F-D in a bit. The loop might be a bit tricky—it's basically flattened out recursion—but I'm sure you can figure it out. :)

(defun normalize-arg (arg)
  "Returns (list var gensym actual-type parsed-type)"
  (cond
    ((eq arg '&rest) arg)
    ((listp arg)
     (destructuring-bind (var type) arg
       (let ((parsed-type (parse-type type)))
         (cond
           ((typep parsed-type 'output-arg)
            (list var (gensym (symbol-name var)) :pointer parsed-type))
           (t
            (list var (gensym (symbol-name var)) type parsed-type))))))
    (t (error "oh noez!"))))

NORMALIZE-ARG is not particularly interesting. Just calculates some necessary gensyms and parses types for us.

The remaining magic is in E-T-F-D. Except it's not really magical, it's pretty straightforward.

(defmethod expand-to-foreign-dyn (value var body (type output-arg))
  (let* ((real-type (real-type type))
         (canonical-type (cffi::canonicalize real-type)))
    `(with-foreign-object (,var :pointer)
       ,@body
       (push ,(expand-from-foreign `(mem-ref ,var ,canonical-type) real-type) ,*values-accumulator*))))

We make a pointer, run some code, and then add a translated copy of the value being pointed at to the list of values to be returned.

(defmethod expand-to-foreign-dyn (value var body (type place))
  (let* ((real-type (real-type type))
         (canonical-type (cffi::canonicalize real-type)))
    (expand-to-foreign-dyn
     value var
     `(,(call-next-method var value `((setf (mem-ref ,value ,canonical-type) ,var) ,@body) type))
     real-type)))

PLACE types are only slightly more interesting. First, we convert the Lisp value given into a foreign value. Then, it's pretty much the same as for RETURN values, with the exception of pointing our pointer at the foreign value we just made.

Also note that we end up swapping VALUE and VAR for the pointer. This is because the gensymed symbol is already in use for the converted foreign value. (A second gensym may have been preferable, but this should be safe.)

%CHOOSE-SYMBOL takes this into account, for when we generate the call to CFFI's C function wrapper.

(defun %choose-symbol (arg)
  (typecase (fourth arg)
    (return (second arg))
    (place (first arg))
    (t (second arg))))

And that's pretty much it. (If I left anything out, you can find it in cffi-output-args.lisp from burgled-batteries.) After that

(defcfun* "foo" :void ((a :int) (b (return :boolean)) (c (place :boolean))))

Gives us a function #'FOO, with arguments A and C, which returns (values nil B C).

An arguably better implementation would either have it return (values B C), or have it return (values B) and modify C (the latter necessitating a macro, of course). I'll leave those as exercises for the reader.

CFFI Tricks: Creating a Lisp Version of a Missing C Function

2011-10-13T14:17:05Z

Working on burgled-batteries, I found that a non-trivial portion of Python's C API isn't actually exported functions, but preprocessor macros disguised as functions. In the interests of continuing to work if they go back and forth on any given function, I thought it would be a good idea to automatically detect if the function exists, and use an alternative Lisp implementation if it didn't.

Now, the thing about replacing C code with Lisp code is that I want the Lisp code to run as if it were C. In other words, for all of the type-translation of CFFI to have already occurred by the time the Lisp code has been called, and to also occur on the return value. CFFI, naturally, does not anticipate this use-case. Fortunately, Lisp is not a bondage-and-discipline language; also fortunately, CFFI's internals are perfectly aligned for this particular, ah, "creative restructuring".

(let* ((realfn (symbol-function 'cffi::defcfun-helper-forms))
       (lisp-args ...)
       (alt-body ...)
       (ourfn (make-defpyfun-helper-forms lisp-args alt-body decl))
       (doc+decl ...))
  ;; We temporarily replace cffi::defcfun-helper-forms and call
  ;; cffi::%defcfun to create the defun so we can ensure the body
  ;; of our alternate function runs having been given pointers as
  ;; if it were C.
  (unwind-protect
       (progn
         (setf (symbol-function 'cffi::defcfun-helper-forms) ourfn)
         (cffi::%defcfun lisp-name c-name return-type c-args nil doc+decl))
    (setf (symbol-function 'cffi::defcfun-helper-forms) realfn)))

If you're not familiar with CFFI's internals, #'cffi::%defcfun is the function which produces the macroexpansion for DEFCFUN. Quite handy that it was already split out for us to call. #'cffi::defcfun-helper-forms is the function #'%defcfun uses to determine what form to use to call a C function. Fate smiles upon us: #'%defcfun produces the type translation code, and wraps it around the form returned by #'defcfun-helper-forms. Since we want to run code in place of calling a C function, all we have to do is get D-H-F to spit out the code we want to run.

So we replace D-H-F with a call to our own function (produced by #'make-defpyfun-helper-forms) and let #'%defcfun do its thing none the wiser.

#'make-defpyfun-helper-forms isn't particularly complicated. It binds the gensym'd variables from CFFI back into the user-provided variables, and includes any relevant user-specified declarations.

(defun make-defpyfun-helper-forms (wrapped-args wrapped-forms declarations)
  "When called, produces a function which is intended to take the place of
#'CFFI::DEFCFUN-HELPER-FORMS within a CFFI:DEFCFUN expansion.  This is used when
creating an alternate definition for a C function which either doesn't exist in
the currently-included Python library (e.g., because it belongs to a newer
version of Python, or because it's really a preprocessor macro)."
  (lambda (name lisp-name rettype args types options)
    (declare (ignore name lisp-name rettype types options))
    (values '()
            `(let ,(mapcar #'cl:list wrapped-args args)
               ,@(%filter-declarations declarations :for wrapped-args)
               ,@wrapped-forms))))

It's somewhat awkward to write C-equivalent code within Lisp; largely because it's necessary to keep in mind that the return value must also be a C value (e.g., 1 and 0 for true and false, rather than t and nil). But it also means library users have less to worry about, because some of the minor differences between Python versions can be papered over.

Lots of scary internal machinations allow us to provide a (hopefully) straightforward interface to create Lisp-side reimplementations of C code:

(defcvar ("Py_Py3kWarningFlag" *err.warn-py3k* :read-only T) :boolean)
(defpyfun "PyErr_WarnPy3k" 0-on-success ((message :string) (stacklevel :int))
  (:implementation
   (if *err.warn-py3k* (err.warn-ex +exc.deprecation-warning+ message stacklevel) 0)))
...
(defpyfun "PyErr_NewExceptionWithDoc" object ((name :string) (doc :string) (base object) (dict dict))
  (:implementation
   (declare (ignore doc))
   (err.new-exception* name base dict)))

Preview: Burgled-Batteries, Yet Another Python-Lisp Bridge

2011-09-27T09:52:06Z

(push :burgled-batteries.guess-not-grovel *features*) ; optional; avoids the groveller (asdf:load-system "burgled-batteries") (in-package #:burgled-batteries) (run "1+1") ; ⇒ 2 (run "[1, 2, 3]") ; ⇒ #(1 2 3) (run "(1, 2, 3)") ; ⇒ (1 2 3) (run "dict(a=1, b=2, c=3)") ; ⇒ #<HASH-TABLE> (alexandria:hash-table-alist *) ; ⇒ (("b" . 2) ("c" . 3) ("a" . 1)) (import "feedparser") (defpyfun "feedparser.parse" (url)) (documentation 'feedparser.parse 'function) ; ⇒ "Parse a feed from a URL, file, stream, or string" (feedparser.parse "http://pinterface.livejournal.com/data/atom") ; ⇒ #<HASH-TABLE> (gethash "title" (first (gethash "entries" *))) ; ⇒ "Preview: Burgled-Batteries, Yet Another Python-Lisp Bridge"

To answer the most obvious question: CL-Python seems to require more resources than I can give it, Python-on-Lisp is incomplete and poorly integrated (also: I'll probably steal code from it), and pyffi is ... what I started from. If you count Python-on-Lisp-ex, this is the fifth Lisp/Python mashup I know about. Hopefully, it will be the last. ;)

Edit Oct. 11, 2011: Lots more to do yet, but if anybody is interested in throwing things at it, `darcs get http://repo.kepibu.org/burgled-batteries/`. Feel free to e-mail me directly, or ping the mailing list.

Parsing robots.txt Files with Common Lisp

2011-08-31T07:51:50Z

Do you use Common Lisp? Do you scrape web pages, and need to deal with the robots.txt file to play nicely?

Well have I got a deal for you! For the low, low price of absolutely free you too can parse robots.txt files with my Common Lisp library machina-policy¹. See the readme for a brief overview.

Don't forget to join the mailing list to offer ideas, submit bugs, and make requests for improvement.

For more web-scraping fun, you might try some libraries with which I am completely unaffiliated and haven't yet tried: cl-web-crawler, cl-kappa, and css-selectors; or my own Oh, Ducks!.

machina-policy was very briefly named robots.txt. How not to name a library, ladies and gentlemen.

kepibu.org: Now with Project Mailing Lists!

2011-08-21T02:46:32Z

As you may be aware, because it's primarily what I post about, I host a small number of libraries on my personal site. In the unlikely event you are interested in any of those libraries, most of them now have mailing lists.

Well, in theory. I'm pretty new at administering mailman, and it's entirely possible I missed something or other.

I hate mailing lists, actually—netnews is a much better interface—but an NNTP server is a bit harder to set up, so that's on the "FUTURE" list for now.

I expect the combined traffic of all the lists to be pretty low: over the past several years, I have received two e-mails total about the small number of libraries I maintain.

Anyway, now you can publicly berate me and my code, amongst an audience of similarly-interested parties. No longer must you withhold your lamentations merely because nobody else could applaud you!

Compliments and questions are okay, too. ;)

Finding the Cause: Hunting Down a Spike in IO Usage

2011-08-11T05:24:11Z

A little over a week ago, my VPS started a significant jump in IO usage. This had cascading effects, spiking context switches and interrupts, load average, and a very unwelcome pinching of memory availability. (I noticed these other things first, actually—it wasn't until I hit the IO graph that I figured out how I could track down the problem.)

A little iotop and some luck on timing, and I discovered the culprit: htcacheclean. In other words, the periodic drop in performance was caused by mod_disk_cache, something I implemented to improve performance. Vexing!

So I did some investigating into what exactly my caching solution was doing.

cd /var/cache/apache2/mod_disk_cache/

I started by seeing how many files it was caching.

find . -name '*.data' -print | wc -l

Over 5000 files. By very rough count, I have ~170 pages, which I expect to be cached four times each (ASCII or UTF-8, compressed or not-compressed). That's a little over 7 times as many files as I'd expect. Clearly, something is amiss.

First, I suspected the problem was duplication. Maybe the same things are getting cached multiple times.

find . -name '*.data' -exec sha1sum {} + | \
  awk '{print $1}' | sort | uniq -c | awk '{print $1}' | \
  sort | uniq -c | sort

Provides me with a listing of how many pages share the same sha1sum, sorted by duplicates. 4310 files have unique sha1s, 30 sha1s are shared across five or more files. So duplication is clearly not worth worrying about right now.

Next, I checked the pages being cached.

find . -name '*.header' -exec \
 grep -hosaE '([a-z0-9.-]+:80/[^?]*)' {} + | \
 sort | uniq -c | sort

Most pages are cached once. Several dozen are cached up to 37 times. That's excessive, but here's the real shocker: two pages are cached several thousand times each (one 1300 times, another 3600).

What's so special about these pages? Query arguments. Search boxes. In other words, I probably shouldn't be caching them anyway, because they're only going to be seen once.

So I stopped.

I then turned my eye to another site on the same server, that happens to implement its own caching layer, and turned off mod_disk_cache for that site. Testing before and after revealed: mod_disk_cache made no statistical difference in speed for that site. So it can stay disabled.

I also disabled it for some smaller, very low-traffic sites I host, because it was essentially irrelevant for those sites. Leaving only one site still using mod_disk_cache.

One site remaining, I started checking if that site really needed it. Inside the application's internal caching layer, I noticed a comment mentioning that mod_cache fails to cache URLs ending in "/" (in other words, anywhere DirectoryIndex applies). Man, I had forgotten about that! A little URL checking confirms: yup, not caching those pages. Those pages, as it happen, are roughly every page I care about being really fast, which means mod_cache isn't caching anything I value. Turning it off and testing again reveals... well, it isn't slowing anything down by being in the middle, but it also isn't making things faster.

So no more mod_cache. For me at least, it just isn't worthwhile. And that means no more htcacheclean, which means my I/O can return to its nice, normal, near-zero level. As can everything else affected by htcacheclean's periodic runs.

That's one fewer thing running on my server. And a lesson learned: if you've already noted something doesn't help performance, disable the things that don't help so they don't later make it worse. Whoops!

The Opinions Expressed Herein are My Own

Upgrading the Linux Kernel when Using a USB Boot Disk

A Public Service Announcement About rm and tildes

Live from the DMV! (This Program was Recorded Earlier)

Getting a Job, a Retrospective

The Reunion

Introducing clavatar

Video of Burgled-Batteries Talk is Available

burgled-batteries: The Talk (July 10th!)

burgled-batteries: The Talk (Coming Soon?)

Firefox has Gone Rogue

The Many and Horrifying UI Sins of Firefox

Frikkin' Scavengers

Passwordless Drive Encryption for Virtual Machines on Linux

Set up Drive Encryption

Configuring the VM

A Note about Fibre Channel

Footnotes

DLNA and Netflix: A Review

DLNA

PS3 Media Server

MediaTomb

Serviio

Netflix Streaming

Summary

Accept-Charset is Still Necessary

Accidental Load Testing

Nashville, 2010

Table of Contents

Strategies for Automatic Release of Foreign Resources

Release by GC

Release by Barrier

Conclusions

Footnotes

CFFI Tricks: Automatic Freeing of C Resources

Footnotes

Aside: Greetings Planet Lisp!

CFFI Tricks: Instrumenting the Groveller

CFFI Tricks: Mapping C Arguments to Lisp Return Values

CFFI Tricks: Creating a Lisp Version of a Missing C Function

Preview: Burgled-Batteries, Yet Another Python-Lisp Bridge

Parsing robots.txt Files with Common Lisp

kepibu.org: Now with Project Mailing Lists!

Finding the Cause: Hunting Down a Spike in IO Usage