Wednesday, February 21, 2007

Living the sysadmin dream

There's a legendary threat of sysadmins: "Go away, or I will replace you with a very small shell script".

Well, today I got a taste of that.

At work, we've got a new web site almost ready to go. The content is stored the plainest-possible XHTML. That makes it somewhat future-proof. We process that through Kid templates to make it look snazzy.

We wanted to reuse a lot of the content from our existing site, which is a mix of HTML and PHP with some abuse of divs and other incidental wackiness. We knew the error of our ways, and we wanted to get pure, so we hired a contractor to convert our site over. He knew Python. He seemed to have a good grasp of advanced web site architectures.

Boy, were we ever wrong.

Originally expected to take one month, the job ended up taking three. In the process, I had to write a script to do automated testing of the guy's work, because he was screwing up on the mechanical stuff. Problems that can be solved with search and replace, he was choosing to solve by hand, and getting wrong. It was insulting getting work with the same mechanical errors time and time again.

What I had, perhaps naively, expected was that he would write a conversion script, eyeball the results, and fix up any problems.

There were a few files we'd asked him not to convert, but we eventually decided to convert some of them after all. The task fell to me, and it was a real joy to set to it. After all that time spent poring over his work, I'd developed a very clear idea of what I thought he should have done in the first place.

Phase 1: Text Substitution
In this phase, we convert PHP to HTML. In the general case, you can't do this without implementing PHP. But our existing site uses PHP only as a templating system, so there are only a couple of stock phrases to worry about. re.sub handles this nicely.

Another thing we do is remove any ® symbols we encounter, because our house style has changed, and we no longer want to affix this to every mention of our brand.

Phase 2: Tidy HTML
This phase consists of shoving all our files through Tidy, so that they become well-formed, almost-valid HTML. It's wonderful to have a utility like Tidy, because programs can't feel pain.

Phase 3: Ending div abuse

Our existing content uses div classes for headings all over the place. To tackle these problems, we should really operate in the XML domain, so that we avoid parsing mistakes. Now that tidy has made the documents well-formed, we can parse them with ElementTree's HTML parser. Then we process it through a Kid Template with a series of match rules. These turn divs into h2s, and so on. We can also clean up our document title (which has an inexplicable trailing non-breaking spaces)

Phase 4: Whitespace cleanup
The kid processing has fixed the semantics of the document and exported it as XHTML, but we're left with swaths of whitespace and line breaks in unreasonable places. Another run through Tidy fixes that.

Automation is wunnerful
Not only did I manage to get a basic conversion routine in place in about three hours, I actually stayed at long past my notional end-of-shift, because I was enjoying what I was doing. Writing a script to convert files to XHTML is fun. Doing the conversion yourself is not. That's a problem our contractor faced. Not only was he slow when he was actually working, but he had trouble getting motivated to work, so he didn't put in as much time as we wanted him to.

They say that good programmers are 10x more productive than average programmers, and truly excellent programmers are 10x more productive again. I suspect automation is one of the factors in that difference. Unless you stop them, an excellent programmer will find a way to eliminate the boring bits. Even if automating the job takes just as long as doing it by hand, a programmer will be happier and more motivated because the work is much more stimulating.

So today, I replaced a contractor with a reasonably small Python script. I wish I'd done it months ago.


Anonymous said...

This is an affair of too much magnitude to be fully described in the small space yet remaining in which to bring out our veracious history.. The fates were again propitious for a brief period, but again a trivial incident interfered.. a thought reinforced by a transference and distorted in expression by the censor.. You are quite welcome, however, to make use of the furnishings in the small room over the kitchen.. During the usual Sunday-evening singing Champe laughed till Aunt Missouri threatened to send her to bed.. I beg your pardon, sir, said Mr.. Take it, honey, said Uncle Mose.. I did the agreeable to Mrs.. For a time the Major showed an inclination to discourage the advances of the play actor, as he privately termed him; but soon the young man's agreeable manner and indubitable appreciation of the old gentleman's stories completely won him over.. Pork-chop, eh? Is that the best you can think of? I know what you mean, though.. Just come right on up and see.. Podington began to totter.. The male genital symbolized by persons and the female by a landscape.. Nor will it seem a matter of chance that in the course of interpretation one always happens upon subjects of which one does not like to speak or think.. Men may not remember much of what they are told about children; but any man knows the difference between an aunt and a mother-in-law. Titbottom to dine? Titbottom smiled gently, and answered: He might have brought his spectacles with him, and I have been a happier man for it.. It was as excruciatingly funny as it had ever been, when his boat nosed its way into a great flock of ducks idling upon the water, to see the mad paddling haste of those nearest him, the reproachful turn of their heads, or, if he came too near, their spattering run out of water, feet and wings pumping together as they rose from the surface, looking for all the world like fat little women, scurrying with clutched skirts across city streets.. I couldn't make a call any more than a cow.. Afterwards the children are permitted to indulge themselves in rational play, for I permit no amusement that is not also instructive.. He then opened a cupboard and brought out a bottle of whiskey and two glasses...

Anonymous said...

black mold exposureblack mold symptoms of exposurewrought iron garden gatesiron garden gates find them herefine thin hair hairstylessearch hair styles for fine thin hairnight vision binocularsbuy night vision binocularslipitor reactionslipitor allergic reactionsluxury beach resort in the philippines

afordable beach resorts in the philippineshomeopathy for big with great mineral makeup bargainsmineral makeup wholesalersprodam iphone Apple prodam iphone prahacect iphone manualmanual for P 168 iphonefero 52 binocularsnight vision Fero 52 binocularsThe best night vision binoculars here

night vision binoculars bargainsfree photo albums computer programsfree software to make photo albumsfree tax formsprintable tax forms for free craftmatic air bedcraftmatic air bed adjustable info hereboyd air bedboyd night air bed lowest pricefind air beds in wisconsinbest air beds in wisconsincloud air beds

best cloud inflatable air bedssealy air beds portableportables air bedsrv luggage racksaluminum made rv luggage racksair bed raisedbest form raised air bedsaircraft support equipmentsbest support equipments for aircraftsbed air informercialsbest informercials bed airmattress sized air beds

bestair bed mattress antique doorknobsantique doorknob identification tipsdvd player troubleshootingtroubleshooting with the dvd playerflat panel television lcd vs plasmaflat panel lcd television versus plasma pic the bestThe causes of economic recessionwhat are the causes of economic recessionadjustable bed air foam The best bed air foam

hoof prints antique equestrian printsantique hoof prints equestrian printsBuy air bedadjustablebuy the best adjustable air bedsair beds canadian storesCanadian stores for air beds

migraine causemigraine treatments floridaflorida headache clinicdrying dessicantair drying dessicantdessicant air dryerpediatric asthmaasthma specialistasthma children specialistcarpet cleaning dallas txcarpet cleaners dallascarpet cleaning dallas

vero beach vacationvero beach vacationsbeach vacation homes veroms beach vacationsms beach vacationms beach condosmaui beach vacationmaui beach vacationsmaui beach clubbeach vacationsyour beach vacationscheap beach vacations