Wednesday, February 21, 2007

Living the sysadmin dream

There's a legendary threat of sysadmins: "Go away, or I will replace you with a very small shell script".

Well, today I got a taste of that.

At work, we've got a new web site almost ready to go. The content is stored the plainest-possible XHTML. That makes it somewhat future-proof. We process that through Kid templates to make it look snazzy.

We wanted to reuse a lot of the content from our existing site, which is a mix of HTML and PHP with some abuse of divs and other incidental wackiness. We knew the error of our ways, and we wanted to get pure, so we hired a contractor to convert our site over. He knew Python. He seemed to have a good grasp of advanced web site architectures.

Boy, were we ever wrong.

Originally expected to take one month, the job ended up taking three. In the process, I had to write a script to do automated testing of the guy's work, because he was screwing up on the mechanical stuff. Problems that can be solved with search and replace, he was choosing to solve by hand, and getting wrong. It was insulting getting work with the same mechanical errors time and time again.

What I had, perhaps naively, expected was that he would write a conversion script, eyeball the results, and fix up any problems.

There were a few files we'd asked him not to convert, but we eventually decided to convert some of them after all. The task fell to me, and it was a real joy to set to it. After all that time spent poring over his work, I'd developed a very clear idea of what I thought he should have done in the first place.

Phase 1: Text Substitution
In this phase, we convert PHP to HTML. In the general case, you can't do this without implementing PHP. But our existing site uses PHP only as a templating system, so there are only a couple of stock phrases to worry about. re.sub handles this nicely.

Another thing we do is remove any ® symbols we encounter, because our house style has changed, and we no longer want to affix this to every mention of our brand.


Phase 2: Tidy HTML
This phase consists of shoving all our files through Tidy, so that they become well-formed, almost-valid HTML. It's wonderful to have a utility like Tidy, because programs can't feel pain.

Phase 3: Ending div abuse

Our existing content uses div classes for headings all over the place. To tackle these problems, we should really operate in the XML domain, so that we avoid parsing mistakes. Now that tidy has made the documents well-formed, we can parse them with ElementTree's HTML parser. Then we process it through a Kid Template with a series of match rules. These turn divs into h2s, and so on. We can also clean up our document title (which has an inexplicable trailing non-breaking spaces)

Phase 4: Whitespace cleanup
The kid processing has fixed the semantics of the document and exported it as XHTML, but we're left with swaths of whitespace and line breaks in unreasonable places. Another run through Tidy fixes that.

Automation is wunnerful
Not only did I manage to get a basic conversion routine in place in about three hours, I actually stayed at long past my notional end-of-shift, because I was enjoying what I was doing. Writing a script to convert files to XHTML is fun. Doing the conversion yourself is not. That's a problem our contractor faced. Not only was he slow when he was actually working, but he had trouble getting motivated to work, so he didn't put in as much time as we wanted him to.

They say that good programmers are 10x more productive than average programmers, and truly excellent programmers are 10x more productive again. I suspect automation is one of the factors in that difference. Unless you stop them, an excellent programmer will find a way to eliminate the boring bits. Even if automating the job takes just as long as doing it by hand, a programmer will be happier and more motivated because the work is much more stimulating.

So today, I replaced a contractor with a reasonably small Python script. I wish I'd done it months ago.

1 comment:

Anonymous said...

This is an affair of http://startso11.info/excel+2000-pobierz.html too much magnitude to be fully described in the small space yet remaining in which to bring out our veracious history.. The fates were again propitious for a brief period, but again a http://startso11.info/anczelotti.html trivial incident interfered.. a thought http://wowyesaf.info/system+podw%C3%B3jnej+wi%C4%99kszo%C5%9Bci+polega+na.html reinforced by a transference and distorted in expression by the censor.. You are quite welcome, however, to make use of the furnishings in the http://startso11.info/filary.html small room over the kitchen.. During the usual Sunday-evening singing Champe laughed till Aunt Missouri threatened to send her to http://startso11.info/studia+filmowe.html bed.. I beg http://startso11.info/szopka+narz%C4%99dziowa.html your pardon, sir, said Mr.. Take it, honey, said Uncle http://storyah44.info/pogl%C4%85dy+Wincentego+Okoia.html Mose.. I did the agreeable to http://startso11.info/czerwone+legginsy.html Mrs.. For a time the Major showed an inclination to discourage the advances of the play http://startso11.info/nik+polsko-wegierski.html actor, as he privately termed him; but soon the young man's agreeable manner and indubitable appreciation of the old gentleman's stories completely won him over.. Pork-chop, eh? Is that the best you can think of? I know what http://startso11.info/myka1.html you mean, though.. Just come right http://startso11.info/pkp+bielsko.html on up and see.. Podington began to http://startso11.info/PODSTAWOWE+WZORY+CA%C5%81KOWANIA.html totter.. The male genital symbolized by persons and the female by a http://startso11.info/www.3maja37A.pl.html landscape.. Nor will it seem a matter of chance http://startso11.info/nowe+komunikatory.html that in the course of interpretation one always happens upon subjects of which one does not like to speak or think.. Men may not remember much of what they are told about children; but any man knows the difference between an aunt and a mother-in-law. http://startso11.info/d-1.html. Titbottom to dine? Titbottom smiled gently, and answered: He might have http://startso11.info/amatorskie+holly+dolly.html brought his spectacles with him, and I have been a happier man for it.. It was as excruciatingly funny as it had ever been, when his boat nosed its way into a great flock of ducks idling upon the water, to see the mad paddling haste of those nearest him, the reproachful turn of their heads, or, if he came too near, their spattering run out of water, feet and wings pumping together as they rose from the surface, looking http://startso11.info/sync+master+710.html for all the world like fat little women, scurrying with clutched skirts across city streets.. I couldn't make a call any http://startso11.info/www.motosmok.pl%2Fquadysportowe.html more than a cow.. Afterwards the children are permitted to indulge themselves in rational play, for I permit no http://startso11.info/wwwfoto.plus.pl.html amusement that is not also instructive.. He then opened a cupboard and http://startso11.info/spellforce+poradnik.html brought out a bottle of whiskey and two glasses...