tag:blogger.com,1999:blog-173566192024-02-08T11:34:20.450-05:00Blog me BentleyAaron Bentleyhttp://www.blogger.com/profile/05272759295389762492noreply@blogger.comBlogger15125tag:blogger.com,1999:blog-17356619.post-62093217091952086392012-01-27T15:02:00.002-05:002012-01-27T15:15:20.519-05:00TDD as path to slothOne of the lesser-sung attributes of Test-Driven Development is that it can actually save you development effort.<br /><br />In TDD, you start by creating a failing test. Then you write code to make it pass. Sometimes the test will not fail. Perhaps you're written the test incorrectly, but other times, the functionality you planned to implement is already implemented elsewhere (e.g. in the framework / library you're using). That means you can skip implementing useless code. Remember, functionality is an asset, but code is a liability.<br /><br />This happened to me yesterday, when I was refactoring some <a href="https://code.launchpad.net/%7Eabentley/launchpad/data-download-view/+merge/90269">Launchpad code</a>. Turned out Zope sets the Content-length http header if you don't. It could have happened today, because Zope also encodes Unicode strings to utf-8 for you, but I was unit-testing at a different level, so I implemented, then removed, manual utf-8 encoding.Aaron Bentleyhttp://www.blogger.com/profile/12107238330872108981noreply@blogger.com0tag:blogger.com,1999:blog-17356619.post-76317294683120207132011-06-01T16:39:00.003-04:002011-06-01T16:58:53.659-04:00Guessing relevant test modules with Fault LineLaunchpad's test suite takes a long time to run. Far too long to wait for. And the likelihood that you've broken any given test is pretty small. So you want to test a small subset, at least at first.<br /><br />You can usually guess which tests to run; if you've changed "archive.py", you should probably run "test_archive". But some connections are easier to miss, so I've hacked up a <a href="http://launchpad.net/fault-line">Fault Line</a>, a bzr plugin that uses past changes to guess which test files correlate to the files you've changed. You can run it like so:<br /><span style="font-family:monospace;"><br /></span><span style="font-family:courier new;">bin/test -m $(bzr fault-line --module-regex)<br /><br /></span>This will look at all the files you've changed, look at their recent history, and see which files tended to change in the last 100 revisions where you changed the specified files. The --module-regex option causes it to output a regular expression, assuming that the test files are modules. Otherwise, it would just output a list of the test files it found.<span style="font-family:courier new;"><br /></span><br />Thanks to Jelmer Vernoij, this is even easier to achieve for testing bzr. You just need to can run "<span style="font-family:courier new;">bzr selftest --auto</span>".<span style="font-family:courier new;"><br /><br /></span>Yes, Fault Line is a hacky, limited tool. But they have their place. Who knows, maybe it will become more general.<span style="font-family:courier new;"><br /></span>Aaron Bentleyhttp://www.blogger.com/profile/12107238330872108981noreply@blogger.com0tag:blogger.com,1999:blog-17356619.post-13623949269352105282010-12-20T20:19:00.009-05:002010-12-20T23:57:40.474-05:00Early impressions of the Viewsonic G tablet<span style="font-weight: bold;font-size:130%;" >I got a tablet</span><br />I used to dream about pervasive Internet. I used to dream about using a tablet to read web sites on a streetcar. That future is here, but it's in beta.<br /><br />More recently, a colleague of mine got an <a href="http://www.apple.com/ipad/">iPad</a>. I think the coolest thing about it is its long battery life. I sometimes take long trips, and I need my entertainment. I deal with jet-lag by staying awake until it's night at my destination, so I'm often too dazed for reading. Video is crucial. On long flights, I bring a laptop, an extra battery and a netbook. Cutting back to a tablet and a laptop would be nice.<br /><br />I know <a href="http://www.android.com/">Android</a>'s not really ready for tablets-- in six months, 3.0 devices should be plentiful and wonderful. But I've got travel coming up, and I lost my patience. So I got a <a href="http://www.viewsonic.com/gtablet/">Viewsonic G tablet</a>. The premier Android tablet is <a href="http://galaxytab.samsungmobile.com/">Samsung's Galaxy Tab</a>, but its screen is 7 inches, and I think that's too small for enjoying video. I considered the <a href="http://www.archos.com/products/ta/archos_101it/index.html">Archos 101</a>, but it was hard to find. I didn't find the Advent Vega until after I'd ordered.<br /><br />So what's the G tablet like? Is it worth it?<br /><br /><span style="font-weight: bold;font-size:130%;" >Viewing Angle</span><br />If you use a landscape orientation, slight changes in viewing angle affect the brightness a lot, but it is easy to find angles that look good. If you use a portrait orientation, each of your eyes has a different viewing angle. The screen has a simultaneously bright-dark look that may be familiar if you've used red-blue 3D glasses or polarized glasses. This makes the tablet an inferior e-book reader, but a decent video player.<br /><br /><span style="font-weight: bold;font-size:130%;" >Rectangular pixels</span><br />The G tablet's 1024x600 screen has slightly rectangular pixels; they are roughly 87:84. This means that the launcher icons, and presumably other raster art, are slightly distorted. It's fairly subtle, though, and I didn't notice for a while. It probably won't affect your enjoyment of the device.<br /><br /><span style="font-weight: bold;font-size:130%;" >Size and weight</span><br />At 1.8 lbs, this is an object that you don't want to hold in one hand. You can do it if you hold the top corner in your fingers and rest the opposite corner against your inner elbow, but it feels like a stunt. It's fine to hold two handed, or reclining and resting it on your body, or lying in bed.<br /><br /><span style="font-size:130%;"><span style="font-weight: bold;">Processor and chipset</span></span><br />Performance is very good in most apps, like Angry Birds or video playback. Bonsai Blast is unusably slow. The G Tablet is based on Nvidia's Tegra 2 chipset, which is rumoured to be the reference platform for Android 3.0, so things can only improve as more software is optimized for Tegra 2.<br /><br /><span style="font-weight: bold;font-size:130%;" >Software</span><br />Before buying the G tablet, I knew that the software it ships with was unusable, so I bought it knowing I would have to install a hacked firmware from xda-developers. That's getting routine. I've also installed <a href="http://openwrt.org/">OpenWRT</a> on a router and <a href="http://ubuntu.com/">Ubuntu</a> on my new laptop in the past couple of weeks.<br /><br />I've installed the<a href="http://http//forum.xda-developers.com/showthread.php?t=842004"> TnT lite</a> firmware and it works. The Android Market works, and I've got Flash installed. Most things either work or aren't available on the Market. "On Air", a program I sometimes use to wirelessly transfer files, doesn't work. "Tape Machine", which I've used for rehearsal recordings, seems confused about whether I've purchased it or not.<br /><br />I get a lot of "Force close" messages, though. I think some of it is because it's easy to accidentally hard-reset the tablet, leaving broken data files. The problem is that sometimes the "shutdown menu" takes too long to show up, and if you hold the power button down long enough, it does a hard shutdown (just like a modern desktop or laptop).<br /><br />Occasionally, the sound stops functioning correctly, and just makes white noise. This is unpleasant, and can't be controlled with the volume control.<br /><br />One point of confusion with the stock and TnT lite firmwares is that the internal storage location is labelled "sdcard". This made me think it was the (micro-)SD card.<br /><br />I wish the status bar wasn't on the top; in a 16:9 device, every pixel of vertical real estate is precious.<br /><br /><span style="font-weight: bold;font-size:130%;" >Ports, etc.</span><br />The tablet has two usb ports; one for accessing its storage like a USB stick, and one so that <span style="font-weight: bold;">it</span> can access USB devices like USB sticks. It's a shame that neither port can be used for charging, like my Nexus One. Instead, it has a 12V/2A "wall-wart" adapter. I've managed to get it to work with USB sticks, but neither of my USB hard disks works. Oh, and it takes mini-USB instead of micro-USB, like my Nexus One, so that's one more cable to take when I travel.<br /><br />It would be nice to have an HDMI port, like the Archos 101 or the Advent Vega, but I can't really see a need. I can use my HTPC or laptop for that. I do wish it took a line-level input, so I could use it for live recording. <span style="font-style: italic;">Hmm, would it make a good control surface for mixing?</span><br /><br />It has a micro-SD slot, but I kinda wish it took regular SD. My laptop and camera both use regular SD, so it would be easier to swap in. That said, I can use my laptop to access the generous internal storage like a USB stick, and I guess I can use an SD reader in the USB port if I need to transfer photos directly from my camera. (Or plug my camera in as a USB device?)<br /><br /><span style="font-weight: bold;font-size:130%;" >Battery life</span><br />So far, I've only gotten it down to 58% battery life in two days of occasional use. This is pretty good in my opinion.<br /><br /><span style="font-size:130%;"><span style="font-weight: bold;">Touchscreen</span></span><br />The touch screen seems a little bit flaky when I'm playing Angry Birds. (When I tap to make the blue bird split into multiple birds, it often doesn't work.) I'm not sure if this is a general problem.<br /><br /><span style="font-weight: bold;font-size:130%;" >Hard Keys</span><br />Unlike my Nexus One, the hard keys aren't backlit. This means they're hard to use in the dark, but the status bar provides soft keys you can use. Well, mostly. The soft "Home" key won't launch the task switcher if you long-press it. The real problem is that without backlighting, it's easy to hit the hard keys accidentally.<br /><br /><span style="font-size:130%;"><span style="font-weight: bold;">Conclusion</span></span><br />Wait six months or so for a proper Android tablet if you can. If you need a tablet today, consider an iPad or Galaxy Tab. You should only get a G tablet if you can stomach poor performance as portrait display, and put up with immature hacked firmwares. The software situation is improving, but the screen may be a deal-breaker if you want to use it as a portrait device.Anonymousnoreply@blogger.com0tag:blogger.com,1999:blog-17356619.post-33945147308138010482009-03-31T19:56:00.002-04:002009-03-31T20:18:06.303-04:00Upgrading to Jaunty: kill fglrxI've just upgraded one of my boxen to Ubuntu Jaunty Jackalope, and I wish it had been easier. The problem is that the new version ATI's proprietary fglrx driver doesn't seem to support my video card. For other reasons, single user mode was also broken. So I even had trouble getting a root shell.<br /><br />But on the bright side, the open source drivers have "gotten there" for me-- this is a Home Theatre PC (HTPC), and I now get smooth, tear-free playback at 1920x1080. I don't get 3D acceleration, though.<br /><br />fglrx is a greedy constellation of software that tries to control your hardware even when you've selected the open source drivers, and this stops the open source drivers from working correctly. Even if they show the display correctly, fglrx's Direct Rendering Manager can interfere with the standard Direct Rendering Manager, preventing video acceleration. So I recommend uninstalling fglrx before upgrading. Then you'll know whether the Free drivers are up to scratch on your hardware, and you can try fglrx if they're not.<br /><br />To remove fglrx, open a terminal and type: "sudo apt-get remove --purge xorg-driver-fglrx fglrx-kernel-source; sudo reboot"<br /><br />After the reboot, the open source drivers should kick in, and you'll be able to upgrade to Jaunty while they're in use. Once you've got Jaunty working, you can try your luck with fglrx again if you like. I probably won't :-)Anonymousnoreply@blogger.com0tag:blogger.com,1999:blog-17356619.post-73040292390514080462007-12-16T11:34:00.000-05:002007-12-16T12:23:36.948-05:00Bazaar 1.0: not just a numberI've been working on Bazaar since before it started. Back before Martin had started coding Bazaar, I worked on an implementation of the same ideas (as I understood them), called "BaZing". Now, more than 2.5 years later, we've finally reached 1.0.<br /><br />In a sense, these numbers are arbitrary. Bazaar has been usable for many years, depending on your needs. So to us, 1.0 is a signal that says "Hey, look at us. We're ready for you." It's a number we could choose to use at any time. We've chosen to use it now, but that doesn't mean it's meaningless.<br /><br />The act of declaring 1.0 meant that several people pushed back <span style="font-weight: bold;">hard</span>. We have a new storage format called "packs". It's safer and faster, and almost lockless. People said we weren't ready until our new, faster, safer format was the default. They said we couldn't reasonably expect people to benchmark a non-default format. They were right. So Robert Collins pushed hard on getting "packs" up to scratch, and we delayed the release until they were ready.<br /><br />Meanwhile, for me, 1.0 meant I decided I should fix some problems that I'd known about for a while, but had never been enthusiastic about fixing: problems with case-insensitive (or case-preserving) filesystems. Bazaar can now detect case-insensitive filesystems and avoid trying to create files whose names differ only in case.<br /><br />Others also put work into solving the kind of bugs that give bad first impressions.<br /><br />Originally, this release was called "0.93". But if we'd released it that way, it would have been a completely different release. We may not have done everything we wanted to do, but I'm proud of Bazaar 1.0.Anonymousnoreply@blogger.com0tag:blogger.com,1999:blog-17356619.post-88950502380541825212007-02-21T20:57:00.000-05:002007-02-21T22:04:01.429-05:00Living the sysadmin dreamThere's a legendary threat of sysadmins: "Go away, or I will replace you with a very small shell script".<br /><br />Well, today I got a taste of that.<br /><br />At work, we've got a new web site almost ready to go. The content is stored the plainest-possible XHTML. That makes it somewhat future-proof. We process that through Kid templates to make it look snazzy.<br /><br />We wanted to reuse a lot of the content from our existing site, which is a mix of HTML and PHP with some abuse of divs and other incidental wackiness. We knew the error of our ways, and we wanted to get pure, so we hired a contractor to convert our site over. He knew Python. He seemed to have a good grasp of advanced web site architectures.<br /><br />Boy, were we ever wrong.<br /><br />Originally expected to take one month, the job ended up taking three. In the process, I had to write a script to do automated testing of the guy's work, because he was screwing up on the mechanical stuff. Problems that can be solved with search and replace, he was choosing to solve by hand, and getting wrong. It was insulting getting work with the same mechanical errors time and time again.<br /><br />What I had, perhaps naively, expected was that he would write a conversion script, eyeball the results, and fix up any problems.<br /><br />There were a few files we'd asked him not to convert, but we eventually decided to convert some of them after all. The task fell to me, and it was a real joy to set to it. After all that time spent poring over his work, I'd developed a very clear idea of what I thought he should have done in the first place.<br /><br /><span style="font-size:130%;">Phase 1: Text Substitution</span><br /><blockquote></blockquote>In this phase, we convert PHP to HTML. In the general case, you can't do this without implementing PHP. But our existing site uses PHP only as a templating system, so there are only a couple of stock phrases to worry about. re.sub handles this nicely.<br /><br />Another thing we do is remove any ® symbols we encounter, because our house style has changed, and we no longer want to affix this to every mention of our brand.<br /><br /><br /><span style="font-size:130%;">Phase 2: <a href="http://www.w3.org/People/Raggett/tidy/">Tidy HTML</a><br /></span>This phase consists of shoving all our files through Tidy, so that they become well-formed, almost-valid HTML. It's wonderful to have a utility like Tidy, because programs can't feel pain.<br /><span style="font-size:130%;"><br />Phase 3: Ending div abuse</span><br />Our existing content uses div classes for headings all over the place. To tackle these problems, we should really operate in the XML domain, so that we avoid parsing mistakes. Now that tidy has made the documents well-formed, we can parse them with <a href="http://effbot.org/zone/element-index.htm">ElementTree's</a> HTML parser. Then we process it through a <a href="http://kid-templating.org/">Kid Template</a> with a series of match rules. These turn divs into h2s, and so on. We can also clean up our document title (which has an inexplicable <span style="font-style: italic;">trailing</span> non-breaking spaces)<br /><br /><span style="font-size:130%;">Phase 4: Whitespace cleanup</span><br />The kid processing has fixed the semantics of the document and exported it as XHTML, but we're left with swaths of whitespace and line breaks in unreasonable places. Another run through Tidy fixes that.<br /><br /><span style="font-size:130%;">Automation is wunnerful</span><br />Not only did I manage to get a basic conversion routine in place in about three hours, I actually stayed at long past my notional end-of-shift, because I was enjoying what I was doing. Writing a script to convert files to XHTML is fun. Doing the conversion yourself is not. That's a problem our contractor faced. Not only was he slow when he was actually working, but he had trouble <span style="font-style: italic;">getting motivated</span> to work, so he didn't put in as much time as we wanted him to.<br /><br />They say that good programmers are 10x more productive than average programmers, and truly excellent programmers are 10x more productive again. I suspect automation is one of the factors in that difference. Unless you stop them, an excellent programmer will find a way to eliminate the boring bits. Even if automating the job takes just as long as doing it by hand, a programmer will be happier and more motivated because the work is much more stimulating.<br /><br />So today, I replaced a contractor with a reasonably small Python script. I wish I'd done it months ago.Anonymousnoreply@blogger.com1tag:blogger.com,1999:blog-17356619.post-1162082791820679822006-10-28T20:40:00.000-04:002006-10-28T23:38:55.506-04:00Workarounds for SQLAlchemy and Turbogears 1.0bI've started work on a new <a href="http://turbogears.org">TurboGears</a> project to track merge requests for <a href="http://bazaar-vcs.org">Bazaar</a>. I was already familiar with earlier versions of TurboGears, but I decided to upgrade to 1.0b, and try out this new <a href="http://sqlalchemy.org">SQLAlchemy</a> thing that everyone's raving about. It's supposed to be the official object-relational-mapper in TurboGears 1.1.<br /><br />You have the option of using it in 1.0b, but the integration is not smooth.<br /><br />Although you can use the identity framework with SQLAlchemy, the default model.py that TurboGears provides is broken. TurboGears uses the ActiveMapper extension, but the default model.py defines relationship between groups and users, and between groups and permissions in two places. Even though the definitions are equivalent, this is not allowed, and it fails silently.<br /><br />So to use groups/permissions effectively, comment out the two <tt>many_to_many</tt> lines in Group. You'll still get <tt>users</tt> and <tt>properties</tt> attributes on <tt>Group</tt>, but they'll be created by the 'backref' parameters on the many_to_many statements in <tt>User</tt> and <tt>Property</tt>.<br /><br />I've learned to like unit-testing. But the Turbogears TestCase object does not handle SQLAlchemy objects. So I wrote my own:<br /><br /><pre>class TestMessages(TestCase):<br /><br />def iter_active_mappers(self):<br />for key, item in model.__dict__.items():<br /> try:<br /> if issubclass(item, ActiveMapper):<br /> if item is not ActiveMapper:<br /> yield item<br /> except TypeError:<br /> pass<br /><br />def setUp(self):<br />for active_mapper in self.iter_active_mappers():<br /> active_mapper.table.create()<br /><br />def tearDown(self):<br />for active_mapper in self.iter_active_mappers():<br /> active_mapper.table.drop()</pre><br />Finally, I also had to force it to provide my own database configuration:<br /><tt>config.update({'sqlalchemy.dburi': 'sqlite:///:memory:'})<br /></tt><br />The <tt>tg-admin</tt> tool has quite limited support for SQLAlchemy: it only does <tt>sql create</tt>. I've done sql drop quite a lot in past projects. I could adapt the <tt>tearDown</tt> method above, but I'm currently using SQLite, so <tt>rm devdata</tt> is an adequate subsitute.<br /><br />I hope these tips are helpful.<br /><br />Update:<br />SQLAlchemy emits reams and reams of SQL kipple; To disable that, place <tt>sqlalchemy.echo = False</tt> in your config file.<br /><br />It's great that TurboGears is so clearly committed to delivering best-of-breed components that they'll switch ORMs and templating libraries for their 1.1 release. But 1.1 isn't here yet, and while you can integrate SQLAlchemy and TurboGears today, there's a certain amount of pain involved.Anonymousnoreply@blogger.com0tag:blogger.com,1999:blog-17356619.post-1149966680881909902006-06-10T15:01:00.000-04:002006-06-10T17:13:29.203-04:00Bloom filters and Smart ServersI just got back from a joint <a href="http://www.selenic.com/mercurial">Mercurial</a> / <a href="http://bazaar-vcs.org">Bazaar</a> meetup. One of the neat things about it was the bloom filter collaboration. Bryan O'Sullivan described <a href="http://en.wikipedia.org/wiki/Bloom_filter">bloom filters</a> to us as a mechanism for <a href="http://en.wikipedia.org/wiki/Greylisting">greylisting</a>. They're a way of cheaply storing large sets of keywords, with the tradeoff that testing whether a keyword is in a set may produce false positives.<br /><br />Later on, they were discussing their <a href="http://www.selenic.com/mercurial/wiki/index.cgi/hgserve">smart server</a> implementation. This is especially interesting to the Bazaar crew, because we are now planning to create our own smart server.<br /><br />One of the tricky things about smart servers is that they must determine which revisions the server has that are not present on the client side. Mercurial currently does this using a binary search, but this can have up to 15 roundtrips, and roundtrips in network protocols can slow them down dramatically.<br /><br />It came to me that bloom filters could be applied to this problem, because they allow a set to be represented compactly, and the revisions in a repository can be treated as a set.<br /><br />Robert Collins stated that bloom filters can also be inverted, so that instead of the risk of false positives, you have a risk of false negatives. This is useful if the receiver sends its filter to the sender, and the sender decides which revisions to send. That way, the sender can never send revisions that depend on unknown revisions. So far, we haven't found a description of how to invert a bloom filter to produce false negatives, however.<br /><br />If the sender sends its filter, then the receiver decides which tips the two machines have in common, and so false positives don't produce a risk of sending revisions that depend on unknown revisions.<br /><br />Matt Mackell investigated the possibility of using blooms with Mercurial's smart server, and concluded that a bloom filter of a million revisions with a 1% error rate would take about a megabyte of data-- too much to send all at once. I pointed out that we could accept a higher error rate for this data, because revisions in a set have strong correspondances: if a revision is in a set, then its direct ancestor should also be in the set. Using this rule, we can eliminate matches where the direct ancestor is not in the set, reducing the error rate to 1% again. Unfortunately, a bloom filter with a 10% error rate is still 500k or so.<br /><br />However, it's worth noting that it's rarely necessary to send a million revisions; instead, a bloom filter of a smaller number of revisions can be sent, since repositories are usually very similar.Anonymousnoreply@blogger.com0tag:blogger.com,1999:blog-17356619.post-1143173436334780932006-03-23T22:39:00.000-05:002006-03-23T23:10:36.373-05:00More tree transformsHere's a follow-up on tree transforms, now that I've finished implementing them.<br /><br />One thing I found as I implemented tree transforms in bzr was that the modification step is probably the smallest and most boring. Content changes can be structured as delete/create pairs, so they don't need to be modifications. So the only modifications bzr does are file permission changes.<br /><br />Another thing I found is that you can structure a tree transform so that all the risky stuff happens up-front, making it very unlikely that you'll run into an error that leaves your tree in an inconsistent state. The trick is to do all your creation operations into a temp directory beforehand. Then, when you're applying the transform, you rename-into-place instead of creating. <br /><br />That means you're not subject to 'disk full' errors or failures in the text merge. It also means that you can safely perform merge operations that make reference to the working tree, like using '.' as BASE or OTHER in a merge. And the simpler you can make the tranform application, the better.<br /><br />You can even take this a step farther, and defer deletions until you've completeliy updated the working tree. Just rename files into a temporary directory, and then delete it when the working tree has been successfully updated. That should make it relatively easy to roll back to the previous state, should you encounter an error.<br /><br />Another thing we do before applying the transform is conflict resolution. After all, there may already be files where you want to put them. Or a text merge may need to emit .BASE, .THIS and .OTHER files. Or a merge may produce a directory loop. Etc. Bzr has a set of fairly simple rules for detecting and resolving conflicts, and doesn't try to predict how different conflict resolutions could interact with each other. It just runs the resolver up to 10 times, and if that doesn't fix the problem, it gives up.<br /><br />But again, with all of this happening before a single move or deletion, there isn't a huge penalty for failure.Anonymousnoreply@blogger.com0tag:blogger.com,1999:blog-17356619.post-1142559699768158122006-03-16T20:31:00.000-05:002006-03-16T20:41:41.700-05:00A satisfying morning's hackSo standards-conformant web sites are mother-and-apple-pie kinda stuff. Everyone knows they're a good thing. The w3c ha<span style="font-style: italic;"></span>s even been so helpful as to provide a way to validate anything you might have online.<br /><br />Problem is, a lot of your stuff <span style="font-style: italic;">isn't</span> online. Once you know it's valid, <span style="font-style: italic;">then</span> it'll go online. Sure, the w3c validator will accept file uploads, but if you're doing any kinda dynamic content generation, you have to save the rendered output, and upload <span style="font-style: italic;">that</span>. Every time.<br /><br />So yesterday, (or was it Tuesday?) I hacked up a validator proxy. It's TurboGears, but it just uses CherryPy. It pulls down a specified URL, which can be a LAN-only URL, shoves it up to the w3c validator, and returns a slightly-rewritten version of the validator's response. For an encore, I hacked up the w3c validation bookmarklet, to teach it to use my proxy.<br /><br />Validation happens a lot more when it's easy. And it's so validating...Anonymousnoreply@blogger.com0tag:blogger.com,1999:blog-17356619.post-1135212510849080382005-12-21T18:27:00.000-05:002005-12-21T19:58:53.856-05:00Tree transforms on POSIX filesystems and related structuresThis post comes from some discussion I had with Denys Duchier about reimplementing the inventory-rewriting step of the bzr merge procedure.<br /><br />The bzr inventory system has a lot of the same constraints as a POSIX filesystem does. You can't have two files with the same name, at any point. You must never delete a directory before deleting its children. You must always create a directory before putting files inside it.<br /><br />On the other hand, you can simply replace one inventory with another, updating the whole tree structure in one fell swoop. This is fairly straightforward, so that's what I did.<br /><br />On the filesystem, there's no such shortcut. But there is a rather elegent way of doing it, one that I learned from reading the the GNU Arch source code.<br /><br />There are 4 things you can do with a file:<br /><ol> <li>Create it</li> <li>Delete it</li> <li>Rename it</li> <li>Modify it</li> </ol> For regular files and directories, "modify" may mean permission changes. For files, it may mean contents changes. And for symlinks, it may mean target changes. It also includes changing a file from one type to another.<br /><br />In order to allow operations to happen in the required order, we need to split rename into two operations: rename FROM, and rename TO. This means that we can classify the operations like so:<br /><ol> <li>Name removals (delete, rename FROM)</li> <li>Name insertions (create, rename TO)</li> <li>File modifications</li> </ol> If you do all your removals before insertions, you can be certain that insertions will never try to use a name that is pending removal. So as long as you don't try to insert the same name twice, you'll never violate the POSIX requirement of only one file per name.<br /><br />So for name operations, we need two phases<br /><ol> <li>removals</li> <li>insertions</li> </ol> Splitting rename into two operations means that, during the removal phase, we need to keep renamed files somewhere else. We can keep them in a temp directory, then move them into place afterward.<br /><br />It's important to do rename-FROM and delete at the same time, because of the ordering issues. Children must always be removed before parents. If you delete the parent first, your delete will fail. If you rename the parent first, you'll lose track of the child, and be unable to perform your operation on it. Because the inheritance hierarchy may include both deletes and rename-FROMs, you can't do renames or deletes as separate steps. They must be intermingled.<br /><br />The inverse is true of insertions. If you try to create a child before creating its parents, the operation will fail. If you try to rename a child to its new name, the operation will also fail.<br /><br />Modifications are largely free of naming concerns. However, there is one kind of modification that can affect the availability of names: file type changes. Turning a file into a directory enables it to have children. Turning a directory into something else makes it impossible for it to have children. Since file type modification requires that removals be done first, and since file type modification is required before insertions are safe, the logical place for it is in the middle.<br /><br />Unfortunately, this means that you must do contents changes while renamed files are still in the temp dir.<br /><br />So to recap, the three phases are:<br /><ol> <li>Do deletions and rename-to-temp in child-to-parent order</li> <li>Do file modifications</li> <li>Do creations and rename-to-final in parent-to-child order</li> </ol> I hope this is as fascinating to you as it is to me.Anonymousnoreply@blogger.com0tag:blogger.com,1999:blog-17356619.post-1133499008794228682005-12-01T23:17:00.000-05:002005-12-02T00:03:47.183-05:00When they say "Turbo", they mean itI've been fiddling around with <a href="http://www.turbogears.org/">TurboGears</a> recently. One web app I've worked on is a service that lets you know when the local bands you like are playing in a calendar format. So that you can say "I feel like seeing a band tonight" and see what's on. Another was a help system.<br /><br />Well, I've just spent a fun few hours putting together a web front end to my <span style="font-weight: bold;">Bugs Everywhere </span>bug tracker. It was definitely a "turbo" experience. Went from zero to "does a lot and looks decent" in mere hours. It was also a "gears" experience, because half the fun was how easily everything fit together. Rock on, Turbogears.<br /><br />There have been a spate of web interfaces for bzr, and I wonder whether they'd be easier using TurboGears or Django. My guess is yes-- lots easier.<br /><br />These new web megaframeworks interest me a lot, partly because writing <a href="http://www.panoramicfeedback.com/">Panoramic Feedback</a> required coming up with one. Yes, PF does templates, data persistence, url dispatch and JavaScript (not to mention having its own XML parser and PDF report generation subsystem). But if I was starting out today, there's no way I'd build our own megaframework. They're easy, they're fun, and they let you focus on what you really want to do.<br /><br />TurboGears isn't the only megaframework in town, and I wouldn't turn up my nose at Django or Rails. But I'd prefer to write in Python, since that's compatible with most of my software. Ruby seems too similar to Python for it to be worthwile knowing both.<br /><br />What I especially like about TurboGears is its philosophy of collaboration, and Kid templates. The fact that they're valid XHTML is neat. The fact that they pack the whole expressive power of Python (should you need it) is great. In my bug tracker front end, I'm actually passing list of Bug objects to the display template, and the template is retrieving the members it needs. Nice.<br /><br />I'd like it if CherryPy had better support for RESTful URLs. SQLObject seems a little buggy to me. And CatWalk will be very nice, when it happens.<br /><br />But it's already come a very long way.Anonymousnoreply@blogger.com1tag:blogger.com,1999:blog-17356619.post-1132117886887675152005-11-15T23:43:00.000-05:002005-11-16T00:11:26.896-05:00LandedI'm back from Ubuntu Below Zero. It was a great experience. Lots of stimulating conversations with smart people. Lots of people appreciating what we've done with bzr. Great progress on our future plans.<br /><br />On the sunday, there was a dinner where we told them something people would never guess about us, and then everyone had to figure out who the facts were related to. I couldn't think of anything else, so I told them I started programming when I was eight. Turns out, that's not very unusual for Canonical employees. It may even make me a late bloomer.<br /><br />Perhaps what I should have said was, "I've worked on five VCS clients in the past year".<br /><ul> <li>Gnu Arch</li> <li>Friendly Arch Interface (My python front-end to tla)</li> <li>Bazaar (Canonical's fork of tla)</li> <li>BaZing (My attempt to implement Martin Pool's idea for a new VCS)<br /> </li> <li>Bazaar-NG (Martin Pool implementation of his idea)</li> </ul> Totally nuts.<br /><br />Of course, they all factor into each other.<br />Early BaZing code was usable from Fai, the BaZing merge code was merged into bzr, and it took some ideas from tla. Lately, I've adapted Fai's shell mode for use with bzr.<br /><br />Hopefully, things will settle down in the next few years. Funny how, after being a stagnant backwater for so long, VCSes are thriving now.Anonymousnoreply@blogger.com0tag:blogger.com,1999:blog-17356619.post-1130109030473158942005-10-23T16:59:00.000-04:002005-10-23T19:10:30.483-04:00Implementing mergeIt seems like I've been implementing merge forever, but it's only been a year.<br /><br />Merge is a fundamental operation when you have groups of people working together on a single piece of source code. It essentially means "take these other changes and combine them with mine."<br /><br />The merge tech you'll see in CVS, SVN, Bazaar and Gnu Arch is called "three-way merging", because it looks at three versions of the file. The one you had before any changes were made ("BASE"), the one containing the changes the other person made ("OTHER"), and the one containing the changes you made ("THIS").<br /><br />If THIS or OTHER (or both) introduced a change, it goes into the final version. But if THIS and OTHER made different changes to the same code, that's considered a conflict.<br /><br />In CVS, "update" was used to merge other peoples' committed changes with your recent work. In distributed systems, it's typically its own command, and unlike CVS, it's typically invoked <span style="font-style: italic;">after</span> you commit, not <span style="font-style: italic;">before</span>.<br /><br />My first work on merge was implementing three-way behaviour for baz merge-- at the Baz Code Sprint last November. At the time, baz and tla only supported three-way merging of text, and only when a particular command was used. I was making baz merge follow three-way behaviour.<br /><br />At that code sprint, Martin Pool introduced his ideas for a new revision control system, now known as Bazaar-NG or bzr. I decided to take a stab at implementing those ideas, myself. My project, BaZing, got as far as being able to do merges and apply Arch changesets. Then I threw in with Martin to work on bzr.<br /><br />I ported over the merge code to bzr. For a while, that meant a system of shims and adaptors to make one codebase speak to the other. Lately, I've been working on integrating the code better.<br /><br />Martin implemented the actual text-merging portion of bzr, so we don't depend on diff3. But it generates more conflicts than I think it should, so I've been playing around with my own three-way algorithm.<br /><br />And I went ahead and integrated weave merge, a technique used by SCCS and (we believe) BitKeeper, into bzr. Again, Martin had written the code, but I hooked it up to bzr's merge tech.<br /><br />It's important that VCS creators not get hung up on merge. It's only one aspect of usability.<br /><br />Normal, boring three-way merges are good enough quite a lot of the time. Merge has quite a lot of strange corner cases, but they're not hit all that often. Improvements are welcome, but we will never get it 100% right, because merge doesn't address combining <span style="font-style: italic;">programs</span>, but combining <span style="font-style: italic;">text</span>. Most merge tools don't understand what that text means. What we need, ideally, is artificial intelligence. Since we don't have that, we need tricks that make the program <span style="font-style: italic;">seem</span> intelligent without actually <span style="font-style: italic;">being</span> intelligent.<br /><br />Merging is an art, not a science. So merge tech is a tar pit. It can take as much time as you're willing to throw at it, and still not be perfect. There are other things bzr needs, like better remote operations, central storage, better Windows support. So at times, it can be frustrating working on merging yet again.Anonymousnoreply@blogger.com0tag:blogger.com,1999:blog-17356619.post-1128228553400078252005-10-02T00:00:00.000-04:002005-10-21T11:02:22.813-04:00The Cathedral is bizarreI can't friggin' believe Larry McVoy. I mean, I just don't understand him.<br /><br />Here he is, lead designer of a powerful version control system(VCS). For a long time, BitKeeper had very good buzz in the open source world. (And, perhaps, even in the Free Software one.)<br /><br />You'd think he would be proud. You'd think he'd focus on how to do even better. The last thing I expect from someone who's done great service to Linux is anticompetitive behaviour.<br /><br />But lately, that's the kind of stuff we've seen. A while back, he cut Linus out because Tridge was writing an open-source Bitkeeper client. How does that work again? Now he's forced Brian O'Sullivan to stop working Mercurial, claiming he <a href="http://www.selenic.com/pipermail/mercurial/2005-September/004745.html">fears O'Sullivan will copy</a> Bitkeeper's secret sauce.<br /><br />Well, last time I checked, BitKeeper was a proprietary, closed-source program. Since Brian can't copy the source code, it can't be an issue of copyright infringement. No, Larry fears that Brian will copy <span style="font-style: italic;">ideas</span> from BitKeeper.<br /><br />In the first place, isn't that totally wrong? You shouldn't build a better mousetrap if you know how current mousetraps work? Edison has to invent lightbulbs in the dark? The hell?<br /><br />In the second place, if Larry thinks his ideas are so special, why doesn't he patent them?<br /><br />One possible reason is that not all the ideas are his own. BK is heavily based on SCCS, a 30 year-old VCS. It uses SCCS files to store its data. From what we can tell, its merge technology is also based on SCCS.<br /><br />So Larry can base his VCS on someone else's, but Brian can't base his VCS on Larry's? Sure, that seems fair.<br /><br />Look at the FOSS side of things, and there are no secrets. There's more than a few projects to build a great distributed VCS in progress at the moment, like Bazaar-NG (the one I'm with), Monotone, Codeville, Mercurial, Darcs, SVK, and more. Not only is the code open, but we're always chatting on IRC about merge algorithms, the merits and demerits of GUIDs for files, and other technology. IRC's where I first heard about Larry's latest escapades.<br /><br />Maybe you think I should be happy that Mercurial's hit a bump in the road? Don't they say the enemy of my enemy is my friend? Maybe they do, but Brian O'Sullivan isn't my <span style="font-style: italic;">enemy</span>, he's a <span style="font-style: italic;">competitor</span>. We both want the better open-source VCSes. Why shouldn't we copy each others' best ideas?<br /><br />Larry, he could have been another friendly competitor. But with anticompetitive behaviour and his talk of "<a href="http://www.forbes.com/execpicks/2005/05/26/cz_dl_0526linux.html">innovation</a>", he's starting to remind me of another of Free Software's enemies. But Microsoft has Visual Source Safe, which makes them BitKeeper's enemy, too. So I guess the enemy of my enemy is my enemy.Anonymousnoreply@blogger.com0