I've just upgraded one of my boxen to Ubuntu Jaunty Jackalope, and I wish it had been easier. The problem is that the new version ATI's proprietary fglrx driver doesn't seem to support my video card. For other reasons, single user mode was also broken. So I even had trouble getting a root shell.
But on the bright side, the open source drivers have "gotten there" for me-- this is a Home Theatre PC (HTPC), and I now get smooth, tear-free playback at 1920x1080. I don't get 3D acceleration, though.
fglrx is a greedy constellation of software that tries to control your hardware even when you've selected the open source drivers, and this stops the open source drivers from working correctly. Even if they show the display correctly, fglrx's Direct Rendering Manager can interfere with the standard Direct Rendering Manager, preventing video acceleration. So I recommend uninstalling fglrx before upgrading. Then you'll know whether the Free drivers are up to scratch on your hardware, and you can try fglrx if they're not.
To remove fglrx, open a terminal and type: "sudo apt-get remove --purge xorg-driver-fglrx fglrx-kernel-source; sudo reboot"
After the reboot, the open source drivers should kick in, and you'll be able to upgrade to Jaunty while they're in use. Once you've got Jaunty working, you can try your luck with fglrx again if you like. I probably won't :-)
Tuesday, March 31, 2009
Sunday, December 16, 2007
Bazaar 1.0: not just a number
I've been working on Bazaar since before it started. Back before Martin had started coding Bazaar, I worked on an implementation of the same ideas (as I understood them), called "BaZing". Now, more than 2.5 years later, we've finally reached 1.0.
In a sense, these numbers are arbitrary. Bazaar has been usable for many years, depending on your needs. So to us, 1.0 is a signal that says "Hey, look at us. We're ready for you." It's a number we could choose to use at any time. We've chosen to use it now, but that doesn't mean it's meaningless.
The act of declaring 1.0 meant that several people pushed back hard. We have a new storage format called "packs". It's safer and faster, and almost lockless. People said we weren't ready until our new, faster, safer format was the default. They said we couldn't reasonably expect people to benchmark a non-default format. They were right. So Robert Collins pushed hard on getting "packs" up to scratch, and we delayed the release until they were ready.
Meanwhile, for me, 1.0 meant I decided I should fix some problems that I'd known about for a while, but had never been enthusiastic about fixing: problems with case-insensitive (or case-preserving) filesystems. Bazaar can now detect case-insensitive filesystems and avoid trying to create files whose names differ only in case.
Others also put work into solving the kind of bugs that give bad first impressions.
Originally, this release was called "0.93". But if we'd released it that way, it would have been a completely different release. We may not have done everything we wanted to do, but I'm proud of Bazaar 1.0.
In a sense, these numbers are arbitrary. Bazaar has been usable for many years, depending on your needs. So to us, 1.0 is a signal that says "Hey, look at us. We're ready for you." It's a number we could choose to use at any time. We've chosen to use it now, but that doesn't mean it's meaningless.
The act of declaring 1.0 meant that several people pushed back hard. We have a new storage format called "packs". It's safer and faster, and almost lockless. People said we weren't ready until our new, faster, safer format was the default. They said we couldn't reasonably expect people to benchmark a non-default format. They were right. So Robert Collins pushed hard on getting "packs" up to scratch, and we delayed the release until they were ready.
Meanwhile, for me, 1.0 meant I decided I should fix some problems that I'd known about for a while, but had never been enthusiastic about fixing: problems with case-insensitive (or case-preserving) filesystems. Bazaar can now detect case-insensitive filesystems and avoid trying to create files whose names differ only in case.
Others also put work into solving the kind of bugs that give bad first impressions.
Originally, this release was called "0.93". But if we'd released it that way, it would have been a completely different release. We may not have done everything we wanted to do, but I'm proud of Bazaar 1.0.
Wednesday, February 21, 2007
Living the sysadmin dream
There's a legendary threat of sysadmins: "Go away, or I will replace you with a very small shell script".
Well, today I got a taste of that.
At work, we've got a new web site almost ready to go. The content is stored the plainest-possible XHTML. That makes it somewhat future-proof. We process that through Kid templates to make it look snazzy.
We wanted to reuse a lot of the content from our existing site, which is a mix of HTML and PHP with some abuse of divs and other incidental wackiness. We knew the error of our ways, and we wanted to get pure, so we hired a contractor to convert our site over. He knew Python. He seemed to have a good grasp of advanced web site architectures.
Boy, were we ever wrong.
Originally expected to take one month, the job ended up taking three. In the process, I had to write a script to do automated testing of the guy's work, because he was screwing up on the mechanical stuff. Problems that can be solved with search and replace, he was choosing to solve by hand, and getting wrong. It was insulting getting work with the same mechanical errors time and time again.
What I had, perhaps naively, expected was that he would write a conversion script, eyeball the results, and fix up any problems.
There were a few files we'd asked him not to convert, but we eventually decided to convert some of them after all. The task fell to me, and it was a real joy to set to it. After all that time spent poring over his work, I'd developed a very clear idea of what I thought he should have done in the first place.
Phase 1: Text Substitution
In this phase, we convert PHP to HTML. In the general case, you can't do this without implementing PHP. But our existing site uses PHP only as a templating system, so there are only a couple of stock phrases to worry about. re.sub handles this nicely.
Another thing we do is remove any ® symbols we encounter, because our house style has changed, and we no longer want to affix this to every mention of our brand.
Phase 2: Tidy HTML
This phase consists of shoving all our files through Tidy, so that they become well-formed, almost-valid HTML. It's wonderful to have a utility like Tidy, because programs can't feel pain.
Phase 3: Ending div abuse
Our existing content uses div classes for headings all over the place. To tackle these problems, we should really operate in the XML domain, so that we avoid parsing mistakes. Now that tidy has made the documents well-formed, we can parse them with ElementTree's HTML parser. Then we process it through a Kid Template with a series of match rules. These turn divs into h2s, and so on. We can also clean up our document title (which has an inexplicable trailing non-breaking spaces)
Phase 4: Whitespace cleanup
The kid processing has fixed the semantics of the document and exported it as XHTML, but we're left with swaths of whitespace and line breaks in unreasonable places. Another run through Tidy fixes that.
Automation is wunnerful
Not only did I manage to get a basic conversion routine in place in about three hours, I actually stayed at long past my notional end-of-shift, because I was enjoying what I was doing. Writing a script to convert files to XHTML is fun. Doing the conversion yourself is not. That's a problem our contractor faced. Not only was he slow when he was actually working, but he had trouble getting motivated to work, so he didn't put in as much time as we wanted him to.
They say that good programmers are 10x more productive than average programmers, and truly excellent programmers are 10x more productive again. I suspect automation is one of the factors in that difference. Unless you stop them, an excellent programmer will find a way to eliminate the boring bits. Even if automating the job takes just as long as doing it by hand, a programmer will be happier and more motivated because the work is much more stimulating.
So today, I replaced a contractor with a reasonably small Python script. I wish I'd done it months ago.
Well, today I got a taste of that.
At work, we've got a new web site almost ready to go. The content is stored the plainest-possible XHTML. That makes it somewhat future-proof. We process that through Kid templates to make it look snazzy.
We wanted to reuse a lot of the content from our existing site, which is a mix of HTML and PHP with some abuse of divs and other incidental wackiness. We knew the error of our ways, and we wanted to get pure, so we hired a contractor to convert our site over. He knew Python. He seemed to have a good grasp of advanced web site architectures.
Boy, were we ever wrong.
Originally expected to take one month, the job ended up taking three. In the process, I had to write a script to do automated testing of the guy's work, because he was screwing up on the mechanical stuff. Problems that can be solved with search and replace, he was choosing to solve by hand, and getting wrong. It was insulting getting work with the same mechanical errors time and time again.
What I had, perhaps naively, expected was that he would write a conversion script, eyeball the results, and fix up any problems.
There were a few files we'd asked him not to convert, but we eventually decided to convert some of them after all. The task fell to me, and it was a real joy to set to it. After all that time spent poring over his work, I'd developed a very clear idea of what I thought he should have done in the first place.
Phase 1: Text Substitution
In this phase, we convert PHP to HTML. In the general case, you can't do this without implementing PHP. But our existing site uses PHP only as a templating system, so there are only a couple of stock phrases to worry about. re.sub handles this nicely.
Another thing we do is remove any ® symbols we encounter, because our house style has changed, and we no longer want to affix this to every mention of our brand.
Phase 2: Tidy HTML
This phase consists of shoving all our files through Tidy, so that they become well-formed, almost-valid HTML. It's wonderful to have a utility like Tidy, because programs can't feel pain.
Phase 3: Ending div abuse
Our existing content uses div classes for headings all over the place. To tackle these problems, we should really operate in the XML domain, so that we avoid parsing mistakes. Now that tidy has made the documents well-formed, we can parse them with ElementTree's HTML parser. Then we process it through a Kid Template with a series of match rules. These turn divs into h2s, and so on. We can also clean up our document title (which has an inexplicable trailing non-breaking spaces)
Phase 4: Whitespace cleanup
The kid processing has fixed the semantics of the document and exported it as XHTML, but we're left with swaths of whitespace and line breaks in unreasonable places. Another run through Tidy fixes that.
Automation is wunnerful
Not only did I manage to get a basic conversion routine in place in about three hours, I actually stayed at long past my notional end-of-shift, because I was enjoying what I was doing. Writing a script to convert files to XHTML is fun. Doing the conversion yourself is not. That's a problem our contractor faced. Not only was he slow when he was actually working, but he had trouble getting motivated to work, so he didn't put in as much time as we wanted him to.
They say that good programmers are 10x more productive than average programmers, and truly excellent programmers are 10x more productive again. I suspect automation is one of the factors in that difference. Unless you stop them, an excellent programmer will find a way to eliminate the boring bits. Even if automating the job takes just as long as doing it by hand, a programmer will be happier and more motivated because the work is much more stimulating.
So today, I replaced a contractor with a reasonably small Python script. I wish I'd done it months ago.
Saturday, October 28, 2006
Workarounds for SQLAlchemy and Turbogears 1.0b
I've started work on a new TurboGears project to track merge requests for Bazaar. I was already familiar with earlier versions of TurboGears, but I decided to upgrade to 1.0b, and try out this new SQLAlchemy thing that everyone's raving about. It's supposed to be the official object-relational-mapper in TurboGears 1.1.
You have the option of using it in 1.0b, but the integration is not smooth.
Although you can use the identity framework with SQLAlchemy, the default model.py that TurboGears provides is broken. TurboGears uses the ActiveMapper extension, but the default model.py defines relationship between groups and users, and between groups and permissions in two places. Even though the definitions are equivalent, this is not allowed, and it fails silently.
So to use groups/permissions effectively, comment out the two many_to_many lines in Group. You'll still get users and properties attributes on Group, but they'll be created by the 'backref' parameters on the many_to_many statements in User and Property.
I've learned to like unit-testing. But the Turbogears TestCase object does not handle SQLAlchemy objects. So I wrote my own:
Finally, I also had to force it to provide my own database configuration:
config.update({'sqlalchemy.dburi': 'sqlite:///:memory:'})
The tg-admin tool has quite limited support for SQLAlchemy: it only does sql create. I've done sql drop quite a lot in past projects. I could adapt the tearDown method above, but I'm currently using SQLite, so rm devdata is an adequate subsitute.
I hope these tips are helpful.
Update:
SQLAlchemy emits reams and reams of SQL kipple; To disable that, place sqlalchemy.echo = False in your config file.
It's great that TurboGears is so clearly committed to delivering best-of-breed components that they'll switch ORMs and templating libraries for their 1.1 release. But 1.1 isn't here yet, and while you can integrate SQLAlchemy and TurboGears today, there's a certain amount of pain involved.
You have the option of using it in 1.0b, but the integration is not smooth.
Although you can use the identity framework with SQLAlchemy, the default model.py that TurboGears provides is broken. TurboGears uses the ActiveMapper extension, but the default model.py defines relationship between groups and users, and between groups and permissions in two places. Even though the definitions are equivalent, this is not allowed, and it fails silently.
So to use groups/permissions effectively, comment out the two many_to_many lines in Group. You'll still get users and properties attributes on Group, but they'll be created by the 'backref' parameters on the many_to_many statements in User and Property.
I've learned to like unit-testing. But the Turbogears TestCase object does not handle SQLAlchemy objects. So I wrote my own:
class TestMessages(TestCase):
def iter_active_mappers(self):
for key, item in model.__dict__.items():
try:
if issubclass(item, ActiveMapper):
if item is not ActiveMapper:
yield item
except TypeError:
pass
def setUp(self):
for active_mapper in self.iter_active_mappers():
active_mapper.table.create()
def tearDown(self):
for active_mapper in self.iter_active_mappers():
active_mapper.table.drop()
Finally, I also had to force it to provide my own database configuration:
config.update({'sqlalchemy.dburi': 'sqlite:///:memory:'})
The tg-admin tool has quite limited support for SQLAlchemy: it only does sql create. I've done sql drop quite a lot in past projects. I could adapt the tearDown method above, but I'm currently using SQLite, so rm devdata is an adequate subsitute.
I hope these tips are helpful.
Update:
SQLAlchemy emits reams and reams of SQL kipple; To disable that, place sqlalchemy.echo = False in your config file.
It's great that TurboGears is so clearly committed to delivering best-of-breed components that they'll switch ORMs and templating libraries for their 1.1 release. But 1.1 isn't here yet, and while you can integrate SQLAlchemy and TurboGears today, there's a certain amount of pain involved.
Saturday, June 10, 2006
Bloom filters and Smart Servers
I just got back from a joint Mercurial / Bazaar meetup. One of the neat things about it was the bloom filter collaboration. Bryan O'Sullivan described bloom filters to us as a mechanism for greylisting. They're a way of cheaply storing large sets of keywords, with the tradeoff that testing whether a keyword is in a set may produce false positives.
Later on, they were discussing their smart server implementation. This is especially interesting to the Bazaar crew, because we are now planning to create our own smart server.
One of the tricky things about smart servers is that they must determine which revisions the server has that are not present on the client side. Mercurial currently does this using a binary search, but this can have up to 15 roundtrips, and roundtrips in network protocols can slow them down dramatically.
It came to me that bloom filters could be applied to this problem, because they allow a set to be represented compactly, and the revisions in a repository can be treated as a set.
Robert Collins stated that bloom filters can also be inverted, so that instead of the risk of false positives, you have a risk of false negatives. This is useful if the receiver sends its filter to the sender, and the sender decides which revisions to send. That way, the sender can never send revisions that depend on unknown revisions. So far, we haven't found a description of how to invert a bloom filter to produce false negatives, however.
If the sender sends its filter, then the receiver decides which tips the two machines have in common, and so false positives don't produce a risk of sending revisions that depend on unknown revisions.
Matt Mackell investigated the possibility of using blooms with Mercurial's smart server, and concluded that a bloom filter of a million revisions with a 1% error rate would take about a megabyte of data-- too much to send all at once. I pointed out that we could accept a higher error rate for this data, because revisions in a set have strong correspondances: if a revision is in a set, then its direct ancestor should also be in the set. Using this rule, we can eliminate matches where the direct ancestor is not in the set, reducing the error rate to 1% again. Unfortunately, a bloom filter with a 10% error rate is still 500k or so.
However, it's worth noting that it's rarely necessary to send a million revisions; instead, a bloom filter of a smaller number of revisions can be sent, since repositories are usually very similar.
Later on, they were discussing their smart server implementation. This is especially interesting to the Bazaar crew, because we are now planning to create our own smart server.
One of the tricky things about smart servers is that they must determine which revisions the server has that are not present on the client side. Mercurial currently does this using a binary search, but this can have up to 15 roundtrips, and roundtrips in network protocols can slow them down dramatically.
It came to me that bloom filters could be applied to this problem, because they allow a set to be represented compactly, and the revisions in a repository can be treated as a set.
Robert Collins stated that bloom filters can also be inverted, so that instead of the risk of false positives, you have a risk of false negatives. This is useful if the receiver sends its filter to the sender, and the sender decides which revisions to send. That way, the sender can never send revisions that depend on unknown revisions. So far, we haven't found a description of how to invert a bloom filter to produce false negatives, however.
If the sender sends its filter, then the receiver decides which tips the two machines have in common, and so false positives don't produce a risk of sending revisions that depend on unknown revisions.
Matt Mackell investigated the possibility of using blooms with Mercurial's smart server, and concluded that a bloom filter of a million revisions with a 1% error rate would take about a megabyte of data-- too much to send all at once. I pointed out that we could accept a higher error rate for this data, because revisions in a set have strong correspondances: if a revision is in a set, then its direct ancestor should also be in the set. Using this rule, we can eliminate matches where the direct ancestor is not in the set, reducing the error rate to 1% again. Unfortunately, a bloom filter with a 10% error rate is still 500k or so.
However, it's worth noting that it's rarely necessary to send a million revisions; instead, a bloom filter of a smaller number of revisions can be sent, since repositories are usually very similar.
Thursday, March 23, 2006
More tree transforms
Here's a follow-up on tree transforms, now that I've finished implementing them.
One thing I found as I implemented tree transforms in bzr was that the modification step is probably the smallest and most boring. Content changes can be structured as delete/create pairs, so they don't need to be modifications. So the only modifications bzr does are file permission changes.
Another thing I found is that you can structure a tree transform so that all the risky stuff happens up-front, making it very unlikely that you'll run into an error that leaves your tree in an inconsistent state. The trick is to do all your creation operations into a temp directory beforehand. Then, when you're applying the transform, you rename-into-place instead of creating.
That means you're not subject to 'disk full' errors or failures in the text merge. It also means that you can safely perform merge operations that make reference to the working tree, like using '.' as BASE or OTHER in a merge. And the simpler you can make the tranform application, the better.
You can even take this a step farther, and defer deletions until you've completeliy updated the working tree. Just rename files into a temporary directory, and then delete it when the working tree has been successfully updated. That should make it relatively easy to roll back to the previous state, should you encounter an error.
Another thing we do before applying the transform is conflict resolution. After all, there may already be files where you want to put them. Or a text merge may need to emit .BASE, .THIS and .OTHER files. Or a merge may produce a directory loop. Etc. Bzr has a set of fairly simple rules for detecting and resolving conflicts, and doesn't try to predict how different conflict resolutions could interact with each other. It just runs the resolver up to 10 times, and if that doesn't fix the problem, it gives up.
But again, with all of this happening before a single move or deletion, there isn't a huge penalty for failure.
One thing I found as I implemented tree transforms in bzr was that the modification step is probably the smallest and most boring. Content changes can be structured as delete/create pairs, so they don't need to be modifications. So the only modifications bzr does are file permission changes.
Another thing I found is that you can structure a tree transform so that all the risky stuff happens up-front, making it very unlikely that you'll run into an error that leaves your tree in an inconsistent state. The trick is to do all your creation operations into a temp directory beforehand. Then, when you're applying the transform, you rename-into-place instead of creating.
That means you're not subject to 'disk full' errors or failures in the text merge. It also means that you can safely perform merge operations that make reference to the working tree, like using '.' as BASE or OTHER in a merge. And the simpler you can make the tranform application, the better.
You can even take this a step farther, and defer deletions until you've completeliy updated the working tree. Just rename files into a temporary directory, and then delete it when the working tree has been successfully updated. That should make it relatively easy to roll back to the previous state, should you encounter an error.
Another thing we do before applying the transform is conflict resolution. After all, there may already be files where you want to put them. Or a text merge may need to emit .BASE, .THIS and .OTHER files. Or a merge may produce a directory loop. Etc. Bzr has a set of fairly simple rules for detecting and resolving conflicts, and doesn't try to predict how different conflict resolutions could interact with each other. It just runs the resolver up to 10 times, and if that doesn't fix the problem, it gives up.
But again, with all of this happening before a single move or deletion, there isn't a huge penalty for failure.
Thursday, March 16, 2006
A satisfying morning's hack
So standards-conformant web sites are mother-and-apple-pie kinda stuff. Everyone knows they're a good thing. The w3c has even been so helpful as to provide a way to validate anything you might have online.
Problem is, a lot of your stuff isn't online. Once you know it's valid, then it'll go online. Sure, the w3c validator will accept file uploads, but if you're doing any kinda dynamic content generation, you have to save the rendered output, and upload that. Every time.
So yesterday, (or was it Tuesday?) I hacked up a validator proxy. It's TurboGears, but it just uses CherryPy. It pulls down a specified URL, which can be a LAN-only URL, shoves it up to the w3c validator, and returns a slightly-rewritten version of the validator's response. For an encore, I hacked up the w3c validation bookmarklet, to teach it to use my proxy.
Validation happens a lot more when it's easy. And it's so validating...
Problem is, a lot of your stuff isn't online. Once you know it's valid, then it'll go online. Sure, the w3c validator will accept file uploads, but if you're doing any kinda dynamic content generation, you have to save the rendered output, and upload that. Every time.
So yesterday, (or was it Tuesday?) I hacked up a validator proxy. It's TurboGears, but it just uses CherryPy. It pulls down a specified URL, which can be a LAN-only URL, shoves it up to the w3c validator, and returns a slightly-rewritten version of the validator's response. For an encore, I hacked up the w3c validation bookmarklet, to teach it to use my proxy.
Validation happens a lot more when it's easy. And it's so validating...
Subscribe to:
Posts (Atom)
