Daniel Nouri's Blog

This weekend, I decided I'd finally make good use of my M-Audio Axiom 25 and wire it up to SuperCollider. What I did to achieve this was write a small Python program that listens to incoming MIDI input and writes OpenSound Control to SuperCollider.

The piece of Python's called midi2sc.py and it's actually become a somewhat high-level library for assigning MIDI controls to output OSC. It makes use of the excellent pyrtmidi and scosc libraries both developed by Patrick Kidd Stinson.

An example of wiring controls to SuperCollider is provided in the module. Imagine you have a SynthDef defined in SuperCollider that looks like this:

SynthDef("funk", {
  arg freq=700, amp=0.7, gate=1, cutoff=20000, rez=1, lfospeed=0;
  // for the full source of the example, see midi2sc.py
}).send(s);

Note that this "funk" synth takes 6 arguments, all of which can be controlled while the synth is playing. To control the cutoff frequency with a MIDI control, we define this:

cutoff_control = AbsoluteControl(
    synthdef, min=300, max=7000, param_name='cutoff')

Then, to assign it to a MIDI command, we register a GroupControl to handle commands from a group of continous controllers:

midi = rtmidi.RtMidiIn()
port = ask_for_port(midi)

midi_in = MidiIn(midi, port,
                 handlers={0Xb0: GroupControl({71: cutoff_control})})

The Essentials of the MIDI protocol explains that the 0xb0 MIDI command means we're dealing with a continuous controller, and 71 happens to be the number of the knob that I'm using.

Adding another control is easy. This adds a turning knob for the resonance:

rez_control = AbsoluteControl(synthdef, min=0.10, max=1.5, param_name='rez')
midi_in.handlers[0xb0].controls[91] = rez_control

There's also examples of Note-On, Note-Off, Pitch bend and Channel Pressure controls in the module.

Thanks to the MIDI standard, midi2sc can take input from all kinds of MIDI controllers, including software sequencers like seq24. In fact, I'm having loads of fun with this particular combination; using seq24 to add and remove loops that I prepared and turning knobs on the keyboard to influence the timbre of the notes played.

Sure, there's also MIDI classes in SuperCollider, but I'm still a relative dork in SuperCollider's language sclang. And I like to pretend that this is more compact and readible (and debuggable) than an implementation in sclang.

After going through part two of this Arduino sound tutorial, I made this little synthesizer.

Click here if you can't see the video.

You can see that it starts out with a constant frequency that can be varied with the potentiometer.

An LFO is added to the frequency once I push the button. Successive button pushes will change the amplitude of the LFO.

While the aforementioned tutorial explains how to connect Arduino with your PC sound card to get sound, I simply plugged in a Piezo buzzer. The advantage being that this synth is totally self-contained and portable, so I could use it in the train (I won't!). The downside is that the audio quality is bad, and it's even worse in the video.

Everything I needed to build this came with the Arduino workshop kit (50 EUR). This includes the Arduino Diecimila itself, potentiometer, buzzer, breadboard, and button.

The button press detection in the Wiring code doesn't work properly. A good implementation would probably need to use interrupt handlers. I ignored this issue for now:

int buttonPin = 7;
int buzzerPin = 8;
int potPin = 2;

void setup() {
  pinMode(buzzerPin, OUTPUT);
  pinMode(buttonPin, INPUT);
}

void inner_loop(int j) {
  digitalWrite(buzzerPin, HIGH);
  delayMicroseconds(j*16 + analogRead(potPin));
  digitalWrite(buzzerPin, LOW);
  delayMicroseconds(j*16 + analogRead(potPin));
}

void loop() {
  int val = digitalRead(buttonPin);
  int factor = 0;

  for (int i = 100; i > 0; i--) {
    inner_loop(i * factor);
  }

  // XXX: This doesn't work very well
  if (digitalRead(buttonPin) != val && val == HIGH) {
    factor++;
    if (factor > 3)
      factor = 0;
  }

  for (int i = 0; i < 100; i++) {
    inner_loop(i * factor);
  }
}

Vanessa and I did some fun experiments lately with Arduino and SuperCollider. We're building an alarm clock that wakes you up with quite an unusual sound...

Read the story here.

I just uploaded a new tutorial to plone.org that explains how to use z3c.form with Plone. z3c.form is the next generation Zope form library and excels through ease of use and very good documentation.

Your feedback is welcome. See also the project's homepage and the discussion on the Plone Product Developers list.

I'm now serving the SuperCollider help files. These are available as part of the SuperCollider source distribution (version 3.2) and are licensed under the GPL. Although these files are also available from SourceForge, Google doesn't seem to index them there.

You can search the documentation here:

Apparently, SuperCollider is no longer available in the official package repositories for recent versions of Debian and Ubuntu, despite the official SuperCollider download page pointing us there. The lack of a Debian package is quite annoying; besides it can easily bugger a rookie's SuperCollider experience on these systems.

So this is the Debian package that I've built using the infrastructure (namely the debian/ directory) that's already in the source distribution:

Download SuperCollider 3.2 deb package for Debian and Ubuntu

Download the .deb package and run sudo dpkg -i supercollider3.deb to install it! If you have a graphical Debian package installer like the GDebi Package installer, you can use that.

I applied two small patches:

• The first allows the Emacs Lisp contained in the distribution to be compiled on my system, with Emacs 23.
• The second deals with an issue with class library paths.

I use Emacs for pretty much everything, that's why the Emacs mode for SC works pretty well for me (although it can be quirky at times...). If you're unfamiliar with Emacs, I suggest you use the fabulous and super easy sced plugin for gedit. The steps for installing sced are a bit cumbersome, but it's totally worth it. Maybe I should try and make a Debian package of sced next.

Update: Ha! A minute after I publish this, I find out that the nice guy behind sced (sorry, I still can't figure out how to write non-ASCII characters here ;-), quite recently released SuperCollider packages for Ubuntu Hardy (which I suppose work for Debian just as well). And his packages look much better than mine. Oh well! :-)

Update: Another update. Our friend behind gedit released a Debian package for gedit. And he's written a short tutorial on how to get started. Go and read it!

We've been working hard recently. And today I'm pleased to announce the first official release of Singing & Dancing, the next-generation newsletter and notification solution for Plone.

Thanks to Thomas Clement Mogensen and all contributors, and kudos to Headnet for their support. And Giuseppe Zizza for making the very cool logo!

Ship it!!

If you're still looking for something cool to do at next week's Sorrento sprint besides lying in the sun and sipping on that Limoncello: Why not join us with Singing & Dancing?

I finally fixed my problem with Gnus. I'm writing this, hoping that it'll maybe help someone out there with the same issue.

The symptom: Gnus checks your IMAP mailboxes forever. The problem I had was with one of my mailbox names: I had a ampersand (&) in there, which had Gnus hanging forever trying to escape it. At least that's what looking at *imap-log* suggested. Activating the logging is as simple as executing (setq imap-log t).

The fix: Change your mailbox's name to something without & and update the .newsrc.eld accordingly. And wait for a proper fix. I'm submitting a bug report now.

Thanks to fakezope2eggs, I can now in my Zope 2 buildout depend on packages like z3c.form and zc.queue. These would otherwise pull in incompatible versions of packages already in the Zope 2 lib/python directory.

Kudos to Jean-Francois!

Edit: It turns out that the order of the parts in your buildout is very important. Make sure that the fakezope2eggs part comes right after the zope2 part, and before your Zope 2 instance part!

This project is frustrating the hell out of me.

If you want to customize your Plone site, don't do this:

• Go around wildly in your Products folder in customization frenzy and patch files like there's no tomorrow; and not even write down what you changed where. Monkey patches are a hundred times better than this!
• Make customizations in the ZODB that your filesystem code depends on, so that you can't use the filesystem code anymore without that exact same database. Have fun reproducing your environment!

Please, these things are totally disastrous. You might just as well throw your Plone site out the window!

Gaah!!

(Thanks to Florian Schulze for being an inspiration to the title for this blog post.)

Will McGugan recently blogged about his timed caching decorator. And today I found out about gocept.cache, which allows you to memoize a function for a given amount of seconds.

This is how you would cache a function's value for an hour using plone.memoize:

>>> from time import time
>>> from plone.memoize import ram
>>> @ram.cache(lambda *args: time() // (60 * 60))
... def fun():
...     return "Something that takes awfully long"

In the same way we're making the cache depend on time here, we could have it depend on anything else, like certain arguments provided to the function. So, really, we can do more than just memoization with plone.memoize; an example is this time-out application.

See the plone.memoize docs for more examples.

I attended the SuperCollider Symposium 2007 in The Hague last week, which was fantastic. So many interesting people with their interesting projects gathered in one place! Too bad I missed out the last part of the symposium because of a cold.

Dan Stowell blogged about the symposium. There's some other cool things to discover on his page, like his Evolutionary Sound System (which he developed with SuperCollider), and him getting his hair cut.

Gregorio Garcia Karman took some nice photos at the conference.

You can learn more about SuperCollider on the SC home page and on Wikipedia. You might find these links particularly useful:

• SuperCollider on Ubuntu Linux, which describes how to install SC from subversion and how to install the sced SC editor for Linux, which you might find easier to use than the Emacs bindings.
• The ixi SuperCollider tutorial is a concise resource that will get you started quickly if you're already a programmer.
• If you speak German, there are excellent lecture notes from Hanns Rutz to get started.
• A list of tutorials on the SC wiki. Get David Cottle's book!
• And finally, my list of supercollider links.

After finding out about plone.recipe.bundlecheckout yesterday, I thought I'd mention infrae.subversion, a similar recipe for zc.buildout. These are the key differences:

• p.r.bundlecheckout allows you to specify one URL per part, whereas with infrae.subversion you can specify a list of URLs. Why does this matter? Because it helps you keep the exact URLs and therefore versions of the components that you use in your buildout configuration, which is better than keeping them in a svn:externals property.

• infrae.subversion takes care not to wipe any changes that you might have done in the checkout. That is, you can safely use its checkouts for development.

  Why not instead make a separate products directory and use svn:externals for development? Because again, we want to keep all dependencies in the buildout configuration. And it's good to keep the development buildout as close as possible to the deployment one, to minimize the chance of error. With infrae.subversion, you also have the advantage of being able to run bin/buildout and have all dependencies updated, instead of having to run svn up in some directory manually, which is a source of confusion.

• p.r.bundlecheckout works with both SVN and CVS, while infrae.subversion only works with SVN.

The Silva buildout is an example of a buildout that uses infrae.subversion.

[my-products]
recipe = infrae.subversion
urls = 
  https://svn.plone.org/svn/collective/FCKeditor/trunk FCKeditor
  ...

[prep-fckeditor]
recipe = plone.recipe.command
command =
    ${buildout:executable} ${buildout:directory}/parts/my-products/FCKeditor/utils/base2zope.py
update-command = ${prep-fckeditor:command}

If you need to speed up your Python application, you should take a look at plone.memoize, in particular its new cache decorator that's really easy to use:

>>> from plone.memoize.ram import cache
>>> def cache_key(fun, first, second):
...     return (first, second)
>>> @cache(cache_key)
... def pow(first, second):
...     print 'Someone or something called me'
...     return first ** second

>>> pow(3, 2)
Someone or something called me
9
>>> pow(3, 2)
9
>>> pow(3, 3)
Somone or something called me
27

You can see that the cache key function determines which input values result in the same output and thus can be cached. Cache key functions receive exactly the same arguments as the functions that they cache, plus the function itself. Thus, key functions for methods can also use self for determining a cache key. A cache key function may raise the DontCache exception to signal that no caching should take place.

The cache storage backend can be freely chosen using the second argument to the cache decorator:

>>> @cache(cache_key, cache_storage)

where cache_storage is a function that again takes all arguments of the original function and returns a dict-like object for use as a cache storage. plone.memoize has built-in support for memcached and zope.app.cache as storages.

See the doctests for volatile.py for more examples.

For now, you'll need to use the SVN version. A release containing the cache decorator will follow soon.

Tom thought it'd be a good idea to write down what we were up to at the PIKtipi sprint (2nd to 7th of June), so here's my wrap-up:

PIKtipi was my third Plone sprint so far in 2007. I had an excellent, and productive time, thanks to all organizers. Here's what I did:

• Develop further plone.memoize and make use of it in the portlets implementation of Plone 3. Together with Hanno, we were able to speed up Plone for anonymous users by about 50%, and for logged in users by about a third. Stefan helped me implement the memcached support.
• Fix some bugs for the 3.0 release of Plone. By the way, if you're one of the 145 Plone providers earning money with Plone, you should consider helping out. Plone 3 will have loads of new features, but that also means that the bugtracker needs increased attention to get all these features stable.
• Have a fun time with fellow Plone developers, and attend the DZUG conference that was running in parallel.
• In Berlin, run into a 3 hour DJ set by Ellen Allien by accident. Thanks to Siebo and Saskia for their great company. :-)

My iPod shuffle recently broke completely, a couple of days after the warranty had expired. I'm a bit angry I had to trash it so early. Apparently, I'm not the only one having trouble with the iPod shuffle. Also, while trying to get support for my problem I've found that Apple's customer service sucks.

So, I needed to replace the iPod with something else. I noticed that they were dumping the same shuffle model that I had (first generation) at the electronics store for 30 EUR. I resisted, and went for the somewhat similar Samsung YP-U2 for 70 EUR.

And the Samsung is really nice so far:

• it has a display with very useful configuration options, unlike the shuffle,
• it has a nice equalizer with many presets, unlike the shuffle,
• it's a USB mass storage device that lets you drop your music right in, unlike the shuffle, which requires you to jump through hoops to have transfer of music working without using iTunes,
• it's got USB that you can directly plug in, no need for an adapter, like with the shuffle,
• it supports Ogg Vorbis, unlike the shuffle. Although I've kinda given up on Vorbis quite a while ago after realizing that my MP3 player back then wouldn't support it, it's nice to see that Samsung apparently cares and supports this free format.

Five Easy Pieces - Simple Python Non-Patterns from Alex Martelli was something that I had tagged toread for quite a while now, and it turned out to be a very interesting read.

The most important point that Martelli makes in this paper is that in Python, some of the traditional software design patterns don't apply, because the Python programmer has facilities that other OO languages lack, the lack of which makes certain patterns necessary in other languages that seem superfluous in Python.

It's nice how Martelli puts context into this topic, by explaining some of the background of patterns. Most notably, he references a book called A Pattern Language - Towns, Buildings, Construction by Christopher Alexander and a paper called The Structure of Pattern Languages by Nikos Salingaros, which is available online.

There's quite a nice analogy between software patterns and architectural design patterns. The dictionary of design patterns of Christopher Alexander lists examples for patterns found in architecture like SMALL PARKING LOTS, for which the summarizing statement is Vast parking lots wreck the land for people.

Interestingly, both Alexander and Salingaros are harsh critics of modern architecture. Salingaros writes:

Architecture has changed in this century from being a trade serving humanity with comfortable and useful structures, to an art that serves primarily as a vehicle for self-expression for the architect. In the current architectural paradigm, the emotional and physical comfort of the user are of only minor importance.

Well, this reminds me of software architects who care more about their design than users. The Big Design Up Front page in the PPR has some nice quotes and statements that go with this.

It makes sense to me when Martelli explains how some patterns are unnecessary in Python and others that tend to be involved in other languages are very easy in Python. Need a Singleton? Just create an instance of a class and have clients access that instance by its name. Need a registry? Make a module global instance of a list or dict, and have append and remove be your register and unregister functions. In Eby's article Python is not Java, one reads about a software project:

So, the sad thing is that these poor folks [a team of former Java developers who were new to Python] worked much, much harder than they needed to, in order to produce much more code than they needed to write, that then performs much more slowly than the equivalent idiomatic Python would.

Simplicity is the key to quality, and simple definitely beats complex. Test-first is probably a good way to create simple designs. The KISS principle represents:

If you have a choice between a simple solution and a complex one, then opting for complexity would be stupid.

And:

All design should be as simple as possible, but no simpler. [...] the more elegant designs are usually the more simple ones. Simple also does not mean quick and dirty. In fact, it often takes a lot of thought and work over multiple iterations to simplify.

Python code is by nature simpler than code in many other languages, because it lets you focus on your actual problem -- making simplicity a built-in feature, and apparent that the source code is the design. Finally, a nice quote from Kent Beck found in the PPR:

As a consultant, 80% of the time my job involves taking out premature abstraction so we can get the damned thing out the door.

For me, the PPR is an enormous repository of insightful ideas about software development. I've already emptied my cup and I'm surfing for my next zen slap! :-)

If you've tried the eXtremeManagement tool (or: XM) before, you may have noticed that it's not exactly an interaction design masterpiece. Doing a simple thing like booking your hours for the last week easily took you a few dozen clicks, and that with a web application that takes its time to respond. Getting an overview over your iteration (XM's loosely based on Extreme Programming ideas) used to be hard too. Because of the hierarchical way XM lays out iterations, getting from an project overview to a task that you were interested in would easily involve three clicks. And that's for getting an overview. What if you forgot the details about the task when you were back on the overview page? Considering that a project easily spans half a dozen stories per iteration, this was an overview nightmare.

At Jazkarta, we've been thinking a lot about what the right project management tool would be for us. We were fed up with XM and its sluggishness -- one project tried out Trac combined with some booking plug-in, but it turned out that Trac lacked the project management view of things, or that we were unable to produce the right reports, something that the eXtremeManagement tool is considered to do well out of the box. We looked at other projects and services on the web, but it looked like XM, despite being sluggish, was still the best thing there was. (If I had my way, we'd try to strip XP to the bones first, not thinking about existing tools, and try to understand the process more before we try to modify it.)

So, we looked at improving the tool. XM's trunk now features a simple expand button in the iteration view that lets you drill down to tasks. That is, when you're looking at the listing of stories, you can click an arrow to see the list of associated tasks inline, without any loading time. Malthe, a fellow from Jazkarta, thinks it's cool:

<malthe>  Oh my god! What happened to XM?
<malthe>  It's a UI revolution!

Another missing link in XM used to be the tracker integration. When your task was about fixing a bug in the tracker, your XM task would have no notion of the Poi issue that you were fixing (-- Poi is the bugtracker that we use). Consequently, you had to manually take care of linking the one with the other in the respective text fields, and that, of course, is more than dull, and it breaks easily (think moving around tasks in the system) and not doing it meant no good overview of things.

Now, the way I like to work with XM and Poi is that almost everything everything that I do goes into an issue. Because Poi has much better facilities for maintaining a dialogue with the customer, and Poi issues can have a longer lifetime than tasks -- they can be relevant for more than one iteration. (With stories and tasks that are bound to an iteration in XM's model, that's a bit hairy.)

With the new Poi Task on trunk, connecting tasks with issues finally makes sense. A Poi Task can be connected to one or more issues. The view of the Poi Task will show you all the issues that are associated with it, together with their status (unconfirmed, open, closed, etc.). On the other side, an issue will show you the tasks it's associated with. Plus, you now have a one-click way of adding a task for our issue to any open story in your project.

Also, Poi trunk now has auto-linking to other issues and Trac. Like in Trac itself, you can now simply write #123 in your report and get a link to that issue. The same goes for r123 and revisions in Trac.

The next thing my TODO for the XM tool is gtimelog integration. I want to be able to retrieve a task list from a XM project, and book my hours directly from gtimelog. This is by the way how the folks at Infrae have been booking their hours for a while now into their in-house timesheet app. This integration will mean that I'll no longer need to write down my hours into three different places by hand.

So... If you're looking for a project management tool that's flexible and XP, or if you've looked at XM before and you thought it's dull or doesn't give you enough overview, try it (again). I think it's grown to quite a usable solution for project management. By the way, thanks to Zest and Jazkarta for supporting this work.

Two weeks ago, I went to Apenheul, which is not your typical zoo: In Apenheul, apes and monkeys walk around freely. Apparently, the Rotterdam Zoo is trying out the same concept lately, with a Gorilla hanging out in the zoo's restaurant.

For those of you who say they don't like doctests because they don't give you isolation between tests. Here's an example of an integration doctest for Plone that gives you exactly that:

from Testing import ZopeTestCase as ztc
from Products.PloneTestCase import PloneTestCase as ptc
from Products.PloneTestCase.layer import PloneSite

ZOPE_DEPS = ['MyZopeProductDependecy']
PLONE_DEPS = ['MyPloneProduct',
              'MyPloneDependency']

for x in ZOPE_DEPS + PLONE_DEPS:
    ztc.installProduct(x)
ptc.setupPloneSite(products=PLONE_DEPS)

class MyTest:
    def test_one():
        r"""
        Check if one and one is two:

          >>> 1 + 1
          2
        """

    def test_two():
        r"""
        Check if one minus one is zero:

          >>> 1 - 1
          0
        """

def test_suite():
    suite = ztc.ZopeDocTestSuite(test_class=ptc.PloneTestCase)
    suite.layer = PloneSite
    return suite

Look at G. Writing Tests in the (somewhat out of date) The Zope 3 Developer's Book for more inspiration.

Don't do traditional-style unit tests, they're ugly. Is this a matter of test, err taste? I don't think so.

Of course, if you want real isolation, you go for unit tests (i.e. no ZopeTestCase involved). People are usually scared and say, but then I need to set so much up by hand, or, don't make me write a mock object for everything, but those arguments aren't really valid most of the time. This is how easy a pure unit test using doctest for a Plone Product (or any Python program) can look like:

import unittest
from zope.testing import doctest

class Add:
    r"""
    >>> 1 + 1
    2
    """

class Subtract:
    r"""
    >>> 1 - 1
    0
    """

def test_suite():
    return unittest.TestSuite((doctest.DocTestSuite()))

If you have a hard time creating mock objects to test against, try Mocky. This is an example of mocking a CMF site that gives you getPhysicalPath, has a portal_catalog and some special site_properties for use with your content type:

>>> from mocky import Mocky
>>> import Acquisition
>>> class MockySite(Mocky, Acquisition.Explicit):
...     def getPhysicalPath(self):
...         return tuple(self.name.split('.'))
...     def portal_catalog(self, **kwargs):
...         return []
>>> site = MockySite('site')
>>> props = site.portal_properties.site_properties
>>> props.getProperty = lambda x: 'utf-8' # doctest: +ELLIPSIS
Set site.portal_properties.site_properties.getProperty to <function ...>

>>> from Products.MyProduct.content import MyContentType
>>> content = MyContentType('someid').__of__(site)
>>> content.something_that_involves_using_portalcatalog_and_siteproperties()
'your result here'

Update: The python-in-testing list has an interesting discussion about doctest vs. unittest.

I never leave home without these, at least not when I leave home to do development:

• Emacs

  Emacs of course isn't specific to Zope or Python. Nevertheless, Emacs is the editor for doing Zope development. Check out Philipp von Weitershausen demo pdbtrack. You need nothing more than the usual python-mode for this. (If you don't know what pdb is, read this gentle intro to debugging in Python.)

  I hear that other editors can do this too, but it can't possibly work as well. :-P

  Start with Emacs basics right now!

• Firebug

  You've certainly been living under a rock if you haven't heard of Firebug before. Never was trying out and debugging JavaScript so joyful. Firebug has an interactive JavaScript shell that since version 1.0 also has access to the variables in the scope of any breakpoint that you might have set.

• PDBDebugMode

  What it does: It loads the Python Debugger (pdb) whenever Zope encounters an exception that was not filtered in the error_log. In combination with pdbtrack this means: Don't read tracebacks to find out where the exception has occurred and where you might want to hook in to debug. And don't restart after you figured out where to put that pdb.set_trace(). Instead: Have your pdb point to the failing line of code and do your debugging right away. Works similar to --debug-inplace of the Zope testrunner.

  This might be not as useful for development of new applications (think tests) as for situations where you have to debug something quickly, possibly on a live deployment.

• ScriptDebugging

  Before, I totally hated debugging code in Script (Python) objects. That was until I found Lennart Regebro's ScriptDebugging, which helps pdbtrack find out where in the script you are, which means that you can debug scripts just like any other Python code.

• DeadlockDebugger

  Stuck with a Zope that's stuck? If your Zope stops responding, use DeadlockDebugger to find out where exactly Zope's threads are deadlocked. From the author:

  This product adds a hook so that a deadlocked Zope process can be debugged, by dumping a traceback of all running python processes. The dump is sent to the event log (at the DEBUG level) and returned to the browser (even though the Zope is deadlocked and doesn't answer any other requests!).

  DeadlockDebugger can of course also be used to debug Zope in non-deadlock situations, when a Zope process is taking a long time and you wish to know what code is being executed.

What's the Zope 2 tool that you'd take with you on an island?

Luke Slater's party on Second Life is starting now in the Alpha Box. He'll be playing his DJ set in less than an hour!

Last weekend, I went to my first FOSDEM (video recordings). I took my girlfriend with me, which I believe helped her understand my nerdy side better. ;-)

Paul Everitt did a nice presentation on Plone 3, where he showed how you set up a Plone 3 instance using ploneenv, and then an easy example of using KSS, and some other Plone 3 niceties. The ploneenv screencast is available now, and so is the rest of the screencasts that Paul demos at the presentation.

Paul was followed by Dries Buytaert's Drupal presentation, which was also really good. It was more a demo of what Drupal is and what it can do for the end user, as opposed to how you can develop for the platform and some legal background, which Paul concentrated on.

Unfortunately, I missed the OLPC talk, but nevertheless I was able to get my hands on one of those laptops, and they really do look cool.

Elixir, a library built on top of SQLAlchemy that's recently gone into beta, looks like a very convenient way to talk to relational databases.

Amazing how a conference with so many good talks can be totally free. Next FOSDEM? I'll be there...

Update: Added a link to the ploneenv screencast.

This year's Baarn sprint was way cool! And we got a lot of stuff done.

I really like the multi-cultural aspect to Plone sprints. Where else do you get to meet and work face to face with people from Norway, the Netherlands, Scotland, Great Britain, Germany, United States, Hungary, Belgium, Finland and Austria in the course of a couple of days?

Seeing some people that I've never met in real life before was very nice, too. That'll help with communicating with those people through IRC and mailing lists.

Thanks everyone for a fantastic time! I'm totally looking forward to the Sorrento sprint, where over 60 people are expected!

I've updated the experimental Plone 3.0 setuptools distribution and ploneenv today a bit and fixed some bugs along the way.

Here are some small recipes that show you how ploneenv works, and how you can make use of it.

A requirement for all of these recipes is that you have ploneenv installed, which you can install like this:

sudo easy_install ploneenv

The recipes will also assume that you have two shell variables set. One is MKZO, the path to the mkzopeinstance.py script of your Zope 2.10 installation. The other is INSTANCE_HOME, the folder where you want your Zope instance to go into. This is an example of setting those variables:

MKZO=~/lib/Zope-2.10/bin/mkzopeinstance.py
INSTANCE_HOME=~/myplone3.0

ploneenv $INSTANCE_HOME -m $MKZO --no-requirements

source $INSTANCE_HOME/bin/activate
easy_install yolk

yolk -l

easy_install Plone==dev

ploneenv $INSTANCE_HOME -m $MKZO

easy_install ZopeSkel
paster create -t plone mypackage

source $INSTANCE_HOME/bin/activate
cd mypackage
python setup.py develop

Check out this new tutorial that helps you get started quickly with Product development in Plone 2.5 and higher. Why another tutorial? Because I myself needed a simple resource where I could copy and paste the most important snippets from. And because Plone needs more easy and practical get-started tutorials.

Thanks to Infrae and Wim who helped me write it!

Ideas I have to extend this tutorial:

• How to convert the Product into a ZopeSkel template
• How to use a formlib form for the search page

LovelySync is a small and quite flexible library for importing data into the ZODB, the object database that Plone uses. In theory it can be used as a library for anything that needs to be imported into any database (yeah, sure I hear you say). But I've only ever used it with the ZODB as the target database.

One of Lovely's customers needed to synchronize a Filemaker database regularly with their Plone site. The database was quite complex; it had different languages (which we mapped into LinguaPlone content), images, and all kinds of references between the objects. At the time I began working for the project, we didn't know exactly how the export format would look like, so we had to develop something that was easily adjustable: The input format is easily customized using schema files. All the actual writing is delegated to small specialized components called WriteHandlers. Reading is done by the Reader, which supplies the Writer (which is the object delegating to WriteHandlers) with Records. You might want to check out the interfaces file for the technical details.

LovelySync, originally written more than a year ago, but refactored a lot lately, was the first project where I used the Component Architecture of Zope 3 extensively, and quite successfully (and without ZCML ;). Using LovelySync, I've been doing imports for hundreds of members from a CSV file into Plone, screen scraping contents into Plone sites for migration, and even an import of usenet (NNTP) groups into Listen through nntp2listen, which will be in a public SVN shortly.

So if ArcheCSV doesn't exactly do what you want, and PLIP 112 is too far away for you, you might want to try LovelySync.

What a nice Zope Product I found today. Ross Patterson's PDBDebugMode allows you to:

• post-mortem debug any Zope exceptions while Zope is running in foreground,
• debug traversal when you provide the special query parameter pdb_runcall in the URL and
• import pdb in restricted code.

The README has more details.

Yesterday, I watched Guido van Rossum's talk about Python 3000 along with some Monty Python sketches that turned up in my search results page. It was quite interesting to see what Python 3.0 will be and what it will not be.

One thing that occured to me some months ago after I watched the TurboGears Ultimate DVD was that those generic functions (in the form of RuleDispatch) are an alternative to adaptation, as it exists in Zope. That is, you could do everything that you wanted to do with adaptation also with generic functions, and that in an arguably less verbose way. Also, Guido explains in his Google talk that adapters are only a special case of generic functions.

I tried it out. In plone.checksum (which is not that interesting by itself), I'm using a generic function instead of adaptation. I have a class called Checksum that returns an md5 checksum in its do_checksum method:

import dispatch

class Checksum:
    @dispatch.generic()
    def do_checksum(self, value):
        """Return md5 object that has the calculated checksum.
        """

You can see that the generic function doesn't actually have an implementation. It's meant to be overloaded like this:

import md5

@Checksum.do_checksum.when('isinstance(value, object)')
def do_checksum(self, value):
    checksum = md5.new()
    checksum.update(str(value))
    return checksum

This is the most general implementation that I could come up with. It'll be called when the argument value is an instance of object, that is, always, unless there's a more specialized generic function. I needed a different implementation of do_checksum for OFS.Image.File, which was easy to hook in:

import OFS.Image.File

@Checksum.do_checksum.when('isinstance(value, OFS.Image.File)')
def do_checksum(self, value):
    checksum = md5.new()
    value = value.data
    if isinstance(value, str):
        checksum.update(value)
    else:
        while value is not None:
            checksum.update(value.data)
            value = value.next
    return checksum

Interfaces are a generally a good thing, but not in this case where we would have to mark the File class just to serve our adaptation.

Guido blogged about adaptation versus generic functions half a year ago. There are also some nice additional links in there.

Update: I found out that setting the label for your USB device is the less painful way to do this, see below.

The naming of my USB hard disks in Ubuntu differed from time to time, which became more and more annoying recently. When I booted up the system with my external hard disk attached, the disk would be mounted as /media/sdb6. However, when I plugged the disk in when the system was already up and I was logged in, it would mount as /media/usbdisk. This was very nasty, as it broke links and libraries of applications like F-Spot and Quod Libet.

The cure for this problem was to have proper /etc/fstab entries for these devices. I found out about udev, which is the device manager for Linux since 2.16: udev manages the device nodes in /dev. In addition to the usual names like /dev/sdb, udev in Edgy is configured to also create symbolic links that will never change their name, like /dev/disk/by-id/usb-WD_3200JB_External_574D414D5231343637323730-part5 or /dev/disk/by-path/pci-0000:00:1f.2-scsi-0:0:0:0. (/etc/udev/rules.d/65-persistent-storage.rules does this.)

Knowing this, I went and put the right paths into /etc/fstab. The following is an excerpt from my configuration. I have an external WD hard disk with two partitions, one vfat and the other ext3. The last entry mounts a SanDisk USB stick, which uses vfat again:

# USB WD
/dev/disk/by-id/usb-WD_3200JB_External_574D414D5231343637323730-part5 /media/WD_3200JB-part5 ext3 auto,users,defaults 0 0
/dev/disk/by-id/usb-WD_3200JB_External_574D414D5231343637323730-part6 /media/WD_3200JB-part6 vfat rw,nosuid,nodev,quiet,shortname=mixed,auto,users,iocharset=utf8 0 0

# SanDisk
/dev/disk/by-id/usb-SanDisk_U3_Cruzer_Micro_0000060326115761-part1 /media/SanDisk_Cruzer vfat rw,nosuid,nodev,quiet,shortname=mixed,auto,users,iocharset=utf8 0 0

The Gnome Disk Mounter, which I don't use, seems quite confused by this. However, the disks are now automatically mounted with nice names, and I can reliably link into my external disks.

Update: I found out that setting the label for your USB device is the less painful way to do this. E.g. after renaming your FAT32 device to MYDATA, Ubuntu will mount your device at /media/MYDATA. This page explains how to rename your device for all kinds of filesystem types: https://help.ubuntu.com/community/RenameUSBDrive

The Document Library document management system has seen lots of improvements lately. There'll be a new release shortly. We've been busy adding:

• easy installation, using Zope Buildout,
• LDAP support, based on ldappas and ldapadapter,
• filesystem storage support, based on Tramline and hurry.file and
• a more reliable conversion between file formats, provided by the OooConv conversion server, which now has an easy way to install it, too.

Quite a number of different components are involved. And that's only the list of recently added dependencies. Code reuse is good. The working together of these components, however, needs extra care, because components will change over time and we need a way to make sure that the Document Library doesn't break, or that its breakage is apparent when we update its dependencies to newer versions. Testing the Document Library's APIs in isolation isn't enough here. Neither is leaving the testing to the user.

The optional filesystem storage support requires Apache in front of Zope. The conversion mechanism requires a running OooConv process (which in turn controls a number of Open Office worker processes). And then we allow mass uploading of documents using ZIP files, the contents of which may want filesystem storage and conversion, too. This leaves us with quite a number of variables, and lots of ways to screw up.

Manual testing of all of these combinations is tedious. Let's say a human tester prepares a testing session by writing down all the possible combinations he can think of (which is what we did, too). He tests document adding, editing, validation errors and uploading with Tramline and without and conversion with Tramline and without and with ZIP files.

Lots of functionality is cool, but it's also worrying. Will feature A combined with B under circumstance C still work when I change this line of code? Doing all the human testing after we do a change isn't an option. But not doing it feels bad.

The answer to this is to write automated functional tests. For that, we decided to use the Zope Testbrowser (with doctest), which has already proven itself a very useful tool in the past.

Now you might already know that Testbrowser has a mode for testing against any HTTP server. In our case, this mode is the only way to test conversion and filesystem storage, because both of these features require help from external processes.

Before running these Testbrowser tests now, the tester start up Apache and the conversion server and points Testbrowser to Apache. The tests work very similarly to human testing: A Document Library is created through the web browser, documents are added, edited, files are uploaded, mistakes are made to trigger validation error etc. And all that in a well-defined manner which leaves little place for human testers to make errors or forget about a certain test. (They can still forget to run the tests, though. ;)

At this point, I stopped worrying about breakage. And I learned that these functional tests are the bomb.

One thing worth noticing is that in these remote functional tests, we can't use the Zope Testrunner's -D option. This would normally put us (= the debugger) right at the relevant point of the application code where the test failure occurs. However, we can still run Zope in foreground and put breakpoints into the application code, or into the test code. Which means that at any time into the tests we can introspect the state of the application by either looking at variables in the debugger, or by simply browsing the Document Library instance that's being tested in Firefox.

If you want to read more about automated tests, check out this big list of interesting articles and links from Grig Gheorghiu.

Big thanks to Daniel T. Chen a.k.a crimsun on #alsa (freenode) who helped me get audio capture (recording) working on the Sony Vaio VGN-FE31M on Ubuntu Edgy.

Edgy comes with ALSA 1.0.12. What you need to do is get ALSA 1.0.13 driver sources and compile them:

$ wget ftp://ftp.alsa-project.org/pub/driver/alsa-driver-1.0.13.tar.bz2
$ tar xjvf alsa-driver-1.0.13.tar.bz2
$ cd alsa-driver-1.0.13
$ ./configure --with-cards=hda-intel,usb-audio --with-oss=yes \
  --with-sequencer=yes --with-kernel=/lib/modules/`uname -r`/build \
  --with-debug=full
$ make && sudo make install-modules && sudo depmod -e

Now reboot and have sound recording working!

For turning on and off the internal mic you must run alsamixer, then in the catpure view select Line and hit space. The letters CAPTUR should appear in red above the word Line. If you want to activate the internal microphone, select Mic and hit the space bar. (Thanks to gnubien for explaining this!)

A feature that I missed in Metacity (the default Window Manager of GNOME nowadays) since my old FVWM days is focus follows mouse. This is easily configured in Metacity as I found out today:

gconf-editor /apps/metacity/general/focus_mode

Set this to sloppy and your windows are focused whenever your mouse enters them.

While you're at it, you might want to also check auto_raise and set auto_raise_delay to 0, so that the focused windows raise to the front.

Update: Jasper just tells me that you can configure the same by going to System -> Preferences -> Windows in the Gnome menu.

Our prachtig cat, Sultan.

Yesterday I found an excellent tutorial for setting up Jack on Ubuntu. I read on the Jack homepage that it also runs on Mac OS X. So this might even be interesting for the I have to have a Mac because... umm, well... because types among us.

What is Jack for? It's a low-latency audio server. And it's for connecting audio and midi signals between programs. With Jack, you can have a virtual midi keyboard drive a sequencer and have the sequencer drive a synthesizer and a drum machine, for example. Actually, that's exactly the setup that this tutorial describes.

I figure I should only start up Jack when I make music. Because I listen to music with amaroK, which cannot output to Jack. I can imagine lots of my other everyday applications don't output to Jack. But that doesn't hurt. qjackctl is a tool for starting up Jack and configuring it. It comes with a status icon in the tray bar of which the background color turns red if you had a buffer overflow or underflow, which is useful for finding out which JACK settings work for you: The more performant your system and your soundcard, the less latency you can have. Latency is the time between playing a note and hearing it.

The last time I set up Jack was on a Debian box. The memories I have from the Jack installation on that box are that it took me quite a while to figure out things. Hence, I was hesitating but now I finally installed Jack on this Ubuntu box, and it was a snap.

After reading the beforementioned tutorial, seq24 finally starts to make sense to me. I tried seq24 before, but frankly, I didn't know what to do with it. Its user interface scared me away.

seq24 is a sequencer, as you might have guessed. It has a live mode that is very similar to what Ableton Live has: While your (MIDI) loops are playing (which drive other synthesizer or drum machine software -- you get the idea) you can switch between them by mouseclick. You can have the switch happen in such a way that the previously selected loop continues playing until it comes to the end after which your new loop starts. This mode of play is very nice for experimenting and finding out which of your loops sound good together.

All I need now is get into a creative mood... (waiting)

Update: The old Ubuntu Studio wiki was merged with the Community Ubuntu Documentation for Sound.