This weekend, I decided I'd finally make good use of my M-Audio Axiom 25 and wire it up to SuperCollider. What I did to achieve this was write a small Python program that listens to incoming MIDI input and writes OpenSound Control to SuperCollider.

The piece of Python's called midi2sc.py and it's actually become a somewhat high-level library for assigning MIDI controls to output OSC. It makes use of the excellent pyrtmidi and scosc libraries both developed by Patrick Kidd Stinson.

An example of wiring controls to SuperCollider is provided in the module. Imagine you have a SynthDef defined in SuperCollider that looks like this:

SynthDef("funk", {
  arg freq=700, amp=0.7, gate=1, cutoff=20000, rez=1, lfospeed=0;
  // for the full source of the example, see midi2sc.py
}).send(s);

Note that this "funk" synth takes 6 arguments, all of which can be controlled while the synth is playing. To control the cutoff frequency with a MIDI control, we define this:

cutoff_control = AbsoluteControl(
    synthdef, min=300, max=7000, param_name='cutoff')

Then, to assign it to a MIDI command, we register a GroupControl to handle commands from a group of continous controllers:

midi = rtmidi.RtMidiIn()
port = ask_for_port(midi)

midi_in = MidiIn(midi, port,
                 handlers={0Xb0: GroupControl({71: cutoff_control})})

The Essentials of the MIDI protocol explains that the 0xb0 MIDI command means we're dealing with a continuous controller, and 71 happens to be the number of the knob that I'm using.

Adding another control is easy. This adds a turning knob for the resonance:

rez_control = AbsoluteControl(synthdef, min=0.10, max=1.5, param_name='rez')
midi_in.handlers[0xb0].controls[91] = rez_control

There's also examples of Note-On, Note-Off, Pitch bend and Channel Pressure controls in the module.

Thanks to the MIDI standard, midi2sc can take input from all kinds of MIDI controllers, including software sequencers like seq24. In fact, I'm having loads of fun with this particular combination; using seq24 to add and remove loops that I prepared and turning knobs on the keyboard to influence the timbre of the notes played.

Sure, there's also MIDI classes in SuperCollider, but I'm still a relative dork in SuperCollider's language sclang. And I like to pretend that this is more compact and readible (and debuggable) than an implementation in sclang.

After going through part two of this Arduino sound tutorial, I made this little synthesizer.

Click here if you can't see the video.

You can see that it starts out with a constant frequency that can be varied with the potentiometer.

An LFO is added to the frequency once I push the button. Successive button pushes will change the amplitude of the LFO.

While the aforementioned tutorial explains how to connect Arduino with your PC sound card to get sound, I simply plugged in a Piezo buzzer. The advantage being that this synth is totally self-contained and portable, so I could use it in the train (I won't!). The downside is that the audio quality is bad, and it's even worse in the video.

Everything I needed to build this came with the Arduino workshop kit (50 EUR). This includes the Arduino Diecimila itself, potentiometer, buzzer, breadboard, and button.

The button press detection in the Wiring code doesn't work properly. A good implementation would probably need to use interrupt handlers. I ignored this issue for now:

int buttonPin = 7;
int buzzerPin = 8;
int potPin = 2;

void setup() {
  pinMode(buzzerPin, OUTPUT);
  pinMode(buttonPin, INPUT);
}

void inner_loop(int j) {
  digitalWrite(buzzerPin, HIGH);
  delayMicroseconds(j*16 + analogRead(potPin));
  digitalWrite(buzzerPin, LOW);
  delayMicroseconds(j*16 + analogRead(potPin));
}

void loop() {
  int val = digitalRead(buttonPin);
  int factor = 0;

  for (int i = 100; i > 0; i--) {
    inner_loop(i * factor);
  }

  // XXX: This doesn't work very well
  if (digitalRead(buttonPin) != val && val == HIGH) {
    factor++;
    if (factor > 3)
      factor = 0;
  }

  for (int i = 0; i < 100; i++) {
    inner_loop(i * factor);
  }
}

Vanessa and I did some fun experiments lately with Arduino and SuperCollider. We're building an alarm clock that wakes you up with quite an unusual sound...

Read the story here.

I just uploaded a new tutorial to plone.org that explains how to use z3c.form with Plone. z3c.form is the next generation Zope form library and excels through ease of use and very good documentation.

Your feedback is welcome. See also the project's homepage and the discussion on the Plone Product Developers list.

We've been working hard recently. And today I'm pleased to announce the first official release of Singing & Dancing, the next-generation newsletter and notification solution for Plone.

Thanks to Thomas Clement Mogensen and all contributors, and kudos to Headnet for their support. And Giuseppe Zizza for making the very cool logo!

Ship it!!

If you're still looking for something cool to do at next week's Sorrento sprint besides lying in the sun and sipping on that Limoncello: Why not join us with Singing & Dancing?

Thanks to fakezope2eggs, I can now in my Zope 2 buildout depend on packages like z3c.form and zc.queue. These would otherwise pull in incompatible versions of packages already in the Zope 2 lib/python directory.

Kudos to Jean-Francois!

Edit: It turns out that the order of the parts in your buildout is very important. Make sure that the fakezope2eggs part comes right after the zope2 part, and before your Zope 2 instance part!

This project is frustrating the hell out of me.

If you want to customize your Plone site, don't do this:

• Go around wildly in your Products folder in customization frenzy and patch files like there's no tomorrow; and not even write down what you changed where. Monkey patches are a hundred times better than this!
• Make customizations in the ZODB that your filesystem code depends on, so that you can't use the filesystem code anymore without that exact same database. Have fun reproducing your environment!

Please, these things are totally disastrous. You might just as well throw your Plone site out the window!

Gaah!!

(Thanks to Florian Schulze for being an inspiration to the title for this blog post.)

Will McGugan recently blogged about his timed caching decorator. And today I found out about gocept.cache, which allows you to memoize a function for a given amount of seconds.

This is how you would cache a function's value for an hour using plone.memoize:

>>> from time import time
>>> from plone.memoize import ram
>>> @ram.cache(lambda *args: time() // (60 * 60))
... def fun():
...     return "Something that takes awfully long"

In the same way we're making the cache depend on time here, we could have it depend on anything else, like certain arguments provided to the function. So, really, we can do more than just memoization with plone.memoize; an example is this time-out application.

See the plone.memoize docs for more examples.

After finding out about plone.recipe.bundlecheckout yesterday, I thought I'd mention infrae.subversion, a similar recipe for zc.buildout. These are the key differences:

• p.r.bundlecheckout allows you to specify one URL per part, whereas with infrae.subversion you can specify a list of URLs. Why does this matter? Because it helps you keep the exact URLs and therefore versions of the components that you use in your buildout configuration, which is better than keeping them in a svn:externals property.

• infrae.subversion takes care not to wipe any changes that you might have done in the checkout. That is, you can safely use its checkouts for development.

  Why not instead make a separate products directory and use svn:externals for development? Because again, we want to keep all dependencies in the buildout configuration. And it's good to keep the development buildout as close as possible to the deployment one, to minimize the chance of error. With infrae.subversion, you also have the advantage of being able to run bin/buildout and have all dependencies updated, instead of having to run svn up in some directory manually, which is a source of confusion.

• p.r.bundlecheckout works with both SVN and CVS, while infrae.subversion only works with SVN.

The Silva buildout is an example of a buildout that uses infrae.subversion.

[my-products]
recipe = infrae.subversion
urls = 
  https://svn.plone.org/svn/collective/FCKeditor/trunk FCKeditor
  ...

[prep-fckeditor]
recipe = plone.recipe.command
command =
    ${buildout:executable} ${buildout:directory}/parts/my-products/FCKeditor/utils/base2zope.py
update-command = ${prep-fckeditor:command}

If you need to speed up your Python application, you should take a look at plone.memoize, in particular its new cache decorator that's really easy to use:

>>> from plone.memoize.ram import cache
>>> def cache_key(fun, first, second):
...     return (first, second)
>>> @cache(cache_key)
... def pow(first, second):
...     print 'Someone or something called me'
...     return first ** second

>>> pow(3, 2)
Someone or something called me
9
>>> pow(3, 2)
9
>>> pow(3, 3)
Somone or something called me
27

You can see that the cache key function determines which input values result in the same output and thus can be cached. Cache key functions receive exactly the same arguments as the functions that they cache, plus the function itself. Thus, key functions for methods can also use self for determining a cache key. A cache key function may raise the DontCache exception to signal that no caching should take place.

The cache storage backend can be freely chosen using the second argument to the cache decorator:

>>> @cache(cache_key, cache_storage)

where cache_storage is a function that again takes all arguments of the original function and returns a dict-like object for use as a cache storage. plone.memoize has built-in support for memcached and zope.app.cache as storages.

See the doctests for volatile.py for more examples.

For now, you'll need to use the SVN version. A release containing the cache decorator will follow soon.

Tom thought it'd be a good idea to write down what we were up to at the PIKtipi sprint (2nd to 7th of June), so here's my wrap-up:

PIKtipi was my third Plone sprint so far in 2007. I had an excellent, and productive time, thanks to all organizers. Here's what I did:

• Develop further plone.memoize and make use of it in the portlets implementation of Plone 3. Together with Hanno, we were able to speed up Plone for anonymous users by about 50%, and for logged in users by about a third. Stefan helped me implement the memcached support.
• Fix some bugs for the 3.0 release of Plone. By the way, if you're one of the 145 Plone providers earning money with Plone, you should consider helping out. Plone 3 will have loads of new features, but that also means that the bugtracker needs increased attention to get all these features stable.
• Have a fun time with fellow Plone developers, and attend the DZUG conference that was running in parallel.
• In Berlin, run into a 3 hour DJ set by Ellen Allien by accident. Thanks to Siebo and Saskia for their great company. :-)

Five Easy Pieces - Simple Python Non-Patterns from Alex Martelli was something that I had tagged toread for quite a while now, and it turned out to be a very interesting read.

The most important point that Martelli makes in this paper is that in Python, some of the traditional software design patterns don't apply, because the Python programmer has facilities that other OO languages lack, the lack of which makes certain patterns necessary in other languages that seem superfluous in Python.

It's nice how Martelli puts context into this topic, by explaining some of the background of patterns. Most notably, he references a book called A Pattern Language - Towns, Buildings, Construction by Christopher Alexander and a paper called The Structure of Pattern Languages by Nikos Salingaros, which is available online.

There's quite a nice analogy between software patterns and architectural design patterns. The dictionary of design patterns of Christopher Alexander lists examples for patterns found in architecture like SMALL PARKING LOTS, for which the summarizing statement is Vast parking lots wreck the land for people.

Interestingly, both Alexander and Salingaros are harsh critics of modern architecture. Salingaros writes:

Architecture has changed in this century from being a trade serving humanity with comfortable and useful structures, to an art that serves primarily as a vehicle for self-expression for the architect. In the current architectural paradigm, the emotional and physical comfort of the user are of only minor importance.

Well, this reminds me of software architects who care more about their design than users. The Big Design Up Front page in the PPR has some nice quotes and statements that go with this.

It makes sense to me when Martelli explains how some patterns are unnecessary in Python and others that tend to be involved in other languages are very easy in Python. Need a Singleton? Just create an instance of a class and have clients access that instance by its name. Need a registry? Make a module global instance of a list or dict, and have append and remove be your register and unregister functions. In Eby's article Python is not Java, one reads about a software project:

So, the sad thing is that these poor folks [a team of former Java developers who were new to Python] worked much, much harder than they needed to, in order to produce much more code than they needed to write, that then performs much more slowly than the equivalent idiomatic Python would.

Simplicity is the key to quality, and simple definitely beats complex. Test-first is probably a good way to create simple designs. The KISS principle represents:

If you have a choice between a simple solution and a complex one, then opting for complexity would be stupid.

And:

All design should be as simple as possible, but no simpler. [...] the more elegant designs are usually the more simple ones. Simple also does not mean quick and dirty. In fact, it often takes a lot of thought and work over multiple iterations to simplify.

Python code is by nature simpler than code in many other languages, because it lets you focus on your actual problem -- making simplicity a built-in feature, and apparent that the source code is the design. Finally, a nice quote from Kent Beck found in the PPR:

As a consultant, 80% of the time my job involves taking out premature abstraction so we can get the damned thing out the door.

For me, the PPR is an enormous repository of insightful ideas about software development. I've already emptied my cup and I'm surfing for my next zen slap! :-)

If you've tried the eXtremeManagement tool (or: XM) before, you may have noticed that it's not exactly an interaction design masterpiece. Doing a simple thing like booking your hours for the last week easily took you a few dozen clicks, and that with a web application that takes its time to respond. Getting an overview over your iteration (XM's loosely based on Extreme Programming ideas) used to be hard too. Because of the hierarchical way XM lays out iterations, getting from an project overview to a task that you were interested in would easily involve three clicks. And that's for getting an overview. What if you forgot the details about the task when you were back on the overview page? Considering that a project easily spans half a dozen stories per iteration, this was an overview nightmare.

At Jazkarta, we've been thinking a lot about what the right project management tool would be for us. We were fed up with XM and its sluggishness -- one project tried out Trac combined with some booking plug-in, but it turned out that Trac lacked the project management view of things, or that we were unable to produce the right reports, something that the eXtremeManagement tool is considered to do well out of the box. We looked at other projects and services on the web, but it looked like XM, despite being sluggish, was still the best thing there was. (If I had my way, we'd try to strip XP to the bones first, not thinking about existing tools, and try to understand the process more before we try to modify it.)

So, we looked at improving the tool. XM's trunk now features a simple expand button in the iteration view that lets you drill down to tasks. That is, when you're looking at the listing of stories, you can click an arrow to see the list of associated tasks inline, without any loading time. Malthe, a fellow from Jazkarta, thinks it's cool:

<malthe>  Oh my god! What happened to XM?
<malthe>  It's a UI revolution!

Another missing link in XM used to be the tracker integration. When your task was about fixing a bug in the tracker, your XM task would have no notion of the Poi issue that you were fixing (-- Poi is the bugtracker that we use). Consequently, you had to manually take care of linking the one with the other in the respective text fields, and that, of course, is more than dull, and it breaks easily (think moving around tasks in the system) and not doing it meant no good overview of things.

Now, the way I like to work with XM and Poi is that almost everything everything that I do goes into an issue. Because Poi has much better facilities for maintaining a dialogue with the customer, and Poi issues can have a longer lifetime than tasks -- they can be relevant for more than one iteration. (With stories and tasks that are bound to an iteration in XM's model, that's a bit hairy.)

With the new Poi Task on trunk, connecting tasks with issues finally makes sense. A Poi Task can be connected to one or more issues. The view of the Poi Task will show you all the issues that are associated with it, together with their status (unconfirmed, open, closed, etc.). On the other side, an issue will show you the tasks it's associated with. Plus, you now have a one-click way of adding a task for our issue to any open story in your project.

Also, Poi trunk now has auto-linking to other issues and Trac. Like in Trac itself, you can now simply write #123 in your report and get a link to that issue. The same goes for r123 and revisions in Trac.

The next thing my TODO for the XM tool is gtimelog integration. I want to be able to retrieve a task list from a XM project, and book my hours directly from gtimelog. This is by the way how the folks at Infrae have been booking their hours for a while now into their in-house timesheet app. This integration will mean that I'll no longer need to write down my hours into three different places by hand.

So... If you're looking for a project management tool that's flexible and XP, or if you've looked at XM before and you thought it's dull or doesn't give you enough overview, try it (again). I think it's grown to quite a usable solution for project management. By the way, thanks to Zest and Jazkarta for supporting this work.

For those of you who say they don't like doctests because they don't give you isolation between tests. Here's an example of an integration doctest for Plone that gives you exactly that:

from Testing import ZopeTestCase as ztc
from Products.PloneTestCase import PloneTestCase as ptc
from Products.PloneTestCase.layer import PloneSite

ZOPE_DEPS = ['MyZopeProductDependecy']
PLONE_DEPS = ['MyPloneProduct',
              'MyPloneDependency']

for x in ZOPE_DEPS + PLONE_DEPS:
    ztc.installProduct(x)
ptc.setupPloneSite(products=PLONE_DEPS)

class MyTest:
    def test_one():
        r"""
        Check if one and one is two:

          >>> 1 + 1
          2
        """

    def test_two():
        r"""
        Check if one minus one is zero:

          >>> 1 - 1
          0
        """

def test_suite():
    suite = ztc.ZopeDocTestSuite(test_class=ptc.PloneTestCase)
    suite.layer = PloneSite
    return suite

Look at G. Writing Tests in the (somewhat out of date) The Zope 3 Developer's Book for more inspiration.

Don't do traditional-style unit tests, they're ugly. Is this a matter of test, err taste? I don't think so.

Of course, if you want real isolation, you go for unit tests (i.e. no ZopeTestCase involved). People are usually scared and say, but then I need to set so much up by hand, or, don't make me write a mock object for everything, but those arguments aren't really valid most of the time. This is how easy a pure unit test using doctest for a Plone Product (or any Python program) can look like:

import unittest
from zope.testing import doctest

class Add:
    r"""
    >>> 1 + 1
    2
    """

class Subtract:
    r"""
    >>> 1 - 1
    0
    """

def test_suite():
    return unittest.TestSuite((doctest.DocTestSuite()))

If you have a hard time creating mock objects to test against, try Mocky. This is an example of mocking a CMF site that gives you getPhysicalPath, has a portal_catalog and some special site_properties for use with your content type:

>>> from mocky import Mocky
>>> import Acquisition
>>> class MockySite(Mocky, Acquisition.Explicit):
...     def getPhysicalPath(self):
...         return tuple(self.name.split('.'))
...     def portal_catalog(self, **kwargs):
...         return []
>>> site = MockySite('site')
>>> props = site.portal_properties.site_properties
>>> props.getProperty = lambda x: 'utf-8' # doctest: +ELLIPSIS
Set site.portal_properties.site_properties.getProperty to <function ...>

>>> from Products.MyProduct.content import MyContentType
>>> content = MyContentType('someid').__of__(site)
>>> content.something_that_involves_using_portalcatalog_and_siteproperties()
'your result here'

Update: The python-in-testing list has an interesting discussion about doctest vs. unittest.

I never leave home without these, at least not when I leave home to do development:

• Emacs

  Emacs of course isn't specific to Zope or Python. Nevertheless, Emacs is the editor for doing Zope development. Check out Philipp von Weitershausen demo pdbtrack. You need nothing more than the usual python-mode for this. (If you don't know what pdb is, read this gentle intro to debugging in Python.)

  I hear that other editors can do this too, but it can't possibly work as well. :-P

  Start with Emacs basics right now!

• Firebug

  You've certainly been living under a rock if you haven't heard of Firebug before. Never was trying out and debugging JavaScript so joyful. Firebug has an interactive JavaScript shell that since version 1.0 also has access to the variables in the scope of any breakpoint that you might have set.

• PDBDebugMode

  What it does: It loads the Python Debugger (pdb) whenever Zope encounters an exception that was not filtered in the error_log. In combination with pdbtrack this means: Don't read tracebacks to find out where the exception has occurred and where you might want to hook in to debug. And don't restart after you figured out where to put that pdb.set_trace(). Instead: Have your pdb point to the failing line of code and do your debugging right away. Works similar to --debug-inplace of the Zope testrunner.

  This might be not as useful for development of new applications (think tests) as for situations where you have to debug something quickly, possibly on a live deployment.

• ScriptDebugging

  Before, I totally hated debugging code in Script (Python) objects. That was until I found Lennart Regebro's ScriptDebugging, which helps pdbtrack find out where in the script you are, which means that you can debug scripts just like any other Python code.

• DeadlockDebugger

  Stuck with a Zope that's stuck? If your Zope stops responding, use DeadlockDebugger to find out where exactly Zope's threads are deadlocked. From the author:

  This product adds a hook so that a deadlocked Zope process can be debugged, by dumping a traceback of all running python processes. The dump is sent to the event log (at the DEBUG level) and returned to the browser (even though the Zope is deadlocked and doesn't answer any other requests!).

  DeadlockDebugger can of course also be used to debug Zope in non-deadlock situations, when a Zope process is taking a long time and you wish to know what code is being executed.

What's the Zope 2 tool that you'd take with you on an island?

Last weekend, I went to my first FOSDEM (video recordings). I took my girlfriend with me, which I believe helped her understand my nerdy side better. ;-)

Paul Everitt did a nice presentation on Plone 3, where he showed how you set up a Plone 3 instance using ploneenv, and then an easy example of using KSS, and some other Plone 3 niceties. The ploneenv screencast is available now, and so is the rest of the screencasts that Paul demos at the presentation.

Paul was followed by Dries Buytaert's Drupal presentation, which was also really good. It was more a demo of what Drupal is and what it can do for the end user, as opposed to how you can develop for the platform and some legal background, which Paul concentrated on.

Unfortunately, I missed the OLPC talk, but nevertheless I was able to get my hands on one of those laptops, and they really do look cool.

Elixir, a library built on top of SQLAlchemy that's recently gone into beta, looks like a very convenient way to talk to relational databases.

Amazing how a conference with so many good talks can be totally free. Next FOSDEM? I'll be there...

Update: Added a link to the ploneenv screencast.

This year's Baarn sprint was way cool! And we got a lot of stuff done.

I really like the multi-cultural aspect to Plone sprints. Where else do you get to meet and work face to face with people from Norway, the Netherlands, Scotland, Great Britain, Germany, United States, Hungary, Belgium, Finland and Austria in the course of a couple of days?

Seeing some people that I've never met in real life before was very nice, too. That'll help with communicating with those people through IRC and mailing lists.

Thanks everyone for a fantastic time! I'm totally looking forward to the Sorrento sprint, where over 60 people are expected!

I've updated the experimental Plone 3.0 setuptools distribution and ploneenv today a bit and fixed some bugs along the way.

Here are some small recipes that show you how ploneenv works, and how you can make use of it.

A requirement for all of these recipes is that you have ploneenv installed, which you can install like this:

sudo easy_install ploneenv

The recipes will also assume that you have two shell variables set. One is MKZO, the path to the mkzopeinstance.py script of your Zope 2.10 installation. The other is INSTANCE_HOME, the folder where you want your Zope instance to go into. This is an example of setting those variables:

MKZO=~/lib/Zope-2.10/bin/mkzopeinstance.py
INSTANCE_HOME=~/myplone3.0

ploneenv $INSTANCE_HOME -m $MKZO --no-requirements

source $INSTANCE_HOME/bin/activate
easy_install yolk

yolk -l

easy_install Plone==dev

ploneenv $INSTANCE_HOME -m $MKZO

easy_install ZopeSkel
paster create -t plone mypackage

source $INSTANCE_HOME/bin/activate
cd mypackage
python setup.py develop

Check out this new tutorial that helps you get started quickly with Product development in Plone 2.5 and higher. Why another tutorial? Because I myself needed a simple resource where I could copy and paste the most important snippets from. And because Plone needs more easy and practical get-started tutorials.

Thanks to Infrae and Wim who helped me write it!

Ideas I have to extend this tutorial:

• How to convert the Product into a ZopeSkel template
• How to use a formlib form for the search page

LovelySync is a small and quite flexible library for importing data into the ZODB, the object database that Plone uses. In theory it can be used as a library for anything that needs to be imported into any database (yeah, sure I hear you say). But I've only ever used it with the ZODB as the target database.

One of Lovely's customers needed to synchronize a Filemaker database regularly with their Plone site. The database was quite complex; it had different languages (which we mapped into LinguaPlone content), images, and all kinds of references between the objects. At the time I began working for the project, we didn't know exactly how the export format would look like, so we had to develop something that was easily adjustable: The input format is easily customized using schema files. All the actual writing is delegated to small specialized components called WriteHandlers. Reading is done by the Reader, which supplies the Writer (which is the object delegating to WriteHandlers) with Records. You might want to check out the interfaces file for the technical details.

LovelySync, originally written more than a year ago, but refactored a lot lately, was the first project where I used the Component Architecture of Zope 3 extensively, and quite successfully (and without ZCML ;). Using LovelySync, I've been doing imports for hundreds of members from a CSV file into Plone, screen scraping contents into Plone sites for migration, and even an import of usenet (NNTP) groups into Listen through nntp2listen, which will be in a public SVN shortly.

So if ArcheCSV doesn't exactly do what you want, and PLIP 112 is too far away for you, you might want to try LovelySync.

What a nice Zope Product I found today. Ross Patterson's PDBDebugMode allows you to:

• post-mortem debug any Zope exceptions while Zope is running in foreground,
• debug traversal when you provide the special query parameter pdb_runcall in the URL and
• import pdb in restricted code.

The README has more details.

Yesterday, I watched Guido van Rossum's talk about Python 3000 along with some Monty Python sketches that turned up in my search results page. It was quite interesting to see what Python 3.0 will be and what it will not be.

One thing that occured to me some months ago after I watched the TurboGears Ultimate DVD was that those generic functions (in the form of RuleDispatch) are an alternative to adaptation, as it exists in Zope. That is, you could do everything that you wanted to do with adaptation also with generic functions, and that in an arguably less verbose way. Also, Guido explains in his Google talk that adapters are only a special case of generic functions.

I tried it out. In plone.checksum (which is not that interesting by itself), I'm using a generic function instead of adaptation. I have a class called Checksum that returns an md5 checksum in its do_checksum method:

import dispatch

class Checksum:
    @dispatch.generic()
    def do_checksum(self, value):
        """Return md5 object that has the calculated checksum.
        """

You can see that the generic function doesn't actually have an implementation. It's meant to be overloaded like this:

import md5

@Checksum.do_checksum.when('isinstance(value, object)')
def do_checksum(self, value):
    checksum = md5.new()
    checksum.update(str(value))
    return checksum

This is the most general implementation that I could come up with. It'll be called when the argument value is an instance of object, that is, always, unless there's a more specialized generic function. I needed a different implementation of do_checksum for OFS.Image.File, which was easy to hook in:

import OFS.Image.File

@Checksum.do_checksum.when('isinstance(value, OFS.Image.File)')
def do_checksum(self, value):
    checksum = md5.new()
    value = value.data
    if isinstance(value, str):
        checksum.update(value)
    else:
        while value is not None:
            checksum.update(value.data)
            value = value.next
    return checksum

Interfaces are a generally a good thing, but not in this case where we would have to mark the File class just to serve our adaptation.

Guido blogged about adaptation versus generic functions half a year ago. There are also some nice additional links in there.

The Document Library document management system has seen lots of improvements lately. There'll be a new release shortly. We've been busy adding:

• easy installation, using Zope Buildout,
• LDAP support, based on ldappas and ldapadapter,
• filesystem storage support, based on Tramline and hurry.file and
• a more reliable conversion between file formats, provided by the OooConv conversion server, which now has an easy way to install it, too.

Quite a number of different components are involved. And that's only the list of recently added dependencies. Code reuse is good. The working together of these components, however, needs extra care, because components will change over time and we need a way to make sure that the Document Library doesn't break, or that its breakage is apparent when we update its dependencies to newer versions. Testing the Document Library's APIs in isolation isn't enough here. Neither is leaving the testing to the user.

The optional filesystem storage support requires Apache in front of Zope. The conversion mechanism requires a running OooConv process (which in turn controls a number of Open Office worker processes). And then we allow mass uploading of documents using ZIP files, the contents of which may want filesystem storage and conversion, too. This leaves us with quite a number of variables, and lots of ways to screw up.

Manual testing of all of these combinations is tedious. Let's say a human tester prepares a testing session by writing down all the possible combinations he can think of (which is what we did, too). He tests document adding, editing, validation errors and uploading with Tramline and without and conversion with Tramline and without and with ZIP files.

Lots of functionality is cool, but it's also worrying. Will feature A combined with B under circumstance C still work when I change this line of code? Doing all the human testing after we do a change isn't an option. But not doing it feels bad.

The answer to this is to write automated functional tests. For that, we decided to use the Zope Testbrowser (with doctest), which has already proven itself a very useful tool in the past.

Now you might already know that Testbrowser has a mode for testing against any HTTP server. In our case, this mode is the only way to test conversion and filesystem storage, because both of these features require help from external processes.

Before running these Testbrowser tests now, the tester start up Apache and the conversion server and points Testbrowser to Apache. The tests work very similarly to human testing: A Document Library is created through the web browser, documents are added, edited, files are uploaded, mistakes are made to trigger validation error etc. And all that in a well-defined manner which leaves little place for human testers to make errors or forget about a certain test. (They can still forget to run the tests, though. ;)

At this point, I stopped worrying about breakage. And I learned that these functional tests are the bomb.

One thing worth noticing is that in these remote functional tests, we can't use the Zope Testrunner's -D option. This would normally put us (= the debugger) right at the relevant point of the application code where the test failure occurs. However, we can still run Zope in foreground and put breakpoints into the application code, or into the test code. Which means that at any time into the tests we can introspect the state of the application by either looking at variables in the debugger, or by simply browsing the Document Library instance that's being tested in Firefox.

If you want to read more about automated tests, check out this big list of interesting articles and links from Grig Gheorghiu.

I'm writing this down because it's annoying me to death: Everytime I do this, I need to learn how to again.

So, Daniel, after you created a tag, get rid of the setup.cfg file that's there and commit. Then register the metadata for this package by running python setup.py register. After that, upload the package: python setup.py sdist bdist_egg upload -s (-s stands for sign). Upload other distribution formats the same way.

If you get this incredibly unhelpful error message, get yourself a pypirc file:

Submitting dist/your.egg to http://www.python.org/pypi
Upload failed (401): Authorization Required

The CheeseShop tutorial has useful pointers.

And next, be annoyed with doing this work again for getting your package into Plone.org's Products. (Sadly, the PloneSoftwareCenter developers decided not to support the PyPI API.)

Yesterday, I got tired with building mock objects for tests. Then I had this idea of creating a universal mock object that would help me with lots of my mocking needs: Mocky was born.

I haven't looked into other mock object libraries, but please feel free and spoil the party by sending me a link to some library where this has been implemented ages ago.

OK, enough words, time for you to take a look at the source (don't be scared, most of it is documentation):

class Mocky(object):
    """Mocky is a class that wants to help you with setting up mock
    objects for your tests.  It helps you observe which functions get
    called (with which parameters) and which attributes are set.

    Unless given a name, a Mocky's name is 'root':

      >>> Mocky().name
      'root'

    Let's start with a simple example that sets some variables so we
    get a feeling of how Mocky works.  Note that attribute accesss
    will never result in AttributeError.  Instead, an attribute access
    to a nonexistent member variable will yield another Mocky
    instance:

      >>> f = Mocky('f')
      >>> f
      f
      >>> unusual = f.unusual
      >>> unusual
      f.unusual
      >>> type(unusual) is Mocky
      True
      >>> unusual is f.unusual
      True
      >>> f.a.c.r = 'Fidelio'
      Set f.a.c.r to 'Fidelio'
      >>> f.a.c.r
      'Fidelio'

    Note that when we set 'f.a.c.r' to 'Fidelio', Mocky printed out
    that the attribute was set.  Suppose we have a function 'fun' that
    sets some fancy variable on a given object:

      >>> def fun(obj):
      ...     if obj.please_process_me:
      ...         obj.there_you = 'go'
      >>> myobj = Mocky('myobj')
      >>> fun(myobj)
      Set myobj.there_you to 'go'
      >>> myobj.please_process_me = False
      Set myobj.please_process_me to False
      >>> fun(myobj)

    Mocky also supports calling.  Another function that does a bit
    more with our test object:

      >>> def starve(character):
      ...     character.getStatus().hitpoints -= 1
      >>> starve(Mocky('Hugo')) # doctest: +ELLIPSIS
      Traceback (most recent call last):
      ...
      TypeError: unsupported operand type(s) for -=: 'Mocky' and 'int'
      >>> ezequiel = Mocky('ezequiel')
      >>> ezequiel.getStatus().hitpoints = 0
      Called ezequiel.getStatus()
      Set ezequiel.getStatus().hitpoints to 0
      >>> starve(ezequiel)
      Called ezequiel.getStatus()
      Set ezequiel.getStatus().hitpoints to -1

    For calls, Mocky will return the same value if the signature is
    the same:

      >>> secret = f.unusual(password='secret')
      Called f.unusual(password='secret')
      >>> secret is f.unusual(password='secret')
      Called f.unusual(password='secret')
      True
      >>> secret is f.unusual(password='unsafe')
      Called f.unusual(password='unsafe')
      False
    """
    def __init__(self, name='root'):
        self.__dict__['name'] = name
        self.__dict__['_calls'] = {}

    def __call__(self, *args, **kwargs):
        argsstr = ', '.join([repr(arg) for arg in args])
        keys = sorted(kwargs.keys())
        kwargsstr = ', '.join(['%s=%r' % (key, kwargs[key]) for key in keys])
        if argsstr and kwargsstr:
            allargs = ', '.join([argsstr, kwargsstr])
        else:
            allargs = argsstr or kwargsstr

        print "Called %s(%s)" % (self.name, allargs)
        if allargs not in self._calls:
            self._calls[allargs] = Mocky('%s(%s)' % (self.name, allargs))
        return self._calls[allargs]

    def __repr__(self):
        return self.name

    def __getattr__(self, name):
        if name not in self.__dict__:
            self.__dict__[name] = Mocky('%s.%s' % (self.name, name))
        return self.__dict__[name]

    def __setattr__(self, name, value):
        print "Set %s.%s to %r" % (self.name, name, value)
        self.__dict__[name] = value

I'm thinking about extending this so it also does dictionary-like access and a quiet mode for when it's obvious that you're setting this and calling that attribute.

I should be getting source code colouring for my blog like Marius. I have to find out how he does it.

I added a new project called pluggablecatalog to the collective.

pluggablecatalog is a replacement (or rather: a wrapper) for Plone's portal catalog. It adds the ability to plug in default search restrictions without the need to subclass or monkey- patch the catalog.

From the docstring of pluggablecatalog/tool.py:

Wraps CMFPlone's CatalogTool to add default parameters
collected from IQueryDefaults utilities.

  >>> from Products.pluggablecatalog.tool import CatalogTool
  >>> catalog = self.portal.portal_catalog
  >>> isinstance(catalog, CatalogTool)
  True

We create two documents and make sure they're indexed:

  >>> len(catalog())
  0
  >>> self.folder.invokeFactory('Document', 'doc1')
  'doc1'
  >>> self.folder.invokeFactory('Document', 'doc2')
  'doc2'
  >>> doc1, doc2 = self.folder.doc1, self.folder.doc2
  >>> doc1.setTitle('First Document')
  >>> doc2.setTitle('Second Document')
  >>> doc1.reindexObject(); doc2.reindexObject()
  >>> len(catalog())
  2

Let's now add a rather stupid IQueryDefaults utility that
restricts searches by default to objects with the Title 'First
Document':

  >>> from zope import component
  >>> from zope import interface
  >>> from Products.pluggablecatalog.interfaces import IQueryDefaults
  >>> def myDefaults(context, request):
  ...     return {'Title': 'First Document'}
  >>> interface.directlyProvides(myDefaults, IQueryDefaults)
  >>> component.getService('Utilities').provideUtility(
  ...     IQueryDefaults, myDefaults)

With this utility in place, we should only retrieve doc1 now,
unless we explicitly provide a 'Title' query parameter:

  >>> len(catalog())
  1
  >>> catalog()[0].getObject().aq_base is doc1.aq_base
  True

  >>> len(catalog(Title='Second Document'))
  1
  >>> (catalog(Title='Second Document')[0].getObject().aq_base is
  ...  doc2.aq_base)
  True

I've just updated the HOWTO for using Testbrowser and Testrecorder with Plone. The Testbrowser/Testrecorder combination is so easy; also for non-programmers. We need more structured and well-documented functional tests like this, because they make our lives easier [*]. Maybe you can help out?

When you get this right, development turns into a very pleasant
cycle of testing, seeing a simple thing to fix, fixing it, testing,
getting positive feedback all the way.

Guaranteed flow. And you go so fast! Try it, you'll like it.

Thanks to Hanno, ZopeSkel now has a template for starting Plone Core packages. Hanno is using ZopeSkel for the new plone.i18n package.

easy_install http://svn.plone.org/svn/collective/ZopeSkel/trunk#egg=ZopeSkel-dev

$ paster create -t plone_core
Selected and implied templates:
  ZopeSkel#plone_core  A Plone Core project

Enter project name: plone.form
Variables:
  package:  ploneform
  project:  plone.form
Creating template plone_core
Enter namespace_package (Namespace package) ['plone']:
Enter package (The package contained namespace package (like i18n)) ['']: form
Enter version (Version) ['0.1']:
Enter description (One-line description of the package) ['']: Plone compatibility layer for zope.app.schema and zope.formlib
Enter long_description (Multi-line description (in reST)) ['']:
Enter author (Author name) ['Plone Foundation']:
Enter author_email (Author email) ['plone-developers@NOSPAMlists.sourceforge.net']:
Enter keywords (Space-separated keywords/tags) ['']:
Enter url (URL of homepage) ['']:
Enter license_name (License name) ['GPL']:
Enter zip_safe (True/False: if the package can be distributed as a .zip file) [False]:
Creating directory ./plone.form
  Recursing into +namespace_package+
    Creating ./plone.form/plone/
    Recursing into +package+
      Creating ./plone.form/plone/form/
      Copying HISTORY.txt_tmpl to ./plone.form/plone/form/HISTORY.txt
      Copying LICENSE.GPL to ./plone.form/plone/form/LICENSE.GPL
      Copying LICENSE.txt_tmpl to ./plone.form/plone/form/LICENSE.txt
      Copying README.txt_tmpl to ./plone.form/plone/form/README.txt
      Copying __init__.py to ./plone.form/plone/form/__init__.py
      Copying configure.zcml to ./plone.form/plone/form/configure.zcml
      Copying version.txt_tmpl to ./plone.form/plone/form/version.txt
    Copying __init__.py to ./plone.form/plone/__init__.py
  Copying setup.cfg to ./plone.form/setup.cfg
  Copying setup.py_tmpl to ./plone.form/setup.py
Running /usr/bin/python setup.py egg_info

$ paster create -t plone_core
Selected and implied templates:
  ZopeSkel#plone_core  A Plone Core project

Enter project name: plone.form
Variables:
  package:  ploneform
  project:  plone.form
Creating template plone_core
Enter namespace_package (Namespace package) ['plone']:
Enter package (The package contained namespace package (like i18n)) ['']: form

[...]

  Recursing into +namespace_package+
    Recursing into +package+
      ./plone.form/plone/form/HISTORY.txt already exists (same content)
      ./plone.form/plone/form/LICENSE.GPL already exists (same content)
      ./plone.form/plone/form/LICENSE.txt already exists (same content)
      ./plone.form/plone/form/README.txt already exists (same content)
      ./plone.form/plone/form/__init__.py already exists (same content)
      ./plone.form/plone/form/configure.zcml already exists (same content)
      ./plone.form/plone/form/version.txt already exists (same content)
    ./plone.form/plone/__init__.py already exists (same content)
  ./plone.form/setup.cfg already exists (same content)
  ./plone.form/setup.py already exists (same content)
Running /usr/bin/python setup.py egg_info

$ paster create -t plone_core --config=plone.form/template_vars.cfg

$ mkdir -p zopeskel/templates/my_package/+namespace_package+/+package+/browser

$ edit zopeskel/templates/my_package/+namespace_package+/+package+/browser/__init__.py_tmpl
$ edit zopeskel/templates/my_package/+namespace_package+/+package+/browser/configure.zcml

class MyPackage(PloneCore):
    _template_dir = 'templates/my_package'
    summary = 'A Plone package that has a browser subpackage'
    required_templates = ['plone_core']

[...]

entry_points="""
[paste.paster_create_template]
basic_zope = zopeskel:BasicZope
plone_core = zopeskel:PloneCore
my_package = zopeskel:MyPackage
""",

[...]

$ python setup.py develop
[...]
$ paster create --list-templates
Available templates:
  basic_package:  A basic setuptools-enabled package
  basic_zope:     A Zope project
  my_package:     A Plone package that has a browser subpackage
  paste_deploy:   A web application deployed through paste.deploy
  plone_core:     A Plone Core project

There's a handy two-line Zope2 Product that allows pdb.set_trace() inside Script (Python) objects. Check it out here:

svn co svn://svn.zope.org/repos/main/Products.enablesettrace/trunk enablesettrace

And then put it into your Products directory. This is only for development, of course.

• tests
• fewer skin customizations and monkeypatches
• more generic code
• motivation
• how to get there

A presentation on how to improve the quality of your Plone development that I gave recently at Zest Software.

For your next Python project, consider using Paste Script. Paste Script helps you set up your project structure quickly so that it's ready for distribution.

easy_install PasteScript

$ paster create
Selected and implied templates:
  PasteScript#basic_package  A basic setuptools-enabled package

Enter project name: MyFancyNewProject
Variables:
  package:  myfancynewproject
  project:  MyFancyNewProject
Creating template basic_package
Enter version (Version (like 0.1)) ['']: 
Enter description (One-line description of the package) ['']: This is a fancy project

...

Creating directory ./MyFancyNewProject
  Recursing into +package+
    Creating ./MyFancyNewProject/myfancynewproject/
    Copying __init__.py to ./MyFancyNewProject/myfancynewproject/__init__.py
  Recursing into +project+.egg-info
    Creating ./MyFancyNewProject/MyFancyNewProject.egg-info/
  Recursing into docs
    Creating ./MyFancyNewProject/docs/
  Copying setup.cfg to ./MyFancyNewProject/setup.cfg
  Copying setup.py_tmpl to ./MyFancyNewProject/setup.py
Running /usr/bin/python setup.py egg_info

$ paster create --svn-repository=http://mysubversionserver/myprojects/

$ paster create --list-templates
Available templates:
  basic_package:  A basic setuptools-enabled package
  basic_zope:     A Zope project
  paste_deploy:   A web application deployed through paste.deploy

My stab at packaging Plone as an egg (this links to the tarball).

easy_install dist/Plone*egg

easy_install --multi-version \
             --install-dir=$INSTANCE_HOME/lib/python/ \
             --site-dirs=$INSTANCE_HOME/lib/python/ \
             dist/Plone*egg

Index: src/pythonproducts/fivedirectives.py
===================================================================
--- src/pythonproducts/fivedirectives.py      (revision 26757)
+++ src/pythonproducts/fivedirectives.py      (working copy)
@@ -16,6 +16,7 @@
 """

 from zope.interface import Interface
+from zope.schema import BytesLine
 from zope.configuration.fields import GlobalObject


@@ -35,4 +36,11 @@
                     u'with a ProductContext instance',
         required=False
         )
-    
\ No newline at end of file
+
+class IRequireDistributionDirective(Interface):
+    """pkg_resources.require() a distribution."""
+
+    name = BytesLine(
+        title=u'Target distribution',
+        required=True,
+        )
Index: src/pythonproducts/meta.zcml
===================================================================
--- src/pythonproducts/meta.zcml      (revision 26757)
+++ src/pythonproducts/meta.zcml      (working copy)
@@ -12,6 +12,12 @@
        handler="Products.pythonproducts.pythonproducts.registerPackage"
        />

+    <meta:directive
+       name="requireDistribution"
+       schema="Products.pythonproducts.fivedirectives.IRequireDistributionDirective"
+       handler="Products.pythonproducts.pythonproducts.requireDistribution"
+       />
+       
   </meta:directives>

 </configure>
Index: src/pythonproducts/pythonproducts.py
===================================================================
--- src/pythonproducts/pythonproducts.py      (revision 26757)
+++ src/pythonproducts/pythonproducts.py      (working copy)
@@ -37,6 +37,13 @@
         args = (package,initialize)
         )

+def requireDistribution(_context, name):
+    """ZCML directive function for requiring a distribution using
+    pkg_resources.
+    """
+    
+    import pkg_resources
+    pkg_resources.require(name)

 def _registerPackage(module_, initFunc=None):
     """Registers the given python package as a Zope 2 style product

<configure xmlns="http://namespaces.zope.org/zope"
           xmlns:five="http://namespaces.zope.org/five">
    <five:requireDistribution name="Plone" />
</configure>