Daniel Nouri's Old Blog

Shane Graber writes me:

Thought you'd be interested in what I did with collective.skinny over the weekend. :) I grabbed an open source theme and whipped it together to see how easy it was and it was really pretty easy. :D

Note the over the weekend!

Here's the screenshot he sent me. You can click on it for a larger size:

Here's what he made it with: collective.skinny together with this theme.

Shane:

One interesting thing in this is that I've always wanted to somehow include smartypants.py and typogrify.py in a theme and with this theme I found it very easy to do.

So I received lots of useful feedback to my last post. I decided to take a look at collective.skinny again, to try and improve its usability. I really think it's super easy to use now. (Alex, if you still don't think so, let me know why ;-).

So if you're doing Plone skinning in your day job and you haven't taken a look at the package yet, now is a good time to do so. Get a download of the package, include it in your buildout, and hack away. You can also browse the source here. Of particular interest might be the main.pt page template, which is where everything starts.

The PyPI page has details on how to get started with this package, along with a screenshot of the example that's provided as part of it.

Oh, and have I mentioned that this way of skinning works in Plone 3, and there's no reason it shouldn't work with Plone 4, too?

So people asked for more details on the supposedly much easier roll your own way of skinning a Plone site in my last post.

I put together an example package called collective.skinny to demonstrate how this could work. You can use this package either as a template to copy and modify, or use it as it is and extend within your own package. Look at the PyPI page for more details.

The package is some 100 lines of Python code with a couple of ZCML statments. As a designer, you'll concentrate on the resources/ and templates/ subdirectories in the package. If you need to extend your browser views with methods, look at base.py and content.py.

To try it out, put collective.skinny in the eggs of your buildout's [instance] part, and add it to the list of zcml includes in the same section. If you look at your Plone site now, you'll see that nothing has changed. To activate the public skin, you'll need to either add a rewrite rule to your proxy as explained in the packages' main configure.zcml file, or activate a subscriber directive that you can find in the same file. This is of course only useful for local development, and even then using a proxy is more ideal, since only then you'll have a way of looking at both the Plone UI and your separate public skin without restarting Zope.

collective.skinny lives in the collective (surprise!). It still has a few rough edges and probably a few bugs. But I'm sure it'll get you started quickly with this easy approach of skinning your Plone site. Let me know what you think of it.

Because Alan mentioned that with this approach your skin will also be much faster, I ran a little benchmark that showed that the standard front page renders about ten times faster than within the Plone skin. Of course the comparison is a bit unfair, since what collective.skinny does is merely get a few attributes from the content object and render them. I don't think that real-world sites have to be much slower than that, though.

Kudos to Sylvain thefunny Viollon for having the idea for the implementation.

Edit: So comments seemed to be temporarily broken on this post. Sorry for that. I really need to switch to Zine. If you have comments and the comments form below doesn't work, please send them to daniel.nouri <at> gmail.com, and I'll put your comment here. I've also made another post describing more improvements to the package.

A few people have already commented:

Alex Limi (who's comment you can actually find below) thinks it's still too hard. Alex, can you tell me what exactly you think is hard?

As to why Deliverance didn't work for us: Since the nature of Deliverance doesn't allow one to add dynamic parts to the site, all the dynamic markup that you can work with is what the Plone UI spews out. That is, if you wanted to display any more data than the Plone default UI gives you (say, a different navigation), you'd have to go back and understand how Plone's UI is put together. Which we decided was too hard for our designer.

David Bain wants some screenshots. Here's one:

Martin Aspeli writes:

I haven't looked at the implementation, but does this not risk "leaking" of the default skin? e.g. what happens if I tack /edit at the end of a URL and you haven't registered an edit view for the context?

Martin, you're right, there's a chance for stuff to leak. I've made some changes to collective.skinny (now: 0.2) so that all pages that the user is not allowed to see will return a 404 instead of the Plone login form. You can still see Plone's UI when you add /view at the end of any URL, though. Some rewriting or reregistering of the main view under those names should help here. I don't think it's a terribly big problem. You don't usually see links to my-content/folder_listing or the like in content.

Michael Mulich writes:

Great job.

This is the best way to make a skin, since it uses the fundamentals of Zope skinning. However, I can't help but notice that collective.skinny does not solve anything for Plone's skinning inefficiencies.

I think this idea would work better if Plone were using zope skinning techniques rather than building off of the CMF's skinning techniques (e.g. SkinsTool). The problem is still that most of the content types people are using are based on ArcheTypes, which is CMF skins based. With zope skinning techniques, which you are using collective.skinny, there is an entry barrier that requires the author to recreate all the content templates, unless ArcheTypes were to take the same approach.

My thoughts on Deliverance are slightly different than most. I love the idea and the ability of Deliverance. However, I believe Deliverance is ignoring Plone's skinning issues. In the end Deliverance is basically candy coating Plone's bitter skinning techniques.

Michael, I think it should be well possible to, in the DefaultView that you can find in the content.py module, fall back to the default view of your content type in Plone (think extracting <div id="content"> from Plone's default HTML). This way one could reuse the standard content view when no specific view for the public skin is available.

I agree with you regarding Deliverance.

Mikko Ohtamaa writes:

Plone 3.0 is very smart piece of software, but it is trapped by its own smartness. It is written by some of the smartest programmers I know, but it they haven't really focused to make things easy for people without the same level of experience.

If we want to make it possible for an average skin jobber to make a Plone skin we need to do it like the other world does out there. This means compromises with the perceived elegancy of the system.

So here is my suggestion.

Let's encapsulate all ZCML, Python, Viewlet, GenericXML, etc. etc. into metacomments of the templates.

You can say:

<html> <!-- @viewlet plone.footer --> <span id="copyright">...

in your template and it will be automatically registered as a footer. No XML or ZCML needed.

TAL has evolved to be a hack ridden piece of unconquerable mountain. nocall: anyone? python:repeat[mymagicvar].end. HTML'ish XML attribute based language may sound good on a paper, but its utterly verbose and rigid nature makes it awful for hand editing. "But we have a Dreamwaver compability!!!1!" Ok, show me a developer who is able to make Plone themes in Dreamwaver.

And for the template language... ask your three friend who work for web design daily what they would want to use. The answer won't please the ear of a hard core software designer, but it is the only right answer if we want to get Plone popularity out of the well of miserably. And it is a decend language if its used as a template language:

I think we should start consider using PHP for Plone templates.

Kamon Ayeva writes:

Hi Daniel,

Have you looked at repoze.bfg.htmlpage (http://pypi.python.org/pypi/repoze.bfg.htmlpage) in the Repoze repository, contributed by Malthe & friends ? From what I understand, it provides an alternative approach to the Deliverance way for a BFG-based site, and it seems similar to what you propose but uses an XSS rules-based technique to handle the way each page part's content is inserted in the main template.

The benefit is that, in their daily work, designers only have to work with the HTML and CSS files that are under a folder similar to /resources (or /html) in the skinning package.

Maybe this is another solution to experiment with for Plone's future skinning story.

Anyway, being also interested by repoze.bfg, I like the idea of repoze.bfg.htmlpage. And it would be great if the community ends up with the best possible compromise that all can benefit from.

Zeitmaschine writes:

@kamon: We've actually implemented something similar for Plone; you can find it at http://svn.plone.org/svn/plone/sandbox/plone.maintemplate/

Sylvain Viollon writes:

I did in October a project using that idea, and Grok skinning technique (with five.grok), and we end up with a really simple and small extension for our skin (even smaller than collective.skinny).

It was porting a Plone 2.1 to 3.0, which was awfully slow.

Building the technique and skin toke us 2 days. We use chameleon templates by default as well, Structured Document for advanced layout / content rendering.

It's blazing fast, and you don't need a super-computer with a ZEO with 8 nodes and an extreme aggressive cache in order to serve a site which actually have quite some traffic, and some data (3.6Go packed, files being stored outside of the Data.fs).

We got as requirement in the project that the loading time of a page should be less than 2 seconds using a regular 8Mb ADSL line, the customer was definitely unhappy with the old response time, and was thinking about changing of technology (i.e. not using Plone anymore).

So if you compare performance and simplicity, I highly prefer to keep things simple and use that instead of Delivrance, and make my customer happy.

http://www.astrium.eads.net/en

Plone 3 theming done the conventional way is very painful. There are countless blog posts that tell this story. Catherine Williams' recent post is one of them.

An often overlooked alternative to creating a nice looking site based on Plone is rolling your own, completely separate skin. Once you hook up your own skin, you can forget all about viewlets, ZCML, GenericSetup, and theme away, leveraging your knowledge about HTML, CSS, and ZPT.

In a reply to Catherine's post, I'm explaining the most important aspects of rolling your own theme from scratch, along with some code to get you started. Look for my comment titled Roll your own public-facing skin, it's much easier! :-)

Edit: I made a new post and another one related to this with an extensive example package called collective.skinny.

Giving a Python tutorial here in Rabat at the Ecole Mohammadia d'Ingenieurs (EMI) turned out to be one of the most amazing experiences I had in a while. Today was the last of two days of Python for about 40 local students in two groups.

The tutorial itself was mostly stealing from Guido's Python tutorial with some modifications and some bits that I added. The clarity and the absence of noise in Python proved to make it very effective for teaching. Thanks to the interactive mode, I was able to demonstrate most concepts on the fly, which works brilliantly.

We used IDLE as the editor, which worked pretty well. Although I didn't have much experience with it, I knew that it would be the most straight-forward to dive in. One thing that I did find annoying about IDLE is that it doesn't print continuation lines (...) in its Python shell. I'm sure it makes it easier to copy and paste text, but it screws badly with indentation, which does trip up a few people.

Some of the students expressed an interest in Python web development. So with only the standard library available and one hour left in the tutorial we tried python -m SimpleHTTPServer, looked at the contents of that module, and at a custom subclass of BaseHTTPServer.BaseHTTPRequestHandler, one that implements a do_GET with a simple way to dispatch URLs.

I sum, I used some 170 slides (with very big fonts), which I'll put online after Morocco. The material turned out to be a tiny bit too much, and thus we skipped some exercises of the last quarter or so. In general, about half of the students were able to finish the examples in time, which gave me lots of confidence. Those who didn't work in pairs were more likely to be in the second half. :-)

I'm still staying in Morocco until Monday, which is way too short. This place is terrific.

Tomorrow we'll see a mini-conference, also at EMI, with some four presentations. Amine Soulaymani talks about Python vs. Java (it seems Java is widely used in schools here also). There'll be a PloneGov presentation, one about OpenERP (Abderrahim El Kafil). The subject will shift away from Python and more to Free Software in general. On Sunday and Monday, there'll be a PloneGov sprint in Casablanca that I'm attending.

Big THANKS go to Kamon and the rest of the Python African Tour team for organizing this, to the wonderful Aida and her hard working colleagues for making everything work out so smoothly here in Rabat, to the rest of the Moroccon students for being super friendly, and last-but-not-least to Joel and the PloneGov team for making this possible!

I think that with the Etape marocaine we've started something great. This is asking to be continued. Hopefully with more couscous, too.

Update: The slides for the Python tutorial are available now.

This weekend, I decided I'd finally make good use of my M-Audio Axiom 25 and wire it up to SuperCollider. What I did to achieve this was write a small Python program that listens to incoming MIDI input and writes OpenSound Control to SuperCollider.

The piece of Python's called midi2sc and it's actually become a somewhat high-level library for assigning MIDI controls to output OSC. It makes use of the excellent pyrtmidi and scosc libraries both developed by Patrick Kidd Stinson.

An example of wiring controls to SuperCollider is provided in the module. Imagine you have a SynthDef defined in SuperCollider that looks like this:

SynthDef("funk", {
  arg freq=700, amp=0.7, gate=1, cutoff=20000, rez=1, lfospeed=0;
  // ...
}).send(s);

Note that this "funk" synth takes 6 arguments, all of which can be controlled while the synth is playing. To control the cutoff frequency with a MIDI control, we define this:

cutoff_control = AbsoluteControl(
    synthdef, min=300, max=7000, param_name='cutoff')

Then, to assign it to a MIDI command, we register a GroupControl to handle commands from a group of continous controllers:

midi = rtmidi.RtMidiIn()
port = ask_for_port(midi)

midi_in = MidiIn(midi, port,
                 handlers={0Xb0: GroupControl({71: cutoff_control})})

The Essentials of the MIDI protocol explains that the 0xb0 MIDI command means we're dealing with a continuous controller, and 71 happens to be the number of the knob that I'm using.

Adding another control is easy. This adds a turning knob for the resonance:

rez_control = AbsoluteControl(synthdef, min=0.10, max=1.5, param_name='rez')
midi_in.handlers[0xb0].controls[91] = rez_control

There's also examples of Note-On, Note-Off, Pitch bend and Channel Pressure controls in the module.

Thanks to the MIDI standard, midi2sc can take input from all kinds of MIDI controllers, including software sequencers like seq24. In fact, I'm having loads of fun with this particular combination; using seq24 to add and remove loops that I prepared and turning knobs on the keyboard to influence the timbre of the notes played.

Sure, there's also MIDI classes in SuperCollider, but I'm still a relative dork in SuperCollider's language sclang. And I like to pretend that this is more compact and readible (and debuggable) than an implementation in sclang.

Update 2009-06-20: I added a GUI and support for configuration files and released midi2sc on PyPI.

After going through part two of this Arduino sound tutorial, I made this little synthesizer.

Click here if you can't see the video.

You can see that it starts out with a constant frequency that can be varied with the potentiometer.

An LFO is added to the frequency once I push the button. Successive button pushes will change the amplitude of the LFO.

While the aforementioned tutorial explains how to connect Arduino with your PC sound card to get sound, I simply plugged in a Piezo buzzer. The advantage being that this synth is totally self-contained and portable, so I could use it in the train (I won't!). The downside is that the audio quality is bad, and it's even worse in the video.

Everything I needed to build this came with the Arduino workshop kit (50 EUR). This includes the Arduino Diecimila itself, potentiometer, buzzer, breadboard, and button.

The button press detection in the Wiring code doesn't work properly. A good implementation would probably need to use interrupt handlers. I ignored this issue for now:

int buttonPin = 7;
int buzzerPin = 8;
int potPin = 2;

void setup() {
  pinMode(buzzerPin, OUTPUT);
  pinMode(buttonPin, INPUT);
}

void inner_loop(int j) {
  digitalWrite(buzzerPin, HIGH);
  delayMicroseconds(j*16 + analogRead(potPin));
  digitalWrite(buzzerPin, LOW);
  delayMicroseconds(j*16 + analogRead(potPin));
}

void loop() {
  int val = digitalRead(buttonPin);
  int factor = 0;

  for (int i = 100; i > 0; i--) {
    inner_loop(i * factor);
  }

  // XXX: This doesn't work very well
  if (digitalRead(buttonPin) != val && val == HIGH) {
    factor++;
    if (factor > 3)
      factor = 0;
  }

  for (int i = 0; i < 100; i++) {
    inner_loop(i * factor);
  }
}

Vanessa and I did some fun experiments lately with Arduino and SuperCollider. We're building an alarm clock that wakes you up with quite an unusual sound...

Read the story here.

I just uploaded a new tutorial to plone.org that explains how to use z3c.form with Plone. z3c.form is the next generation Zope form library and excels through ease of use and very good documentation.

Your feedback is welcome. See also the project's homepage and the discussion on the Plone Product Developers list.

We've been working hard recently. And today I'm pleased to announce the first official release of Singing & Dancing, the next-generation newsletter and notification solution for Plone.

Thanks to Thomas Clement Mogensen and all contributors, and kudos to Headnet for their support. And Giuseppe Zizza for making the very cool logo!

Ship it!!

If you're still looking for something cool to do at next week's Sorrento sprint besides lying in the sun and sipping on that Limoncello: Why not join us with Singing & Dancing?

Thanks to fakezope2eggs, I can now in my Zope 2 buildout depend on packages like z3c.form and zc.queue. These would otherwise pull in incompatible versions of packages already in the Zope 2 lib/python directory.

Kudos to Jean-Francois!

Edit: It turns out that the order of the parts in your buildout is very important. Make sure that the fakezope2eggs part comes right after the zope2 part, and before your Zope 2 instance part!

This project is frustrating the hell out of me.

If you want to customize your Plone site, don't do this:

• Go around wildly in your Products folder in customization frenzy and patch files like there's no tomorrow; and not even write down what you changed where. Monkey patches are a hundred times better than this!
• Make customizations in the ZODB that your filesystem code depends on, so that you can't use the filesystem code anymore without that exact same database. Have fun reproducing your environment!

Please, these things are totally disastrous. You might just as well throw your Plone site out the window!

Gaah!!

(Thanks to Florian Schulze for being an inspiration to the title for this blog post.)

Will McGugan recently blogged about his timed caching decorator. And today I found out about gocept.cache, which allows you to memoize a function for a given amount of seconds.

This is how you would cache a function's value for an hour using plone.memoize:

>>> from time import time
>>> from plone.memoize import ram
>>> @ram.cache(lambda *args: time() // (60 * 60))
... def fun():
...     return "Something that takes awfully long"

In the same way we're making the cache depend on time here, we could have it depend on anything else, like certain arguments provided to the function. So, really, we can do more than just memoization with plone.memoize; an example is this time-out application.

See the plone.memoize docs for more examples.

After finding out about plone.recipe.bundlecheckout yesterday, I thought I'd mention infrae.subversion, a similar recipe for zc.buildout. These are the key differences:

• p.r.bundlecheckout allows you to specify one URL per part, whereas with infrae.subversion you can specify a list of URLs. Why does this matter? Because it helps you keep the exact URLs and therefore versions of the components that you use in your buildout configuration, which is better than keeping them in a svn:externals property.

• infrae.subversion takes care not to wipe any changes that you might have done in the checkout. That is, you can safely use its checkouts for development.

  Why not instead make a separate products directory and use svn:externals for development? Because again, we want to keep all dependencies in the buildout configuration. And it's good to keep the development buildout as close as possible to the deployment one, to minimize the chance of error. With infrae.subversion, you also have the advantage of being able to run bin/buildout and have all dependencies updated, instead of having to run svn up in some directory manually, which is a source of confusion.

• p.r.bundlecheckout works with both SVN and CVS, while infrae.subversion only works with SVN.

The Silva buildout is an example of a buildout that uses infrae.subversion.

[my-products]
recipe = infrae.subversion
urls =
  https://svn.plone.org/svn/collective/FCKeditor/trunk FCKeditor
  ...

[prep-fckeditor]
recipe = plone.recipe.command
command =
    ${buildout:executable} ${buildout:directory}/parts/my-products/FCKeditor/utils/base2zope.py
update-command = ${prep-fckeditor:command}

If you need to speed up your Python application, you should take a look at plone.memoize, in particular its new cache decorator that's really easy to use:

>>> from plone.memoize.ram import cache
>>> def cache_key(fun, first, second):
...     return (first, second)
>>> @cache(cache_key)
... def pow(first, second):
...     print 'Someone or something called me'
...     return first ** second

>>> pow(3, 2)
Someone or something called me
9
>>> pow(3, 2)
9
>>> pow(3, 3)
Somone or something called me
27

You can see that the cache key function determines which input values result in the same output and thus can be cached. Cache key functions receive exactly the same arguments as the functions that they cache, plus the function itself. Thus, key functions for methods can also use self for determining a cache key. A cache key function may raise the DontCache exception to signal that no caching should take place.

The cache storage backend can be freely chosen using the second argument to the cache decorator:

>>> @cache(cache_key, cache_storage)

where cache_storage is a function that again takes all arguments of the original function and returns a dict-like object for use as a cache storage. plone.memoize has built-in support for memcached and zope.app.cache as storages.

See the doctests for volatile.py for more examples.

For now, you'll need to use the SVN version. A release containing the cache decorator will follow soon.

Tom thought it'd be a good idea to write down what we were up to at the PIKtipi sprint (2nd to 7th of June), so here's my wrap-up:

PIKtipi was my third Plone sprint so far in 2007. I had an excellent, and productive time, thanks to all organizers. Here's what I did:

• Develop further plone.memoize and make use of it in the portlets implementation of Plone 3. Together with Hanno, we were able to speed up Plone for anonymous users by about 50%, and for logged in users by about a third. Stefan helped me implement the memcached support.
• Fix some bugs for the 3.0 release of Plone. By the way, if you're one of the 145 Plone providers earning money with Plone, you should consider helping out. Plone 3 will have loads of new features, but that also means that the bugtracker needs increased attention to get all these features stable.
• Have a fun time with fellow Plone developers, and attend the DZUG conference that was running in parallel.
• In Berlin, run into a 3 hour DJ set by Ellen Allien by accident. Thanks to Siebo and Saskia for their great company. :-)

Five Easy Pieces - Simple Python Non-Patterns from Alex Martelli was something that I had tagged toread for quite a while now, and it turned out to be a very interesting read.

The most important point that Martelli makes in this paper is that in Python, some of the traditional software design patterns don't apply, because the Python programmer has facilities that other OO languages lack, the lack of which makes certain patterns necessary in other languages that seem superfluous in Python.

It's nice how Martelli puts context into this topic, by explaining some of the background of patterns. Most notably, he references a book called A Pattern Language - Towns, Buildings, Construction by Christopher Alexander and a paper called The Structure of Pattern Languages by Nikos Salingaros, which is available online.

There's quite a nice analogy between software patterns and architectural design patterns. The dictionary of design patterns of Christopher Alexander lists examples for patterns found in architecture like SMALL PARKING LOTS, for which the summarizing statement is Vast parking lots wreck the land for people.

Interestingly, both Alexander and Salingaros are harsh critics of modern architecture. Salingaros writes:

Architecture has changed in this century from being a trade serving humanity with comfortable and useful structures, to an art that serves primarily as a vehicle for self-expression for the architect. In the current architectural paradigm, the emotional and physical comfort of the user are of only minor importance.

Well, this reminds me of software architects who care more about their design than users. The Big Design Up Front page in the PPR has some nice quotes and statements that go with this.

It makes sense to me when Martelli explains how some patterns are unnecessary in Python and others that tend to be involved in other languages are very easy in Python. Need a Singleton? Just create an instance of a class and have clients access that instance by its name. Need a registry? Make a module global instance of a list or dict, and have append and remove be your register and unregister functions. In Eby's article Python is not Java, one reads about a software project:

So, the sad thing is that these poor folks [a team of former Java developers who were new to Python] worked much, much harder than they needed to, in order to produce much more code than they needed to write, that then performs much more slowly than the equivalent idiomatic Python would.

Simplicity is the key to quality, and simple definitely beats complex. Test-first is probably a good way to create simple designs. The KISS principle represents:

If you have a choice between a simple solution and a complex one, then opting for complexity would be stupid.

And:

All design should be as simple as possible, but no simpler. [...] the more elegant designs are usually the more simple ones. Simple also does not mean quick and dirty. In fact, it often takes a lot of thought and work over multiple iterations to simplify.

Python code is by nature simpler than code in many other languages, because it lets you focus on your actual problem -- making simplicity a built-in feature, and apparent that the source code is the design. Finally, a nice quote from Kent Beck found in the PPR:

As a consultant, 80% of the time my job involves taking out premature abstraction so we can get the damned thing out the door.

For me, the PPR is an enormous repository of insightful ideas about software development. I've already emptied my cup and I'm surfing for my next zen slap! :-)

If you've tried the eXtremeManagement tool (or: XM) before, you may have noticed that it's not exactly an interaction design masterpiece. Doing a simple thing like booking your hours for the last week easily took you a few dozen clicks, and that with a web application that takes its time to respond. Getting an overview over your iteration (XM's loosely based on Extreme Programming ideas) used to be hard too. Because of the hierarchical way XM lays out iterations, getting from an project overview to a task that you were interested in would easily involve three clicks. And that's for getting an overview. What if you forgot the details about the task when you were back on the overview page? Considering that a project easily spans half a dozen stories per iteration, this was an overview nightmare.

At Jazkarta, we've been thinking a lot about what the right project management tool would be for us. We were fed up with XM and its sluggishness -- one project tried out Trac combined with some booking plug-in, but it turned out that Trac lacked the project management view of things, or that we were unable to produce the right reports, something that the eXtremeManagement tool is considered to do well out of the box. We looked at other projects and services on the web, but it looked like XM, despite being sluggish, was still the best thing there was. (If I had my way, we'd try to strip XP to the bones first, not thinking about existing tools, and try to understand the process more before we try to modify it.)

So, we looked at improving the tool. XM's trunk now features a simple expand button in the iteration view that lets you drill down to tasks. That is, when you're looking at the listing of stories, you can click an arrow to see the list of associated tasks inline, without any loading time. Malthe, a fellow from Jazkarta, thinks it's cool:

<malthe>  Oh my god! What happened to XM?
<malthe>  It's a UI revolution!

Another missing link in XM used to be the tracker integration. When your task was about fixing a bug in the tracker, your XM task would have no notion of the Poi issue that you were fixing (-- Poi is the bugtracker that we use). Consequently, you had to manually take care of linking the one with the other in the respective text fields, and that, of course, is more than dull, and it breaks easily (think moving around tasks in the system) and not doing it meant no good overview of things.

Now, the way I like to work with XM and Poi is that almost everything everything that I do goes into an issue. Because Poi has much better facilities for maintaining a dialogue with the customer, and Poi issues can have a longer lifetime than tasks -- they can be relevant for more than one iteration. (With stories and tasks that are bound to an iteration in XM's model, that's a bit hairy.)

With the new Poi Task on trunk, connecting tasks with issues finally makes sense. A Poi Task can be connected to one or more issues. The view of the Poi Task will show you all the issues that are associated with it, together with their status (unconfirmed, open, closed, etc.). On the other side, an issue will show you the tasks it's associated with. Plus, you now have a one-click way of adding a task for our issue to any open story in your project.

Also, Poi trunk now has auto-linking to other issues and Trac. Like in Trac itself, you can now simply write #123 in your report and get a link to that issue. The same goes for r123 and revisions in Trac.

The next thing my TODO for the XM tool is gtimelog integration. I want to be able to retrieve a task list from a XM project, and book my hours directly from gtimelog. This is by the way how the folks at Infrae have been booking their hours for a while now into their in-house timesheet app. This integration will mean that I'll no longer need to write down my hours into three different places by hand.

So... If you're looking for a project management tool that's flexible and XP, or if you've looked at XM before and you thought it's dull or doesn't give you enough overview, try it (again). I think it's grown to quite a usable solution for project management. By the way, thanks to Zest and Jazkarta for supporting this work.

For those of you who say they don't like doctests because they don't give you isolation between tests. Here's an example of an integration doctest for Plone that gives you exactly that:

from Testing import ZopeTestCase as ztc
from Products.PloneTestCase import PloneTestCase as ptc
from Products.PloneTestCase.layer import PloneSite

ZOPE_DEPS = ['MyZopeProductDependecy']
PLONE_DEPS = ['MyPloneProduct',
              'MyPloneDependency']

for x in ZOPE_DEPS + PLONE_DEPS:
    ztc.installProduct(x)
ptc.setupPloneSite(products=PLONE_DEPS)

class MyTest:
    def test_one():
        r"""
        Check if one and one is two:

          >>> 1 + 1
          2
        """

    def test_two():
        r"""
        Check if one minus one is zero:

          >>> 1 - 1
          0
        """

def test_suite():
    suite = ztc.ZopeDocTestSuite(test_class=ptc.PloneTestCase)
    suite.layer = PloneSite
    return suite

Look at G. Writing Tests in the (somewhat out of date) The Zope 3 Developer's Book for more inspiration.

Don't do traditional-style unit tests, they're ugly. Is this a matter of test, err taste? I don't think so.

Of course, if you want real isolation, you go for unit tests (i.e. no ZopeTestCase involved). People are usually scared and say, but then I need to set so much up by hand, or, don't make me write a mock object for everything, but those arguments aren't really valid most of the time. This is how easy a pure unit test using doctest for a Plone Product (or any Python program) can look like:

import unittest
from zope.testing import doctest

class Add:
    r"""
    >>> 1 + 1
    2
    """

class Subtract:
    r"""
    >>> 1 - 1
    0
    """

def test_suite():
    return unittest.TestSuite((doctest.DocTestSuite()))

If you have a hard time creating mock objects to test against, try Mocky. This is an example of mocking a CMF site that gives you getPhysicalPath, has a portal_catalog and some special site_properties for use with your content type:

>>> from mocky import Mocky
>>> import Acquisition
>>> class MockySite(Mocky, Acquisition.Explicit):
...     def getPhysicalPath(self):
...         return tuple(self.name.split('.'))
...     def portal_catalog(self, **kwargs):
...         return []
>>> site = MockySite('site')
>>> props = site.portal_properties.site_properties
>>> props.getProperty = lambda x: 'utf-8' # doctest: +ELLIPSIS
Set site.portal_properties.site_properties.getProperty to <function ...>

>>> from Products.MyProduct.content import MyContentType
>>> content = MyContentType('someid').__of__(site)
>>> content.something_that_involves_using_portalcatalog_and_siteproperties()
'your result here'

Update: The python-in-testing list has an interesting discussion about doctest vs. unittest.

I never leave home without these, at least not when I leave home to do development:

• Emacs

  Emacs of course isn't specific to Zope or Python. Nevertheless, Emacs is the editor for doing Zope development. Check out Philipp von Weitershausen demo pdbtrack. You need nothing more than the usual python-mode for this. (If you don't know what pdb is, read this gentle intro to debugging in Python.)

  I hear that other editors can do this too, but it can't possibly work as well. :-P

  Start with Emacs basics right now!

• Firebug

  You've certainly been living under a rock if you haven't heard of Firebug before. Never was trying out and debugging JavaScript so joyful. Firebug has an interactive JavaScript shell that since version 1.0 also has access to the variables in the scope of any breakpoint that you might have set.

• PDBDebugMode

  What it does: It loads the Python Debugger (pdb) whenever Zope encounters an exception that was not filtered in the error_log. In combination with pdbtrack this means: Don't read tracebacks to find out where the exception has occurred and where you might want to hook in to debug. And don't restart after you figured out where to put that pdb.set_trace(). Instead: Have your pdb point to the failing line of code and do your debugging right away. Works similar to --debug-inplace of the Zope testrunner.

  This might be not as useful for development of new applications (think tests) as for situations where you have to debug something quickly, possibly on a live deployment.

• ScriptDebugging

  Before, I totally hated debugging code in Script (Python) objects. That was until I found Lennart Regebro's ScriptDebugging, which helps pdbtrack find out where in the script you are, which means that you can debug scripts just like any other Python code.

• DeadlockDebugger

  Stuck with a Zope that's stuck? If your Zope stops responding, use DeadlockDebugger to find out where exactly Zope's threads are deadlocked. From the author:

  This product adds a hook so that a deadlocked Zope process can be debugged, by dumping a traceback of all running python processes. The dump is sent to the event log (at the DEBUG level) and returned to the browser (even though the Zope is deadlocked and doesn't answer any other requests!).

  DeadlockDebugger can of course also be used to debug Zope in non-deadlock situations, when a Zope process is taking a long time and you wish to know what code is being executed.

What's the Zope 2 tool that you'd take with you on an island?

Last weekend, I went to my first FOSDEM (video recordings). I took my girlfriend with me, which I believe helped her understand my nerdy side better. ;-)

Paul Everitt did a nice presentation on Plone 3, where he showed how you set up a Plone 3 instance using ploneenv, and then an easy example of using KSS, and some other Plone 3 niceties. The ploneenv screencast is available now, and so is the rest of the screencasts that Paul demos at the presentation.

Paul was followed by Dries Buytaert's Drupal presentation, which was also really good. It was more a demo of what Drupal is and what it can do for the end user, as opposed to how you can develop for the platform and some legal background, which Paul concentrated on.

Unfortunately, I missed the OLPC talk, but nevertheless I was able to get my hands on one of those laptops, and they really do look cool.

Elixir, a library built on top of SQLAlchemy that's recently gone into beta, looks like a very convenient way to talk to relational databases.

Amazing how a conference with so many good talks can be totally free. Next FOSDEM? I'll be there...

Update: Added a link to the ploneenv screencast.

This year's Baarn sprint was way cool! And we got a lot of stuff done.

I really like the multi-cultural aspect to Plone sprints. Where else do you get to meet and work face to face with people from Norway, the Netherlands, Scotland, Great Britain, Germany, United States, Hungary, Belgium, Finland and Austria in the course of a couple of days?

Seeing some people that I've never met in real life before was very nice, too. That'll help with communicating with those people through IRC and mailing lists.

Thanks everyone for a fantastic time! I'm totally looking forward to the Sorrento sprint, where over 60 people are expected!

I've updated the experimental Plone 3.0 setuptools distribution and ploneenv today a bit and fixed some bugs along the way.

Here are some small recipes that show you how ploneenv works, and how you can make use of it.

A requirement for all of these recipes is that you have ploneenv installed, which you can install like this:

sudo easy_install ploneenv

The recipes will also assume that you have two shell variables set. One is MKZO, the path to the mkzopeinstance.py script of your Zope 2.10 installation. The other is INSTANCE_HOME, the folder where you want your Zope instance to go into. This is an example of setting those variables:

MKZO=~/lib/Zope-2.10/bin/mkzopeinstance.py
INSTANCE_HOME=~/myplone3.0

ploneenv $INSTANCE_HOME -m $MKZO --no-requirements

source $INSTANCE_HOME/bin/activate
easy_install yolk

yolk -l

easy_install Plone==dev

ploneenv $INSTANCE_HOME -m $MKZO

easy_install ZopeSkel
paster create -t plone mypackage

source $INSTANCE_HOME/bin/activate
cd mypackage
python setup.py develop

Check out this new tutorial that helps you get started quickly with Product development in Plone 2.5 and higher. Why another tutorial? Because I myself needed a simple resource where I could copy and paste the most important snippets from. And because Plone needs more easy and practical get-started tutorials.

Thanks to Infrae and Wim who helped me write it!

Ideas I have to extend this tutorial:

• How to convert the Product into a ZopeSkel template
• How to use a formlib form for the search page

LovelySync is a small and quite flexible library for importing data into the ZODB, the object database that Plone uses. In theory it can be used as a library for anything that needs to be imported into any database (yeah, sure I hear you say). But I've only ever used it with the ZODB as the target database.

One of Lovely's customers needed to synchronize a Filemaker database regularly with their Plone site. The database was quite complex; it had different languages (which we mapped into LinguaPlone content), images, and all kinds of references between the objects. At the time I began working for the project, we didn't know exactly how the export format would look like, so we had to develop something that was easily adjustable: The input format is easily customized using schema files. All the actual writing is delegated to small specialized components called WriteHandlers. Reading is done by the Reader, which supplies the Writer (which is the object delegating to WriteHandlers) with Records. You might want to check out the interfaces file for the technical details.

LovelySync, originally written more than a year ago, but refactored a lot lately, was the first project where I used the Component Architecture of Zope 3 extensively, and quite successfully (and without ZCML ;). Using LovelySync, I've been doing imports for hundreds of members from a CSV file into Plone, screen scraping contents into Plone sites for migration, and even an import of usenet (NNTP) groups into Listen through nntp2listen, which will be in a public SVN shortly.

So if ArcheCSV doesn't exactly do what you want, and PLIP 112 is too far away for you, you might want to try LovelySync.

What a nice Zope Product I found today. Ross Patterson's PDBDebugMode allows you to:

• post-mortem debug any Zope exceptions while Zope is running in foreground,
• debug traversal when you provide the special query parameter pdb_runcall in the URL and
• import pdb in restricted code.

The README has more details.

Yesterday, I watched Guido van Rossum's talk about Python 3000 along with some Monty Python sketches that turned up in my search results page. It was quite interesting to see what Python 3.0 will be and what it will not be.

One thing that occured to me some months ago after I watched the TurboGears Ultimate DVD was that those generic functions (in the form of RuleDispatch) are an alternative to adaptation, as it exists in Zope. That is, you could do everything that you wanted to do with adaptation also with generic functions, and that in an arguably less verbose way. Also, Guido explains in his Google talk that adapters are only a special case of generic functions.

I tried it out. In plone.checksum (which is not that interesting by itself), I'm using a generic function instead of adaptation. I have a class called Checksum that returns an md5 checksum in its do_checksum method:

import dispatch

class Checksum:
    @dispatch.generic()
    def do_checksum(self, value):
        """Return md5 object that has the calculated checksum.
        """

You can see that the generic function doesn't actually have an implementation. It's meant to be overloaded like this:

import md5

@Checksum.do_checksum.when('isinstance(value, object)')
def do_checksum(self, value):
    checksum = md5.new()
    checksum.update(str(value))
    return checksum

This is the most general implementation that I could come up with. It'll be called when the argument value is an instance of object, that is, always, unless there's a more specialized generic function. I needed a different implementation of do_checksum for OFS.Image.File, which was easy to hook in:

import OFS.Image.File

@Checksum.do_checksum.when('isinstance(value, OFS.Image.File)')
def do_checksum(self, value):
    checksum = md5.new()
    value = value.data
    if isinstance(value, str):
        checksum.update(value)
    else:
        while value is not None:
            checksum.update(value.data)
            value = value.next
    return checksum

Interfaces are a generally a good thing, but not in this case where we would have to mark the File class just to serve our adaptation.

Guido blogged about adaptation versus generic functions half a year ago. There are also some nice additional links in there.

The Document Library document management system has seen lots of improvements lately. There'll be a new release shortly. We've been busy adding:

• easy installation, using Zope Buildout,
• LDAP support, based on ldappas and ldapadapter,
• filesystem storage support, based on Tramline and hurry.file and
• a more reliable conversion between file formats, provided by the OooConv conversion server, which now has an easy way to install it, too.

Quite a number of different components are involved. And that's only the list of recently added dependencies. Code reuse is good. The working together of these components, however, needs extra care, because components will change over time and we need a way to make sure that the Document Library doesn't break, or that its breakage is apparent when we update its dependencies to newer versions. Testing the Document Library's APIs in isolation isn't enough here. Neither is leaving the testing to the user.

The optional filesystem storage support requires Apache in front of Zope. The conversion mechanism requires a running OooConv process (which in turn controls a number of Open Office worker processes). And then we allow mass uploading of documents using ZIP files, the contents of which may want filesystem storage and conversion, too. This leaves us with quite a number of variables, and lots of ways to screw up.

Manual testing of all of these combinations is tedious. Let's say a human tester prepares a testing session by writing down all the possible combinations he can think of (which is what we did, too). He tests document adding, editing, validation errors and uploading with Tramline and without and conversion with Tramline and without and with ZIP files.

Lots of functionality is cool, but it's also worrying. Will feature A combined with B under circumstance C still work when I change this line of code? Doing all the human testing after we do a change isn't an option. But not doing it feels bad.

The answer to this is to write automated functional tests. For that, we decided to use the Zope Testbrowser (with doctest), which has already proven itself a very useful tool in the past.

Now you might already know that Testbrowser has a mode for testing against any HTTP server. In our case, this mode is the only way to test conversion and filesystem storage, because both of these features require help from external processes.

Before running these Testbrowser tests now, the tester start up Apache and the conversion server and points Testbrowser to Apache. The tests work very similarly to human testing: A Document Library is created through the web browser, documents are added, edited, files are uploaded, mistakes are made to trigger validation error etc. And all that in a well-defined manner which leaves little place for human testers to make errors or forget about a certain test. (They can still forget to run the tests, though. ;)

At this point, I stopped worrying about breakage. And I learned that these functional tests are the bomb.

One thing worth noticing is that in these remote functional tests, we can't use the Zope Testrunner's -D option. This would normally put us (= the debugger) right at the relevant point of the application code where the test failure occurs. However, we can still run Zope in foreground and put breakpoints into the application code, or into the test code. Which means that at any time into the tests we can introspect the state of the application by either looking at variables in the debugger, or by simply browsing the Document Library instance that's being tested in Firefox.

If you want to read more about automated tests, check out this big list of interesting articles and links from Grig Gheorghiu.