plone.recipe.command

This is another buildout recipe that I should quickly mention. It can be used to run arbitrary shell commands at install or update time. Here is an example that uses the beforementioned infrae.subversion recipe to install the latest Plone's FCKeditor Product from SVN. The reason for using plone.recipe.command here is that we need to call the base2zope.py script to bootstrap the Product after doing a checkout:

[my-products]
recipe = infrae.subversion
urls =
  https://svn.plone.org/svn/collective/FCKeditor/trunk FCKeditor
  ...

[prep-fckeditor]
recipe = plone.recipe.command
command =
    ${buildout:executable} ${buildout:directory}/parts/my-products/FCKeditor/utils/base2zope.py
update-command = ${prep-fckeditor:command}

You can find more buildout recipes in PyPI.

Create a pristine Zope instance

ploneenv $INSTANCE_HOME -m $MKZO --no-requirements

source $INSTANCE_HOME/bin/activate
easy_install yolk

yolk -l

Create an instance with Plone 3 from Subversion in it

easy_install Plone==dev

Create an instance with a Plone 3 release in it

ploneenv $INSTANCE_HOME -m $MKZO

Develop packages for Plone 3

easy_install ZopeSkel
paster create -t plone mypackage

source $INSTANCE_HOME/bin/activate
cd mypackage
python setup.py develop

Status quo

Plone 3.0, ploneenv and the Plone egg distribution are under heavy development still. Therefore you might encounter errors while trying out these recipes. Please report those!

There is one known problem with ploneenv, where easy_install attempts to compile Script (Python) files and spews a lot of error messages. However, those error messages are not critical, and the installation succeeds nevertheless.

If you're developing Plone 3 itself, you might want to go for ploneout, which is the mechanism used by most Plone developers who actively develop Plone.

My previous blog on ploneenv has some more pointers and info.

Update: Thanks to Paul Everitt, there is now a ploneenv screencast.

What is ploneenv?

ploneenv builds a Zope instance that is also a workingenv and installs (by default) Plone 3.0 in it.

There are other packages out there that can do the same:

• ploneout
• instancemanager
• zopebuilder (anyone have a link?)
• skeletor

The most prominent of these packages is ploneout, a Zope Buildout configuration (including add-ons) that allows you to install Zope and Plone 3.0 in one step. While this is a very appealing solution in most situations, sometimes it is more comfortable for a developer to work more interactively, in an environment where packages can be installed and tested out, removed, replaced and queried using the standard easy_install and tools like Yolk.

So what is ploneenv? ploneenv is a one module Python script that builds heavily on workingenv and setuptools. What it does:

• It creates a Zope instance for you. You always provide the mkzopeinstance.py script that you want to use as an argument. E.g.:

  ploneenv ~/myzopeinstance --mkzo=~/lib/Zope-2.10/bin/mkzopeinstance.py
  

• It creates a workingenv in the Zope instance for you.

• It installs the Plone egg by default. However, you could just as well install something else in your new Zope instance using the --requirements argument. ploneenv is not Plone specific.

These steps are quite similiar to what you do manually when you make your Zope instance a workingenv.

Give me some examples

This is how you use ploneenv to install Plone:

easy_install ploneenv

MKZO=~/lib/Zope-2.10/bin/mkzopeinstance.py
INSTANCE_HOME=$HOME/myplone30

ploneenv $INSTANCE_HOME -m $MKZO

At this point, you can install any extra packages. This would install simplegeneric:

source $INSTANCE_HOME/bin/activate
easy_install simplegeneric

Old-style Products that aren't wrapped in eggs are installed as usual:

cd $INSTANCE_HOME/Products
wget http://plone.org/products/ploneformgen/releases/1.0.3/ploneformgen_1-0-3.tgz
tar xzfv ploneformgen_1-0-3.tgz

You can also override Products that come with Plone. For example, you could set a symlink from your local CMFPlone checkout into the $INSTANCE_HOME/Products directory and hack away.

To use a local SVN checkout to develop an existing or new new-style Plone package, you would simply do:

cd ~/myproject
python setup.py develop

Note that except for the activation of the environment, this is exactly how you would install a package for development in Python (=setuptools). That is, this is not ploneenv nor workingenv specific.

Try it out!

Please try it out and give me feedback. As mentioned before, this is all you need to type:

easy_install ploneenv

MKZO=~/lib/Zope-2.10/bin/mkzopeinstance.py
INSTANCE_HOME=$HOME/myplone30

ploneenv $INSTANCE_HOME -m $MKZO

That's it! Just make sure that you set the MKZO and INSTANCE_HOME according to where your mkzopeinstance.py script is and where you want to create your instance respectively.

Now you can start Zope using $INSTANCE_HOME/bin/zopectl fg.

What's inside the Plone egg and why

The Plone egg goes back to some early experiments I did to package Plone. Plone is a meta-package that has a Products namespace and currently all old-style Products that Plone requires contained in it. On top of that, it defines a number of requirements, like elementtree and plone.openid.

By not mixing installation code with the package itself, the Plone egg makes sure that it can be installed in any context, for example in a ploneout.

If the CMF were to became available as an egg, Plone would remove it from its own contents and just define another requirement.

The tagged and released Plone egg (or bundle, if you like) should arguably not have svn externals that point to other Products' SVN trunks. Instead it should either use svn externals to SVN tags of Products where possible or include the Product itself if it's not maintained in subversion. This might seem a bit hacky, and Rocky says I'm cheating, but it effectively brings the Plone package more inline with other Python packages out there, with all the benefits that this brings. IMO, this should be the way to release Plone 3.0.

Discussion happens on the Plone development list.

Update: I've updated this entry to reflect the fact that ploneenv and Plone are now in the Cheeseshop, so the installation becomes a lot easier.

Update: I've retired ploneenv in favour of repoze (especially repoze.plone). repoze is what ploneenv wanted to be, and more.

Of cardinals and ambassadors

It's now almost one month that I work for Infrae, and I enjoy it a lot. I have a nice working environment with friendly colleagues and interesting projects. And I get to share my workplace with Martijn Faassen, which rocks, because I can learn so much from this guy!

The other day I went to the COM.lounge TV page to check out Martijn's keynote at this month's DZUG conference. It was fun to watch. And then I discovered four other very interesting videos from this year's Europython conference in Geneva. I wasn't there, so I was eager to see the recording of the Web framework shootout which took place there. I was particularly impressed about Kevin Dangoor's presentation of TurboGears. If this is not the incarnation of agile development, then I don't know what! Kevin upstaged the other two contestants Simon Willison (representing Django) and Philipp von Weitershausen (of Zope 3).

Drawing the bow back to Martijn's presentation, I totally agree with him that Zope 3 needs another face on the web. Finally, he and some others are doing something about this. They're still looking for help from you!

Also, the eggification of Zope 3 packages, which is landing slowly, will help in making Zope 3 popular in the wider Python community.

Zope 3 = Ninja Power?

But Zope 3 has also another problem IMO: It's hard to grasp. Even if it's less confusing than the huge inheritance trees in Zope 2, Zope 3 has still to go a long way to be as newbie-friendly as, say, TurboGears. Again, Martijn comes to the resuce: In October he'll be sprinting with the people from Gocept with the goal to flatten Zope 3's learning curve. The project is called Grok, and it sounds really promising.

Honestly, I think that if I hadn't known the concepts of Zope 3 before, I wouldn't have understood much of what Philipp explained at the shootout. I don't buy the it's complex because it solves complex problems argument. Python is a good example of a very approachable technology that can scale sky high.

When following the mailing lists of TurboGears, I noticed that their team considers lack of documentation, especially in the form of approachable tutorials, a bug. Now, Zope 3 comes with lots of doctests, which I find great. But they're not exactly approachable for newbies. Not because of their style of writing, but because they are in places where newbies don't look, i.e. in the source tree.

Benji York created the very nice Zope 3 Quick Start Guide, which I personally recommend to anyone who starts with Zope 3. Also, zeapartners.org has two nice little screencasts. We definitely need more tutorials like this! And fewer Zope 3 packages with the hidden ninja wisdom feel to them.

And Now for Something Completely Different

While I played around with Testrecorder, I wrote a small script for text match and replace using regular expressions. This is arguably one of the most unreadable pieces of code I've written in a while. But to the rescue come the doctests, which I've used to develop this test-first.

I don't know anything about Ron Jeffries but this is how he describes it and how it felt:

When you get this right, development turns into a very pleasant
cycle of testing, seeing a simple thing to fix, fixing it, testing,
getting positive feedback all the way.

Guaranteed flow. And you go so fast! Try it, you'll like it.

Update: Martin incorporated the Testbrowser and Testrecorder HOWTO in his tutorial. I updated the link to point to that.

Get ZopeSkel

ZopeSkel has moved to the Collective. Get the latest and greatest of ZopeSkel via this command:

easy_install http://svn.plone.org/svn/collective/ZopeSkel/trunk#egg=ZopeSkel-dev

Create a package

Remember from last time that Paste Script is useful for creating a consistent directory structure and files for you so that you can get to the actual work quickly. Plus, your projects will be ready for distribution without any additional work.

Now let's say we want to start a plone.form package. What we need to do is invoke paster create, telling it that we want to use the plone_core template. This is done using the -t command-line option. After invoking the command, we're asked for some variables:

$ paster create -t plone_core
Selected and implied templates:
  ZopeSkel#plone_core  A Plone Core project

Enter project name: plone.form
Variables:
  package:  ploneform
  project:  plone.form
Creating template plone_core
Enter namespace_package (Namespace package) ['plone']:
Enter package (The package contained namespace package (like i18n)) ['']: form
Enter version (Version) ['0.1']:
Enter description (One-line description of the package) ['']: Plone compatibility layer for zope.app.schema and zope.formlib
Enter long_description (Multi-line description (in reST)) ['']:
Enter author (Author name) ['Plone Foundation']:
Enter author_email (Author email) ['plone-developers@NOSPAMlists.sourceforge.net']:
Enter keywords (Space-separated keywords/tags) ['']:
Enter url (URL of homepage) ['']:
Enter license_name (License name) ['GPL']:
Enter zip_safe (True/False: if the package can be distributed as a .zip file) [False]:
Creating directory ./plone.form
  Recursing into +namespace_package+
    Creating ./plone.form/plone/
    Recursing into +package+
      Creating ./plone.form/plone/form/
      Copying HISTORY.txt_tmpl to ./plone.form/plone/form/HISTORY.txt
      Copying LICENSE.GPL to ./plone.form/plone/form/LICENSE.GPL
      Copying LICENSE.txt_tmpl to ./plone.form/plone/form/LICENSE.txt
      Copying README.txt_tmpl to ./plone.form/plone/form/README.txt
      Copying __init__.py to ./plone.form/plone/form/__init__.py
      Copying configure.zcml to ./plone.form/plone/form/configure.zcml
      Copying version.txt_tmpl to ./plone.form/plone/form/version.txt
    Copying __init__.py to ./plone.form/plone/__init__.py
  Copying setup.cfg to ./plone.form/setup.cfg
  Copying setup.py_tmpl to ./plone.form/setup.py
Running /usr/bin/python setup.py egg_info

Note that we typed in values for the project, package and descriptions variables only. For the rest we decided that the default was good enough. We're ready to start working on plone.form now.

Non- Plone Core packages

There's no reason why we wouldn't use the plone_core template for our own packages. After all, the name of the namespace package and that of the contained package are variables, and the directory structure might well be useful for your own project.

Updating a ZopeSkel package

Whenever the plone_core template changes, we can simply rerun the paster create command to update our project:

$ paster create -t plone_core
Selected and implied templates:
  ZopeSkel#plone_core  A Plone Core project

Enter project name: plone.form
Variables:
  package:  ploneform
  project:  plone.form
Creating template plone_core
Enter namespace_package (Namespace package) ['plone']:
Enter package (The package contained namespace package (like i18n)) ['']: form

[...]

  Recursing into +namespace_package+
    Recursing into +package+
      ./plone.form/plone/form/HISTORY.txt already exists (same content)
      ./plone.form/plone/form/LICENSE.GPL already exists (same content)
      ./plone.form/plone/form/LICENSE.txt already exists (same content)
      ./plone.form/plone/form/README.txt already exists (same content)
      ./plone.form/plone/form/__init__.py already exists (same content)
      ./plone.form/plone/form/configure.zcml already exists (same content)
      ./plone.form/plone/form/version.txt already exists (same content)
    ./plone.form/plone/__init__.py already exists (same content)
  ./plone.form/setup.cfg already exists (same content)
  ./plone.form/setup.py already exists (same content)
Running /usr/bin/python setup.py egg_info

If something had changed, we would have been presented a diff of our version of the file and the new version inside the template.

An inconvenience here is that when running paster create, we're asked for all variables again. Fortunately, there's the --config=CONFIG option that'll store all our variables in a file. Both for initial creation and for updating, our command becomes:

$ paster create -t plone_core --config=plone.form/template_vars.cfg

Extending ZopeSkel

Extending the plone_core package to fit our own needs is really easy. Let's say we want to create a new template called my_package that's based on plone_core. We want our new template to use what's already there in plone_core and add a browser package in there with __init__.py and configure.zcml files.

Inside our SVN checkout of ZopeSkel, we create this directory:

$ mkdir -p zopeskel/templates/my_package/+namespace_package+/+package+/browser

Then we add our two files:

$ edit zopeskel/templates/my_package/+namespace_package+/+package+/browser/__init__.py_tmpl
$ edit zopeskel/templates/my_package/+namespace_package+/+package+/browser/configure.zcml

Note that we create an __init__.py_tmpl file, with the _tmpl suffix because setuptools would currently not include our .py file in the distribution if the file ended with .py. So although our __init__.py_tmpl file isn't necessarily a template (nothing is substituted), we make it a Paste Script template so that it gets picked up by setuptools.

The next thing we do is wire up the my_package template so that it's available from the command-line. Inside zopeskel/__init__.py we add this:

class MyPackage(PloneCore):
    _template_dir = 'templates/my_package'
    summary = 'A Plone package that has a browser subpackage'
    required_templates = ['plone_core']

Finally, we define an entry point for our template in setup.py:

[...]

entry_points="""
[paste.paster_create_template]
basic_zope = zopeskel:BasicZope
plone_core = zopeskel:PloneCore
my_package = zopeskel:MyPackage
""",

[...]

Note that we only added the my_package entry point here.

After calling setup.py develop we're ready to create a new project with our own template:

$ python setup.py develop
[...]
$ paster create --list-templates
Available templates:
  basic_package:  A basic setuptools-enabled package
  basic_zope:     A Zope project
  my_package:     A Plone package that has a browser subpackage
  paste_deploy:   A web application deployed through paste.deploy
  plone_core:     A Plone Core project

Easy as pie!

Getting Paste Script

Just easy_install Paste Script like so:

easy_install PasteScript

This will download the latest Paste Script and all its dependencies from the Cheese Shop.

If you don't want easy_install to put the downloaded packages into your standard site-packages directory, consider using workingenv or look at Custom Installation Locations.

Using Paste Script

This is where it gets interesting. When we invoke the paster create command, we'll be asked a couple of questions, most of which we can answer by hitting enter and thereby using the default value:

$ paster create
Selected and implied templates:
  PasteScript#basic_package  A basic setuptools-enabled package

Enter project name: MyFancyNewProject
Variables:
  package:  myfancynewproject
  project:  MyFancyNewProject
Creating template basic_package
Enter version (Version (like 0.1)) ['']:
Enter description (One-line description of the package) ['']: This is a fancy project

...

After answering the questions, Paste Script will set up the files for our project:

Creating directory ./MyFancyNewProject
  Recursing into +package+
    Creating ./MyFancyNewProject/myfancynewproject/
    Copying __init__.py to ./MyFancyNewProject/myfancynewproject/__init__.py
  Recursing into +project+.egg-info
    Creating ./MyFancyNewProject/MyFancyNewProject.egg-info/
  Recursing into docs
    Creating ./MyFancyNewProject/docs/
  Copying setup.cfg to ./MyFancyNewProject/setup.cfg
  Copying setup.py_tmpl to ./MyFancyNewProject/setup.py
Running /usr/bin/python setup.py egg_info

Your newly started project is now ready for distribution!

Paste Script and Subversion

By the way, there's also a handy --svn-repository=REPOS argument to the paster create command which will create a basic Subversion layout for your project. Pass the Subversion directory that you want to be used as the parent of your project's directory like so:

$ paster create --svn-repository=http://mysubversionserver/myprojects/

This will create a MyFancyNewProject directory under myprojects/ and it'll create trunk, branches and tags subdirectories therein.

Pluggability

Adding your own set of templates to Paste Script is fairly simple. The Paste Script developer documentation explains how to do that.

Based on that, I've had the idea of creating a simple set of templates for quickstarting a Zope project. There's already the skeletor project which aims to do something similar for Zope 2. When I looked at skeletor though, I decided that it does a bit much and it's fairly complex. I also ran into a few tracebacks while trying to use the tool. Which led me to trying out Paste Script.

ZopeSkel project

I've started a small project which I call ZopeSkel. The only purpose of ZopeSkel is to add a couple of templates to the existing ones of Paste Script. Right now, ZopeSkel contains only one template called basic_zope, which is very bare bones. Take a look at the project's templates directory.

You can think of ZopeSkel as a minimal example of creating your own templates package for use with Paste Script. I want to look and see if I can create a good set of templates. Please tell me what kinds of templates would be helpful for you!

To install the latest ZopeSkel from Subversion do easy_install http://svn.plone.org/svn/collective/ZopeSkel/trunk#egg=ZopeSkel-dev. After installing, you can see that the list of templates available to paster create has been extended:

$ paster create --list-templates
Available templates:
  basic_package:  A basic setuptools-enabled package
  basic_zope:     A Zope project
  paste_deploy:   A web application deployed through paste.deploy

Try paster create --help to find out more about the create command.

setuptools Development Mode

When developing setuptools enabled projects, setuptool's Development Mode is essential. In Development Mode, you can edit code inside the checkout directory and immediatley see the effects of your changes when running the program. (If you make any changes to the project's setup script or C extensions, you need to run python setup.py develop).

Building the egg

1. Checkout a bundle of Plone that contains all the necessary products in the Products/ subdirectory
2. Run setup.py bdist_egg. This requires you to have setuptools installed.

The egg should be now in dist/Plone*egg, ready for installation. You can distribute your egg at this point.

Note: Currently there is a problem that prevents ATContentTypes' thirdparty package from being included. This should probably be fixed in ATContentTypes. For now, you need to put an empty __init__.py file into the Products/ATContentTypes/thirdparty directory.

Installing the egg

You can install the egg site-wide. With the effect that the Products will be available for import implicitly for every Zope 2 instance in the site. Doing this is very simple:

easy_install dist/Plone*egg

Because Zope 2 will automatically scan all packages inside Products, there is no way to control exclusion of a Product in your instance. This is less than ideal. If our Product were not to use Zope2's Products namespace (which is well doable using Rocky Burt's pythonproducts), we could have more control of what is included, using Five's new registerPackage directive. (Note that pythonproducts was merged into Five.)

Now that all packages in Plone use the Products namespace, we'll have to think about a way to deal with this. Fortunately, easy_install comes with the --multi-version command line option, which allows you to have more control over which packages are available at runtime.

Using --multi-version, we can install our egg into the instance's lib/python directory to effectively make our egg only available in that instance:

easy_install --multi-version \
             --install-dir=$INSTANCE_HOME/lib/python/ \
             --site-dirs=$INSTANCE_HOME/lib/python/ \
             dist/Plone*egg

When you install something --multi-version, client code that uses your package has to be aware of that and do an explicit import pkg_resources; pkg_resources.require(name). In our example, we need to require('Plone') somewhen in our Zope process. I had the idea of requiring packages in zcml, similiar to how pythonproducts registers ordinary Python packages as Products. There's a pending patch for pythonproducts that enables this, although pythonproducts is maybe the wrong place to implement this:

Index: src/pythonproducts/fivedirectives.py
===================================================================
--- src/pythonproducts/fivedirectives.py      (revision 26757)
+++ src/pythonproducts/fivedirectives.py      (working copy)
@@ -16,6 +16,7 @@
 """

 from zope.interface import Interface
+from zope.schema import BytesLine
 from zope.configuration.fields import GlobalObject


@@ -35,4 +36,11 @@
                     u'with a ProductContext instance',
         required=False
         )
-
\ No newline at end of file
+
+class IRequireDistributionDirective(Interface):
+    """pkg_resources.require() a distribution."""
+
+    name = BytesLine(
+        title=u'Target distribution',
+        required=True,
+        )
Index: src/pythonproducts/meta.zcml
===================================================================
--- src/pythonproducts/meta.zcml      (revision 26757)
+++ src/pythonproducts/meta.zcml      (working copy)
@@ -12,6 +12,12 @@
        handler="Products.pythonproducts.pythonproducts.registerPackage"
        />

+    <meta:directive
+       name="requireDistribution"
+       schema="Products.pythonproducts.fivedirectives.IRequireDistributionDirective"
+       handler="Products.pythonproducts.pythonproducts.requireDistribution"
+       />
+
   </meta:directives>

 </configure>
Index: src/pythonproducts/pythonproducts.py
===================================================================
--- src/pythonproducts/pythonproducts.py      (revision 26757)
+++ src/pythonproducts/pythonproducts.py      (working copy)
@@ -37,6 +37,13 @@
         args = (package,initialize)
         )

+def requireDistribution(_context, name):
+    """ZCML directive function for requiring a distribution using
+    pkg_resources.
+    """
+
+    import pkg_resources
+    pkg_resources.require(name)

 def _registerPackage(module_, initFunc=None):
     """Registers the given python package as a Zope 2 style product

After applying this patch to your pythonproducts installation, you can go ahead and include Plone in your instance using zcml. Create a $INSTANCE_HOME/etc/package-includes/plone-configure.zcml file and put the following in it:

<configure xmlns="http://namespaces.zope.org/zope"
           xmlns:five="http://namespaces.zope.org/five">
    <five:requireDistribution name="Plone" />
</configure>

Please refer to the pythonproducts README for details on where to add zcml and why.

Now start up your Zope and hopefully see Plone getting loaded.

Note that for packages which are not in the Products namespace package, i.e. for Product-less Python packages you would need to use pythonproducts' registerPackage directive as well. (Which is a good thing.) For these packages, the whole process becomes a lot easier, because you can just easy_install away and be done, or if you need instance- specific versions/packages, use --multi-version and the requireDistribution directive.