kotti_tinymce: new version

Vanč <ferewuz> worked on upgrading Kotti's WISYWIG editor TinyMCE to version 4.0.2. This new version features a much nicer looking user interface, which fits a lot better into the overall Kotti style. It also works better with smaller displays, as the editor will now scale down along with the browser window's width. Our existing image-upload and document-linking pop-ups were updated to work with the new version. You can test drive on our demo site.

deform_bootstrap: tabbed forms

Natan <nightmarebadger> worked on adding tabbed forms support to deform_bootstrap. With this change, each nested (Mapping)Schema in your Colander schema will be rendered inside its own tab. As an example, consider this Client schema:

import colander

class Person(colander.Schema):
    name = colander.SchemaNode(
        colander.String(),
        title='Name',
        )

class Car(colander.Schema):
    horsepower = colander.SchemaNode(
        colander.Integer(),
        title='Horsepower',
        )

class Client(colander.Schema):
    person = Person(title='Person data')
    car = Car(title='Car stuffs')

A deform_bootstrap form using the Client schema will render with two tabs 'Person data' and 'Car stuffs'.

kotti_multilingual: translated content

kotti_multilingual was started by Andreas <disko> earlier this year. It includes basic support for language roots and switching between languages. The sprinters decided to use this package as the starting point for adding more advanced content translation support.

Before getting down to coding, we first had a round of discussion with Domen <iElectric>, Ramon <bloodbare>, Jure <ibi> and me. In particular, we discussed how plone.multilingual handles translations and 'language independent fields' and compared it to the venerable LinguaPlone. We decided that we'd use the LinguaPlone model, where translations derive from a so-called canonical document. This is the only document that holds the actual data for any of the language independent fields, and where all translated documents refer to when they look up those fields.

This is still a work in progress, but so far we've managed to h̶a̶c̶k̶ convince SQLAlchemy to return the value of a language independent attribute from the canonical document, if it exists.

We've also implemented linking between translations using a separate translations table. This, too, works quite similar to how LinguaPlone does it when you configure it to use one root folder per language (e.g. /en, /de, and so on). For locating the destination folder for new translations, we look up the parent document's translation, and put the new translation in there. Thus, a translation from /en/food/mexican will automagically be created at /de/food/mexican, given that /de/food is already a translation of /en/food.

Versioning support

Andreas pointed us to the SQLAlchmey Versioned Objects example. He had used it in a Pyramid project and it worked very well for him. So Vanč took this up on the last day of the sprint and implemented a prototype for history support for Kotti.

Vanč reports that his prototype saves the old version of a document anytime you edit it, and keeps all the old versions. Next on his list is adding a user interface for viewing older versions (there's merely a list right now), and adding options for deciding which content types to version and how many old versions to keep. Other things to work on are a separate button for when you want to save a new version, so that small changes can be made without creating a new version. And possibly a comment field to describe what the changes in your new version are about.

Finishing the tutorial

Natan took on the important task of finishing the developer tutorial. Previously, the tutorial introduced concepts such as setting up a new project, configuring Kotti, adding Poll and Choice content types, and simple views. But it did not have forms to actually vote on the poll. This is now being added.

During the sprint, we also discussed which are the most important topics that the docs don't cover well enough yet. We decided we should add more documentation for users and security, workflows, and a more advanced forms example.

Search and kotti_solr

Andi (witsch), Marco and Nuno worked on adding search to Kotti. Kotti now has a search box at the top right of the site that searches the 'title' and 'description' fields of content in the database. A setting called kotti.search_content allows the search function to be overridden. Which is where kotti_solr hooks in to provide search through Solr. kotti_solr has a buildout that installs Solr, and it's already working, but will currently only search the same 'title' and 'description' fields. (That is, it will need to grow adapters so that we can extract all searchable text from content.)

The basic search has already landed in Kotti's master branch in Git and will be available in the next version. There's still a few issues that need to be worked on here before that can happen though:

• Do permission checks with search results; at the moment search will return all items, whether the user has permission to see them or not. (Maybe this is a good time to consider adding something like CMF's allowedUsersAndRoles index to Kotti's Node.)
• Document the kotti.search_content setting.

We could also use some help in styling the search box and search results better. You can test drive the search on the demo site.

Snippets, slots and pyramid_snippets

Christian, Florian, Krille and Ibi worked on adding support for what Wordpress calls Shortcodes and MoinMoin calls macros.

With this, users will be able to insert codes like [gallery] into their pages and have these render to dynamic elements in the public view. Shortcodes may also take arguments like [table-of-contents level=3]. We discussed this idea back in February on the list.

The team decided they would move the core functionality behind what they call snippets into a separate package pyramid_snippets. The idea is that snippets behave like views with the advantage that they can be easily pulled in via AJAX and be used in other places in the site, like in the portlet columns. The integration with Kotti is happening in the snippets branch. Plans are to add a button to TinyMCE that allows users to add snippets instead of having to remember the codes and syntax.

We also want to eventually deprecate Kotti's slots and unify them with snippets so that slot renderers will become views and can be used as snippets and vice versa.

Images

Anreas (disko) added a new Image type and image scaling to Kotti core. He extracted these out of the nice kotti_imagegallery add-on that he had written earlier.

Predefined image scales include span1 (60x120) to span12 (1160x2320). Additional image scales can be configured through the kotti.image_scales variables.

Rich text editing

Andreas (disko) and Daniel worked on improving the rich text editing story for Kotti. Based on the work done on images and scaling by Andreas, they created kotti_tinymce, which is an add-on that provides image and file upload, browsing and insertion through TinyMCE.

kotti_tinymce is actually based on the Products.TinyMCE add-on for Plone, and uses the exact same JavaScript resources. Only page templates and CSS are overridden and the server-side AJAX hooks implemented to work with Kotti. (We were quite amazed at how well this worked out.)

One problem with kotti_tinymce is that there's no functional tests. We've just verified that things are working by clicking through all the dialogues manually. Considering that Products.TinyMCE is evolving separately and that we want to maintain upstream compatibility, we really want to implement some browser-based functional tests to be able to guarantee a reasonable level of stability. These tests will also help us fix the remaining bugs in kotti_tinymce.

You can try out the rich text editing improvements on the demo site!

HTTP Caching

Andreas (disko) and Daniel added HTTP caching into the Kotti core. There's now a default caching policy which may be overridden using the kotti.caching_policy_chooser setting. The default policy is to not cache HTML, cache media for four hours, cache static resources for a month and so on.

The http caching code is based on code previously written by Daniel.

Paster templates

Marco and Tom worked on kotti_paster which Marco initially started. They've been working on a set of accumulative templates for Kotti that are based around Buildout and try to define best practices for different aspects of Kotti and Pyramid development for beginners.

Problem

Now imagine that in our application we're using some fancy notifications plug-in that displays messages as a nice pop-up. This is useful when working with Ajax since it'll guarantee that the user actually sees the notification even when they have scrolled way to the bottom of the page. Typically, we'd have some code to turn the contents of <div id="message"> into a pop-up in our document ready handler:

$(document).ready(function() {
    displayNotification($("#message"));
});

What's the problem with this? Well, when we update our <div id="message"> through Ajax, the notifications pop-up won't display. This is because the document ready handler is not triggered for mere updates to the page.

We need to add a call to displayNotification to our Ajax success callback from before to make notifications work for the HTML that we inject dynamically:

function success(data) {
    var html = $(data);
    $("#message").replaceWith(("#message", html));
    $("#content").replaceWith(("#content", html));
    displayNotification($("#message"));
}

Then imagine that you're using more progressive enhancement to turn <ul> list elements elements with a class dropdown into a dropdowns. Again, we need to add a bit of code to both our success handler and to the document ready handler:

$(document).ready(function() {
    displayNotification($("#message"));
    makeDropdowns($("ul.dropdown")); // new!
});

function success(data) {
    var html = $(data);
    $("#message").replaceWith(("#message", html));
    $("#content").replaceWith(("#content", html));
    displayNotification($("#message"));
    makeDropdowns($("ul.dropdown", $("#content"))); // new!
}

Notice how the newly added call to makeDropdowns in function success passes on only ul.dropdown elements inside #content, that is, only the dropdown lists that were injected just now:

makeDropdowns($("ul.dropdown", $("#content")));

By passing only the element that's changed we avoid having to somehow remember in function makeDropdowns which lists were already turned into dropdowns and which not. These takes quite some burden off of these functions.

The problem that emerges as we add these more and more enhancement functions like makeDropdowns and displayNotification is that we keep adding slightly different code to both our document ready handler and to a number of success handlers that we might have created in our app. This sort of duplication of code is bad. Let's try to generalize a bit more.

Solution

Let's first create a unified interface for handlers like displayNotification and makeDropdowns. All of these should have the form:

function handler(node) {
  // do our progressive enhancement here, but only inside node
}

As discussed before, we decide to pass in only the node that has changed. The individual handlers can then apply their enhancements only to the updated parts of the page. We'll then create an array of handlers so that later, when the DOM has been updated, we can call the handlers one by one:

var node_changed_handlers = new Array();
node_changed_handlers.push(displayNotification);
node_changed_handlers.push(makeDropdowns);

function node_changed(node) { // call handlers with 'node' one by one
    $.each(node_changed_handlers, function(index, func) { func(node) });
  }

We can now go back to our document ready handler and to our success handlers and simplify them substantially:

$(document).ready(function() {
    node_changed($('html'));
});

function success(data) {
    var html = $(data);
    $("#message").replaceWith(("#message", html));
    $("#content").replaceWith(("#content", html));
    node_changed($("#message"));
    node_changed($("#content"));
}

No longer do we now need to change the code of both of these whenever we add a new handler to node_changed_handlers. That's good.

Still, writing individual success handlers for all sorts of different actions and Ajax requests, of which there are usually many in a modern web app, is cumbersome. Ideally, our success handler and the server could use some more intelligent protocol that would allow us to reuse the same success callback function for all our application's Ajax requests. In short, we want a success handler that we can use for all our Ajax needs.

In order to achieve this, we add a little bit more information to the HTML that is returned from the server and processed in our handlers. We add a class ajax-replace to all those elements that need to be updated in the page. Here's the example from before with the class added:

<html>
  ...
  <div id="message" class="ajax-replace">Successfully saved.</div>
  ...
  <div id="content" class="ajax-replace"> ... some updated content ... </div>
  ...
</html>

We can now strip out the bit in our success handler that looks for specific ids and generalize it to this:

function success(data) {
    var html = $(data);
    $(".ajax-replace", html).each(function() {
        var selector = "#" + this.id;
        $(selector).replaceWith(this);
        node_changed($(selector));
    });
}

How does this work exactly? It looks for all elements in the server response's HTML with the class ajax-replace (line 3) and replaces elements with a corresponding id in the current DOM (lines 4 and 5). It then calls all node changed handlers with the newly added element (line 6).

Voila! What we have here is a very powerful and simple success handler that's reusable for all our Ajax requests.

Demo

I've recently added Ajax forms and Growl-like notifications (using the jquery-toastmessage-plugin) to the Kotti CMS, a user-friendly light-weight CMS that I'm building on top of Pyramid and jQuery.

Take a look at Kotti's JavaScript code, which implements just the approach I've presented here. In particular, take a look at function messages and function dropdowns which correspond to the two handlers described in this tutorial.

To see this code in action, log in to the Kotti demo server with the username owner and password secret and try the reorder form.