libblas and liblapack issues and speed, with SciPy and Ubuntu

December 19, 2012 | categories: Python, Programming | View Comments

The problem

You've built from source a package that uses the linear algebra libraries BLAS and LAPACK, but now you're getting: cannot open shared object file: No such file or directory

In my case, I was trying to install SciPy from source using pip install scipy, but then ran into this issue when I tried to use it. I also ran into this other problem:

ImportError: scipy/linalg/ undefined symbol: clapack_sgesv

To fix it

  1. Make sure you've got libatlas3-base installed:

    sudo apt-get install libatlas3-base
  2. Make sure you're using the right implementation of, that is, the one that libatlas3-base provides. This implementation lives in /usr/lib/atlas-base/atlas/

    To do this, run the following command to set your system's default Atlas implementation:

    sudo update-alternatives --config

    An example of what this might produce, and what number you might need to enter:

    Selection      Path                                     Priority   Status
    * 0            /usr/lib/openblas-base/   40        auto mode
      1            /usr/lib/atlas-base/atlas/    35        manual mode
      2            /usr/lib/libblas/             10        manual mode
      3            /usr/lib/openblas-base/   40        manual mode
     Press enter to keep the current choice[*], or type selection number: 1
  3. Use from /usr/lib/atlas-base/atlas/ as your default:

    sudo update-alternatives --config

That's it. Now you shouldn't be getting the aforementioned errors anymore.

More information

The Debian Wiki has an overview of linear algebra libraries available, and it also describes how to use update-alternatives to switch between BLAS and LAPACK implementations.

Speed it up! Build an optimized Atlas for your architecture.

The README.Debian file of libatlas3-base explains:

Building your own optimized packages of Atlas is straightforward. Just get the sources of the package and its build-dependencies:

# apt-get source atlas
# apt-get build-dep atlas
# apt-get install devscripts dpkg-dev dch

and type the following from the atlas source subdir:

# fakeroot debian/rules custom

it should produce a package called:


which is optimized for the architecture Atlas has been built on. Then install the package using dpkg -i.

If you, like me, get this "classical" error when building Atlas:


then you'll need to read this: Your install dies with "unable to get timings in tolerance".

Lastly, you might need to remove any existing libopenblas installation from your system for the building of the Debian package to finish successfully:

sudo aptitude purge libopenblas-{base,dev}
Read and Post Comments

Use apt-get to install Python dependencies for Travis CI

November 23, 2012 | categories: Python, Programming | View Comments

Travis CI is used by more and more open source Python projects to do their continous testing. scikit-learn is the latest project to adopt it.

With Travis, usually you'll use pip or setuptools to get your project's Python dependencies installed before running the test script. Here's the minimal .travis.yml example from the Travis docs for Python, one that will install dependencies using pip before it runs nosetests:

language: python
  - "2.6"
  - "2.7"
  - "3.2"
# command to install dependencies
install: "pip install -r requirements.txt --use-mirrors"
# command to run tests
script: nosetests

There is however some dependencies where this is problematic. Two of those are numpy and scipy, which contain a lot of C code that, with the method just discussed, needs to be compiled every time you run the Travis tests.

Travis allows you to install system packages through apt-get, which is quite cool. And there's already the binary python-numpy and python-scipy packages in Ubuntu, so why not use them?

The problem is that simply installing them via apt-get does not work for the same reason it doesn't work when you do this locally: The default virtualenv that Travis sets up for you to run the tests in is isolated from the system packages, so it won't see those globally installed numpy and scipy packages.

The solution for this is use virtualenv with the --system-site-packages option, which allows you to also import packages from the global site packages directory.

How it works

Add these lines to your Travis configuration to use a virtualenv with --system-site-packages:

  system_site_packages: true

You can thus install Python packages via apt-get in the before_install section, and use them in your virtualenv:

 - sudo apt-get install -qq python-numpy python-scipy

A real-world use of this approach can be found in nolearn.

Read and Post Comments

python-mode gone wrong

November 16, 2012 | categories: Python, Emacs, Programming | View Comments


If you recently upgraded Emacs' python-mode to version 6.0 or higher, and it feels wrong (indentation seems broken, the menu is confusing), downgrade to version 5. In Debian or Ubuntu you can do it like so:

sudo aptitude install python-central pymacs
sudo dpkg -i python-mode_5.1.0-1_all.deb

Broken hearts

Something terrible happened. My beloved python-mode for Emacs has been infested with featuritis, and it doesn't look like it's coming back.

Yesterday I upgraded to Xubuntu 12.10 for my work laptop. That was after it turned out that I'm not manly enough to install Debian Wheezy proper. (What the f--k is wrong with your WLAN support, Debian!!)

With Xubuntu and a few tweaks you can luckily ignore all the less fortunate decisions that Ubuntu has made recently. No Unity means no Dash means no broken privacy out of the box.

Synaptic and GDebi are quickly installed to replace the ad-ridden and sluggish Ubuntu Software Center. And packages rock again and are fast. Also, I've come to really like Xfce; but I digress...

So far so good. Then I installed Emacs and python-mode and started editing my first Python file. Which was when I fell into a deep shock. A few things felt just plain wrong. In fact so wrong that I was sure I had the wrong mode active (namely the other python.el, which ships with Emacs). For hours and hours I searched for a way to disable Emacs' own Python mode and enable the "good one", tried some hacks, but it wouldn't work. Because, really I was using the right mode all the time, but python-mode had changed so much I couldn't recognize it anymore.

Firstly, the menu entries are completely different. Instead of the two entries "IM-Python", and "Python" (which I hardly ever used), there's now five different menus called "PyShell", "PyEdit", "PyExec", "PyTools" and "Outline", each with a shitload of entries inside, with mostly either trivial things, or stuff I have no idea what they're good for.

Take an example: The newer python-mode added skeletons. So that now instead of writing a for statement, you can now call up a macro that will ask you what to put into the name and expression parts of the statement, and then it'll write it. Yes, friends, imagine that instead of writing for foo in baz:, you can now call up this insanely useful macro and it will ask you to enter foo and then you hit enter and then it asks for baz, enter, and then it will write the line for foo in baz:.

What the f--k?

Surely, you could say I can ignore all of this seemingly useless functionality. After all, it might be useful to someone (it's NOT!!). But it still leaves a very bad taste in my mouth. I feel alarmed that my beloved python-mode seems to have taken a step into a very bad direction. And I wonder if with all these added features, is it going to be as well-maintained? Is it going to make similar decisions in the future, ones that will affect me more? (This is software that I rely on for my daily work.)

And it looks like my fears are already confirmed. It turns out python-mode gets indentation wrong, arguably the most fundamental feature of any Python editing mode, or has anyway changed the way it works. As an example, if you hit enter where the X is below, it will indent four characters:

class Bar:
    def foo(self):

Xclass Baz:

So that you'll end up with:

class Bar:
    def foo(self):

    class Baz:

(Where what you really wanted is put a blank line between them, or maybe add a new method to Bar.)

So what I did after I realized what was going on was to downgrade to python-mode 5.1, and finally my blood pressure went back to normal.

I'd be interested to hear other people's experiences with python-mode 6, I wonder if I'm alone.


Here's the two versions with their respective menus.

python-mode version 5

Simple is better than complex

python-mode version 5

python-mode version 6

F--k simple, let's do a million features because we can

python-mode version 6
Read and Post Comments

Pyramid EuroPython tutorial video

August 16, 2012 | categories: Python, Web, Pyramid, Programming | View Comments

Here's the videos and material for the Writing a Pyramid application tutorial that I gave at last month's EuroPython 2012.

The tutorial is around 3,5 hours long. The accompanying material includes code and documentation for the seven individual steps of the tutorial (ex1_hello through ex7_final).

(Head here if the videos aren't displayed.)

This is the tutorial's complete description of the tutorial from the EuroPython website:

Pyramid is a very general open source Python web framework. As a framework, its primary job is to make it easier for a developer to create an arbitrary web application. The type of application being created isn’t really important; it could be a spreadsheet, a corporate intranet, or a social networking platform. Pyramid is general enough that it can be used in a wide variety of circumstances.

In this tutorial you will build from scratch a simple app in which users each have their individual workspaces, and can invite other users to collaboratively edit content.

While writing the application, you will learn about many aspects of Pyramid. Concretely, you will learn how to:

  • Install Pyramid within a virtualenv
  • Use Pyramid's scaffolds to quickstart your project
  • Work efficiently with templates (using Chameleon)
  • Create models and views
  • Write your own user authentication
  • Make use of Pyramid's Access Control Lists support
  • Use Pyramid's Traversal to map URL paths to your objects
  • Create forms using the deform form library
  • Work with the ZODB, a convenient database to store your Python objects in.

The tutorial will be split into steps with each step accomplishing a certain set of goals. Documented code for all the individual steps, including the final application, will be made available to participants.

Prerequisites: If you're on Mac OSX or Windows and you don't have Python 2.6 or Python 2.7 installed, you might want to run through the Pyramid installation instructions prior to the tutorial.

Read and Post Comments

Kotti Werkpalast Sprint Wrap Up

May 28, 2012 | categories: Python, Web, Pyramid, Programming, Kotti | View Comments

Last week's Kotti sprint in Berlin was full of win. Here's a summary of what we worked on. (Sprinters: If I forgot to mention anything, please let me know and I'll update.)


Search and kotti_solr

Andi (witsch), Marco and Nuno worked on adding search to Kotti. Kotti now has a search box at the top right of the site that searches the 'title' and 'description' fields of content in the database. A setting called kotti.search_content allows the search function to be overridden. Which is where kotti_solr hooks in to provide search through Solr. kotti_solr has a buildout that installs Solr, and it's already working, but will currently only search the same 'title' and 'description' fields. (That is, it will need to grow adapters so that we can extract all searchable text from content.)

The basic search has already landed in Kotti's master branch in Git and will be available in the next version. There's still a few issues that need to be worked on here before that can happen though:

  • Do permission checks with search results; at the moment search will return all items, whether the user has permission to see them or not. (Maybe this is a good time to consider adding something like CMF's allowedUsersAndRoles index to Kotti's Node.)
  • Document the kotti.search_content setting.

We could also use some help in styling the search box and search results better. You can test drive the search on the demo site.

Snippets, slots and pyramid_snippets

Christian, Florian, Krille and Ibi worked on adding support for what Wordpress calls Shortcodes and MoinMoin calls macros.

With this, users will be able to insert codes like [gallery] into their pages and have these render to dynamic elements in the public view. Shortcodes may also take arguments like [table-of-contents level=3]. We discussed this idea back in February on the list.

The team decided they would move the core functionality behind what they call snippets into a separate package pyramid_snippets. The idea is that snippets behave like views with the advantage that they can be easily pulled in via AJAX and be used in other places in the site, like in the portlet columns. The integration with Kotti is happening in the snippets branch. Plans are to add a button to TinyMCE that allows users to add snippets instead of having to remember the codes and syntax.

We also want to eventually deprecate Kotti's slots and unify them with snippets so that slot renderers will become views and can be used as snippets and vice versa.


Anreas (disko) added a new Image type and image scaling to Kotti core. He extracted these out of the nice kotti_imagegallery add-on that he had written earlier.

Predefined image scales include span1 (60x120) to span12 (1160x2320). Additional image scales can be configured through the kotti.image_scales variables.

Rich text editing

Andreas (disko) and Daniel worked on improving the rich text editing story for Kotti. Based on the work done on images and scaling by Andreas, they created kotti_tinymce, which is an add-on that provides image and file upload, browsing and insertion through TinyMCE.

kotti_tinymce is actually based on the Products.TinyMCE add-on for Plone, and uses the exact same JavaScript resources. Only page templates and CSS are overridden and the server-side AJAX hooks implemented to work with Kotti. (We were quite amazed at how well this worked out.)

One problem with kotti_tinymce is that there's no functional tests. We've just verified that things are working by clicking through all the dialogues manually. Considering that Products.TinyMCE is evolving separately and that we want to maintain upstream compatibility, we really want to implement some browser-based functional tests to be able to guarantee a reasonable level of stability. These tests will also help us fix the remaining bugs in kotti_tinymce.

You can try out the rich text editing improvements on the demo site!

HTTP Caching

Andreas (disko) and Daniel added HTTP caching into the Kotti core. There's now a default caching policy which may be overridden using the kotti.caching_policy_chooser setting. The default policy is to not cache HTML, cache media for four hours, cache static resources for a month and so on.

The http caching code is based on code previously written by Daniel.

Paster templates

Marco and Tom worked on kotti_paster which Marco initially started. They've been working on a set of accumulative templates for Kotti that are based around Buildout and try to define best practices for different aspects of Kotti and Pyramid development for beginners.

Read and Post Comments

« Previous Page -- Next Page »