Tag Archives: Buildout

Dear Plone, welcome to year 2014

TL;DR: Production-level Plone on free-tier Heroku: https://github.com/niteoweb/heroku-buildpack-plone

First, a bit of history: it was year 2006 and I was realizing that I was not made to be an academic. I made my first strides into entrepreneurship and being in IT, the first logical step was to create a few websites and try to get paid for my work. I did have some programming experience but haven’t done any web development yet. I heard PHP was not the ideal solution so I started looking elsewhere. In my student association I heard about this Plone thingy and gave it a go: I purchased a 20€ per month shared Plone hosting account at Nidelven IT and started hacking. It was the Plone 2.x era and boy was I productive! I threw in content, installed a ready-made theme and did some TTW tweaks. Done! First customer happy! Rinse & repeat, upgrade to a beefier server, rinse & repeat. NiteoWeb Ltd. was born.

Fast forward to a couple of months ago: we used GeckoBoard to drive a wall-mounted dashboard display. GeckoBoard works fine, but they want almost $60 per month if you want to drive more than one display. Sounds quite expensive for a bit of HTML and JavaScript, doesn’t it? So I looked around for alternatives and one of them was the Dashing dashboard from Spotify. I was reluctant to even give it a try as it was a self-hosted Ruby app. And I didn’t know *any* Ruby. But there, in their documentation, I found a short guide on how to deploy your very own version of a sample dashbord to your personal Heroku account. And when I say short, I mean short! I copy/pasted 6 simple commands into my console and that was it! My very own dashboard! After it was running I was motivated enough to read through Dashing’s docs and did a few minor tweaks. One “git push heroku master” later, my changes were again deployed to Heroku and showing up on my dashboard display. Wow, is this developer-friendly or what!

My mind drifted and I thought: Boy wouldn’t it be nice if a non-Plone person could come to a Plone add-on page and create their very own Plone instance with the add-on installed, and they could immediately start using it. Uhm … why not? Why don’t we, as a community, provide “private” demos that people can use? Is there something that prevents us from using Heroku for demos, the same was as Dashing, and many other Ruby products, use it?

Turns out, there isn’t. During the Plone dinner at EuroPython 2014 conference last week, I ordered a few beers and got hacking. Goal: get Plone to run on a free-tier Heroku account.

There have been attempts to run Plone on Heroku before, but they failed because they took the wrong approach. Look, Heroku, by default, supports Python apps that are installed with “pip“. Previous attempts were all focused on fixing Plone so it could be installed with pip. And they failed, Plone’s ecosystem is just too complex.

I decided to take a different approach: buildpacks. Heroku allows you to build *anything* on it. So I created a buildpack that supports zc.buildout. Once I got that done, it was not far from getting Plone installed on Heroku.

The next roadblock came in the form of Heroku’s ephemeral filesystem. Everytime your Heroku “dyno” is restarted, the filesystem is recreated. And you lose your Data.fs. Humpff. Wait, but what about the PostgreSQL that Heroku offers? A production-quality PostgreSQL, for free, with a 10k row limit. That could work! So, add in support for RelStorage and you have a production-ready Plone site running on Heroku. For free. And you are not limited to one, you can have as many as you wish, on the same account. Heroku really is awesome!

So, Plone is suddenly again a viable option for college dropouts starting their businesses! No need for system administration knowledge, how to setup Nginx in front of Plone, how to do proper backups, just a few command-line commands and your site is online!

And, our add-ons can now have demos. If you use Data.fs instead of PostgreSQL, the demo instance’s data will be recreated at least once per day, giving visitors an up-to-date demo instance, displaying what your Plone add-on does and how it looks.

Does this really works? Hell yeah it does! This blog has been running on Heroku since last week! And here’s a Plone 4.3 demo, a Plone 5 demo and a collective.cover demo. Wanna see plone.app.mosaic in action?

Why doesn’t your add-on have a demo yet? Follow instructions on https://github.com/niteoweb/heroku-buildpack-plone and showcase your add-on to the world!

Plone 4 dev on Lion

Introduction

Recently I upgraded to OS X Lion and here are my notes on how I got my working environment working.

XCode

First things first, you need to setup build tools (gcc, make and the like). On OS X these come as a part of XCode. Even if you had XCode installed on Snow Leaopard before upgrading to Lion, do take time and reinstall it completely. Stuff changed since the white cat and it’s safer to have your build tools up-to-date.

collective.buildout.python

Same story as with XCode. Remove your collective.buildout.python directory entirely and recreate it from scrach. It takes a while, go make yourself some nice tea.

UPDATE: In OSX 10.7.1 readline does not get built because it’s configuration script does not correctly recognizes the OS. You can fix this by appending the following two lines to the bottom of src/readline.cfg in your local checkout of collective.buildout.python and re-running buildout.

make-options =
    SHOBJ_LDFLAGS="-dynamiclib"

egg cache

If you are using a local egg cache it might get tricky. Some people report everything works fine, but several people also report that things were randomly breaking until they had removed all eggs from their egg cache and re-ran buildout. My suggestion? Clear the cache! It takes a while to download all those eggs again, but it beats losing a day chasing readline related SEGFAULTS causing Zope crashes on seemingly random requests:

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0x0000000000000000
0x00000001131ccf09 in call_readline ()

lxml

If you are using z3c.recipe.staticlxml you are in more trouble. The latest version uses libxml2-2.6.32 and libxslt-1.1.24 and these two break in a very nasty way when you try to access a Plone site themed with Diazo. Zope crashes with no traceback. Attaching gdb to the process spits out the following error:

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0x0000001400001c4f
[Switching to process 89119]
0x00007fff8dbcd403 in strncmp ()
(gdb) bt
#0  0x00007fff8dbcd403 in strncmp ()
#1  0x00000001088e5830 in __xmlParserInputBufferCreateFilename (URI=0x1400001c4f <Address 0x1400001c4f out of bounds>, enc=163580032) at xmlIO.c:2412
#2  0x00000001088bc951 in xmlNewInputFromFile (ctxt=0x10985e5b0, filename=0x10407fd30 "/Users/zupo/work/example.project/eggs/diazo-1.0rc3-py2.6.egg/diazo/defaults.xsl") at parserInternals.c:1462
[...snip...]

One solution is to add the following two lines to the [lxml] part of your buildout.cfg:

libxml2-url = ftp://xmlsoft.org/libxml2/libxml2-2.7.8.tar.gz
libxslt-url = ftp://xmlsoft.org/libxml2/libxslt-1.1.26.tar.gz

These tell lxml to build against newer versions of libxml2 and libxslt making those nasty errors go away.

UPDATE: You can just as well use the latest version of z3c.recipe.staticxml which includes URLs to newer versions of both libxml2 and libxslt.

The other solution is simply to not use z3c.recipe.staticlxml at all. It seems that you can now use your system’s libxml provided to you by XCode (located at /usr/include/libxml2). To do this, remove [lxml] from your buildout.cfg and re-run buildout.

Assertion `t_size >= b_size’ failed

Recently, when migrating a Plone 4 site from one VPS server instance to another, we had the following problem:

We were not able to start Zope with Supervisord because it crashed every time we tried to start it. Running Zope in foreground (bin/instance fg) produced the following error:

Python: Objects/typeobject.c:1736: extra_ivars: Assertion `t_size >= b_size' failed.
Aborted

Luckily, after some Googling around I discovered that Hanno had already commited a patch to fix this. Pinning eggs to patched versions by adding the lines below to versions.cfg fixed the problem entirely.

# Handle tp_basicsize correctly
ExtensionClass = 2.13.2
Persistence = 2.13.2

Deploying Cyn.in 3.1.3 on CentOS 5.4

Last week we were upgrading Cyn.in to the latest version. After a few days of testing on a local server it was time to deploy it to the server.

Since Cyn.in needs quite some RAM to operate normally, we chose Virpus VPS instance with 2 gigs of RAM, running CentOS 5.4.

Running buildout on the server produced this error:

...
Modules/constants.c: In function ‘LDAPinit_constants’:
Modules/constants.c:184: error: ‘LDAP_OPT_X_TLS_NEWCTX’ undeclared (first use in this function)
Modules/constants.c:184: error: (Each undeclared identifier is reported only once
Modules/constants.c:184: error: for each function it appears in.)

After a bit of googling around it seemed that the problem was caused by a bug in python-ldap. A quick look at CHANGES revealed that 2.3.10 version of python-ldap mentions work being done around LDAP_OPT_X_TLS_NEWCTX. Maybe it was fixed in the next version, 2.3.11?

I tried pinning the version of python-ldap to 2.3.11 and buildout finished without errors. Great!

Adding the following lines to buildout.cfg makes buildout more repeatable and future-proof:

versions = versions

[versions]
python-ldap = 2.3.11

Order of ‘parts’ when compiling lxml

CentOS’s repos don’t have a working version of libxslt (you need 1.1.20, repos have 1.1.17) so we need to statically compile it for collective.xdv to work.

But, there is a catch! You need to be careful about how you order your parts in your buildout.cfg.

For examle, the following buildout.cfg works perfectly fine, it downloads libxml and libxslt and compiles them in your buildout environment for your use:

[buildout]
extends = http://dist.plone.org/release/4.0/versions.cfg
parts =
  lxml
  zopepy

[zopepy]
recipe = zc.recipe.egg
interpreter = zopepy
eggs =
  lxml

[lxml]
recipe = z3c.recipe.staticlxml
egg = lxml
static-build = true

Problems start when you change the order of ‘parts’ section to be like this:

[buildout]
parts =
  zopepy
  lxml

...

In this case, zopepy wants to fetch the lxml egg and compile lxml immediately. Since the ‘lxml’ part has not been run yet, there is no libxml2 and libxslt available in your buildout and the whole thing crashes. The point being: always put the lxml part at the top of your parts list.

The fix is a one-liner and I just spent 2 days figuring it out. I wish I knew this earlier.

UPDATE: Florian Schulze pointed out that “an even better fix is to reference the lxml part from zopepy.

[buildout]
extends = http://dist.plone.org/release/4.0/versions.cfg
parts =
  lxml
  zopepy

[zopepy]
recipe = zc.recipe.egg
interpreter = zopepy
eggs =
  ${lxml:egg}

[lxml]
recipe = z3c.recipe.staticlxml
egg = lxml
static-build = true 

That way buildout always installs the lxml part first, because it installs a part before resolving any references to it. This way you actually only need zopepy in parts and can leave out lxml completely.”