Tag Archives: Plone

Dear Plone, welcome to year 2014

TL;DR: Production-level Plone on free-tier Heroku: https://github.com/niteoweb/heroku-buildpack-plone

First, a bit of history: it was year 2006 and I was realizing that I was not made to be an academic. I made my first strides into entrepreneurship and being in IT, the first logical step was to create a few websites and try to get paid for my work. I did have some programming experience but haven’t done any web development yet. I heard PHP was not the ideal solution so I started looking elsewhere. In my student association I heard about this Plone thingy and gave it a go: I purchased a 20€ per month shared Plone hosting account at Nidelven IT and started hacking. It was the Plone 2.x era and boy was I productive! I threw in content, installed a ready-made theme and did some TTW tweaks. Done! First customer happy! Rinse & repeat, upgrade to a beefier server, rinse & repeat. NiteoWeb Ltd. was born.

Fast forward to a couple of months ago: we used GeckoBoard to drive a wall-mounted dashboard display. GeckoBoard works fine, but they want almost $60 per month if you want to drive more than one display. Sounds quite expensive for a bit of HTML and JavaScript, doesn’t it? So I looked around for alternatives and one of them was the Dashing dashboard from Spotify. I was reluctant to even give it a try as it was a self-hosted Ruby app. And I didn’t know *any* Ruby. But there, in their documentation, I found a short guide on how to deploy your very own version of a sample dashbord to your personal Heroku account. And when I say short, I mean short! I copy/pasted 6 simple commands into my console and that was it! My very own dashboard! After it was running I was motivated enough to read through Dashing’s docs and did a few minor tweaks. One “git push heroku master” later, my changes were again deployed to Heroku and showing up on my dashboard display. Wow, is this developer-friendly or what!

My mind drifted and I thought: Boy wouldn’t it be nice if a non-Plone person could come to a Plone add-on page and create their very own Plone instance with the add-on installed, and they could immediately start using it. Uhm … why not? Why don’t we, as a community, provide “private” demos that people can use? Is there something that prevents us from using Heroku for demos, the same was as Dashing, and many other Ruby products, use it?

Turns out, there isn’t. During the Plone dinner at EuroPython 2014 conference last week, I ordered a few beers and got hacking. Goal: get Plone to run on a free-tier Heroku account.

There have been attempts to run Plone on Heroku before, but they failed because they took the wrong approach. Look, Heroku, by default, supports Python apps that are installed with “pip“. Previous attempts were all focused on fixing Plone so it could be installed with pip. And they failed, Plone’s ecosystem is just too complex.

I decided to take a different approach: buildpacks. Heroku allows you to build *anything* on it. So I created a buildpack that supports zc.buildout. Once I got that done, it was not far from getting Plone installed on Heroku.

The next roadblock came in the form of Heroku’s ephemeral filesystem. Everytime your Heroku “dyno” is restarted, the filesystem is recreated. And you lose your Data.fs. Humpff. Wait, but what about the PostgreSQL that Heroku offers? A production-quality PostgreSQL, for free, with a 10k row limit. That could work! So, add in support for RelStorage and you have a production-ready Plone site running on Heroku. For free. And you are not limited to one, you can have as many as you wish, on the same account. Heroku really is awesome!

So, Plone is suddenly again a viable option for college dropouts starting their businesses! No need for system administration knowledge, how to setup Nginx in front of Plone, how to do proper backups, just a few command-line commands and your site is online!

And, our add-ons can now have demos. If you use Data.fs instead of PostgreSQL, the demo instance’s data will be recreated at least once per day, giving visitors an up-to-date demo instance, displaying what your Plone add-on does and how it looks.

Does this really works? Hell yeah it does! This blog has been running on Heroku since last week! And here’s a Plone 4.3 demo, a Plone 5 demo and a collective.cover demo. Wanna see plone.app.mosaic in action?

Why doesn’t your add-on have a demo yet? Follow instructions on https://github.com/niteoweb/heroku-buildpack-plone and showcase your add-on to the world!

NiteoWeb attended a Pyramid sprint in Halle, Germany

Gocept, a company based in Halle (Saale), Germany, organized a Pyramid sprint, which lasted from 15th to 17th August 2013. The sprint took place at their headquarters which, by the way, has a lovely garden perfectly suited for relaxation, eating, drinking and development (not necessarily in that order!).

A bunch of NiteoWeb former and present employees took part, too. Despite the fact that Halle is about 10 hours away from Ljubljana if you go by car (quite a long ride indeed!), but it was well worth coming there. Gocept did an excellent job of feeding and entertaining all the sprint participiants and it was a great pleasure to meet Chris McDonough, author of the Pyramid web framework, in person. An interesting and amusing dude I must say.

Happy sprint participants were quite productive and a whole bunch of bug fixes and enhancements have been made – see the wrap up for more details. The only real downside was that the sprint ended so early, three days really passed in the blink of an eye. But hey, isn’t that always the case when you’re having fun?

So Gocept, thanks again for everything and hope to see you in 2014!

Write a Plone CLI maintenance script

This is a quick tip on how to write your own command line maintenance scripts for your Plone application.

Recently one of our clients expressed a need for a maintenance script which would walk the database of a Plone application and, for every object, print out all the roles users have assigned on that object. Since we were dealing with a large database, processing it would take a considerable amount of time, rendering a solution with a browser view (or similar) not particularly useful. We needed some kind of a command line utility which would occasionally be run in the background (e.g. as a cron job) and would output results to a file.

The question which arised next was how to write such a Python script so that it would hook up to the Plone application, its database and would at the same time already have access to all the Plone utilities it might need (i.e. Products.CMFCore). I searched around the internet quite a bit before I finally stumbled upon the missing part.

It turned out to be pretty easy actually. The key is to define a zopectl.command entry point in your package’s setup.py (i.e. in a package containing your maintenance script):

setup(
    ...

    entry_points="""
    [zopectl.command]
    dump_roles = your.package.module:some_callable
    """,

    ...
)

You can then add this package as a dependency of the bigger Plone project, run the buildout and there it is – your maintenance script (named dump_roles in our example) can be called like this:

$ bin/instance dump_roles <arguments>

When you run the command, some_callable from the your.package.module (as defined in setup.py) will be invoked. The callable will receive two arguments – the Zope root application object (as a starting point of object traversal) and the additional command-line arguments that might have been used on script invocation.

For the sake of completeness, here’s a skeleton of the actual maintenance script (defined in your.package.module):

def some_callable(app, cmd_args):
    """Module's entry point (zopectl.command).

    :param app: Zope root application object
    :param cmd_args: list of command line arguments used to invoke the command
    """
    pass  # your maintenance code goes here ...

Happy coding!

Load overrides.zcml in plone.app.testing

Today I was working on a project where we use overrides.zcml to easily override some default Plone behavior. All was working fine (in the browser, that is) until I started writing tests for our custom behavior.

First thing I noticed was that the overrides.zcml was not loaded in our test layer. “Doh, I need to load it manually with the loadZCML() method!” I thought to myself. Boy was I wrong :).

The problem with using loadZCML() is that it loads whatever ZCML you tell it to load in a “normal” way. So, obviously, you get conflict errors, since declarations in your overrides.zcml directly map declarations that they override. Hence I needed to find a way to tell plone.app.testing to load my overrides.zcml in an “override” manner and not throw conflict errors.

After quite some research and asking around, I got a tip from JC Brand on the #plone IRC channel: use xmlconfig.includeOverrides(). This indeed got me the exact result I wanted.

Here’s a snippet for my future self and any other plonista struggling with the same problem that happen to stumble upon this blog:

from plone.app.testing import PLONE_FIXTURE
from plone.app.testing import PloneSandboxLayer
from plone.testing import z2
from zope.configuration import xmlconfig


class MyProductLayer(PloneSandboxLayer):

    defaultBases = (PLONE_FIXTURE,)

    def setUpZope(self, app, configurationContext):
        import my.product
        self.loadZCML(package=my.product)
        xmlconfig.includeOverrides(configurationContext, 'overrides.zcml', package=my.product)
        z2.installProduct(app, 'my.product')

Dexterity vs. Archetypes

TL;DR: migrating your Archetypes content to Dexterity shrinks your Data.fs considerably!

I’ve started looking into migrating Archetypes content in one of the sites we’re running to Dexterity. But before I started coding I wanted to make sure that the juice is worth the squeeze.

The site contains roughly 130k content objects. Most of them are default Plone File type, with a few additional string fields (added with archetypes.schemaextender) and a custom workflow. Nothing fancy, just your standard integration project. New content is imported into the site in batches of ~10k items every now and then with the help of the awesome collective.transmogrifier.

To test how Dexterity compares to Archetypes for our use-case I first created a new Dexterity type that matched the feature-set of the Archetypes type. Then I created a fresh instance, used Transmogrifier to import 13k items (10% of our production data) and ran some numbers.

Results are pretty amazing. With Archetypes, an import of 13k items into a fresh Plone 4.2 site took 61 minutes and the resulting Data.fs had 144 MB (details). With Dexterity, the same import took only 18 minutes and the resulting Data.fs had 64 MB (details). That’s a whopping 70% decrease in import time and a 55% decrease in Data.fs size!

More than enough to be a valid reason to invest in rewriting our types and write that migration script.