Category Archives: Tech

IRL#4: Istanbul

Last month we again held our biannual in-person gathering of all Niteans, the IRL. As is customary, we meet somewhere warm for the winter edition of the event. This year we chose Istanbul, Turkey. A stunning city with millennia of heritage.

We’ve gotten good at running IRLs so this one was by far the most productive. We’ve done several improvements to our policy and process (published in our Handbook), set good goals for the year and planned out the coming months. We had technical discussions, we had marketing brainstorms. We had coding sessions in the sun, we listened to lightning talks while sipping Mango Rum (thanks, Marbe!) and we took turns trying to beat Dejan at “foosball”. With a ton of fantastic food in between.

 

One of the outcomes I personally am most proud of is that we are charging ahead with our purpose, to help improve lives, using information and software as the main driving force. We are doing so by moving into a more mainstream market with our upcoming project so that we will be able to help “regular Johns and Janes” and not just hard-core internet marketers. There’s more: in the past, we have donated to great organizations such as LibreOffice, Wikipedia and Let’s Encrypt. Now, we’ve set it as a company goal for this year to make our impact on such causes an order of magnitude greater. In order to do that, we are 10X-ing our donations in 2018!

Since Niteo has grown a lot in the last year I finally got to meet some of the fellow Niteans in real life. It’s always nice to paint a face over that Slack profile. And go deep on some random topic with someone you only had text-based conversations before. If you work in a remote team, I heartily recommend getting everyone in the same location for a few days. It’s pure magic.

AWS EC2 gotchas

Lately, AWS has been giving us a lot of headaches on the Easy Blog Networks project. It turns out only because resources (we pay per how many things we can do per second) are fiercely measured on AWS, but the underlying problems are happening on all EBN providers. So when we go over operations per second limit which is 100, the EC2 hypervisor will limit entire server by lowering CPU resources. We see that as CPU steal. Now in order to figure out what exactly is using more than 100 operations per second, I installed netdata on a few servers. Which gave me the following picture:

In above picture we can see enormous spike in CPU usage by group of apache2 servers.

We had ~1.8k I/O operations(read,open,mmap syscalls), the limit is 100. And we still had a lot of credits to spare.

But AWS has this thing called burstable limit: you can have 100OP per second but you can also spike over that limit (3000 in our case). The burst is limited to 5-seconds moving average window if instance does not lower usage in that timespan, entire instance is punished by taking away CPU resources. One can disable this behavior on all x2 instances (Enable T2 unlimited), but this will incur additional costs, if you have constant usage above limits and not just spikes.

Now what that means is that some scripts are doing more things than anything else (backups don’t even come close to this). Tracking it down I came to something running mysql query:

The process was up for almost 5 minutes, where it can only be up for 90 seconds. To get things in order I limited MySQL runtime from 28k seconds which is the default, to 30 seconds.

[mysqld]
interactive_timeout=30
wait_timeout=30
connect_timeout=30

This is after the lowered runtime limit was applied:

I’m still tracking down what exactly is causing extremely expensive queries because when the server gets spanked by AWS, every query is considered “costly” and then no monitoring helps. But if we catch it just before it happens, we have the culprit! 🙂

To be continued …

Recording Talks

Last year on IRL#2, our biannual in-person get-together, we decided to start recording talks, so that those that could not attend in person would be able to watch them at some later point in time. Turns out, there is another great use case for these recordings: onboarding. Whenever a new person joins our team, these talks help to get them up to speed. The recordings have been a huge success and we’ve started doing them regularly when we give a talk, at local meetups, at conferences without dedicated video teams, etc.

Our setup is opinionated based on what we need and what devices we already have. It tries to be as simple & portable as possible while keeping good audio quality:

  • An iPhone for video: video is not very important when recording talks and iPhone’s camera is good enough.
  • A fantastic external shotgun microphone for audio: we often have more than one speaker (at once or alternating), we have questions from the audience, we have remote attendees pitching in, etc. We wanted our setup to be invisible, without attaching microphones to speakers’ shirts, without passing microphones around for questions and without complicated channels mixing in post-production. It turns out, the Rode VideoMic Pro “shutgun” microphone does the job perfectly. We set it up 7 to 10 meters from the speaker, among the audience, pointed at the speaker. Since the microphone is directional, it picks up the speaker’s voice very well while also allowing the comments from the audience to get recorded with sufficient quality. Set and forget! Do note that you need the SC4 adapter to be able to use the VideoMic with an iPhone.
  • Use FiLMic Pro app on iPhone for best quality audio recording and audio monitoring via headphones.
  • A basic tripod with a custom smartphone mount so that we can mount both the iPhone and the mic on the same tripod, point both at the speaker and hit record.
  • Use Airdrop to send the video file to macOS and use iMovie for basic video editing. No cables involved there and iMovie’s intuitive UI makes the workflow as painless as possible.

Assuming you already have an iPhone and a tripod laying around, a rig like this will cost you around 200€, which is a bargain considering the quality and convenience you get:

  • 150€ Rode VideoMic Pro
  • 11€ Rode SC4 TRS-TRRS adapter
  • 13€ Rode SC6 Dual TRRS adapter
  • 17€ FiLMiC Pro app
  • 10€ cheap smartphone tripod mount
  • 10€ male and female 1/4 to 3/8 thread adapters

Custom smartphone mount

I was hunting around Amazon & eBay and could not find a similar thing that would ship fast enough before IRL#2, so I built one myself. I bought a “smartphone tripod mount” at a local gadget shop and glued in a female 3/8 to 1/4 tripod screw adapter so it has an additional tripod screw insert. Now I can screw the mount into a tripod, mount the iPhone and then screw in the VideoMic into the new insert I added to the top of the mount, using the male 3/8 to 1/4 tripod screw adapter. Works like a charm!

iMovie workflow

First, create images of all slides. If you have the presentation in Keynote, you can export as images. If you have it as PDF, you can use Automator to create images.

OK, fire up iMovie and let’s start:

  • import talk recording
  • start a new project
  • drag clip into the project area
  • click the “magic wand” icon to auto-tune audio and video settings
  • right click -> Show Clip Trimmer -> remove the noise in the beginning and end of the talk
  • insert slide image a few seconds after the slide is changed in the video, as “video overlay“
  • select all “video overlay” images, select “cutaway” and set opacity to ~80%
  • add first slide image to the beginning of the clip, add cross dissolve transition
  • add cross dissolve to the end of the clip
  • click File -> Share -> File to render the final video of the talk (Resolution: 720p, Quality: High, Compress: Better Quality)
  • use Handbrake to compress the video
    • Web optimized ticked
    • Format: MP4 File
    • Video Encoder: H.264
    • Framerate: 24
    • Peak Framerate (VFR) selected
    • Constant Quality selected and RF:20
    • Encoder Options: Preset: slow
    • Picture: Storage Size: 1280 x 720
    • Audio: Mixdown: Mono
    • Upload to videos.niteo.co (AWS S3 static website using github.com/oscherler/pretty-s3-index-html)

Redundancy

After recording 10+ talks we realized that things sometimes go wrong and it’s usually a human error: forgot to turn on the microphone, forgot to press record, etc. We now do the following to decrease the chance of not having a recording.

GoPro backup

Mount a GoPro to one of the tripod’s legs, using the standard GoPro pole mount. Stick a huge SD card in the camera, turn it on while setting things up and keep it running. If all else fails you at least get a backup recording.

Audio monitoring

With recording talks, it’s the audio quality that you should be most concerned with. Therefore it comes super handy to be able to monitor audio that is being recorded. This is how you do it:

  • Plug the Rode SC6 into your iPhone. Then plug the VideoMic + SC4 adapter to one of the mic inputs on the SC6 adapter. Finally, plug your headphones into the headphones output on SC6 adapter.
  • Use the FiLMic Pro app on iPhone to get audio passthrough, i.e. to hear in headphones what the microphone is recording.

Checklist

Print out this recording checklist and run through it before every talk:

  • Enable Airplane Mode on iPhone.
  • Turn off all apps running on iPhone.
  • Plug the charger cable into iPhone.
  • Verify VideoMic is correctly plugged into iPhone.
  • Turn VideoMic on, move switch all the way to the right (“curved line” icon) to enable the high pass filter that reduces background noise.
  • Start FiLMic Pro app and verify its configuration (click the Overview icon):
    • 1280×720 @ Apple Standard
    • 24FPS / 24FPS
    • PCM @ 48.8 Kbps Mic: External Microphone
  • Lightly touch the surface of VideoMic. You should hear scratching in your headphones. If you don’t, then FiLMiC is using iPhone’s internal microphone and you should go a few steps back.
  • Ask someone to act as the speaker for a few seconds and try which volume setting on VideoMic works best for given conditions. The farther away the VideoMic is from the speaker, the more boost it needs (+20).
  • Start recording.
  • Stop recording.
  • Turn off the VideoMic.

Sample

Here’s a few seconds of a recent recording from IRL#4 in Istanbul.

Sample for Recording Talks blog post from Nejc Zupan on Vimeo.

Upgrading our Intranet to Plone 5

While we haven’t done any Plone consulting (well, *any* consulting to be exact) in over 3 years, we still use Plone internally, every single day. It was about time our Intranet got some love, so I put on my disco pants, poured some of Belgium’s finest and got to work.

Our installation was an old Plone 4.1 one. It was about 3 years since we last touched it (apart from applying security hotfixes, obviously). The first task was to clean up and update our buildout environment. Mostly throwing things away. I like my Plone vanilla flavored. What followed was making sure our tests still pass and enabling Travis CI for the project.

Then came the upgrade to a new major version of Plone. The Plone 5. And let me tell you how the upgrade went:

  1. Bump version to 5.0.6, make sure buildout uses version pins that come with Plone 5.0.6.
  2. Run buildout & restart server.
  3. Click a big honking “Upgrade” button.
  4. Crack open another beer.

I mean seriously, people! Why can’t all software have such smooth upgrade paths!? Plonistas, you have my utmost respect.

Back in the Plone 4.1 days we still had Archetypes, an ancient framework for defining types of content, but Plone 5 is all about Dexterity, so I migrated that too. With a click of a button. Zero hassle once again.

What followed was some throwing content around, updating workflows and permissions to match our company structure and that was mostly it. Oh yeah and I released a Plone 5 remake of our old “niteoweb.loginas” add-on, now promptly named “collective.impersonate“.

If felt good to do some Plone after such a long time. I was surprised how polished the version 5 is. The docs are miles better than what I remember, the UI is fantastic and the thing is now really snappy.

That said, there’s a few more things I would like to get done:

  • increase portal max width to ~1200px
  • fix a styling bug in State tab in toolbar
  • red font color for private items in Navigation and Sidebar
  • and a few more minor tweaks here and there

Are you a Plone freelancer who can take care of these for us? Shoot us an email to jobs at niteoweb.com! Bonus karma points for getting your fixes merged upstream.

Our fancy intranet!

Strings in Python 2 and Python 3

The goal of this post is to show you how to properly use encode and decode in python 2 and in python 3. This post will be based on small examples that will (hopefully) make you better understand how strings work in python 2 and python 3.

A bit of background on unicode and UTF-8:

Unicode has a different way of thinking about characters. In Unicode, the letter “A“ is a platonic ideal. It’s just floating in “heaven”. Every platonic letter in every alphabet is assigned a magic number by the Unicode consortium which is written like this: U+0639 (in python “\u0639“).

UTF-8 is a system of storing your string of unicode code points (those magic “U+number“) in memory using 8 bit bytes.

One of the common questions for python 3 is when to use bytestring and when to use strings as an object? When you are manipulating string (e.g. “reversed(my_string)“) you always use string object and newer bytestring. Why? Here is an example:

my_string = "I owe you £100"
my_bytestring = my_string.encode()

>>> print(''.join([c for c in reversed(my_string)]))
001£ uoy ewo I
>>> print(''.join([chr(c) for c in reversed(my_bytestring)]))
001£Â uoy ewo I

 

The first print is what we expect but the second is not. And why is that? Well the “reversed“ function iterates over a sequence which in second case is bytestring which is b’I owe you \xc2\xa3100′. We can also verify this by checking the length of “my_bytestring“ and “my_string“:

>>> print(len(my_string))
14
>>> print(len(my_bytestring))
15

 

If I always just add “.encode()“ everything will be fine right? No! For start you should never call encode without specifying which encoding to use because then the interpreter will pick for you which will “almost” always be UTF-8 but there are some instances where this won’t be so and you will spent a lot of time finding this bug. So ALWAYS specify which encoding to use (e.g. “.encode(‘utf-8’)“). Example:

>>> print('I owe you £100'.encode('utf-8').decode('latin-1'))
I owe you £100

 

The other problem which is even bigger with “sprinkling” “.encode()“ is that if you already have encoded string you will get error (in python 3) or even worse (in python 2), you will do string operations on bytestring.

In python 2 “str“ is for strings of bytes and “unicode“ is for strings of unicode code points. The problem is that python 2 implicitly converts between types… sometimes. It allows you things like this:

>>> print((u'I owe you £100'.encode('utf-8') + 'Plus another $100').decode('latin-1'))
I owe you £100Plus another $100

 

This will quickly raise error when “Plus another $100“ becomes something that is not ASCII. If you try this in python 3 you get “TypeError: can’t concat bytes to str“.

If you need your code to run both on python 2 and python 3 then a rule of thumb is to first write a code for python 3 and then try it in python 2.
References: