IRL#4: Istanbul

Last month we again held our biannual in-person gathering of all Niteans, the IRL. As is customary, we meet somewhere warm for the winter edition of the event. This year we chose Istanbul, Turkey. A stunning city with millennia of heritage.

We’ve gotten good at running IRLs so this one was by far the most productive. We’ve done several improvements to our policy and process (published in our Handbook), set good goals for the year and planned out the coming months. We had technical discussions, we had marketing brainstorms. We had coding sessions in the sun, we listened to lightning talks while sipping Mango Rum (thanks, Marbe!) and we took turns trying to beat Dejan at “foosball”. With a ton of fantastic food in between.

 

One of the outcomes I personally am most proud of is that we are charging ahead with our purpose, to help improve lives, using information and software as the main driving force. We are doing so by moving into a more mainstream market with our upcoming project so that we will be able to help “regular Johns and Janes” and not just hard-core internet marketers. There’s more: in the past, we have donated to great organizations such as LibreOffice, Wikipedia and Let’s Encrypt. Now, we’ve set it as a company goal for this year to make our impact on such causes an order of magnitude greater. In order to do that, we are 10X-ing our donations in 2018!

Since Niteo has grown a lot in the last year I finally got to meet some of the fellow Niteans in real life. It’s always nice to paint a face over that Slack profile. And go deep on some random topic with someone you only had text-based conversations before. If you work in a remote team, I heartily recommend getting everyone in the same location for a few days. It’s pure magic.

AWS EC2 gotchas

Lately, AWS has been giving us a lot of headaches on the Easy Blog Networks project. It turns out only because resources (we pay per how many things we can do per second) are fiercely measured on AWS, but the underlying problems are happening on all EBN providers. So when we go over operations per second limit which is 100, the EC2 hypervisor will limit entire server by lowering CPU resources. We see that as CPU steal. Now in order to figure out what exactly is using more than 100 operations per second, I installed netdata on a few servers. Which gave me the following picture:

In above picture we can see enormous spike in CPU usage by group of apache2 servers.

We had ~1.8k I/O operations(read,open,mmap syscalls), the limit is 100. And we still had a lot of credits to spare.

But AWS has this thing called burstable limit: you can have 100OP per second but you can also spike over that limit (3000 in our case). The burst is limited to 5-seconds moving average window if instance does not lower usage in that timespan, entire instance is punished by taking away CPU resources. One can disable this behavior on all x2 instances (Enable T2 unlimited), but this will incur additional costs, if you have constant usage above limits and not just spikes.

Now what that means is that some scripts are doing more things than anything else (backups don’t even come close to this). Tracking it down I came to something running mysql query:

The process was up for almost 5 minutes, where it can only be up for 90 seconds. To get things in order I limited MySQL runtime from 28k seconds which is the default, to 30 seconds.

[mysqld]
interactive_timeout=30
wait_timeout=30
connect_timeout=30

This is after the lowered runtime limit was applied:

I’m still tracking down what exactly is causing extremely expensive queries because when the server gets spanked by AWS, every query is considered “costly” and then no monitoring helps. But if we catch it just before it happens, we have the culprit! 🙂

To be continued …

Recording Talks

Last year on IRL#2, our biannual in-person get-together, we decided to start recording talks, so that those that could not attend in person would be able to watch them at some later point in time. Turns out, there is another great use case for these recordings: onboarding. Whenever a new person joins our team, these talks help to get them up to speed. The recordings have been a huge success and we’ve started doing them regularly when we give a talk, at local meetups, at conferences without dedicated video teams, etc.

Our setup is opinionated based on what we need and what devices we already have. It tries to be as simple & portable as possible while keeping good audio quality:

  • An iPhone for video: video is not very important when recording talks and iPhone’s camera is good enough.
  • A fantastic external shotgun microphone for audio: we often have more than one speaker (at once or alternating), we have questions from the audience, we have remote attendees pitching in, etc. We wanted our setup to be invisible, without attaching microphones to speakers’ shirts, without passing microphones around for questions and without complicated channels mixing in post-production. It turns out, the Rode VideoMic Pro “shutgun” microphone does the job perfectly. We set it up 7 to 10 meters from the speaker, among the audience, pointed at the speaker. Since the microphone is directional, it picks up the speaker’s voice very well while also allowing the comments from the audience to get recorded with sufficient quality. Set and forget! Do note that you need the SC4 adapter to be able to use the VideoMic with an iPhone.
  • Use FiLMic Pro app on iPhone for best quality audio recording and audio monitoring via headphones.
  • A basic tripod with a custom smartphone mount so that we can mount both the iPhone and the mic on the same tripod, point both at the speaker and hit record.
  • Use Airdrop to send the video file to macOS and use iMovie for basic video editing. No cables involved there and iMovie’s intuitive UI makes the workflow as painless as possible.

Assuming you already have an iPhone and a tripod laying around, a rig like this will cost you around 200€, which is a bargain considering the quality and convenience you get:

  • 150€ Rode VideoMic Pro
  • 11€ Rode SC4 TRS-TRRS adapter
  • 13€ Rode SC6 Dual TRRS adapter
  • 17€ FiLMiC Pro app
  • 10€ cheap smartphone tripod mount
  • 10€ male and female 1/4 to 3/8 thread adapters

Custom smartphone mount

I was hunting around Amazon & eBay and could not find a similar thing that would ship fast enough before IRL#2, so I built one myself. I bought a “smartphone tripod mount” at a local gadget shop and glued in a female 3/8 to 1/4 tripod screw adapter so it has an additional tripod screw insert. Now I can screw the mount into a tripod, mount the iPhone and then screw in the VideoMic into the new insert I added to the top of the mount, using the male 3/8 to 1/4 tripod screw adapter. Works like a charm!

iMovie workflow

First, create images of all slides. If you have the presentation in Keynote, you can export as images. If you have it as PDF, you can use Automator to create images.

OK, fire up iMovie and let’s start:

  • import talk recording
  • start a new project
  • drag clip into the project area
  • click the “magic wand” icon to auto-tune audio and video settings
  • right click -> Show Clip Trimmer -> remove the noise in the beginning and end of the talk
  • insert slide image a few seconds after the slide is changed in the video, as “video overlay“
  • select all “video overlay” images, select “cutaway” and set opacity to ~80%
  • add first slide image to the beginning of the clip, add cross dissolve transition
  • add cross dissolve to the end of the clip
  • click File -> Share -> File to render the final video of the talk (Resolution: 720p, Quality: High, Compress: Better Quality)
  • use Handbrake to compress the video
    • Web optimized ticked
    • Format: MP4 File
    • Video Encoder: H.264
    • Framerate: 24
    • Peak Framerate (VFR) selected
    • Constant Quality selected and RF:20
    • Encoder Options: Preset: slow
    • Picture: Storage Size: 1280 x 720
    • Audio: Mixdown: Mono
    • Upload to videos.niteo.co (AWS S3 static website using github.com/oscherler/pretty-s3-index-html)

Redundancy

After recording 10+ talks we realized that things sometimes go wrong and it’s usually a human error: forgot to turn on the microphone, forgot to press record, etc. We now do the following to decrease the chance of not having a recording.

GoPro backup

Mount a GoPro to one of the tripod’s legs, using the standard GoPro pole mount. Stick a huge SD card in the camera, turn it on while setting things up and keep it running. If all else fails you at least get a backup recording.

Audio monitoring

With recording talks, it’s the audio quality that you should be most concerned with. Therefore it comes super handy to be able to monitor audio that is being recorded. This is how you do it:

  • Plug the Rode SC6 into your iPhone. Then plug the VideoMic + SC4 adapter to one of the mic inputs on the SC6 adapter. Finally, plug your headphones into the headphones output on SC6 adapter.
  • Use the FiLMic Pro app on iPhone to get audio passthrough, i.e. to hear in headphones what the microphone is recording.

Checklist

Print out this recording checklist and run through it before every talk:

  • Enable Airplane Mode on iPhone.
  • Turn off all apps running on iPhone.
  • Plug the charger cable into iPhone.
  • Verify VideoMic is correctly plugged into iPhone.
  • Turn VideoMic on, move switch all the way to the right (“curved line” icon) to enable the high pass filter that reduces background noise.
  • Start FiLMic Pro app and verify its configuration (click the Overview icon):
    • 1280×720 @ Apple Standard
    • 24FPS / 24FPS
    • PCM @ 48.8 Kbps Mic: External Microphone
  • Lightly touch the surface of VideoMic. You should hear scratching in your headphones. If you don’t, then FiLMiC is using iPhone’s internal microphone and you should go a few steps back.
  • Ask someone to act as the speaker for a few seconds and try which volume setting on VideoMic works best for given conditions. The farther away the VideoMic is from the speaker, the more boost it needs (+20).
  • Start recording.
  • Stop recording.
  • Turn off the VideoMic.

Sample

Here’s a few seconds of a recent recording from IRL#4 in Istanbul.

Sample for Recording Talks blog post from Nejc Zupan on Vimeo.

Nitean Recognition

“Bang a gong, get it on.”

While the lyric is pulled from a modestly talented rocker in the 70s, it does hit a key entrepreneurial principle. Some companies and businesses ring a bell when something good happens or an achievement reached.

At Niteo, we give kudos and +1s.

The wall of happy

Laszlo Bock, head of Google’s People Operations, has a Wall of Happy outside his office where kudos are printed and put on a physical wall.

We took the idea and the bell tolling, and spin them into something that would work for our case.

Instead of a bell, we have a Wall of Happy where celebrations and peer recognition are documented. And since we’re a remote-first team, the wall-of-happy is a Slack channel.

The modality used to implement the Nitean recognition was based on the fact that Slack is our main and effective communication tool.

See our Handbook policy on this.

For what the bell tolls

The Nitean recognition aspires to keep the team motivated by creating a work culture where the simplest achievement is celebrated. Moreover, it also promotes higher peer engagement and better work communication. Both of which are key activities that underpin all aspects of business and life in one-way or another.

Whether it’s closing a sale, removing a blocker or helping out a fellow Nitean, the bell tolls.

There is no discrimination between achievements. From a modest to a momentous victory, the team recognizes and heralds it.

Creating a positive workplace culture

Having a multicultural team has a lot of pros to it. On the flip side, it could entail a lot more from everyone. Besides the work tasks and responsibilities, some adjustments are necessary for a conducive and convivial workplace. The Nitean recognition lets us celebrate the little wins as validations of the team’s progress and success. Because sometimes it’s the little things, the nuances that often get dismissed, that have the most impact.

There is so much more to running and being a part of a company than pure economics or technology decisions. A team with a good and fostering relations with one another is aces in most, if not all, company goals.

Conference hiatus is over

Back in the day when Niteo was still a consulting company I had to do a lot of traveling to find and do work. And to attend conferences. I loved to go to conferences. But being away so much, in boring hotel rooms and eating out every day took a toll on my (mental) health so I decided I needed a break.

The hiatus is now over! I’ve been to three conferences this autumn and I gladly found out they are as enjoyable as I remember them. I’ve been to Open Data Science Conference in London, the annual Plone Conference in Barcelona and finally MicroConf EU in Lisbon. They are vastly different conferences: ODSC is huge and very corporate and academical, MicroConf is tiny, lifestyle-oriented and friendly. And then there is PloneConf: consistently the best fun you can have at a conference. 🙂

Here are a few random thoughts:

  • The data science community is very much where the Python community was a decade ago in terms of Open Sourcing their work. The questions “But how do you make money?” and “How did you convince your boss?” came up a lot during the conference. Maybe I need to do a talk to address these next year. We’ve been open sourcing our code for a decade now and recently also published our internal policies and work processes for others to learn from.
  • Conferences absolutely *suck* if you don’t do a talk. As soon as you are a speaker, at least 50 people will have a feeling they know you (since they were listening to you talk for 30 minutes or more) and they will approach you in the hallway, on social events. You are invited to VIP events where the crowd is smaller and the newly formed relationships are stronger (because you interact with a smaller group of people more often). If you haven’t, you gotta try it. You’ll see how much it improves your conference experience. You’ll have a better time, you’ll meet more people and you will learn a ton more.
  • If I’m ever organizing a conference, and we’re doing a social event in the evening, people need to get free drinks. It sucks that we as a community need alcohol to open up, but sadly, we do. Not everyone, but it helps. And conferences are about meeting your peers. You learn the most when discussing problems with people, not when listening to talks. The “hallway track” is the thing that makes conferences great. And that two beers on the social event help jumpstart the conversations. Obviously, make sure you have great non-alcoholic options, you cannot go wrong with lemonade or even freshly pressed juice.
  • Plone is still very much alive and has a vibrant community. A great example of an Open Source project that was able to survive handover from original authors to contibutors that joined years later.
  • Lightning talks are one honking great idea — let’s do more of those!
  • There’s a whole community of self-funded, bootstrapped entrepreneurs out there. So bummed I did not realize this earlier, it’s such a welcoming bunch of people. MicroConf is definitely going on my yearly conference schedule.
  • off-topic: Until recently I couldn’t care less about NFC-powered credit cards. Then London happened. Wow is it convenient to have NFC readers on subways, buses, pubs, everywhere! Any other city you need to consult the map to know which zones are you crossing, what ticket you need to buy, etc. In London you tap your credit card when you get on and off whatever public transport you use, and the system automatically charges you an appropriate amount. Completely hassle-free. Why isn’t every city like this?!

2018 is shaping up to be great! EuroPython in Edinburgh, PloneConf going to Tokio (!!) this year and possibly another MicroConf in the beautiful and surf friendly Lisbon. See you at one of them!