Twice the Mega Power

That garage opener again..

Sat, 26 Oct 2019 14:00:00 +0200

Remember the garage opener I made a couple of years ago?

Coding a rolling Oloid

Sat, 13 Apr 2019 14:00:00 +0200

Did you know there's a whole class rolling objects that's not spherical or cylindrical and still able to roll in a given direction? I didn't!

Some of them have them even have interesting properties like the fact that every point on their surface touch the ground as they're rolling. The Sphericon and the Oloid are two such objects.

As a fun little experiment I wanted to code and print one.

TL;DR, just watch the video - coding a rolling Oloid. Fullscreen it for readability.

The code to construct an oloid is really super simple, you can construct one by creating a convex hull around two perpendicular circles positioned with the centers at radius distance from each other, like this:

(defn olid [radius]
  (let [c (m/cylinder radius 0.1)
        a (m/translate [0 (/ radius -2)] c)
        b (->> c
               (m/rotate [0 (/ Math/PI 2) 0])
               (m/translate [0 (/ radius 2) 0] ))]
    (m/hull a b)))

Print it!

And let it roll!

via Gfycat

If i were to print it again, I would probably use a higher in-fill percentage to make it a bit heavier. You can also buy these made in metal, which probably feels amazing!

Using scad-clj

Some people have asked how I use Clojure and OpenSCAD, how to set up the live reloading, etc, as the documentation is a bit lacking.

(defn render [model]
  (spit "render.scad" (scad-clj.scad/write-scad model)))

I have this simple helper defined that writes a model to a file. Open the file using OpenSCAD. OpenSCAD will watch the file, and every time you call render on something, it will update the display. You can hide all UI elements other than the display in OpenSCAD, put it side by side with the editor, and even set them up to use the same theme - creating the illusion that your editor and OpenSCAD is the same program.

(render (m/union (m/cylinder 10 10)
                 (m/cube 10 20 20)))

I usually leave a call to render at the bottom of the file I'm working on, and evaluate it as needed. That way a quick call to cider-eval-buffer will update the display.

Adereth got some great tips on how to get started over at his blog. Be sure to check out his amazing keyboard!

The y-split function

I got asked about the y-split function used in the video. It's a simple axis split with a hollowed out cube to make it easier to print, no supports required. I didn't want to make the video more complex than necessary, so I simply used it like magic.

It was custom made for this oloid and looked like this. Quick and dirty, and a bit repetitious.

(defn y-split [obj]
  (let [box (m/cube 10 10 10)
        holed-obj (m/difference obj (m/scale [1.005 1.2 1.005] box))
        a (->> (m/cube 100 100 100)
               (m/translate [0 -50 0])
               (m/intersection holed-obj)
               (m/rotate [(/ Math/PI -2) 0 0])
               (m/translate [35 0 0]))
        b (->> (m/cube 100 100 100)
               (m/translate [0 50 0])
               (m/intersection holed-obj)
               (m/rotate [(/ Math/PI 2) 0 0])
               (m/translate [-35 0 0]))]
    (m/union a b (m/translate [0 -20 5] box))))

I just guessed at the tolerances needed, and it worked perfectly, no glue needed. It seems like a useful function for 3D-printing with Clojure, so I think I'll make a general one later.

Bluetooth LE garage opener

Mon, 27 Mar 2017 14:00:00 +0200

The yak I'm trying to shave here is the garage opener (keychain-)remote. You know those big clunky remotes people carry around on their keychains? I try to maintain a minimum (viable?) keychain and I absolutely hate keychain remotes and big keys with a passion.

Remotes are fine in the car, but I don't want to carry one. The garage in question here only have access through the main port - no door. And since I live on a hill, with a garage down by the road, connected by long flight of stairs, it's annoying to have to go all the way to fetch the remote when you just want to grab something real quick from the garage..

My Bluetooth LE garage opener.

I usually carry my iPhone, so my thinking was that it could work as a decent remote.

Requirements:

Connect to iPhone wirelessly somehow..
No wifi from house available (garage is in a wifi-shadow)
Connect to existing garage door opener expansion port.
Powered directly from garage door opener bus (24v max 50ma)

Connectivity

Wifi is out of the question since it's hard to reach without running cables and putting up antennas on both locations (and no, I don't want to connect to a garage-AP from my phone).

A cellular connected microcontroller would probably work, but I wanted to avoid that. And then, there's Bluetooth. And more importantly - Bluetooth Low Energy (BLE).

Microcontroller

I looked around for a microcontroller with integrated bluetooth (and preferably an available cheap dev board with an external antenna). The nRF51822 sparked my interest. It's cheap, packs a 32 bit ARM Cortex M0 core and supports BLE.

I found a small PCB with all the required supporting electronics from CJMCU (probably designed for use in a cheap drone? as it includes a LIS3DH accelerometer). By adding a 24v -> 3.3v step down voltage regulator, a switch, a led, an optocoupler (for triggering the garage door), and a few inputs for magnetic reed switches (for detecting door open or closed) I pretty much had a working prototype.

Microcontroller board and the iOS app.

App

I've never really done any serious iOS development. But I launched Xcode and managed to hobble together some lines of swift code that connects to a bonded BLE device, exchanges some secrets, and makes a "Sesame!"-button go solid blue when the garage is within range. It's not pretty, but it works.

Now - the bad parts. Unless you pay Apple, then your app will stop working after 7 days (free signing cert only lasts for a week). Paying for proper membership costs $99. Refreshing it every 7 days is too much of a chore, so I ended shelling out for a proper developer account. (This was the most expensive part of the whole project)

I also looked into CarPlay, would be pretty cool if you could open the garage from a car's interface, but sadly it seems that Apple have locked this down and requires you to register for some kind of special entitlement license. :(

Firmware

I might release the source code for the firmware, but there's a few things I want to iron out first. Overall this was probably the biggest learning experience, I've done some bluetooth stuff before, but BLE is a completely different beast (with it's own nuances, own lingo, etc..)

Making it secure was also harder than it should be, IMHO, there was way too much fidling around to make a decently secure BLE device.

I implemented security the following way:

I've hooked up a physical button on the device to put it in pairing mode. You have to manually implement "pairing"-mode and "secure"-mode by selectively broadcast your services, and once in secure mode maintain a whitelist of authorized devices.
After pairing you can exchange long term secure keys, this is called bonding. This makes future communication encrypted.
And, in an effort of cooking bacon in butter, I exchange an application layer secret while opening the door (just as an extra layer of security in case i fucked something up in the BLE communication layer).

I'm not going to go much into details about how to do this, but I recommend looking at this BBC micro:bit firmware (also based on nRF51822) for how to set up a whitelist, pairing/bonding, broadcast services, etc.

And to be honest, what's easier if you want to break into a garage? A 2 minute job with a crowbar or hours of sig int and analysis?

Conclusion

The opener has been running for a few months now, and it's been working great for the most part. Sometimes the app is a bit slow to connect, or even require me to get quite close before it manages to detect the device. I've a few ideas how to iron this out in software (or the nuclear option of hooking it up to an outside 2.4 ghz panel antenna) - but it hasn't annoyed me enough yet to make me put in the effort :)

TLDR; Just watch the video...

Office coffee alarm WiFi-button for Slack (ESP8266)

Mon, 20 Jul 2015 14:00:00 +0200

Update: The coffee alarm system no longer work like described below. The non-coffee-club office neighbors moved out, so we changed from a manual to an automatic system. We've reprogrammed a wifi power plug to notify the system when the coffee maker is turned on. The good old button described below is still operational, but now listens for events over MQTT and does the LED light dance when a new pot is brewing.

We have a Slack integrated coffee notification service at work. It's basically just a few lines of Clojure w/ PostgreSQL (for brew logs and stats), notifying our Slack channel whenever someone put on a brew. It recently got upgraded with a WiFi-connected hardware button button, and now the system operates like this:

Internet connected Coffee alarm button.

So.. how was this made? Let's start with a few of the requirements. Originally I wanted this to be automatic - maybe triggered by an inductive current-sensor connected to the coffee-maker - but since not everyone on the office floor is part of the light-roast-brew-slack-club we decided to go with an Internet connected button to avoid false alarms.

Requirements:

Triggered via a big physical button.
Notify office floor via slack:
- Connect to WiFi
- Do HTTP POST to a service running on heroku.
Battery powered.
- Power down after use to conserve battery power.
Made only using parts I have available. I do not want to wait to complete this project, so we're just going to wing it.

It makes a lot of sense to base this project on an ESP8266. It's a newish Chinese chip that's been quickly gaining traction with hardware hackers. 80 MHz 32 bit RISC CPU, integrated 802.11 b/g/n WiFi, reasonable amount of IO options. Best of all, it costs less than $5, and if you ask me, it's the first real glimpse of the promised Internet-of-Things wave (or should I say tsunami?). Internet connected 80 MHz CPUs (at the cost of a couple of dollars each) mean they can be added to everything - including your toaster.

Let's get building!

Building the wifi button.

Power

I want the project to be battery-powered (by a 18650 li-ion cell), sadly the ESP8266 is quite power hungry while connected to WiFi. We could have the module stay in deep sleep most of the time, but I want to eliminate as much power drain as possible, so instead, we're going to power down the entire circuit, and only power the module when the big red button is pressed.

If we wire the button between our battery and the ESP8266, it will power up while the button is pressed, and power back down when the button is released. However, connecting to WiFi, receiving an IP via DHCP, doing a HTTP POST, etc - could take like 20-30 sec+ on a bad day, and I'm not going to stand around holding a big red button for 30 sec every time I make a pot of coffee.

We need a latch, a circuit that once triggered stays powered. And we also want the module to power down the circuit after finishing its' duties. Let's build a..

Soft Latch Power Switch

Latches are often made using a P-channel MOSFET and NPN transistor pair. Sadly, I only have big N-channel MOSFETS available. So let's try to make one using a PNP and NPN-transistor pair instead.

The theory is as follows. Hitting the button will connect the base of the PNP transistor (Q1) to ground, the transistor will start to conduct, thus powering the micro-controller. Once booted, the micro-controller should immediately put one of it's GPIO pins high, GPIO pin will be connected to the base of NPN transistor (Q2). This will take over the role of the button and circuit will continue conducting until micro-controller sets GPIO pin low - at which point everything powers down.

Problems and solutions

Turns out the circuit works, but the ESP8266 has a pretty slow boot-time before pin is set high (1+ sec - WiFi calibration?). Meaning it works if you press and hold the button for at least 1-2 second. This is a problem, we need to give the micro-controller more time to finish booting before it can take over control of the power flow. I'm a programmer, not a miracle analog magician, but let's see if we can't figure this one out. We need power flow to continue through the base of Q1 for some time even after a very short button press, and how can we do that? We can try adding a capacitor right after the R2 resistor. In theory, when the button connects that part of the circuit to ground, power should flow quickly from the capacitor. Once button is released, power will flow slowly back into capacitor (through R2 resistor and Q1's base), keeping Q1 active for a while until the micro-controller can take over.

We add an overkill 1000uF capacitor, and it works. Now even a really short button press keeps the micro-controller powered for several seconds.

Voltage regulation

The ESP8266 can't really handle voltages above ~3.6v without risking damage, and I want to power the circuit from a regular 18650 li-ion cell (the battery cells often used in laptops, and even in the Tesla cars). Li-ion cells have a max voltage of 4.2v and a working voltage of ~3.7v. This turned out to be a bit of a problem, because I didn't really have any suitable voltage regulators laying around. I've a few AMS1117 3.3v, but their drop-out voltage (~1) is simply too much when running from a 3.7v cell.

What I really need is something like a MCP1702, but I don't want to wait around for parts - we're going to run unregulated from a partially depleted 18650. Hopefully I remember to not charge it all the way up if it ever gets depleted :) I might add a MCP1702 or similar modern LDO later.

Spinning LED indicator

My big red button from sparkfun comes with a single led, I want more, I want to add 3 so that I can display a spinning animation while the button is busy. I'm going to steal a few of Andy Frey's ideas, and hack my button similar as his, but with discrete control over each LED.

3 LED Busy indicator driven by an ULN2003 darlington array.

The ESP8266 can only source like 12 mA current per pin, so to drive the 3 LEDs I add a ULN2003A. The ULN2003A is a NPN (darlington) transistor array, so instead of adding discrete transistors we get 7 of them in one package. Connect 3 GPIO from the ESP8266 to the transistor array's base side, and 3 LEDs (with resistors) to the collector side. This leaves 4 transistors for future expansion.

To make a spinning led indicator we'll use the Ticker library to receive a callback every 0.2 seconds or so.

Ticker spinticker;
unsigned int spincount;

void spinLEDs() {
    spincount++;

    for (int led = 0; led < LEDCOUNT; led++) {
        if (spincount % LEDCOUNT == led) {
            digitalWrite(LEDPINS[led], HIGH);
        } else {
            digitalWrite(LEDPINS[led], LOW);
        }
    }
}

spinticker.attach(SPININTERVAL, spinLEDs);

Enclosure

Spray painted red junction box

I didn't really have any suitable sized electronic boxes laying around, only an ugly junction box. I hate ugly beige boxes, so, let's at least give it some color. Spray painted red looks OKish I guess.

Code

Next up, let's add a bit of code to make it work properly, it's all available on github at revolverhuset/mokkameister and revolverhuset/mokkameister-button. But let's have a look at the important stuff here anyway, it's quite small.

ESP8266

On the micro-controller we connect to wifi at the push of the button, and then do a simple hand crafted HTTP request. We also send an auth token, doesn't really add that much security since we're not doing HTTPS, but it's better than nothing I guess.

const char* host = "myservice.herokuapp.com";
const char* path = "/coffee-button/";
const int httpPort = 80;
const char* secret = "secret";

boolean notifyButtonPush() {
    WiFiClient client;

    if (!client.connect(host, httpPort)) {
        return false;
    }

    client.print(String("POST ") + path + " HTTP/1.1\r\n" +
                 "Host: " + host + "\r\n" +
                 "Content-Type: application/x-www-form-urlencoded\r\n" +
                 "Connection: close\r\n"
                 "Content-Length: " + String(7 + String(secret).length) + "\r\n" +
                 "\r\n" +
                 "secret=" + secret + "\r\n" +
                 "\r\n");

    while(client.available()){
        String line = client.readStringUntil('\r');
    }

    client.stop();
    return true;
}

Heroku/Clojure service

Over at heroku we then create simple liberator resource that does something like the following: Validate token, persist brew-data to db, notify slack, wait 5 minutes, notify slack again. We do the waiting in an core.async go-thread.

(defn valid-button-token? [ctx]
  (let [secret (get-in ctx [:request :params :secret])]
    (= secret (env :button-secret))))

(defn button-post! [_]
  (db/persist-brew! ... brewdata ...)
  (go (slack/notify! "Good news everyone! Coffee is brewing!")
      (<! (timeout (* 5 60 1000)))
      (slack/notify! "Coffee is ready!"))
  "OK")

(defresource coffee-button
  :available-media-types ["text/plain"]
  :allowed-methods [:post]
  :authorized? valid-button-token?
  :post! button-post!)

Mount the resource on a suitable route, and be sure to add the default ring api-middleware (so that params are parsed).

Conclusion

Overall it's been a fun tiny side-project, I need to do more of these. Often I want to make something, order up the parts - but when they arrive one month later I've already moved on to other things - so I'm def going to build more with the parts I've at hand, even if they aren't optimal. Also, having a tiny assortment of ESP8266s around is awesome, they are quite useable for things you would want to Internet-able without much effort..

Enjoy a fresh light roast brew!

Oh.. and we seems to have a robot infestation problem (again..) :(

Capacitive touch banana piano in Clojure / Overtone.

Fri, 24 Apr 2015 14:00:00 +0200

Making of a banana piano, with a bit of swedish jazz from my co-worker Håvard Sørbø

This weekend my niece-in-law is staying over, and to maintain my image as the crazy scientist uncle I've planned to make a banana piano (and lots of weird ice creams). In clojure there's a pretty cool programmable audio environment called Overtone. Overtone features a decent sampled piano, and I'm thinking this could be a great basis for a banana-piano.

Fun fact: a full piano requires about 11 kg bananas - at $3 per kg that's still way cheaper (and lighter) than a Steinway.

There's a couple of ways we can make bananas act as tangents, one of them is to use the bananas as capacitive touch sensors. Using a nice little hack it's possible to do this using regular digital pins on a microcontroller. The hack is (afaik) originally from Mario Becker, Fraunhofer IGD, 2007 (website dead). Check out the article on capacitive sensors over at arduino.cc.

Hook up a pin to something.
Disable interrupts.
Set pin as input.
Active pull-up on pin.
Count cycles until pin is high.
Enable interrupts
Set pin as output, low.

The number of cycles needed before the pin goes high will depend on the capacitance of whatever connected to the pin. A banana might require 8 cycles before the pin goes high, while a touched banana might require 15.

We can connect a number of bananas to a microcontroller, iterate over the bananas, read their cycle count before banana-pin goes high, and then transmit a message over a serial connection indicating the pin number if this cycle count is above a given threshold.

const int PORTS[8] = { 2, 3, 4, 5, 6, 7, 8, 9 };
const int THRESHOLDS[8] = { 13, 13, 13, 13, 13, 13, 13, 13 };

bool touched[8];

uint8_t readCapacitivePin(int pinToMeasure) {
  volatile uint8_t* port;
  volatile uint8_t* ddr;
  volatile uint8_t* pin;
  byte bitmask;

  port = portOutputRegister(digitalPinToPort(pinToMeasure));
  ddr = portModeRegister(digitalPinToPort(pinToMeasure));
  bitmask = digitalPinToBitMask(pinToMeasure);
  pin = portInputRegister(digitalPinToPort(pinToMeasure));

  // Discharge the pin first by setting it low and output
  *port &= ~(bitmask);
  *ddr  |= bitmask;
  delay(1);

  // Prevent the timer IRQ from disturbing our measurement
  noInterrupts();

  // Make the pin an input with the internal pull-up on
  *ddr &= ~(bitmask);
  *port |= bitmask;

  // Now see how long the pin to get pulled up. This manual unrolling of the loop
  // decreases the number of hardware cycles between each read of the pin,
  // thus increasing sensitivity.

  uint8_t cycles = 17;
       if (*pin & bitmask) { cycles =  0;}
  else if (*pin & bitmask) { cycles =  1;}
  else if (*pin & bitmask) { cycles =  2;}
  else if (*pin & bitmask) { cycles =  3;}
  else if (*pin & bitmask) { cycles =  4;}
  else if (*pin & bitmask) { cycles =  5;}
  else if (*pin & bitmask) { cycles =  6;}
  else if (*pin & bitmask) { cycles =  7;}
  else if (*pin & bitmask) { cycles =  8;}
  else if (*pin & bitmask) { cycles =  9;}
  else if (*pin & bitmask) { cycles = 10;}
  else if (*pin & bitmask) { cycles = 11;}
  else if (*pin & bitmask) { cycles = 12;}
  else if (*pin & bitmask) { cycles = 13;}
  else if (*pin & bitmask) { cycles = 14;}
  else if (*pin & bitmask) { cycles = 15;}
  else if (*pin & bitmask) { cycles = 16;}

  // End of timing-critical section
  interrupts();

  // Discharge the pin again by setting it low and output
  *port &= ~(bitmask);
  *ddr  |= bitmask;

  return cycles;
}

void setup() {
  Serial.begin(57600);
}

void handlePort(int index) {
  int cycles = readCapacitivePin(PORTS[index]);

  if (!touched[index] && cycles >= THRESHOLDS[index]) {
    touched[index] = true;
    Serial.print(index);
  }

  if (touched[index] && cycles < THRESHOLDS[index]) {
    touched[index] = false;
  }
}

void loop() {
  for (int i = 0; i < 8; i++) {
    handlePort(i);
  }
  delay(30); // cheap-ass debounce..
}

Your computer must be grounded for this to work reliably, and it also helps to add a ground plane under the bananas (see video).

When touching a banana, the microcontroller will transmit a character from 0 to 7, we can then receive this value in clojure using a serial-port library, turn it into an int, map the value to a scale to get a note. And play this using an instrument - in this case - the excellent sampled piano available in Overtone.

(ns musikk.core
  (:require [serial-port :as serial]
            [overtone.live :refer :all]
            [overtone.inst.sampled-piano :refer :all]))

(def port (serial/open "/dev/tty.usbserial-A800F185" 57600))

(defn chr->int [c]
  (-> (char c)
      (str)
      (Integer.)))

(defn banana-touch [input]
  (let [index    (chr->int input)
        my-scale (scale :C4 :major)
        note     (nth my-scale index)]
    (sampled-piano :note note :sustain 0.2)))

(serial/on-byte port banana-touch)

Now enjoy and experiment with scales and different instruments in Overtone - your efforts will not be fruitless. Maybe I'll make a broccoli theremin next time my niece visits. :)

Update:

Since a few people have asked, here's a wiring diagram. It's super easy, only wires and no extra components. The ground plane (alu foil) is optional. If you're having problems in very noisy environments you can put a 1nF capacitor in line with the banana. I used an arduino nano, but any atmega328 based arduino can be used (nano, uno, etc). (And probably all others with minor modifications). Any wires you can connect to a banana and an arduino works. My arduino had male headers, I used 4p dupont female-female cables, which i broke out to 1p male-male for each individual banana. These come in all shapes, genders and lengths on ebay for very little money.

Clojure libraries used: [serial-port "1.1.2"] and [overtone "0.9.1"]

Wiring setup for arduino nano.

Boiling Sous-Vide Eggs using Clojure's Transducers

Mon, 06 Oct 2014 09:00:00 +0200

I love cooking, especially geeky molecular gastronomy cooking, you know, the type of cooking involving scientific knowledge, -equipment and ingredients like liquid nitrogen and similar. I already have a sous-vide setup, well, two actually (here is one of them: sousvide-o-mator), but I have none that run Clojure. So join me while I attempt to cook up some sous-vide eggs using the new transducers coming in Clojure 1.7. If you don't know what transducers are about, take a look here before you continue.

To cook sous-vide we need to keep the temperature at a given point over time. For eggs, around 65C is pretty good. To do this we use a PID-controller.

(defrecord Pid [set-point k-p k-i k-d error-sum error-last output-max output])

(defn make-pid
  "Create a new PID-controller.
   Requires: target temperature, kp, ki, kd gain.
   Optional: output-max=100 (error-sum=0, error-last=0, output=0)"
  [set-point k-p k-i k-d
   & {:keys [error-sum error-last output-max output]
      :or   {error-sum 0 error-last 0 output-max 100 output 0}}]
  (Pid. set-point k-p k-i k-d error-sum error-last output-max output))

(defn calculate-pid
  "Calculate next PID iteration"
  [{:keys [set-point error-last error-sum k-p k-i k-d output-max] :as pid} input]
  (let [error     (- set-point input)
        error-dv  (- error error-last)
        error-sum (+ error-sum error)
        output    (min output-max
                       (+ (* k-p error)
                          (* k-i error-sum)
                          (* k-d error-dv)))]
    (assoc pid :error-last error :error-sum error-sum :output output)))

Let's start by creating a record type for our PID-controller. The PID algorithm requires a few values: set-point - the target temperature, example: 65C for a perfect sous-vide egg. k-p - the proportional gain, k-i - the integral gain and k-d - the derivative gain. The proportinal gain is the most important part of the Algorithm, it controls how hard we push the pedal to the floor depending on the current error (distance to target-temperature). The integral factor looks at error over time. It's what tries to keep the output steady when we've reached our target. The derivative factor tries to counteract overshooting by looking at the error derivative (the change in error rate) - it dampens the other factors when we close on our target.

We also need to keep track of error-sum, the previous error - error-last, and we need a place to put the recommended output.

Next, we create make-pid - a constructing function, this simply sets a few default values if they're not provided.

Then we implement the PID algorithm as a function - calculate-pid. It accepts a PID-controller and the current input sample (ex: temperature). It returns the PID-controller record for the next iteration.

(defn pid-transducer [set-point k-p k-i k-d]
  (fn [xf]
    (let [pid (volatile! (make-pid set-point k-p k-i k-d))]
      (fn
        ([] (xf))
        ([result] (xf result))
        ([result input]
           (vswap! pid (fn [p] (calculate-pid p input)))
           (xf result (:output @pid)))))))

We can then use this to create a "stateful" transducer for doing PID operations on an input (sequence, channel, stream, etc..). (Clarification from Alex Miller: The transducer isn't stateful, but the returning reducing function is, ofc - thanks!) Transducers look pretty weird (mostly because of their nested multi-arity lambdas), but they smell pretty nice, so I guess they're ok.

To help keep state in a transducer reducing function, clojure 1.7 introduces volatiles - volatiles work just like atoms - but with some limitations (rely on thread isolation for compound atomic swap! operations, etc). They are faster than atoms, but otherwise work just the same. If you're used to atoms, their use should be pretty self explanatory.

All values received by the transducer is given to the PID-controller. The PID-controller state is kept in a volatile, and updated for every iteration.

(into [] (pid-transducer 100 0.2 0.05 0)
      [0 1 3 6 10 20 31 45 61 81 91 98 100 105 110 120 110 100])
; [5.0 9.95 14.8 19.5 24.0 28.0 31.45 34.2 36.15 37.1 37.55 37.65 37.65 37.4 36.9 35.9 35.4 35.4]

Let's test it out by pretend we're trying to make something reach 100C, we give the PID-controller a fake sequence of measured temperatures over time, and transduce into a vector. The output describes how hard the PID-algorithm (with these settings) would have pushed on the throttle if the system responded like the input temperature sequence.

The great thing about transducers is that they're general implementations of operations over data, they really do not care how the data arrives or how it should be delivered. It works with data structures, streams, channels, and so on.

So, imagine that we have an electric kettle. We can read the temperature of this kettle and we can control the heating element. We receive temperatures on a channel, and can push instructions to the heating element on another channel. If we have this setup, then we can simply plug in our PID transducer in the middle, and everything should work!

;; Temperatures arrive via this channel
(def temperatures (chan))

;; This channel accepts temperatures, and supplies
;; PID outputs, trying to achieve a temperature of 65C
(def pid-output (chan 1 (pid-transducer 65 0.1 0.02 0.01)))

;; This channel is used to ask the kettle for the next
;; temperature sample (once our PID cycle is done.
(def fetch-next (chan))

;; We pipe temperatures into the pid-controller:
(pipe temperatures pid-output)

Next we create a few channels, one for temperatures, one for pid outputs, and one to ask for new temperature samples. We connect the temperatures to the pid channel.

(defn control-heater [pid-output fetch-next]
  (go-loop []
    (when-let [pid-time (<! pid-output)]
      (let [time-on  (int (* 300 pid-time))
            time-off (int (- 30000 time-on))]
        (when (< 0 time-on)
          (heater-on!)
          (<! (timeout time-on))
          (heater-off!))
        (<! (timeout time-off))
        (>! fetch-next :next)
        (recur)))))

(control-heater pid-output fetch-next)

The above function will wait for a value from the PID-controller, calculate how long (ms) the power needs to be on, and how long (ms) it needs to be off in the next cycle. It then turns the heater on, waits a bit, turns it off, waits a bit, and requests the next value.

(zmq-send! conn {:system "power"
                 :msgtype "command"
                 :location "kitchen-water"
                 :command "on"}

In my case, the heater-on! and heater-off! functions simply send a message to my zeromq based home automation system. I already have power control of my water kettle and my coffee cooker - so all that's required is simply a JSON-formatted message on a message bus. Another way to do this would for example be an arduino with a solid state relay. See intro, sensors and power for more details of my home automation setup.

;; Open serial port
(def port (serial/open "/dev/tty.usbserial" 115200))

;; Put values into the temperatures channel
(serial/on-value port (partial >!! temperatures))

;; Ask arduino for next temperature when we're ready for one..
(go-loop [_ (<! fetch-next)]
  (serial/write-str "next")
  (recur (<! fetch-next)))

The input temperatures are received over the serial port from an arduino with a DS18B20 temperature probe.

Start the system by running: (>!! fetch-next :next), allow the system to stabilize, drop in a couple of eggs and wait 45 minutes. You should be awarded with the most perfect creamy egg yolk you have ever tasted. I like mine on a piece of bread with avocado and a sprinkle of salt. Enjoy, now you're cooking with clojure!

Transduce me up, Hickey!

Thu, 14 Aug 2014 09:00:00 +0200

Transducers (reducing function transformers) are apparently coming to Clojure in v1.7 - but what is a transducer? Are they some sort of a continuum transfunctioner? Some say they are perverse lenses, others that they are monoidals. I have no idea, but I want to find out! So let's figure out what's changed and play with them for a bit.

Here's the work in progress commit that added them to clojure.core. A few signatures were changed, and a few functions were added. In particular, it seems like most of the higher order functions that works on top of the sequence abstractions were altered - map, filter, reduce + friends. So, what's the new signature of map?

clojure.core/map
([f] [f coll] [f c1 c2] [f c1 c2 c3] [f c1 c2 c3 & colls])
Added in 1.0
  Returns a lazy sequence consisting of the result of applying f to
  the set of first items of each coll, followed by applying f to the
  set of second items in each coll, until any one of the colls is
  exhausted.  Any remaining items in other colls are ignored. Function
  f should accept number-of-colls arguments. Returns a transducer when
  no collection is provided.

Okay! The first signature here is new. If you apply map to a function - and NO collection, it will return a transducer.

(def increment (map inc))

What do we actually have now? Is this simply partial function application (partial map inc) or currying or something like that? Let's naively give it a collection and see what happens.

(increment [1 2 3 4 5]) ;; Don't use them like this..
#<core$map$fn__4338$fn__4339 clojure.core$map$fn__4338$fn__4339@2e2f4944>

Hmm, no, this is defiantly not partial application of map, we get a function in return here, so I'm pretty sure this is not how they should be used. Let's look at how you are supposed to use them.

;; We can get a sequence of something via a transducer function.
(sequence increment [1 2 3 4 5])
;; => (2 3 4 5 6)

;; We can transduce into a collection:
(into [] increment [1 2 3 4 5])
;; => [2 3 4 5 6]

;; We can transform and reduce a collection:
(transduce increment + [1 2 3 4 5])
;; => 20

But.. I can already do all of these steps using the existing sequence abstractions.. Example:

(reduce + (map inc [1 2 3 4 5]))
;; => 20

Transducers got one nice trick up their sleeve, they have a really nice composability.

;; Transducer functions:
(def add-2 (map (partial + 2)))
(def keep-odd (filter odd?))
(def square (map #(* % %)))

;; Composing multiple transducers
(def add-2-keep-odd-squared (comp add-2
                                  keep-odd
                                  square))

(sequence add-2-keep-odd-squared (range 1 100))
;; => (9 25 49 81 121 169 225 289 361 441 529 625 729 841 961 1089 ....

But.. similar stuff can already be done using existing functionality..

(->> (range 1 100)
     (map (partial + 2))
     (filter odd?)
     (map #(* % %)))
;; => (9 25 49 81 121 169 225 289 361 441 529 625 729 841 961 1089 ....

So, what's the point here really? I'm going to assume that there is a really good point, since the people creating this stuff are way smarter than me.. So let's dig deeper. Why couldn't we use the transducers directly earlier? Let's have a look at the signatures of these functions:

map f: (a->b)->(x->b->x)->(x->a->x)

filter pred: (a->bool)->(x->a->x)->(x->a->x)

flatmap f: (a->[b])->(x->b->x)->(x->a->x)

Aha, that's a bit clearer. Filter might be the easiest to look at, it takes a predicate and returns a transducer function. The transducer itself has signature (x->b->x)->(x->a->x) where a = b for filter. It takes another function that takes something and an item type b, and returns something. The result of this again is another function that takes something and item a.. and output something. However, I still find this a bit confusing.

It might be easier if we have a look at what x might be here: collections, sequences, and so on. So, for a vector, I need to give a transducer a function that operates on a vector and on an item, and outputs a vector. Let's try to do that!

;; let's call the x->b->x part "builder"..
;; and the x->a->x "applier".. for a lack of better names.

;; Filtering transducer, remove even numbers:
(def keep-odd (filter odd?))

;; A simple vector builder, takes x and b, returns x.
(defn builder [v b]
  (vec (conj v b)))

;; Now, let's give the builder to the transducer..
(def applier (keep-odd builder))

We now have access to the inner function with signature x->a->x, let's try to use it to filter a list of numbers, manually. It takes a something, and an item. We don't have this something yet, so let's give it nil, and let's give it the first number from our list as b.

;; Vector of numbers: [1 2 3 4 5]
(applier nil 1)
;; => [1]
(applier [1] 2)
;; => [1]
(applier [1] 3)
;; => [1 3]
(applier [1 3] 4)
;; => [1 3]
(applier [1 3] 5)
;; => [1 3 5]

Do you see what's going on? We are driving the vector collection logic here (deconstruction and construction). The transducer itself has zero knowledge about the collection it's working on!

Why is this a good thing?

In clojure there are a couple of different abstractions over collections of data. We have the sequence abstraction, allowing us to work on lazy sequences of data using operations like map, filter, reduce, take, etc - all chilled out working lazily hammock-style. Then we have reducers, a highly efficient (potentially concurrent/parallel) way of operating on collections through r/map, r/fold, r/filter, r/flatten, r/reduce, etc. We also have core.async, which work by shuffling data through channels (CSP-style), core.async has functions like map<, map>, filter<, filter>.

Do you see a pattern? Yes? Cool! Let's play with core.async a bit. The latest development-version of core.async that is. Let's look up the documentation for chan

clojure.core.async/chan
([] [buf-or-n] [buf-or-n xform] [buf-or-n xform ex-handler])
  Creates a channel with an optional buffer, an optional transducer
  (like (map f), (filter p) etc or a composition thereof), and an
  optional exception-handler.  If buf-or-n is a number, will create
  and use a fixed buffer of that size. If a transducer is supplied a
  buffer must be specified. ex-handler must be a fn of one argument -
  if an exception occurs during transformation it will be called with
  the Throwable as an argument, and any non-nil return value will be
  placed in the channel.

Hey! there's an optional transducer here now.

(require '[clojure.core.async :as async :refer [chan onto-chan <!!]])

;; Create a channel, buffer size 10, with a transducer added
(def c (chan 10 add-2-keep-odd-square))

;; Dump all the numbers from 1 to 100 into the channel (this runs in a go-block)
(onto-chan c (range 1 100))

;; Pull all the transformed numbers out of the channel, into a vector.
(<!! (async/into [] c))
;; => [9 25 49 81 121 169 225 289 361 441 529 625 729 841 961 1089 .... ]

Before transducers we would have had to express the add-2-keep-odd-square logic using core async's higher order functions, but now we can instead reuse the transducer we've already created. Here the transducer is used to map and filter over something that isn't even a collection/sequence. The transducer has no idea that it's handling data on a core.async channel, we didn't have to change the map/filter/etc logic in any way to make it work on channels instead of sequences.

This means that all the code for higher order convenience functions in core.async (map<>, mapcat<>, filter<>, etc) now basically is deprecated - and it has now been labeled as such in the documentation. Boom!

Conclusion

Transducers makes it so that that the logic you express using filter, map, reduce, take, random-sample, partition-by, etc, etc, etc, can be re-used in completely different contexts. The logic is all isolated, transducers are oblivious to the underlying collection, stream, observable, iterateable, whatever context. That's an useful abstraction if you ask me :)

For an example of how to make a transducer, head over to: Boiling Sous-Vide Eggs using Clojure's Transducers.

GL.iNet Openwrt router

Tue, 22 Jul 2014 09:00:00 +0200

I recently bought an interesting little router on eBay, a GL.iNet router. Pretty cheap, about $25 + shipping.

It sports the same form factor as the sweet little TL-WR703n - everyones favorite hackable cheap linux router. Not only is the form factor the same, it's the same SOC - a AR9330.

Here the similarites end however. Many people (me included) modify their WR703n routers, upgrading the hardware with larger flash, more ram, break out GPIO, etc, etc. And guess what - the GL.iNet router has all of these modifications - out of the box!

AR9330 SOC (same).
64 Mb ram (vs 32 Mb).
16 Mb nand flash (vs 4 Mb).
Serial UART on 3 header pin (vs serial on pads).
Friendly row of gpio broken out (vs pads spread all over the board).
2 ethernet (vs 1 ethernet).
USB2 (same)

It's like a pre-modded WR703n on steroids! Awesome!

The backside of the device has a sticker with all the settings you need, IP, SSID, passwords, etc, they even set up a ddns address for your device, making it availble from their domain (many people will probably want to disable this..) which I guess can be of great convenience to some.

If you enter the WebUI, it actually looks pretty nice, it's preloaded with stuff for downloading (web, torrent, etc) and sharing an external drive, or streaming from a web camera, etc. You're met with a decent wizard that helps you set up the device.

But.. guess what happens if you press the advanced settings button - boom! The OpenWRT webui.. It's actually running Barrier Breaker from OpenWRT Trunk, and a pretty recent version at that. They just add their own stuff on top of the immensly powerful OpenWRT distribution. All the goodies are right under the hood - and you can easily do a clean OpenWRT-install if you do not want any of the GL-iNet stuff.

This is the device to get if you need a cheap embedded low-powered linux device, with usb2, gpio, lan, wifi, etc for a project.

They even offer instructions on how to build a custom image for their router - how cool is that?

Though, I do wonder if the device is up to the best practices for the SOC, because the component count feel quite low vs the WR703N. One of the big components missing is the H1601CG ethernet transformer. Maybe they do the job using discrete passives, but the passive count is pretty low as well. This is a field where my knowhow is sorely lacking, so I really have no idea - take this with a metric grain of salt - I've had no issues with the device thus far.

Banana Phones, Clojure and the Expression Problem

Sun, 13 Jul 2014 09:00:00 +0200

Clojure is a Functional First language, and I usually get by simply by passing around native datastructures such as maps, lists and vectors. Other than doing some simple Java interop I haven't really dug deep into the Object-Oriented features of Clojure until now. But, just because a language is FF doesn't mean that it's OO is weak sauce.

The Banana

(ns oop.fruits)

(defprotocol Fruit
  (peel [this])
  (describe-fruit [this]))

(defrecord Banana [degree peeled?]
  Fruit
  (peel [this] (Banana. degree true))
  (describe-fruit [this]
    (str "A "
         (if peeled? "peeled" "unpeeled")
         " banana with a " degree " degrees bend")))

(defn get-a-nice-banana []
  (Banana. 30 false))

Imagine that we have a namespace called fruits, in this ns we have a Protocol defined - called Fruit - anything that satisfied being a fruit needs two methods - a function to peel the fruit, and a function to describe the fruit. You can think of Protocols as being something pretty close to "interfaces" in OO languages such as Java and C#. (This whole setup works pretty much as traits).

We also describe a record, a Banana. Think of this as a class, the banana accepts two values - a degree saying how bent the banana is, and a bool saying if it's peeled or not.

Records are often method-less, this one however has two methods peel and describe-fruit, this makes the Banana satisfy the requirements for being a Fruit.

We also define a public get-a-nice-banana function that returns an instance of Banana - a unpeeled 30 degree bend banana - since bananas with 30 degree bends are the best - as we all know.

The Apple iPhone

(ns oop.phone)

(defprotocol Phone
  (call [this number]))

(defrecord iPhone [model]
  Phone
  (call [this number]
    (str "Calling " number " using apple phone model " model "..")))

(defn call-using [phone number]
  (call phone number))

(defn get-a-nice-phone []
  (iPhone. "5s"))

Next, let's define a new namespace. This time we create a Phone protocol, phones should be able to call numbers.

Then we create an iPhone, which fulfills the requirements of being a phone since it can actually call people (not just toot and instaface and..).

We create two functions, call-using which allows you to call a number using a given phone. And a convenience function for getting a nice shiny new iPhone 5s.

We now have a couple of things defined.. We have Phones and Fruits, iPhones and Bananas. They have absolutely nothing to do with each other, and are isolated in their own namespaces.

Apples and Bananas - to the core

(ns oop.core
  (:require [oop.fruits :as fruit]
            [oop.phone :as phone])
  (:import [oop.fruits Banana]))

Let's start a new namespace, oop.core, this namespace imports the two other namespaces we've defined.

(def iphone (phone/get-a-nice-phone))

(phone/call-using iphone 80020123)
;; => "Calling 80020123 using apple phone model 5s.."

(type iphone)
;; => oop.phone.iPhone

(satisfies? phone/Phone iphone)
;; => true

We can use the convenience function from phone to get a shiny iPhone, we see that we can give this phone to call-using which then can use it to call someone. No surprises here, as the iphone satisfies the Phone protocol.

(def banana-phone (fruit/get-a-nice-banana))

(type banana-phone)
;; => oop.fruits.Banana

(satisfies? fruit/Fruit banana-phone)
;; => true

(fruit/describe-fruit banana-phone)
;; => "A unpeeled banana with a 30 degrees bend"

(fruit/describe-fruit (fruit/peel banana-phone))
;; => "A peeled banana with a 30 degrees bend"

Let's try to create a banana, it's a nice banana of the type oop.fruits.Banana, it satisfies the Fruit protocol from the oop.fruits namespace. We can call describe-fruit on it, and we can peel it. No surprises here either.

Now, what if we could actually take our banana and call someone? Wouldn't that be wonderful? But alas..

(phone/call-using banana-phone 80020123)
;; Booom exception.. no implementation of oop.phone/Phone for class oop.fruits.Banana

(satisfies? phone/Phone banana-phone)
;; => false

The banana blows up, spewing yellow goo all over our system - it turns out that a Banana isn't actually a Phone. That makes me a very sad panda :( I've always wanted a real banana phone.

In a regular OO language like Java or C#, I would probably attack this by encapsulating a banana instance in a new class, one that also satisfies the Phone interface.

But no worries, watch this:

(extend-type Banana
  phone/Phone
  (call [this number]
    (str "Calling " number " using a " (:degree this) " degrees bent banana.. WTF?")))

Still in the oop.core namespace, we extend the oop.fruits/Banana type to also satisfy the oop.phone/Phone protocol.

(type (fruit/get-a-nice-banana))
;; => oop.fruits.Banana

Our banana is still just of type Banana.

(satisfies? phone/Phone (fruit/get-a-nice-banana))
;; => true

A Banana is a Phone, it actually satisfies the Phone Protocol... and..

(phone/call-using (fruit/get-a-nice-banana) 80020123)
;; => "Calling 80020123 using a 30 degrees bent banana.. WTF?"

Holy cow! We can now not only call using the Apple iPhone 5s, but also using the 30 degrees bent Banana! And that's using a fresh banana we just got from the oop.fruits namespace - we didn't monkey patch it or modify it. We didn't change the oop.fruits namespace or the oop.phone namespace, we just passed it along to oop.phone.

Mind blow, that's a really elegant solution to the expression problem if you ask me..

Ring ring ring ring ring ring ring bananaphone.
Ping pong ping pong ping pong ping pananaphone.

Bonus feature

As a bonus, here's another pretty nice feature called reify. Reification means to make something real, bring it into being or make something concrete.

In clojure using reify you can actually implement an interface (protocol) anonymously. Observe:

(def anon-phone (reify phone/Phone
                  (call [this number]
                    (str "Calling " number " anonymously.."))))

(phone/call-using anon-phone 80020123)
;; => "Calling 80020123 anonymously.."

An anonymous implementation of the Phone protocol. Try doing that in C# (Håvard tells me you can do it in F#). I thought this wasn't possible in Java, but apparently my limited J-knowledge had me fooled (Thanks Magnus) - so this bonus feature isn't as impressive as I thought :) (reify can do multiple interfaces in one go though, but I guess that's not something I'll use very often)

CSP-style concurrent prime sieve using core.async

Mon, 30 Jun 2014 09:00:00 +0200

The Sieve of Eratosthenes is a well known old algorithm for finding prime numbers up to a given limit. If you don't know how it works, the short version is that you basically write up every number starting with 2 up to the limit. You then remove all multiples of 2, and the next number in the list (3) is prime. Remove all multiples of 3.. next number (5) is prime.. and so on.

Sieve of Eratosthenes (wikipedia).

Now imagine modeling the sieve using Communicating Sequential Processes (CSP). Start by creating a seed process, all it does it outputing all the numbers from 2 to n.

For every number x received, create a filtering process - a process which only job is to remove all multiples of x and pass all other values along. Daisy-chain these filtering process for every number you receive, each represents a prime number.

There's a typo above, filter / 2 should drop a 4, not 2.. sorry..

We can model this in clojure (and in go-lang) using go-blocks from core.async.

(ns primesieve.core
  (:require [clojure.core.async :as async :refer [go-loop chan >! <! <!! close! filter< to-chan]]))

Start by setting up the namespace, we're going to need a few things from core.async.

(defn gen [upper-limit]
  (to-chan (range 2 upper-limit)))

Now create the generating seed process. to-chan is sugar for creating a channel, return the channel and then continue add all items from a collection to the channel. In this case the collection is a lazy sequence of numbers between 2 and upper-limit. It's all run in a microthread (go-block).

(defn sieve-filter [in prime]
  (filter< #(pos? (rem % prime)) in))

Next we create a filtering process, filter< is sugar for creating a channel, reading from a incoming channel, and outputing all values that pass a predicate test to the output channel. It's also launched in a go-block.

(defn sieve-builder [in]
  (let [out (chan)]
    (go-loop [in in]
      (if-let [prime (<! in)]
        (do (>! out prime)
            (recur (sieve-filter in prime)))
        (close! out)))
    out))

Next we create the function that will daisy-chain filtering blocks, there are no sugar for this, so manually launch a go-block, read a prime number from the input channel, pass this prime along to an output channel, and daisy-chain a new filtering process.

(defn collect [in]
  (<!! (async/into [] in)))

To collect all the results, we can use into, which will collect all received values and add them to the datastructure of your choice, a vector in this case.

(defn primes [upper-limit]
  (-> (gen upper-limit)
      (sieve-builder)
      (collect)))

Create a function for generating primes below upper-limit, it will spin up the generator, connect the generator to the sieve-builder, and connect the sieve-builder to the final result collector.

user> (primes 20)
[2 3 5 7 11 13 17 19]

And it works! But.. is it fast? and what level of concurrency is achieved?

Not at all.. it's actually painfully slow, about 30 seconds to find all the primes below 100000. I aborted the run, since I didn't want to wait 5 minutes to complete 10 runs. It isn't really managing to use all cores (HT) properly, in addition to spending most of it's time context switching between the micro-threads. Go-threads are light weight, but there is still some context needed, and each process is doing so little work before blocking - waiting for the next value to be received.

If the processes could do more work before getting blocked and context-switched, maybe it would be faster. Let's try adding a 512 value buffer between them.

(def bufsize 512)

(defn sieve-filter [in prime]
  (filter< #(pos? (rem % prime)) in (chan bufsize)))

(defn sieve-builder [in]
  (let [out (chan bufsize)]
  .......

Now, that's better, from 30 seconds to 4 seconds to calculate primes below 100000. Now we get full CPU utilization as well.

But, is it any good? No, not at all! It's actually still pretty terrible, there are so many better and faster ways of doing this. The problem here is that we have a massive overhead because of communication, we're barely doing any computation before shuffling values along to the next guy.

"Also note that async channels are not intended for fine-grained computational parallelism, though you might see examples in that vein."

This is just a fun example, and core.async isn't really intended for this, as quoted above by the original announcement post.

Top picture is taken by Ryan Prince (cc by-nc-sa 2.0)