A Chrome extension that hides the details of people’s deaths from Wikipedia, allowing you to rest easy in the knowledge that everyone ever is alive. And not dead.

Add to Chrome

See extension page

Writing a Chrome extension can be a bit tricky initially, due to it’s API and the way you have to structure your code. The docs are fairly good however, so take a look at them when building your tool.

In this post I want to show how you can write a simple extension that modifies the content of the page once it’s loaded. This is a method used by extensions like the various adblockers to identify and remove adverts. It’s a good place to start since it gets you used to the environment without getting too complicated. The whole lot is in javascript, which gives you easy access to the DOM and is essentially like writing client side code except you have a few extra APIs for access to the browser and code is scoped meaning you can’t communicate with code that the page loads.

Setting up

Every extension needs a manifest file. It declares what files are a part of your project and tells Chrome how to use them. You also set metadata like icons and descriptions which will be used when you upload your extension to the Chrome app store (this is the only sensible way to distribute it outside of a corporate environment).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

{
  "manifest_version": 2,

  "name": "My new extension",
  "description": "It does stuff",
  "version": "1",

  "icons": { "16": "ext_16.png",
           "48": "ext_48.png",
          "128": "ext_128.png" },  

  "background": {
    "page": "background.html"
  },  
  "content_scripts": [
    {
      "matches": ["https://www.site-to-run-on.com/*"],
      "js": ["helper_library.js", "content_script.js"]
    }
  ],
}

All the files for your extension should be in the same directory, which will be zipped up when you upload it. The manifest is what binds it all together. If it’s not referenced in there, it won’t be used.

Architecture of an extension

• Background page - not visible by the user. A long running script that handles the state of your extension. Communicates with other parts of the extension with messaging.
• Event page - the same as a background page, but only executes in response to events you define. Preferred to background pages as they are less resource intensive.
• Options page - UI page that allows a user to set options. An HTML page that references javascript files.
• Browser actions/page actions - an icon either in the extension bar or the omnibox respectively. Clicking on the icon will show a UI tab which is an HTML page that can reference javascript. These pages cannot communicate with the website resources/DOM, it needs to message the content script to do that.
• Content scripts - javascript only. What is used to modify the page itself. The only part that can access and modify the DOM but not code loaded by the website. Files are run once for each page that matches the manifest. All the JS files in the list are loaded, so you can use libraries just by including it. This is what we’ll use for our extension.

There are plenty more elements that can be used in an extension, but this list covers the core that will get most of your extension built and most other parts are for specific tasks.

Permissions

When users see ‘This extension can see and modify data on all pages’, they get nervous. So it’s best to limit where your extension will be active by explicitly setting permissions in the manifest.

1
2
3

"permissions": [
    "tabs", "http://www.google.com/*"
  ],

In this case, the extension will load content scripts when on any page on Google and use the tabs API. * is a wildcard matching any number of characters including none, ? matches any single character. So b?b would match bob and bib but not boob. See more.

You’ll also need to specify what pages your content scripts will match and run on. More on this in the next section.

Writing your own extension

I’ll now go through the steps of creating an extension I wrote recently called Undead that removes details about people’s deaths from Wikipedia. I made it because I often watch old films and google the cast and then am shocked and saddened to find out that actors that were young attractive women in 1938 are now dead. So this extension allows me to live in a glorious past where everyone ever is still alive.

Manifest

Let’s start with the manifest. We need to state which version it is, if you’re uploading a new version to the web store, the version number must be greater than the existing one. Don’t confuse this with the manifest_version, which is always 2 and is not extension specific.

Create a new folder called ‘undead’ then a new file called ‘manifest.json’ inside it and copy the following into it.

1
2
3
4
5
6
7
8
9
10
11

{
  "manifest_version": 2,

  "name": "Undead",
  "description": "This extension removes any details of people's deaths from Wikipedia",
  "version": "1.1",

  "icons": { "16": "daisy_16.png",
           "48": "daisy_48.png",
          "128": "daisy_128.png" },  
}

You must supply 3 icons at sizes 16, 48 and 128 px. These are used on the extensions page and the toolbar. Download them here and save them into the directory at the same level as manifest.json:

• 16
• 48
• 128

You can now load your bare bones extension into Chrome. Go to Window -> Extensions to open the extensions page. Tick developer mode on the top right which allows you to load your own extensions. Click the ‘Load unpacked extension’ button and then select the ‘undead’ directory. You should now have something that looks like this:

There you go. Technically that’s all you need for an extension. But it doesn’t do very much of course. To modify a page we need a content script. Add the following to your manifest in addition to the above:

1
2
3
4
5
6

"content_scripts": [
  {
    "matches": ["https://en.wikipedia.org/*"],
    "js": ["undead.js"],
  }
]

Now save a new file called undead.js and the following as the content:

1
2
3

window.onload = function() {
	document.write('Hello world')
}

Go back to the extensions page and hit reload (this is very important. If you ever wonder why nothing changed even though you updated the code this is probably the reason).

Go to a page on Wikipedia and now you should see this:

Great! Now we could write the whole lot in native javascript, but it’s useful to see how we can have multiple files and use libraries so let’s drop jquery in there too. Download it from here and drop it in the top level of the extension directory. Then update the manifest to include it:

1
2
3
4
5
6

"content_scripts": [
  {
    "matches": ["https://en.wikipedia.org/*"],
    "js": ["jquery-2.2.0.min.js", "undead.js"],
  }
]

Now we have jquery available to us so we can select elements in the page and remove them. Let’s start with the ‘Died’ part of the table on the infobox.

Get rid of the code in undead.js and put this there instead:

1

$('th:contains("Died")').parent().hide()

Reload it and have a look at a dead person’s page. Well, it’s a start. Here’s some more comprehensive rules I came up with:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

$('th:contains("Died")').parent().hide()

$('th:contains("Died")').parent().hide()
$('th:contains("death")').parent().hide()
$('th:contains("Resting place")').parent().hide()

$('td:contains("death")').parent().hide()

$('span:contains("Death")').parent().hide()

$('p').not('.infobox p').first().html(function() {
	return $(this).html().replace('was', 'is')
})

var hidden = ['died','remains','funeral','death','autopsy','suicide','cremated','interred','mausoleum']
hidden.forEach(function(word) {
	$('p:contains('+word+')').hide()
	$('li:contains('+word+')').hide()
	$('span:contains('+word+')').hide()	
})

$('p').each(function() {
    var html = $(this).html()
    $(this).html(html.replace('was survived by', 'lives with'))
    $(this).html(html.replace(/( | )– [0-9a-zA-Z, ]+\)/, ')'))
})

Reload. That’s working a bit better. But it’s still a bit slow. When javascript runs in the browser it runs immediately, which is why we normally use $(document).ready or similar to delay the execution of that code until the DOM has been constructed. However Chrome doesn’t run the code until everything has loaded, the equivalent of window.onload, making the window.onload callback in the first example unnecessary.

We can choose when we want the content scripts to be injected. Choices are ‘document_start’, after css is loaded but before DOM. ‘document_end’ after DOM complete but before images and the default ‘document_idle’ after window.onload. If we used the first one we’d end up with no changes, because it would run before it has any content to manipulate. So update manifest.json to include this:

1
2
3
4
5
6
7

"content_scripts": [
  {
    "matches": ["https://en.wikipedia.org/*"],
    "js": ["jquery-2.2.0.min.js", "undead.js"],
    "run_at": "document_end"
  }
]

Now it should run without an annoying gap inbetween the page load and rerendering.

Adding a browser_action

Maybe sometimes the user should face up to reality and find out the truth. We could add a bit more functionality to this by creating an undo button that adds back all the death notes to the page. To do this we’ll need a browser action with a button that sends a message to the content script, which will then revert to the original content.

Create a popup:

1
2

<script src="popup.js"></script>
Goodbye world

and the JS file:

1

console.log('A message')

You’ll see this:

Reload. Check the console of the page and… oh, nothing there. The browser action writes to its own console since its separated from the page. Right click the icon and click ‘Inspect popup’ to see the separate console and you’ll see its message.

Now update these files to create the functionality of a button that sends a message that can be received and acted on by the content script:

1
2
3
4
5
6
7
8
9
10
11

<!doctype html>
<html>
  <head>
    <title>Popup</title>
    <script src="jquery-2.1.4.min.js"></script>
    <script src="popup.js"></script>
  </head>
  <body>
	<button class="deadify">Deadify</button>
  </body>
</html>

1
2
3
4
5

$(document).ready(function() {
	$('.deadify').click(function() {
		chrome.runtime.sendMessage({ undo: true })
	})
})

1
2
3
4
5
6
7

var copy = $('body').clone()

// existing code

chrome.runtime.onMessage.addListener(function() {
	$('body').html($(copy))
})

Now you can open up the popup and reverse the effect of the extension at will. In this case $(document).ready is necessary because javascript loads normally as specified in the HTML rather than injected in by Chrome.

Note: this is a very simplified version of what should happen. If this was a real extension it would be a good idea to add a background page that manages state to prevent too much complexity in the UI files and set a tab target and parameters to your messages to limit its effect. Take a look at message passing to learn more.

Now you know what’s possible, I’m sure your head will start fizzing over with interesting and useful things you can add to the browser. For a reference once you’ve grasped the basics, see the Developer’s guide. The code is a repo on my github. If you need help please email me on tom.chambers@gmail.com.

A necklace I made for Eleanor (obviously). Made from 3mm acrylic fashioned with a laser cutter, chain is from a salvaged necklace.

This is a 3mx2m world map I created for the 2015 Shuffle Festival held in Tower Hamlets Cemetery Park. The theme of the festival was migration and the organisers asked for a world map highlighting the birthplaces of the ancestors of people who now live in Tower Hamlets.

The first thing I did was collate the survey data collected by the organisers and cluster the points, as there were initially 400, far too many to display meaningfully on a map. As expected it mostly fell on major cities, but apart from that there were people from all over the world. Lots of Australians.

One thing I didn’t forsee was that after the migratory generation, people’s parents tend to be from where they’re from. Which makes sense, it’s only recently we’ve had the ability to seek a new life across the world, and people don’t tend to make that move unless there’s some big change or shift in the place where they’re from, which happens rarely.

The map consists of a reinforced plywood backing with a polycarbonate top to diffuse the LEDs which mark out all the places where people are from. There’s an arduino on the back controlling the cycle through the four generations of collected data. All in all there are about 90 LEDs, which made for a pretty good effect from a distance.

The dream of watching a world change its shape according to one’s desires without having to make the barest of efforts. This is what leads a programmer to spend an entire weekend of perfect weather inside a dark stuffy room making a lamp turn on and off when he says ‘lights on’. So I find myself here with as complete a setup as I’ll probably ever have I thought I might share it with the wider world (since no-one else is ever coming up here).

Since I live in London, I rent a room so small I repeatedly hit my head on the ceiling when I try to get to my wardrobe. I’m also limited in what I can tear out of the walls and change. So my setup is based around lights, sound and controllers. I’ve got two wemo motion sensors, two switches which control two extension leads with two and three lamps respectively and an LIFX bulb in a lamp. I’ve got a raspberry pi which has a USB microphone and a set of powered speakers as well as a Leap Motion controller. The raspberry pi also runs most of the services required and the MQTT message broker.

My brief thoughts on the quality of these products:

• LIFX: brilliant setup, has a great HTTP API as well as the LAN communication I’m using (with the excellent python bindings by mclarkk). Rather expensive at £60 though, and the colour changing is of limited use really.
• WEMO: devices themselves are good. They use a relay which makes a loud click when turning on and off. The app is horrendous and makes setup a chore. Switches are now £30 which isn’t too bad but not really at average consumer level yet
• Leap Motion: impractical for everyday computer use. But as a single controller it seems to work well. Mine was £27.50 on ebay and some are going for as low as £18, so pretty well priced. Hard to integrate into this system without a fair of custom software though.

Architectural thoughts

The whole thing is built with a microservices architecture - each function, whether it be input or output, is running as its own separate service. I toyed with the idea of making the services use HTTP as it seemed simpler and the overheads didn’t really matter for such low volume. I wanted to experiment with messaging however and I’m glad I did as it made the whole process much easier and the system is now far more flexible. The mosquitto broker is very simple to setup and works on the pi and there are equally simple clients for python, node and whatever else. Connecting is as simple as this (Python):

client = mqtt.Client(“lifx_controller”)
client.on_connect = on_connect
client.message_callback_add(‘lights/on’, turn_lights_on)

And now you’re listening to events. Each service is dumb to whatever other services do. They’re basically a bunch of idiots running around shouting, “it’s dark now, turn the lights on!” or “somebody moved!” without any concern for who’s picking it up. Which makes it easy to develop those things in isolation and build functionality for future services.

One thing that bothers me is that there’s a complete dependency on the broker. If that goes down then absolutely nothing will happen because there’s no communication at all. Although it’s just a simple message carrier it’s what makes the loose coupling possible. I’m not sure what the solution would be here - HTTP fallbacks where services talk to each other directly would be too difficult to maintain since they would only be used in very rare failure cases. One could simply broadcast packets across the network but this doesn’t seem at all sensible and doesn’t really scale.

I had intended to use Docker to make it easy to transfer projects across and build/start them automatically on git push, but this didn’t turn out to be as useful as I’d hoped. For one, the pi’s ARM architecture prevents me from using the same Dockerfile on my macbook, so projects can only be built once they get there, so there’s no advantage of using the same environment everywhere. Builds also take a horrendously long time on the pi, possibly because it’s I/O bound to the speed of transfer to the cheap SD card.

I also considered a continuous integration solution, which would make it easier to keep the services (about 8 so far) up to date without having to SSH in and manually pull/restart the process. Using something hosted would mean exposing my pi to the internet though, which didn’t seem like a good idea since although nobody could do too much, it would let people do weird things with my lights in the middle of the night. Setting up something locally seemed a bit overkill as well.

List of services:

• docker-mosquitto - a docker container that boots up a raspbian version of Mosquitto - an MQTT broker
• lifx-controller - python
• wemo-controller - python
• voice-controller - python, sends audio to wit.ai and outputs command messages
• leap-motion-controller
• spotify-controller - python, listens to music/playlist events and puts on a random song from that search
• moodbot - node, connects to Harmony mood api and changes lighting and music
• sun-manager - node, checks current sunlight and listens to motion events, issues lighting commands
• app-status - a dashboard in node/javascript, just listens to events. In the future it might send control events as well

The meat of the business

By far the coolest part of the setup is the Leap Motion controller. It’s not really very useful as a general input device, but for this case where there’s only two axes it’s great. Left and right controls the colour, and up and down is power.

Next up is voice control. voice-controller is constantly listening and sends blocks of speech to wit.ai, where it gets processed into intents (lights) and entities (on or off) and then it sends a message based on that command which is picked up by the wemo-controller and lifx-controller services which communicate via LAN. The voice controller can also send messages to spotify-controller.

I used the API of an app I built (Harmony mood tracker) to tie my mood to the colour of the LIFX bulb with moodbot. It makes use of Firebase’s event driven data stream to get updates immediately rather than polling. Every time I enter a new mood it updates the colour of the bulb (but doesn’t turn it on) and updates the chosen playlist. When wemo detects motion, the lamp turns on using that colour and moodbot plays the playlist (given that it’s a reasonable time for music).

I’m intending to add a facial expression reading chrome extension to Harmony, which will allow data to be automatically collected. The system will then be giving me a subtle cue about my mood, bringing it to my attention without requiring active thought.

This is the real value of the internet of things - the manifestation of data. Here you take something intangible, like emotion, and represent it in the world with physical elements. It becomes a real part of the environment and now one can interact with that data layer in a physical human way rather than the artificial terminal-based manner we’ve become used to over the last 30 years.

So that’s it so far. The next element will likely be a doorbell, which will either be a dedicated device or an arduino listening to RF signals from the button which then feeds into the system. The great thing about this architecture is that the addition of any new element barely requires any modification to the system. The doorbot service can simply connect to the broker and send light or sound messages. It can also send a generic doorbell/active message which could then be picked up on by other future services.

I also built a dashboard which just picks up on messages and lets me keep tabs on what’s going on:

It’s mostly for debugging, but it could be expanded into a controller in it’s own right. It’s called Hugo after Hugo Cabret the automaton. Looking at this it’s not clear to me where the business logic should go - for example the moodbot sends commands to play music when there’s motion, and the sun-manager turns lights on and off when there’s motion. These could simply be notifiers of ‘mood’ and ‘light/dark’ events with a separate action-manager service that wraps all configuration into one place.

Automation is clearly the future, and it’s on the verge of becoming cheap enough that it will soon be integrated into every household device. This system is clearly outside of the scope of most people’s skills and interest, but it certainly seems like a useful architectural pattern for connecting services together. IFTTT provides a pretty useful broker service and it might be that we see a more advanced version of that start to take responsibility for the bulk of configuration.

It does still concern me that there is a single pinch point for failure and control by a malicious actor - an alternative might be to have a configuration service which is the single point of user interaction with the system, which then distributes relevant mapping to devices so they know how to act based on what input. I think a proper solution to this problem would probably involve taking a step back and rethinking what one is trying to achieve.

There are quite a lot of pain points currently with IoT devices - a major one for me is attaching devices to wifi. I have about 10 headless devices at the moment, each one insists on my downloading an app and typing in my complex wifi password. They also each have their own quirky API to work with, there’s no standard for communication between them, hence my writing wrappers for most of them.

If there’s any details you’re interested in please email me at tom.chambers@gmail.com

I’ve worked at Guidesmiths for just over a year now and for that time our main client has been TES, an edtech beast with a formerly nightmare inducing .net codebase which we’ve been shuffling over piece by piece to a nodejs microservices architecture that’s only mildly disturbing.

The platform has steadily grown from about 15 services to maybe 40 now. By the time I arrived in my current position, the infrastructure was pretty well established and for the most part I’ve been following patterns that were set out as part of the initial design. I was also fairly new to node.js when I started, so I’ve also learned a lot about backend technologies in this period. I’ve had opportunities to learn about mongodb, RabbitMQ and redis at scale and I’ve really been able to bolster my skills as a full stack web developer, something that I wouldn’t have had the chance to do at other companies that wanted me to have a much more narrow focus.

A caveat in my narrative is that I’ve not had the opportunity to architect a system at scale, so this is a report from the trenches. Which to me seems an appropriate view given that microservices are about giving up the bird’s eye view.

There’s nothing too complex about a microservice system in principle. You have a lot of individual applications that do one thing and communicate via a strict set of interfaces that treat the inner workings of services as a black box. Problems generally don’t start with the services themselves, it’s more about the complexity in the way they are linked together and the ecosystem they reside in. What follows is are a few of the things I’ve learned in the past year about how to avoid those tangles and find elegant solutions to communication.

Services should be independent. This is an old principle but it’s something to keep in mind when you’re adding functionality to your services. It’s easy to layer on functionality where it doesn’t fit, and this is doubly true when services are small and dedicated to one thing. When it’s hard to create and deploy new services, people will take the lazy way out and cram it in an existing nook. A good rule of thumb is that a service should be possible to rewrite in a week according to a basic set of specifications.

On the web you’re generally dealing with a user request that comes through the stack and heads back out in the same direction. Sometimes you’re doing data migration and piping things from one place to another internally, but handling requests tends to be the main thing. It’s easy when you’re designing a system to set up your services in a chain with your data source on one end and the routing on the other end with the various stages of composition in the middle. When you’re designing this, it makes sense and it’s easy because it’s all in your head. When you’re tasked with making a small change (for example, change this bit of copy on a product page where that text is stored in an i18n module and then rendered with a front end framework like React) it can become very difficult to follow that change through various separate repos where the names of the variables are changing and you now have to make 5 commits in those repos just for that one small change.

The solution to this is to make that chain clearly defined in a single service. That one service should serve be responsible for serving the page and by looking at the handler, you should be able to see a chain where it retrieves the data, transforms it with the i18n module, pulls in shared content like header and footer and then delivers it back.

Shared client libraries are something that seems like a good idea at the point of starting out. With a fresh system you can be sure in the knowledge that all of your architectural decisions are ideal and will never need to change. Thus you can abstract away implementation details of database connections and config load because the way this is done is the way this is done forever. 6 months or a year later and you find yourself in edge case hell. The database module you’re using no longer works, but now it’s a part of your templates that developers use for creating new services. So they take the easy way out and use that because it’s there, in spite of the problems caused later on. This is not so much of an issue when parts can be easily swapped out independently, but it starts to cause problems when these modules form the means of communication between services, for example loading shared visual components. This is doubly the case when you then write something in another language that doesn’t have the same capabilities.

When you come onboard a project and tinker with existing things, you learn how to work on top of those fundamentals, but you don’t get to learn those fundamentals. Much like spending time in a building gets you used to how to use it, but not necessarily about how it is actually constructed. If you use complex dependency injection or many custom client libraries that abstract away the intricacies then developers will not have the opportunity to understand. If however, you have to start from scratch each time, you get a chance and indeed are forced to iterate and create something that teaches you about how it works, which leads to faster workflow in the future and means that you are always building on what you know about the system.

As well as limiting opportunities for learning, it also makes it difficult to make small changes. For example, if you have a module that sets config and steps for the build process and you want to make experimental tweaks or small changes for that particular service, it gets difficult because there is a single place where it’s updated from. Shared libraries are useful for obvious reasons, it allows you to easily update the entire codebase at once rather than repetitively making that change individually in every single place. I’ve found though that with the exception of some business logic that absolutely needs to be in sync the cost of stagnation is greater than the cost of updating various services. Microservices are about heterogenity, and making it easy to try different things means finding better development practices in the long run.

On the other side of this you have the ecosystem you are putting your services into. It’s really important that your deployment is as simple as possible in order to allow for fast change and growth/death of services as required. This is where having great DevOps comes into play. You can write a simple service that does something useful in 100 lines and it might stay that way for a month or two before it needs to develop. As before, allowing anyone to create new services from scratch means that everyone gets to do architecture, gets to learn about architecture and gets to enjoy themselves much more doing it, all of which result in a system that changes and gets better faster.

While many aspects of design can be left to the implementers of a particular service, there are some things that you will probably want to have a general policy on. Most importantly the way that services communicate. Data is an important one. Having an API that is the guard for a database makes it easy to make schema changes or change the way that data is stored but it also means n+1 for every call made to that database and you have a bottleneck for those requests in the API service. Alternatively, services can talk directly to the database, which will tax the network less but also mean more maintanance if you want to make schema changes or swap out that database entirely. I’m willing to bet your server costs are a tiny fraction of your dev time, so I don’t see why you’d want to compel developers to spend their time in the weeds laboriously making changes in 5 different places, but there you go. This is absolutely one of the things to watch out for as your system scales. Co-ordination problems that are not apparent with 5 services will become a big deal with 40 services.

To reiterate my point from previously, chaining is good. It allows you to quickly understand the way a request is handled and what happens and it gives you a single point of contact for making changes to a transformation. If you want it to do more you can insert another step with one change, rather than going to two services to fuse another service inbetween them. Services should be stupid and emaciated. They do their one job well and blindly. Once you start splitting out the functionality of a service into various places, it’s now hard to manage without being aware of all of those things at once. You’re now a professional plate spinner. Continually rebuilding is also essential, and having spider like links to other things means that you can’t simply rebuild a service because it means that you have to redo everything else at the same time, and then why bother having microservices?

Static assets are one of the most important policy decisions you should make and the point at which building your system will jar the most with the philosophy of microservices. With the current architecture of the internet, there is still a great advantage to minification, CDNs and single domains for serving client side javascript, CSS and images. If you break up the various areas of your site into sections that are served by separate applications you’ll need a step in your build process that brings them all together and pipes them to the CDN and manages the references to them. This can get very complex quickly. When you add in shared components such as header and footer this can get to become a nightmare.

I don’t know of any good solutions to this problem and it’s one of the places where the browser’s implementation of a website as the rendering of a single document butts up against the decentralized/distributed philosophy of microservices. If you have the luxury of not having to provide consistency, perhaps if your site is a series of loosely interlinked applications, this will make the problem much easier. Really I think we need to change our approach to design, to accommodate what the way the web is transforming, but we’re stuck with this for now. You can of course take the network hit and serve everything from the individual apps, but then there’s a cost imposed on the client which is much more significant than your server costs.

Of course, there’s a limit to the amount of concious design you can do in any complex system. And that is doubly so when you have a whole bunch of independent components that are designed in isolation.

Conway’s Law: “organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations”

A lot of the structure of your system will come about as a result of the structure of your organisation. The technology industry has been moving away from hierarchical structure for some time, but what often results is a largely flat structure with a narrow and powerful layer just on top. The organisation I currently work for has a structure like this, and the architecture of their system reflects this. All the independent services are pushed into a funnel with a single routing and composition service. Just like the human communication, this increases the burden of co-ordination, since all service communication must be performed through this single bottleneck (a limiting factor both in realtime and dev time).

So I think it’s important to pay attention to the human factors in the system. I don’t think that it would be sensible to compose developers into small teams with strict and explicit channels of communication beyond which they don’t communicate. It’s generally a mistake to think that analogy is useful beyond example and understanding. Even in a large organisation you won’t have a direct mapping of people to services and people who have knowledge will come and go.

There are some useful lessons - if you want to have successful microservices, let go of centralised control of people and the architecture will tend to reflect this. Each developer is going to be responsible for the maintanence of a number of services. This requires that the team frequently divides into cross functional subteams that are just large enough to do that job in a simple and elegant way without the burden of co-ordination.

Of course, if you divide people over a whole bunch of services, they’ll develop general shallow knowledge and there’ll be a lack of experts in certain areas. On the other hand, if developers are specifically assigned to a small number, they’ll develop a lot of skills in that area but once they leave there’ll be a dark hole in the system. A balance between the two is probably best. Developers should understand a few points in the system and how they communicate with other points. Trying to understand the system as a whole is futile. Missing knowledge as a result of specialisation or generalisation is also generally mitigated because every service tends to demand a knowledge of the full stack in order to understand how it operates.

Microservices aren’t complex. It’s just about following a principle of breaking your system down into simple independent components that can be rebuilt quickly which allows for painless iteration. The problems that arise are due to the difficulties of co-ordinating between services and maintaining the system as it grows. How do you find elegant solutions for getting data from one place to another without introducing burdensome complexity?

For the most part the answer is to keep things separate. Don’t put two pieces of functionality into a single service. Don’t make services dependent on libraries that make it difficult to change those libraries without having to consider the entire system. Have an ecosystem that allows for smooth service creation and deployment. Compose services in a way that allows a process to be quickly understood and altered. Above all keep a system fresh and healthy by allowing services to be rebuilt based on what you know about your system at present, not what you imagined it would be like at the start.

I recently organised along with Theo Papatheodorou and a couple of others, an Art Hackathon in London. It was a great success and went far better than I expected. The participants were wonderful and their creations were magnificent. I initially thought it had to be really small to work since it was our first time organising this kind of event, but my co-organiser Theo’s enthusiasm swayed me into putting on a hackathon with 70 people in attendance.

I’m primarily writing this blog post for other people who are in the same situation as we were two months ago: trying to figure out what elements of the concept would work out and what resources we’d need to organise. I’ve organised the content by the main areas one has to think about.

Organisation

You’ll need a team. For our 50 person event four people working part time would have been suitable. The workload will move from around 1-2 hours per week initially to 20 hours a week in the final stage. Make sure you assess commitment from the start, how much time are people willing to put in? The more people you have, the higher the burden of coordination, so you should make sure you know somebody is actually going to be there on the day before you make compromises as a group to suit them.

You’ll also need a good communication channel that makes it easy to share ideas, spreadsheets, documents, arrange meetings, emails. We used email, but it gets unwieldy trying to find the right threads amongst everything else. Google docs is good for sharing other information. Make a shared folder and stick everything in it. You can add more people to the folder when they join the team.
The space
The physical space is hugely important and it’s something you need to organise fairly early on, certainly before you get people signed up or start selling tickets. The place you’re in will also have an influence on other aspects like serving food, wifi access and how many people you want to have there.

If you’re running a standard software hackathon, you just need to make sure you have a good connection and plenty of power sockets. Not having either of these things will ruin the event. If it’s something hardware related, you have a much greater burden of things to provide and if the space already has these things - it might be a hackspace, workshop or education facility - then it will be a lot easier and cheaper than buying and transporting it yourself.

Don’t forget to make the space a nice place to be. Spend a little effort and money to make the place bright, colourful and fun. People are going to be there for the entire day or weekend and might then be going back to work straight after. Don’t make it a dismal grey box.

Catering

“Thanks for not just serving pizza”, was what one guest said to me over dinner. Having decent food makes the whole thing a much more pleasant experience. Of course, food is going to be the expense that is equal or more than the physical space. Having food meant that most of the ticket price immediately went towards that. You might prefer to have the participants bring their own food. This will reduce costs a lot, but may lead to people leaving early or going elsewhere for dinner and reducing the vibe. We had a lot of people chat to each other during mealtimes and it was a good way to take the temperature of the day. Definitely don’t imagine that you’ll be able to cater for the event yourself - you’ll be too busy.

Also consider tea, coffee, juice and snacks to be available through the day. A tea urn is not too expensive to rent (we found a tea urn and coffee machine for £40). Having things like this available just make the environment a bit more pleasant and help stave off hunger if dinner is delayed.

Start looking for caterers about 1-2 months in advance. I was surprised that it was harder to find a caterer willing to cover the event than I thought. I advise going for lowest common denominator food. Don’t order stuff that people will have trouble eating like spicy food, wheat based etc. Ask your caterer how much notice they need for dietary needs and then send an email to your participants.

Sponsorship

Getting sponsorship for an event like this is pretty difficult. We managed to get some free hardware from Tessel and Bare Conductive, but apart from that we didn’t get anything back from about 100 emails sent off to various art startups and technology companies. If you have an existing relationship will help and if a company has a community manager or outreach person, that seems to make it a lot more likely that they’ll respond, so try and talk to that person directly.

Our budget was about £2800. £2000 of that came from Ravensbourne, the other £800 from ticket sales. We spent the lot and I’m hoping that we’ll break even. The space and catering cost about £800 a piece. Don’t forget to leave contingency for on the day things - we got taxis to transport our stuff around, snacks and booze for the last day.

Hardware/materials

Our total budget for this was £277. Most of that is non-electronics stuff since that was provided by Tessel, Ravensbourne and us from stuff we owned already. It sounds like a lot but it adds up very quickly. I was in charge of this stuff and I bought a dizzying array of items including silly string, glitter and bubble liquid.

With hindsight it wasn’t sensible to buy a whole lot of random bits and pieces. I imagined someone being inspired by a crossword toy I picked up in a junk shop, but as far as I know none of that stuff got used. If I were doing this again I’d focus more on core items that are really useful and have a good supply of them. Gaffer tape, string, paper, pens, 3.5mm speaker cables - these are all really useful things.

Another mistake I made was in the selection and presentation of the electronics. It would have been more sensible to have an easier to grok smaller selection. The way I set things up, people had to rummage through and try and find useful things or ask for what they needed. I brought along some wireless plugs and a strobe light. These would have been cool together, but they were hidden away and not obviously useful on their own.

Try and provide complete sets of things. A clearly laid out set of arduinos with power, raspberry pis with flashed SD cards and touchboards along with paint is better than a box full of some boards and a bunch of random power supplies that didn’t match up to anything. Don’t try to provide for every need, just supply what 80% of people will recognise as useful and use. Having our hardware better physically organised would have made it much easier to pack away again as well rather than just shoving it all in a bag and hoping for the best. If all the SD cards had been together for instance it would have been easier for participants to see that they existed and could be borrowed and easier for us to visually keep track.

Some of the things you lend out might be expensive. Some of ours certainly were. We relied a lot on the trust of people not to just pick things up and walk off with them. The day was hectic enough that someone could have easily done that and there wouldn’t have been much we could have done.

You need to have a policy about what you do. We decided that we would look at people’s tickets and write their name down but we never discussed how strictly this should be enforced so it ended up only just being enforced. You should answer questions like: what happens if someone comes again, do they still need to show their ticket? Does someone need to be at the table at all times? Can people rummage through things or is the stuff on a table behind the hardware manager? Can people take part of a set of tools for 5 minutes then bring it straight back? What if someone wants to chop up and solder a cable, can they do that? Very important: what’s the liability if someone breaks something. You definitely can’t determine that after the fact. If you don’t decide these things in advance you’ll end up doing whatever’s easiest.

Internet access

Absolutely essential for a hackathon. The place you’re in needs to have absolutely rock solid wifi and preferably wired connections. Test it well in advance of the day because a problem with the internet will ruin the day in a way that most other factors won’t. Things you should test are:
can you connect easily with a range of devices
what’s the bandwidth? will it support 50 people downloading at once?
does the connection persist for the whole day? We had a problem where connections got reset every half hour
is there any corporate authentication system? How will you work around it?
can you connect with a headless device like a Raspberry Pi? Devices like these won’t be able to deal with the splash screen that some wifi providers use. Don’t test on a laptop and assume that covers everything

Guests

I put this section lower because I didn’t feel the guests we had at our hackathon, while great people, were an essential part of the event. We could have removed them without having changed the atmosphere or success of the hackathon to any great degree. The talks added a bit of sophistication to the event, gave a good entry into it and might have inspired a few people, but they were not the bedrock and could have been replaced by some other teambuilding activity.

It also feels a little odd to have this kind of structure in such a self-led activity. Something that helps people get to know each other and is not sitting at a table in a team is definitely a good thing, but if I were doing this again I’d probably go for an unconference where participants pitch talks, set themselves up around the space and see who comes along. This would have the added bonus of giving people a chance to check out what the other participants are like before bonding forever.

I’d also suggest that talks should be very much for the purpose of inspiring participants. Artists shouldn’t talk about themselves, they’re only useful if they kick of thought processes and discussions that lead to cool projects.

We also had our guests wander round and help participants. It’s difficult for me to say with my birds eye view how this worked out, but it seems like the technical help was more important than the conceptual stuff.
Participant experience
Absolutely the number one reason to hold a hackathon is to provide a space where people can have fun and meet new people. It’s easy to lose sight of this in the weeks leading up because a lot of the work is definitely not artistic and creative.

On the day you’ll need to have people who are responsible for certain admin tasks like checking tickets and signing people in and also people who interact and take part in teams to find out who is doing well and who needs help. We had the idea that we’d be massaging participants who weren’t quite a good fit and helping teams move around. Even though we said that people could move around if they wanted to I’m not aware of anyone actually doing so.

Hiring people

If you’ve got a reasonably sized budget it might be useful to pay some people to look after certain aspects for you. Provided you can trust that person to manage that thing and you’re spending money you have this is a great idea. Make sure that it’s something that needs little input from you and can be absolutely forgotten about. If that person needs to ask a lot of questions and needs direction then you’ve forgone the advantage of paying them in the first place. Remember that everything that needs doing like having someone take tickets means that person can’t do anything else, like get an extension lead from the car, so if everyone if busy with big tasks, little occasional tasks will pile up and become a problem.

Trust and proficiency are absolutely essential. Be very careful about who you take on. If it’s creative work, make sure to look at their portfolios and get multiple recommendations first. Sit down with the person face to face and hash out a schedule and your precise expectations.

On the day

Prepare everything in advance. Absolutely everything you can think of. If you want to label your hardware, do that as early as possible. Go in a few days in advance and check the AV system and the internet so you can order cables or resolve problems in advance. These simple things that can be done in a relaxed way will suddenly become a nightmare if you try to sort them out while standing on the podium with a line of 8 people behind you.

The few hours from opening to starting hacking were super busy and it was really hard to keep everything organised with setting up guests, presentations, getting people checked in, setting up the hardware library. And that was with spending all Friday setting up as well.

After people formed into teams however everything settled down and there was almost nothing to do except answer questions every so often. To help resolve questions, make sure everything is communicated multiple times. Even after tweeting and emailing all participants and having a notice on the projector screen people about adding music to the collaborative Spotify playlist (highly recommended by the way) I was still asked who was in charge of the playlist.

Everything was fairly quiet for that whole period of hacking. Quiet enough that it didn’t take all four volunteers to look after it. It was useful however to have people who could spend an hour or two helping teams with technical stuff without being short staffed elsewhere. Tasks during this period included answering questions about what hardware was available and signing it out, refilling the water for the tea urn and adding songs to the playlist.

At first our teams were very intensely involved in what they were doing. I went round to take photos and ask people how they were doing and mostly got ‘mmhmm’ and ‘yeah’ when I asked how things were going. Things started to loosen up in the first evening and people became more relaxed and the atmosphere was more fun. I really didn’t think we were going to have people staying from 9:30am till 10pm and then 8am to 6pm on both days, but the majority of people actually did that and had a great time doing it.

It should be noted that dinner on the first day is the time when people who aren’t having a good time are going to leave and not come back. So if you want to deal with those issues, you need to do it in that quiet period a few hours after people have started hacking and are having a great time/not having a great time. Of course you might put it down to those individuals just not working well with others or maybe having different expectations and it’s for the best that they enjoy their Sunday doing something else rather than being convinced to spend it at your event. Either way it was harder than I expected to spot these problems and deal with them. If someone is having a bad time, what do you do? Putting them in another team is hard because teams are generally on the verge of getting too big and if they’re not working together well they might have dynamics problems of their own. I think it’s far better to extend the time at the start of the hack so that people don’t have to make immediate choices about who they want to spend the next 48 hours with.

A lot of the stuff going on was heads down laptops stuff up or small scale creation until pretty much the very end. It was really exciting to see big things pop up and things get quite frenetic in the final few hours. I would have liked to have seen more of this throughout the hackathon and get people out of their chairs, but I’m not sure of the best way to achieve it. Often the physical structures are just the superficial shell around the cool tech part that the team have been working on all weekend. So who knows.

I’m not really a fan of awards, so I was pretty skeptical about this part. It was great to have people’s presentations though and after that we had an exhibition where people could walk round, see what others had made and chat about them. This part was really important as a lot of the stuff really had to be seen in person like the VR museum.

Try and keep presentations short. 2 minutes is pretty reasonable and it’s only a taste that then lets people walk round. We had about 40 minutes for walking round, which wasn’t really long enough. It was heartbreaking to see people have to tear up the things they’d worked really hard making and literally just finished.

The middle of the hack is sacred. Absolutely don’t interrupt it except for food. People will be annoyed at being torn away from the exciting thing they’re doing. If you want to add that sort of thing make sure it’s on the beginning or the end.

So that’s it really. I highly recommend organising this kind of event. If you’re a company organising this thing, please make it free form rather than locking it down to something that benefits you. You’ll benefit from people having fun at your event much more than if you try to control the experience.

If you want any more advice please email me at tom.chambers@gmail.com. I’m very happy to answer questions on things you’re not sure will work and how to deal with particular problems.

Eyeball) is a node cli tool I built to help you find all the missing packages that you’ve required in your app. Sometimes I want to deploy something and each time I do I get

1
2
3
4
5

Error: Cannot find module 'obscure'
at Function.Module._resolveFilename (module.js:338:15)
at Function.Module._load (module.js:280:25)
at Module.require (module.js:364:17)
at require (module.js:380:17)

meaning I have to either keep deploying until I’ve caught them all or go for the nuclear rm -rf node_modules, npm install. So eyeball will just give you a list of node modules that are required but not listed in package.json. You can also use eyeball to add them with the currently installed version to your package.json.

On NPM | on Github

It’s a book and website yo. The living portraits project is an idea I had to keep a better kind of memory. One that fits the mediums we have around us in our world today. Imagine someone captured for a moment in a photo, and that moment being stretched out over time.

The premise I started with was that when people sit for a portrait, they’re very self concious. Their mind is on how they will be percieved in the photo, whether they’re doing the ‘right thing’ in the way they’re sitting, standing, smiling. Some people have a photo pose they immediately snap into when a camera is pointed at them. It’s really difficult to get an honest portrait of those people.

I had the idea that I’d sit people down for an hour’s portrait session, recording one long video. Long enough to forget the camera, or at least long enough that it would be impossible to hold a rictus smile. In the end I plumped for 10 to 20 minutes, as I realised I couldn’t put my friends through an entire hour of sitting still for my little experiment.

I’m not entirely convinced that sitting for a long time ends in relaxation and honesty. You’ll have to judge for yourself.

The other central themes in this project are the unique moment granted to each viewer and the life of the portrait. Because the video is longer than any one person’s viewing, every person who comes to look at the portrait gets a slightly different view, although they are superficially similar.

I really like the portraits in Harry Potter, they’re household objects with animation and personality. This is an attempt to recreate that spirit - the book is an item that could be in the home, reminding you more viscerally of that person or moment. The only downside for me at the moment is the glow when it’s dark. Colour e-paper screens? Auto off and wake on motion?

Technically this project is actually fairly simple. The book uses a raspberry pi and a cheap rear view camera screen with the casing removed. I cut the book centre out with a jigsaw and glued the remainder of the pages back in to the hardback cover. Power is supplied with a 12v wall adapter and a 5v transformer for the pi.

The website just uses an html5 video with a small amount of javascript to randomly seek every 30 seconds. Have a look at the living portraits website

Another implementation of this project might be a wall mounted frame with a motion sensor, perhaps installed in a home. Whenever a person walks past, the video jumps to a random point, much like the online version.

The heartbeat jumper is a wearable device that measures your heartbeat and emits a soft glow on each beat. It’s an arduino flora connected to a pulse sensor amped with a single red LED with a fabric patch sewn over the top for light diffusion.

Originally I had intended to use an electro luminescent panel. I tested the setup as a prototype, but there’s a complication with the power supply. EL panels require high voltage AC power, which means you can’t hook it straight up to an arduino and power it that way. You need either two separate power sources and a relay inbetween the inverter (converts DC battery voltage to AC) and panel or a transformer, still using a relay/transistor.

Because of the simplicity of using an LED, I went with that instead. It works fairly well, but the light is a bit small, and its not particularly diffuse. You can clearly see that it comes from a single point rather than being a glowing circle.

The biggest challenge in this project, aside from the issues with the abandoned EL idea, was the sewing. Not something I’ve done before. A straight line is fairly simple, but my circular sewing abilities leave something to be desired. I have improved a little though.

The idea behind the jumper is to find more ways of communicating. This isn’t a particularly original project, here’s an example as a pendant and a badge. There are many other implementations.

A second version might be on the cards. If I do go ahead with that, the main improvement will be the lighting (requires sourcing an IC to convert power and learning about transistors to turn the EL circuit on and off on the beat). I might also improve the detection. The current IR sensor requires the wearer to tape the detector to the skin, and if you move a lot it’s not particularly accurate.

The Box of Wonder is a high tech fortune cookie. You put your head inside, the box scans you and prints out a prediction of your personality and date of death. It’s better with the mystery, but for the sake of documentation, I’ll bear all.

There are three spoken word audio tracks that are randomly played along with a set of lights timed to match the voice of the track as it takes you through the process. When you put your head through the curtains, it triggers a PIR sensor connected to an Arduino Uno, which controls an (Adafruit Neopixel ring)[http://www.adafruit.com/product/1463], a thermal printer and an (Adafruit Music Maker amped)[http://www.adafruit.com/product/1788] with a 3w 4ohm speaker. Because the printer and lights need a lot of amps, extra power is provided by a 5v 4 amp wall plug.

In fact, a better choice would have been a 7-12v adapter for the arduino, possibly transformed to 5v for the other components. Although the arduino runs on 5v internally, it actually needs more than this. Once everything was set up, it started resetting when printing due to the power consumption. Very confusing to someone who hasn’t dealt in hardware before.

The box itself is made from plywood and covered in velvet. I also learned that making a sturdy simple box is a bit harder than it looks from the outside. Inside its covered with some reflective silver plastic. Works well, but very expensive compared to the rest of the supplies.

The horoscope prediction is assembled from a set of statements that psychologist Bertram Forer came up with. They work by making statements that are vague enough and qualified enough to appear to describe anybody. I.e. sometimes you’re open with others but sometimes you feel a bit hostile. The date of death is just a random date from now until 2115. The experience is pretty much as intended. The box itself is intriguing. I get a lot of looks when carting it from place to place, and told off by bus drivers when I leave it in the aisle.

Nobody believes it’s doing any scanning, but there’s a tiny part of the person that questions if it really knows something. I think this would be heightened if there were genuinely some questioning that it were really doing something - it could respond to interaction to appear intelligent, or make more of a pretence of doing some scanning.

It would have been nice to have some more ‘scanning’ devices inside the box to enhance the effect. Perhaps a head collar brain scanning thing, a bunch of gears moving around inside. I looked into disassembling a scanner to use the motorised light bar, but controlling it seemed fairly difficult so I didn’t go down that path. In the end, all the ideas I had (eyes on stalks following you around?) were a bit too complex for the scope of this project, so I stuck with the lights and sound.