Category: Articles

“I make stuff move with code.”

When people ask me what I do, that’s what I say. I don’t say I’m a framework author. I don’t say I’m a systems administrator or database administrator. I don’t say devops or infrastructure. I don’t say development tooling. I don’t say optimization.

I say eye candy. Pretty pictures. Interactivity. Programmatic animation. Data visualization.

Ten years ago, in the early days of what we were beginning to call “social media” (back when media in social media meant news), discovered I could use public APIs to build front-end only projects that consumed data from web services and did fun things with them. Sometimes, I needed to run a query that was not the result of a simple API call, so I needed to fetch data, store it somewhere, and then query against my copy of the data. Then, these APIs started introducing restrictions like rate limiting and cumbersome authentication schemes.

As time went on, it became more and more difficult to build 3rd party mashups of data from social networks and other web services. By 2010, after acquiring client apps for most major platforms, Twitter took its 3rd party developer hostility to the next level by placing harsh limits on whether and how 3rd party applications could store or display content. Build something for Twitter and you could easily find yourself locked out, all that work for nothing.

What does this new landscape mean for someone like me? It means I need to put additional work into authorization; manage ever changing API documentation for authentication, parameters, and result schemas; do the part I’m actually good at, the front-end; and then hope after all that my access does not get suddenly get revoked due to an unseen hurdle in the complex and restrictive API Terms of Service. And what if you want to make a mashup of multiple web services? Rinse and repeat that process a few times.

“Build the framework you want to see in the world.”

— Mahatma Gandhi’s nephew who’s really good at computers

What I wanted was a way to make mashups, animations, tools, and visualizations without needing to worry about web service authorization or data acquisition. Let one person (me, at first) solve the problem of connecting to a web service and normalizing its data and then forget about it. After an API has been added, I should only have to worry about how to query the local cache and present it.

With a extensible foundation, I could take it a step further. Plugins could register hooks to further transform data before it gets saved (such as analyzing text content for topics or entities). They could also provide local API endpoints that multiple front-end views could share. The front-end view could be the result of the composition of a series of complementary plugins.

For example, if I wanted to make a calendar travel report (left) or a travel map (right), I would first need to know where I was throughout the year(s). Assuming I have one or more social network API plugin, I could query my local cache for geotagged posts, no matter where they were originally posted, using a plugin that analyzes every calendar date and guesses which metropolitan area I was in that day (aside: time zones suck). Because the data imported from social media might not be 100% accurate, it would also need to provide a way for me to make a few corrections. Then, with the geotagged posts normalized and, if necessary, manually corrected, an API endpoint could expose the daily travel history for any number of derivative plugins to use.

It’s often not worth the trouble for a 3rd party developer to build the entire stack just to build one front-end view. It’s also not very likely for people to sign up to some 3rd party’s web service, grant the developer access to personal data, and then remember it exists days after the first use. As a stand-alone service, it’s not very viable.

If these were features you could simply add to an existing system, it would be an entirely different equation.

The popular term Big Data represents the collection, storage, processing, and analysis of vast quantities of data. At its simplest, it represents a shift in perspective to the belief that just about any piece of data can provide value. More data is better than less data. The trend is mainly apparent in areas like business intelligence and government surveillance, but if you look closely, you can see signs of big data on a personal level.

Fitness trackers log every step of every day (okay, often rolled up into 15+ minute intervals) to provide intelligence on your health, progress, and individual workouts. Using a location based social network to “check in” or rate places yields personalized recommendations. By scanning every email in a GMail account, Google Now can provide automatic reminders for appointments, travel, bills, and more. On many social networks, creating a social graph (connections to friends) results in all data from each connection being combined to surface relevant trends and recommendations.

By providing our data, whether a sensor, access to an account, or information about who we know, people get to easily—and often automatically—reap the benefits of certain types of data analysis.

The problem is too much data is being ignored and we don’t have any ways to prevent meaningful, relevant data from falling through the cracks.

The most obvious area of improvement is location. Location based social networks showed how valuable it can be to know where your friends are. The problem is they only help if everyone actively “checks in” on the same social network. You can see if your friends have checked in nearby, but not if they’ve posted a geotagged post nearby via Twitter or Instagram. Instead, that metadata is lost, missed, forgotten.

Generally, I care more about the words my friends are tweeting and the photos they are sharing than the metadata on those posts, but more data is better than less data.

The root of the problem is silos. Twitter could add location tools to the Twitter app, but it would only work for the subset of my friends that are tweeting. And why would they add that extra functionality to Twitter in the first place? Twitter is not a location based social network.

In order to get value from this content and metadata people are sharing online in one place or another, one must first exfiltrate the data from each silo. Then, geographic data can be analyzed for trends or anomalies based on your current location, text content can be grouped, filtered, or emphasized based on your interests, photos can be browsed in a photostream or gallery, and any type of data could be searched, queried, summarized, grouped, filtered, or highlighted.

We have quite a few useful and magical big data tools operating invisibly on our (and our friends’) data, giving us amazing new insights, alerts, and suggestions. But what they provide is dwarfed by what is possible, at our fingertips, and only just barely out of reach.

When I first got into CoffeeScript, one of the main factors that got me over my resistance to learn a foreign (Ruby-inspired) syntax was that it provided a way to use prototypal inheritance without manually typing `MyObject.prototype.method` or endlessly worrying about what `this` currently means. In a surprising twist, the feature that got me into CoffeeScript is one I no longer use. Instead of writing OOP I have started writing in a more functional style, and have found CoffeeScript to be equally powerful (fat and skinny arrow function shorthand, implicit returns, fewer curly braces and parentheses to close, etc). ES6 has added `class` syntax sugar, but for me, it is a feature I will generally avoid in either language.

Imperative Extensibility

In developing a foundation for an extensible application, I started with a base plugin with helper functions. Plugins would import the base plugin class and call those helper functions to change configuration. From a framework perspective, the goal should be to minimize boilerplate—code that is identical in every place a feature is used. Boilerplate wastes the time of plugin authors, and increases the complexity of plugins with code that is not unique to the problem being solved by the plugin. Helper functions are a great way to reduce boilerplate in an imperative system.

Boilerplate and Housekeeping

Boilerplate is not the only concern. Plugin authors should also be spared of as much housekeeping as possible. Housekeeping is the management of state, through operations to add, update, or remove objects (configuration, functions, etc). In an imperative system, a plugin author is responsible for performing the operations required to add functionality to the system, but must also be concerned with updating or removing functionality. These operations are made even more complex with dependencies, as the order of the operations matters. Suddenly, leaving housekeeping up to the plugin author can turn into a massively complex and error-prone system.

Functional Design in an Imperative World

After studying functional programming languages and concepts, it became clear to me that a more graceful way of handling extensibility and mitigating complexity would be to forgo the object prototype-based imperative approach in favor of a declarative approach.

For a plugin author, the result would not be too different, boilerplate-wise. However, housekeeping would be dramatically different. Instead of performing operations to add or configure functionality, plugins would declare functionality and the framework would be responsible the housekeeping.

The filesystem can be used as boilerplate in a declarative system. That is, by simply being in a specific directory, a file may be defining what it is for or how it should be utilized. This is common with MVC frameworks, where files within a `controllers` directory imply `filename.ext` should handle a request to `/filename` or files within a `models` directory inform the framework to create tables or collections in a database where the table or collection name is the same as the file name.

By composing an object or positioning files in a hierarchy, a plugin simply describes how something is supposed to be. The housekeeping—how to add, update, or remove functionality, and in what order—is left up to the framework.

In my primitive proof-of-concept, I took very naive approaches for each part of the system?—?configuration, event handling, web service route definitions. I also made decisions based on language features in Javascript, like the deep merging of nested objects.

For configuration, a plugin may be designed to have an extensible list by storing a default value in an object `{stuff: {default: ’Default’}}` such that a separate, independent plugin could add an item to that list by specifying `{stuff: {newItem: ’New Item’}}` and allowing the framework to construct the final list by merging the declarations into `{stuff: {default: ’Default’, newItem: ’New Item’}}`. If sort order is important, they could be defined as a more robust object, like `{stuff: {default: {value: ‘default’, display: ‘Default’, sort: 1}}}`.

Drawbacks

The way I designed my architecture, a plugin would synchronously return an object with its configuration. For flexibility, I added an asynchronous initialization function that, if provided, would be called after all configurations had been loaded. The most obvious need for something like this was for configuration that was dependent on an asynchronous resource, such as a database call. A different way to address that problem would be to provide a way for a database query to be expressed in the synchronous result and rely on the framework to trigger the database query and embed the results. Ultimately, I wanted to support other types of asynchronous configuration, and wanted the framework to be decoupled from knowledge of a database (which is itself provided by a plugin).

With a blog, you may have multiple themes installed, and thus selectable in a list, but you will want to store the currently selected theme in a persistent data store. This means a plugin cannot declare “the current blog theme is ME.” This is not necessarily a problem with a declarative design, so much as it’s an age-old challenge with the combination of synchronous and asynchronous systems. Static declaration can only get you so far before you find yourself needing dynamic configuration.

I also worry that by doing things in a different way may alienate plugin authors who are more accustomed to imperative systems, even if they necessitate more complex housekeeping.

Ultimately, I’m very happy with the reduction of complexity I have been able to achieve through this shift in thinking from imperative to declarative designs.

I am curious if there is any literature on the design of extensible systems. There are pros and cons to each method, but perhaps people have discovered clever ways of getting around drawbacks in the declarative approach.

In technology, singularity can be defined as the point in which an artificially intelligent computer, network, or robot is capable of recursively improving itself, leading to a runaway affect of rapid advancement. Another take on it—and I’m not precisely sure how I came to associate the term singularity—is the point in which a computer or network exceeds the computational capacity of the human brain.

There are two ways of looking at this milestone, which we may or may not be able to call a singularity. Hardware and software.

Hardware

At what point will we be able to construct a single system that matches the processing potential of all the neurons in the brain. Given some derivative of Moore’s Law or perhaps even a conservative estimate of how many transistors we’ll be able to cram into a single system, with or without advancements in materials or architecture, we can come up with a practical estimate of when it will be possible to build a computer with the physical computing power of the human brain.

As digital transistors and analog neurons are not directly comparable, I don’t think we can put a specific number of operations per second as a target. The fastest networked clusters of computers we have today, collectively referred to as supercomputers, may already exceed the raw processing power of a single human mind. It may be 5–20 years before the same capacity is available in an individual system, and only if major breakthroughs are made in construction, design, and energy consumption. We are already seeing that the future gains will likely be in concurrency rather than faster serial operations, which means programs that can be broken down into simpler tasks and distributed simultaneously to many cores will benefit more.

Ultimately, with hardware, progress will continue to march forward.

Software

With software, things are not so straightforward. While improvements and optimizations are made, progress is not so steady. Even if we have the physical infrastructure to mimic a human brain, what software would we run on it? Managing complexity, detecting patterns, efficiently storing and recalling memories, and making decisions are very difficult challenges.

State-of-the-art software can use substantial computing power, and yet still struggle with basic challenges despite being specifically designed for that one task.

Some software improves steadily and incrementally. Far too many others don’t. For every long-running project that steadily improves against benchmarks (such as operating systems, video game engines, Javascript engines in web browsers, etc.), many more are simply shuttered, re-written in new languages or for new frameworks with marginal improvements, or not ported to new architectures or platforms.

There is a thriving open source community, contributing to software at every level of the stack, but more development time is spent on closed source, proprietary software that will no longer benefit society after the organization decides to shut it down or replace it. And, by nature of being proprietary, if another organization seeks to provide functionality similar to another organization—competitor or not—it must spend resources on recreating the same functionality.

Programming language specific package managers are perhaps the best way we have to isolate functionality and make it available for use in any other projects, or even compose multiple other packages to create higher-level functionality. Another great option is containerizing standalone services, which can transcend programming language. A Ruby Gem is nearly useless for a Node.js project, but a containerized service can be utilized just as easily by a Python project just as easily as a Go project. That is, until you upgrade your kernel or lxc container software…

We are making progress, but far too often, our progress feels like two steps forward and one step back. If anything, what we need most is to make progress on improving how we make progress in software.