“I make stuff move with code.”
When people ask me what I do, that’s what I say. I don’t say I’m a framework author. I don’t say I’m a systems administrator or database administrator. I don’t say devops or infrastructure. I don’t say development tooling. I don’t say optimization.
I say eye candy. Pretty pictures. Interactivity. Programmatic animation. Data visualization.
Ten years ago, in the early days of what we were beginning to call “social media” (back when media in social media meant news), discovered I could use public APIs to build front-end only projects that consumed data from web services and did fun things with them. Sometimes, I needed to run a query that was not the result of a simple API call, so I needed to fetch data, store it somewhere, and then query against my copy of the data. Then, these APIs started introducing restrictions like rate limiting and cumbersome authentication schemes.
As time went on, it became more and more difficult to build 3rd party mashups of data from social networks and other web services. By 2010, after acquiring client apps for most major platforms, Twitter took its 3rd party developer hostility to the next level by placing harsh limits on whether and how 3rd party applications could store or display content. Build something for Twitter and you could easily find yourself locked out, all that work for nothing.
What does this new landscape mean for someone like me? It means I need to put additional work into authorization; manage ever changing API documentation for authentication, parameters, and result schemas; do the part I’m actually good at, the front-end; and then hope after all that my access does not get suddenly get revoked due to an unseen hurdle in the complex and restrictive API Terms of Service. And what if you want to make a mashup of multiple web services? Rinse and repeat that process a few times.
“Build the framework you want to see in the world.”
— Mahatma Gandhi’s nephew who’s really good at computers
What I wanted was a way to make mashups, animations, tools, and visualizations without needing to worry about web service authorization or data acquisition. Let one person (me, at first) solve the problem of connecting to a web service and normalizing its data and then forget about it. After an API has been added, I should only have to worry about how to query the local cache and present it.
With a extensible foundation, I could take it a step further. Plugins could register hooks to further transform data before it gets saved (such as analyzing text content for topics or entities). They could also provide local API endpoints that multiple front-end views could share. The front-end view could be the result of the composition of a series of complementary plugins.
For example, if I wanted to make a calendar travel report (left) or a travel map (right), I would first need to know where I was throughout the year(s). Assuming I have one or more social network API plugin, I could query my local cache for geotagged posts, no matter where they were originally posted, using a plugin that analyzes every calendar date and guesses which metropolitan area I was in that day (aside: time zones suck). Because the data imported from social media might not be 100% accurate, it would also need to provide a way for me to make a few corrections. Then, with the geotagged posts normalized and, if necessary, manually corrected, an API endpoint could expose the daily travel history for any number of derivative plugins to use.
It’s often not worth the trouble for a 3rd party developer to build the entire stack just to build one front-end view. It’s also not very likely for people to sign up to some 3rd party’s web service, grant the developer access to personal data, and then remember it exists days after the first use. As a stand-alone service, it’s not very viable.
If these were features you could simply add to an existing system, it would be an entirely different equation.