18 August 2016

A 14-Point Framework for Evaluating Programming Libraries & APIs (Part 2 - Final)

In Part 1 of this write-up, I discussed some of the reasons we might want for developing a clear-cut framework
measuring and evaluating programming APIs, and went on to identify the first 7 of the total 14 dimensions useful
in classifying and comparing different, but "competing" APIs. Here are the remainder...

8. Leverage

The 80/20 rule - can the most commonly-used 20% of a library do 80% of what we'll ever need it to do? (Gson is a great example of getting this right.) Chances are good that we might never need the remaining 20%, but it is good to know that it's there come the day that we hit some corner case where it becomes necessary.

9. Discretion

How well hidden are those implementation details that we should not be concerned with? Is it obvious (or even apparent) which stuff is meant to live on the surface of the API (i.e. is part of its UI, and intended for our use) versus the stuff that we're not meant to mess with. This relates a bit to the Opacity of an API, but deserves to be considered separately. Well structured APIs will have explicit, well-advertised points whereby we can customise or extend the behaviour of the API to cater for our own peculiar corner cases, without having to open the Pandora's Box that is its inner workings. (Some programming languages and environments make this easy, some make it difficult or impossible. This is a context you must take into account when evaluating an API''s "discreteness".) If (environment permitting) we are constantly having pure implementation thrown in our faces, it becomes much more difficult for us to sort out the stuff we need to know from the stuff we don't need to know, or worse, stuff we really should not mess with.

10. Documentation

How good is the documentation? Is it up-to-date or it is describing some historic version of which a whole lot of stuff no longer applies (Volley!) Open-source may count as documentation, but then make damn sure you can actually read/navigate that (See Couchbase as a counter-example!) We seldom have the time to dredge through some open-source project's source (of questionable quality) to figure out what it should be doing or how we should be using it. A pointer to some examples buried deep in a library's source tree, and lacking any form of comment or documentation (hello Bouncy Castle!) is no substitute for adequate documentation.
While we're talking about documentation, beware of the simple Hello World Tutorial! All too often tutorial material is so trivially simple that it is effectively useless for communicating the intent and use of an API. As a showcase for features, these tutorials are a long-winded way to achieve nothing that can't be told in a short bullet-list, and they almost universally make the terrible error of dragging in irrelevant and distracting features simply as a way to show off, as opposed to provide instruction.

11. Support

StackOverflow is not support. How responsive are the devs (or support staff if it's a closed, proprietary API) in the various fora, mailing lists, etc. The availability of paid support is no guarantee of quality. Anybody who has spent an hour listening to telephone muzak at international call rates knows what I'm talking about.

12. Churn

How quickly are new releases made; do they frequently make compatibility-breaking changes?
Some reasonable update frequency is, of course, a good thing. Usually. It means that bugs are getting fixed, performance enhanced and new approaches being embraced. On the other side of the spectrum, changes can come too often, and we become code-followers, on an endless treadmill of adaptive changes in our own code. (Hello Android!)
A word of caution, though: It is not easy to distinguish between an API that is moribund and one that has simply reached a level of maturity that near-eliminates the need for updates. Too many projects fall into a trap of creating updates for the sake of seeming active and healthy, when, really, all they're doing is following a fashion industry.

13. Power-to-Weight Ratio

How much heavy lifting does the API do relative to how hard it is to learn, use, maintain?
Sometimes a library might be pretty difficult to learn because it requires us to learn whole new vocabularies, whole new ways of thinking about the world. Is it worth it? Sometimes the answer will be a clear YES, sometimes a NO, but mostly somewhere in between. If the problem it solves is a pretty trivial one, then it becomes much easier for us to evaluate Power-to-Weight, and we are more likely to demand that the API be correspondingly trivial to use. At the other extreme, a library might solve a pretty hard problem (synchronising data among multiple devices; distributed pretty-much-anything algorithms; concurrency) and so worth investing significant effort to learn how to use.

14. Entanglement

How many sub-dependencies does the library pull in? Does it stand alone, or are you in for a "Maven-downloading-the-entire-Internet" priming-build?
How much of a problem this is for you depends on many, many factors. The most pernicious thing that can happen is that transitive dependencies drag in incompatible (or, at least, different) notions for the same things. I recently saw this in a project where one library pulled in one way of doing JSON marshalling and unmarshalling, a second library pulled in a different subordinate library for doing the same thing, and my own preference was for yet a third library (which had actually been pulled in well before the other two, so I already had plenty of code using it — changing that would have been a pain!) We ended up with three slightly different JsonObject classes, all slightly incompatible. Ugh.

In Conclusion

In evaluating a bunch of libraries or frameworks that claim to solve a particular set of problems you might be facing, it helps to separate out the various elements that make them more or less suited to your circumstance. These elements (or dimensions of measure) may or may not carry similar weight in different situations. You might have some use for applying an arbitrary numerical scale (1-5, fibonacci series,...) to each dimension and assigning scores to each API under consideration. Or you might be content with a fuzzier gut-feel judgement. Spider diagrams might be useful. Spreadsheets, too. Some of the dimensions I consider important might not be for you, and that's OK. The important thing is to evaluate our tool-sets dispassionately and with some set of metrics to guide us.  I hope you find these ones useful.

My thanks to friends and colleagues at Polymorph Systems for review and helpful suggestions. Mistakes and idiocies remain all my own.

15 August 2016

A 14-Point Framework for Evaluating Programming Libraries & APIs (Part 1)

Libraries, network-services, virtual machines, platforms and frameworks, all qualify under the umbrella term "API". Some are simply things to be lived with — if we develop Windows applications, if we write Android or iOS applications, then we're blessed or cursed with certain platform-level givens, and there's not a great deal we can do about them apart from, perhaps, wrapping them behind a facade layer that feels a bit "nicer" — that makes life a little easier for us as developers by providing abstractions that more closely match the abstractions defined in our own applications and hiding layers of complexity that must necessarily be handled, but are uninteresting or distracting from the goals for our own development.
Aside from those "givens" we are faced (almost daily) with choosing other utility libraries and services to make our own development faster, simpler, more reliable, more performant and less repetitive.
This then, brings us to the heart of my topic: What exactly is it that makes one programming interface "nicer" than another? What makes one library "better" than another? Is it the expressive power? How would we go about measuring that? Is it how quickly we can churn out useful code, working correctly? Does popularity and coolness matter? There has to be a better way to measure — if only in a fuzzy and inexact way — whether one library or REST service is better suited to our needs and wants than another, and whether using a particular library might be better or worse than writing our own.
In case it is not already clear, let me emphasize: There is no One True Best API for any given task. Every problem lives in a context — a set of forces pushing and pulling on the boundaries of the solution space, warping the texture of the implementation landscape. Costs, time, expertise and past experience, functional requirements, timing and reliability constraints and, not least, developers' penchant for playing with the newest, shiny technologies — their desire to learn and extend their mastery. So: each and every API we choose to employ (as opposed to those that are forced upon us whether we will or no) must be tested against the problems we are trying to solve and the constraints and forces acting upon us and our application. A particular library may be the "right" answer for one project, but be quite inappropriate for the next one. Our desire for "good architecture"[1] suggests that we should, at least, make those choices consciously and deliberately rather than blindly or reflexively.
In what follows, I suggest some ways we might pick apart the various dimensions we might choose to use in evaluating various competing APIs, identifying 14 dimensions that you might want to consider as evaluation metrics in choosing (or avoiding) an API.
I should emphasize that I consider APIs (along with programming languages, platforms and codebase-hygiene) as primarily a UX problem. These things are all first and foremost user-interfaces for us, as tool-manufacturing humans to use, misuse or abuse. The principal question is, "How likely is it that this tool (API) will lead us astray and into the murky swamp of technical-despair?" versus "To what degree will this tool allow us to write less code, more reliable code, more readable (comprehensible, therefore maintainable) code?"
[1] I refrain from trying to nail down just what constitutes "good architecture" and rely, here, on your own intuition and experience. Suffice to say that it extends well beyond the merely technical concerns and encompasses the human, social and business spheres, too.

1. Surface Area

How many types, methods, configuration items do you have to learn in order to use this thing?
This is not unrelated to the ideas of Function Points as a way to "size" software — it attempts to measure the number of inputs and outputs (since that's what types and configuration items are) and use-cases for those moving parts. The absolute number is not important, since different APIs address problems of many sizes and a wide range of complexity, but it can be useful in comparing APIs that purport to solve the same or similar problem-spaces.

2. Coverage

How much of the topic-area does an API address? Is that what you need?

Does the API do all that you expect it to do? Does it do way more than you need? If the API is functionally incomplete, you will find yourself writing supplementary code to make up its deficiencies. That may be acceptable, but it may spell trouble if the API in question is supposed to be solving some complex or difficult problem (e.g. crypto) but is not sufficiently complete.

The case where an API covers way more territory than we really need is a little more subtle. Given appropriate tooling (not always available in all toolchains or environments) this is not primarily a technical problem (of linking too much object code into an application codebase, resulting in codebase bloat) but a cognition problem. Every part of an API wants to put little hooks into our brains. They call out to us, crying, "Me, me. Pay attention to me!" and we truly cannot afford to give them that time or mental space. Take the Google Guava library (for Java.) I use it on almost every Java project I am part of. But I only make really heavy use of maybe two chunks of what it does — the Preconditions and some annotations (for adding contract-like guarantees to classes) and the Collections (particularly Immutable collections) classes. The rest of the library is mostly surplus cognitive baggage most of the time. I'd be better off with it being in a separate library to be pulled in only when truly needed. Indeed, I have seen projects that end up with as many as three separate definitions of methods like isNullOrEmpty(aString) and checkNotNull(anObject) simply because developers did not want to pull in all of the Guava library in the early stages of their project, then acquired another instance of those methods because some other third-party library made those definitions, and, at the end of the day, they ended up using the Guava library anyway for other reasons. What a mess.

3. Composability

How well does this API play with other libraries and tools?
If an API works in terms of platform-compatible types, it is much more likely to play well with other APIs. If it insists upon introducing and using only its own types, it will be much more difficult for us to force it to play well with the other libraries in our armory — we are sure to find ourselves writing endless boilerplate code converting between custom datatypes. And unit tests for that code. Or not, so hurting our code coverage metrics and creating emotions of despondency and discouragement because clearly we suck at doing The Right Stuff.

4. Modularity

How easily can we break this library into pieces so that we can use just the bits we need?
This is (again) about reducing cognitive load. Does a library allow us to just pick and choose the bits that suit us well, leaving the remainder strictly alone, or does it force us to schlepp in all sorts of sundry other parts of the library that do not touch on the problem we're solving. Some frameworks tend to be really bad at this.

5. Openness

How much is this API a black-box?
Can you tweak the under-the-hood stuff if when you need to without delving into the twisty, slippery innards of the implementation? This is simply the Open-Closed Principle in its essence.

6. Opacity

How well does an API hide the details and complexities of the problem-space? A well-thought-out API will shield us (to an appropriate degree) from the concepts and particularities of the underlying domain it deals with, allowing us to work with concepts the ought to be much closer to our own, more familiar application domain. The types and operations exposed at the surface of the API should reflect something more amenable to adaptation to our own conceptual framework than the underlying problem that it hides and manages.

If a library is not making stuff simpler for us, why bother using it? Does it provide a facade that makes sense in the context of the problem your application is attempting to solve?

7. Accessibility

Can you learn just a little bit of the library and be useful (Vertx), or do you need to learn the whole damn thing before you can (safely) use any of it? (Git)

Accessibility is one of the more important dimensions for thinking about APIs because it means we can tackle the (sometimes daunting) task of learning to use a library truly effectively in little bites, and each little bite that we can chew and swallow gives us an ever-increasing confidence in the library, and an ever-increasing confidence in our own abilities to put it to good use.

I shall continue with the remaining seven dimensions in a follow-up post in a couple of days. This thing is already too long for Internet-attention-spans as it is.
Related Posts Plugin for WordPress, Blogger...