Self-healing Systems

When we want to change something, should we do it in one fell swoop – or should we let it happen iteratively?  Should we build an elaborate mechanism of checks and balances that ensures that no mistakes are committed – or should we be lenient and let the system recover from mistakes?  These questions stem from one of the most fundamental challenges any designer encounters: should we design a complex system, or should we develop a simple system that grows organically?

Of course, the answer to this age old dilemma is KISS – Keep it Simple, Stupid!

Let me explain.  In one of our internal systems we keep the information of employees and their managers.  Our employees frequently move between projects.  It is necessary to keep track of who is managing whom in a timely fashion.  How do we do that? As the manager-direct relationship is not trivial, we had a process where the outgoing manager informs the HR department about release of their direct and the new manager requests that employee to be assigned to them.  Now the problem is that for the managers this is a chore.  And not one, but two managers need to synchronize their actions to make this happen.  The direct, for whom this change is very important – as her performance review is at stake – has no control over the process which routinely used to drag for days or even weeks.  The process had been designed to ensure that the changes, whenever they are effected, are always correct. But this process proved to be too painful for the key stakeholder, viz. the direct. 

We decided to invert the process completely. We now allow employees to change their managers by themselves.  The immediate reaction was of panic. Sacrilegious! How can you allow someone to arbitrarily change their own manager without any approvals? What if they play a mischief? If they mess it up what will be the consequences and who will clean the data? However, soon everyone reconciled to the simple “organic” nature of the change. Why would anyone play a mischief? If they do, won’t they get caught quickly? (We have notifications built in.) In case someone messes up – isn’t correcting it also simple?

The ability to spot an aberration quickly and letting it be corrected swiftly, as a part of the process itself, is the hallmark of a “self-healing system”. 

The celebrated Gall’s law (John Gall) states: A complex system that works is invariably found to have evolved from a simple system that worked.  To further quote Wikipedia, this law is essentially an argument in favor of underspecification: it can be used to explain the success of systems like the World Wide Web and Blogosphere, which grew from simple to complex systems incrementally, and the failure of systems like CORBA, which began with complex specifications. Gall's Law has strong affinities to the practice of agile software development.

Gall’s law, fundamental as it is, is applicable to any field you could think of.  In fact, chances are you may already have used it – or may have seen it being used - somewhere. If that’s the case, can you share your experience? In any case, why not put the Gall’s law to use in our everyday life?

- Abhay Patil