How to Rapidly Test New Organisation Designs

By Phanish Puranam

Instead of blindly adopting industry best practice, companies can use gamified randomised control trials to pilot new organisation designs.

It’s no secret that there are no universally applicable organisation designs. What works in one context may not work in another because each organisation has a different history, culture and cast of characters. And yet there is a thriving segment of the management consulting business that specialises in implementing “best practices” – or sometimes “flavour of the month” organisation designs – in companies that vary widely in terms of age, industry and background.

One might hope that research and theory would help perfectly predict which designs work best in a particular context. However, having studied the topic for two decades now, I believe this hope is unlikely to become reality anytime soon. Put simply, organisational contexts are dauntingly complex and vary in ways that we can’t fully observe.

This makes it hard to definitively recommend design interventions based on theory alone – context matters enormously. It is nothing short of foolhardy to adopt a new organisation design or practice without evidence that it will work in your organisational context.

The gold standard: Randomised control trials

Field experiments, also known as randomised control trials (RCTs), are the gold standard for determining whether a design will work in a specific context. Experiments involve randomly assigning some units (i.e., people, teams, projects or departments) to a treatment condition (the new policy you’re thinking of implementing) and others to the control group (where things stay as they were without the new policy). We then check if outcomes are statistically different between the two conditions.

Randomisation is crucial. Imagine you implement, without randomising, a new training policy and find that employees who applied for the training and attended it saw their performance improve. You have no way of knowing whether this is because your training was effective or because the people who applied for it are motivated, high performers whose evaluations were going to rise anyway. You might object that if you make the training mandatory for all employees you could just see if everyone’s performance improves. But the problem is that you can’t rule out other factors (industry cycles, demand spikes) that may have affected all employees.

Randomisation ensures you avoid these problems by creating counterfactuals, that is an understanding of what would have happened without the intervention. This is possible because randomised treatment and control groups are statistical twins: They are similar enough to be treated as identical, so the control group can serve as the counterfactual. We cannot establish causation without counterfactuals, and randomisation is the best way to establish counterfactuals (unless you have a time machine).

But even when an experiment may be desperately needed, it can be logistically very challenging to conduct. Consider the following situation: Your company is considering whether it should adopt agile structures to manage its project teams. Despite the general enthusiasm, we know there are good reasons to be cautious: Agile structures are not a universally superior design.

Ideally you would randomly assign half of the teams in your company to the new agile structure, keep the rest the same, and test for statistically and economically significant differences in their performance at the end of a few months. In practice, the cost, the risk to business continuity and the political challenges of pushing through randomisation can make this daunting.

Does this mean companies are trapped forever in the limbo of adopting industry best practice without any proof it will work, and just hoping for the best?

An alternative: Gamification meets randomisation

Here is an alternative protocol that I believe can beat blind implementation of current “best practices”.

Step one

Find a team task that can be done in a few hours, but which is a reasonable approximation of what your project teams do. This is tricky but by no means impossible. For instance, business school case studies embody exactly this principle – with a combination of a few pages of text and a few hours of discussion, our students get thrown into a simulation of a situation where they must solve a problem which might have unfolded over a period of weeks or months in real life.

Sometimes, you might have small sample sizes – e.g., not enough teams to draw any statistically meaningful conclusions. But the beauty of the gamified approach is that you can select a task in step one that involves a few people, not entire teams. This scales up your sample size. All organisation designs ultimately specify how people interact. With ingenuity and drawing on theory, we can find ways to put just the interactions that matter under the microscope.

Two things are crucial about this gamified task: First, it should be a reasonably valid approximation of what project teams in fact do. Second, there should be a clear metric of successful performance on this task.

Step two

Organise a day-long hackathon. The purpose of the hackathon is to get all the teams in your company to participate at the same time in working on the case study that you came up with in step one.

Step three

Assign half the teams participating in the hackathon to the new agile structure. Keep the remaining teams in their standard structures with the same team leaders and role allocations. It is crucial this is done in a randomised manner – roll a die or flip a coin if you have to.

Step four

Compare how the teams in the agile structure versus those in the traditional structure performed.

That’s it! In one day, you can combine a team building event with a pilot test of the new design you are thinking of implementing.

Why this is a good idea

This approach creates a “toy” version of the work (e.g. projects) you are trying to improve, and organisational design variants (agile vs traditional teams) that can be piloted cheaply and fast, with randomisation. Think of it as the equivalent to what aircraft companies do when building a new model airplane: They first test prototypes in a wind tunnel. The wind tunnel is not the same as real world conditions, but it gives useful signals that can save a lot of money and grief.

Debriefing the results at the end of the hackathon (results can be computed in hours) can lead to a very rich discussion of the intended organisation design change – creating broad understanding, rooted in evidence, of the tradeoffs and buy-in. It’s also worth highlighting that the entire protocol for gamified randomised control trials can be run online too (or even within a metaverse application), both within and across teams. In fact, it could be used to answer questions about whether distributed working within teams will be effective for your company.

In sum: The low success rates of organisational re-design projects suggests that companies have nothing to lose and perhaps a lot to gain by trying out gamified randomised control trials. Start playing!