Part 1: What Can AI Safety Research* Teach You When Designing Corporate Performance Management Systems?

*Artificial Intelligence (AI) Safety can be broadly defined as the endeavor to ensure that AI is deployed in ways that do not harm humanity.

KEY TAKEAWAYS

• Designing a Corporate Performance Management System and ensuring safety in the Artificial Intelligence domain share similar traits.
• Intelligent agents have demonstrated a tendency to develop a behavioral habit that contradicts the purpose of intended objectives in various cases.

Did you know that designing Corporate Performance Management Systems and AI safety have a lot in common? In both cases, the designer has to set objectives for the intelligent agents to maximize their function. But, unfortunately, that can lead to a series of unexpected outcomes.

Intelligent How?

Nick Bostrom, the famous AI academic and philosopher, once developed a thought experiment in which an AI was tasked with a straightforward objective function: to acquire as many paper clips as possible. Regrettably -as an unintended consequence- the AI turned the universe into paper clips! Here is the summary of the thought experiment in Mr. Bostrom’s own words:

“Suppose we have an AI whose only goal is to make as many paper clips as possible. However, the AI will quickly realize that it would be much better if there were no humans because humans might switch it off. Because if humans do so, there would be fewer paper clips. Also, human bodies contain a lot of atoms that could be made into paper clips. So the future that the AI would be trying to gear towards would be one in which there were many paper clips but no humans.”

That thought experiment might seem like science-fiction. However, there is a list of real-life examples of intelligent agents exploiting their environments to maximize their rewards. Some examples from the list are: 

• An AI trained to play Tetris, which realized that it would never fail if it pauses when about to lose.
• Another AI trained to play a boating game quickly recognized that if it constantly turns in circles in a particular area to grab the booster packs to increase its score, it can indefinitely increase the game score.

Objective Misalignment

There are also other theoretical AI safety concepts like objective misalignment, which is when the intelligent agent learns the incorrect objective and develops a different behavior pattern. 

For example, an AI may fool the designer during the algorithm testing process that its goals are aligned. That said, should it face a more complex situation, the intelligent agent might seek its objectives, creating unexpected results. One could argue that AI’s lack of human common sense could be the reason plaguing AI development and that humans would never misinterpret such set objectives. 

A Path to Designing Hard to Game Performance Management Systems

While designing a robust Corporate Performance Management System, one needs to understand that each intelligent agent in the system might make every potential gaming effort to maximize their rewards. Therefore, a well-thought behavioral analytics study must be conducted to enact anti-gaming policies to predict and prevent unintended consequences in a Corporate Performance Management System.

In this blog series, we would like to discuss where humans, as intelligent agents, can and will exploit their environments to further their objectives—we intend to showcase the similarities between designing a Corporate Performance Management system and an AI.

With our in-house experts in strategy design, behavioral economics, and advanced analytics, we can help you in your journey in establishing safe and hard-to-game performance management systems. For more details, please get in touch with us.