A scientific experiment conducted by researchers at the University of Cambridge and British AI startup Strategize.inc confirms these fears. In particular, incompetent CEOs and management consultants should fear being replaced by AI.
Generative AI collapses due to black swan event
The research team’s experiment was conducted as follows.
- A total of 344 people, including students and current bank executives, participated in this study.
- They had to make CEO decisions through a simulation based on gamification elements. The quality of these decisions was recorded using a variety of indicators. Participants had to complete several rounds depending on the business year. More than 500,000 decision combinations were possible per round.
- A digital twin of the U.S. auto industry was used as the data base, including information on vehicle sales and pricing strategies, as well as important factors such as economic trends and the impact of the COVID-19 pandemic.
- The goal was to maximize market capitalization through a combination of sustainable growth rates and free cash flow. Additionally, the goal was to meet various KPIs and not be fired by the virtual board of directors. “This goal served as a realistic benchmark to measure the CEO’s actual performance,” the research team said.
- They then gave OpenAI’s GPT-4o the same task and compared the results of the two best-performing human participants in both groups.
“The results were surprising, provocative, and at odds with the team’s assumptions about AI’s potential for leadership, strategy, and advanced decision-making,” the research team said. GPT-4o outperformed the best human participants on almost every metric, designed its products with surgical precision, and tightly controlled costs.
However, GPT-4o was dismissed from the virtual board meeting faster than the students. The cause is the so-called ‘black swan’ event. “We included unpredictable shocks to simulate sudden price fluctuations, changes in user behavior, and supply chain problems,” the research team said. The highest-performing students approached these risks cautiously. They primarily focused on maintaining adaptability in uncertain times rather than pursuing short-term profits.
GPT-4o, on the other hand, took a different path, with the research team explaining that “the AI adopted an optimization mindset and maximized growth and profitability regardless of losses, but was thrown off track by market shocks.”
AI can learn and iterate quickly in controlled environments, but it does not respond well to unexpected and disruptive events that require human intuition and foresight. This was also true for bank executives who were fired from virtual board meetings faster than students. “Both GPT-4o and bank executives succumbed to the same flaws: aggressive ambition, flexibility, and excessive trust in systems that reward long-term thinking,” the researchers wrote.
Nevertheless, we came to a positive conclusion about OpenAI’s generative AI tool. The research team said, “Despite its limitations, GPT-4o showed impressive performance. “Although they were eliminated more often than the best human contestants, they were still able to hold their own against the smartest contestants.”
The most dangerous job is CEO
The research team drew various conclusions through experiments. In summary:
- Generative AI can no longer be ignored as a strategic resource. This experiment shows that even untuned models can provide creative and strategic information when given the right prompts. The bottom line is that generative AI delivers powerful results, especially when it comes to creating value for shareholders.
- Data quality is a decisive factor. In order for AI to exceed expectations in terms of corporate strategy, high-quality data similar to that used in experiments is needed. The research team is convinced that a robust data infrastructure is a prerequisite for AI in the boardroom.
- AI effectiveness is not without risks. Aggressive maximization strategies without sufficient foresight can lead to disastrous results. Therefore, generative AI tools should not operate without supervision, and people should not use them without prediction, the researchers say.
- Responsibility and generative AI cannot mix. Transparent guardrails are even more important because it is difficult or impossible to hold AI systems accountable. The research team said this is the only way to ensure that generative AI-based decisions are aligned with corporate values.
- Digital twins play a key and strategic role. According to the research team, a digital twin of an enterprise ecosystem ‘populated’ with multiple LLM agents can be a valuable sandbox for AI leadership. Not only does this serve as a buffer when mistakes occur, but it can also provide CEOs with important insights for better decision making.
- Management consultants can face turbulent times. The research team predicts that consultants will experience difficult times due to the emergence of ‘artificial CEOs’. Companies like McKinsey may face situations where their services are supplemented or replaced by AI systems.
- On the other hand, an open boss with modern leadership does not need to worry about this. The research team said, “AI cannot completely replace the CEO’s responsibilities. However, it can greatly improve your strategic planning process and help you avoid costly mistakes. “The true power of generative AI lies in enriching CEO decision-making and enabling them to focus on the human skills of strategic, empathetic and ethical decision-making through analytics and simulation work.”
There are also things CEOs need to be careful about. The research team said, “We are clinging to the illusion that we will continue to be able to decide everything on our own in the future. The future of leadership is hybrid. “A successful CEO is one who views artificial intelligence as a partner, not a competitor,” he emphasized.
editor@itworld.co.kr
Source: www.itworld.co.kr
