LogIn"They might as well be shaking at Magic 8 ball and writing down the answers it displays."
We're only about two years into the widespread AI boom, and already the artificial intelligence models seem to have advanced enough to start promoting their own agenda, making predictions that would benefit AI and its robotic counterparts.
Case in point, a recent research paper published by the Tony Blair Institute for Global Change and presented by Tony Blair himself, which claims that nearly 40% of tasks performed by public-sector workers could be at least partially automated using a combination of AI-enabled hardware and AI-based software, such as machine-learning models and LLMs. If implemented, this automation could lead to significant time and financial savings as a result of, you guessed it, a substantial portion of the public-sector workforce getting fired, the report concludes.
While loud proclamations of AI zealots about how machines will replace humans are nothing new and hardly newsworthy at this point, the report in question did manage to make headlines – not for its findings, but due to the questionable methodology employed by its authors.
As spotted by 404 Media, the prediction shared in the paper was made by feeding data about ~19,000 tasks performed by workers, including work and worker characteristics and skill requirements, into ChatGPT-4, which was then asked to analyze the information and determine which tasks could be performed by AI.
By tweaking the prompts, the authors eventually tasked the machine with identifying how AI could be used to perform specific tasks, estimating potential time savings, and assessing whether it would be cost-effective to use AI, yielding the results listed above.
"Relying on expert judgements to make these individual decisions would make our analysis intractable. Fortunately, this is exactly the sort of task to which LLMs, trained on a vast amount of source material, are well suited. Another approach that has been used by researchers in this area is to use OpenAI's GPT-4 to determine whether each task in the O*NET database can be done using AI. But given the nature of these models – LLMs are not deterministic and it is not possible to trace how they have sourced information – there is a concern that these models are a mysterious 'black box' that may or may not give reliable results," the paper reads, confirming that the authors knew very well that their methodology was faulty.
Emily Bender, a Professor and Director of the Computational Linguistics Laboratory at the University of Washington, was among those who criticized the report, stating that no matter what prompts you use, an AI chatbot can never provide a reliable basis for a research paper:
"This is absurd – they might as well be shaking at Magic 8 ball and writing down the answers it displays," Bender told 404 Media. "They suggest that prompting GPT-4 in two different ways will somehow make the results reliable. It doesn't matter how you mix and remix synthetic text extruded from one of these machines – no amount of remixing will turn it into a sound empirical basis."
"There is no validation in this method that a language model is good at working out what is, in principle, able to be automated," adds Associate Professor at University College London, Michael Veale. "Automation is a complex phenomenon – in government it involves multiple levels of administration, shared standards, changing legislation, very low acceptable cost of failure. These tasks do not exist in isolation, but are part of a much broader set of practices and routines."
So there we have it – an AI making predictions about AI taking even more jobs from human workers, with the former Prime Minister of one of the world's most powerful countries advocating for it. If that doesn't sound dire, I don't know what does.
Don't forget to join our 80 Level Talent platform and our Telegram channel, follow us on Instagram, Twitter, LinkedIn, TikTok, and Reddit, where we share breakdowns, the latest news, awesome artworks, and more.