The rise of the “prompt engineer” and “AI whisperer”: How generative AI is transforming content moderation

Following the emergence of specialized “prompt engineer” positions within artificial intelligence (AI) companies, the need for a new kind of role is being recognized in the user-generated content (UGC) moderation industry too – trained human operators who specialize in understanding artificial intelligence tools. At WebPurify, we call these people “AI whisperers.”

Artificial intelligence has played a pivotal role in content moderation for a number of years and its superpowers are scalability and speed. It can keep pace with vast volumes of UGC, processing decisions in near real-time to proactively remove violative content before it reaches the end-user. It can also learn based on feedback from human moderators and become more accurate over time. Artificial intelligence is an essential tool for any platform that deals with UGC at scale, and has proven to be extremely effective.

Traditionally, harmful content is moderated using a combination of manual and automated systems. In such an approach, machine learning models or artificial intelligence are used to detect patterns in UGC, and then assign a confidence score indicating the likelihood of harmful content being present. They then take a corresponding action – accept, reject, or send for human review. Human moderators are layered on top, to review content the automated systems aren’t sure about, and thus might warrant additional scrutiny.

Human moderators are required because AI isn’t perfect. It can struggle with nuance and context, and can’t readily handle review workflows that might require additional investigation. For these reasons, an AI moderation-only approach will likely result in a high level of false positives, creating a poor user experience.

A particularly good example of this problem is when moderating for misinformation. Artificial intelligence has proved a poor detector of misinformation, and it can take a while for AI to adapt to emergent risks. By contrast, human moderators are better at parsing nuance and discerning fact from fiction, and can also pivot more readily. They’re just slower when it comes to processing masses of UGC at scale.

But as the saying goes, that was then, and this is now. Sophisticated generative AI tools like OpenAI’s ChatGPT and image generators such as DALL·E 2 and Midjourney present serious new challenges for content moderation. Generative AI can be used to create everything from convincing scams and sophisticated disinformation to content that infringes on copyright and intellectual property. And it can do so convincingly, at extraordinary speed and scale.

At WebPurify, we’re monitoring the emergence of AI-generated content (AIGC), and can already see it will require new moderation approaches that incorporate the skills of human experts with a deep understanding of how generative AI works.

A new job category has already emerged in the field of artificial intelligence, within companies that are designing AI tools. “Prompt engineers” are tasked with training AI so that it delivers more accurate, relevant, and less harmful responses. It’s a prompt engineer’s job to devise strategies for instructing AI models in an optimal way to help refine their responses, thereby mitigating the potential for them to generate misleading content or so-called “hallucinations.”

Important skills include a good understanding of language, pattern recognition, risk assessment, and the ability to think creatively and ask open-ended questions. It’s entirely possible that as these roles evolve, candidates may not even be required to write code – in fact, the skill set required may become closer to that of a psychologist. For their part, developers are already being encouraged to extend their existing skill sets to include the writing and engineering of effective AI prompts.

Meanwhile, moderation companies are experimenting with using ChatGPT to build highly accurate, text-based UGC moderation models that require far less training data than traditional methods. Generative AI still can’t match humans when it comes to detection of nuance, but it is able to discern subtleties in UGC that traditional AI-based moderation models may struggle with.

At WebPurify, we think that the prompt engineer / AI whisperer skill set will become increasingly valuable at both ends of the content moderation pipeline – not only in designing better prompts to make generative AI tools safer, but also when it comes to parsing AI-generated content for harmful, misleading or otherwise violative material.

Building and implementing AI is not a set-it-and-forget-it exercise. While “prompt engineers” have an important role to play in designing better and safer AI tools, that can’t be the end of the story. In order to use generative AI effectively, you have to understand it. That’s especially true in the field of content moderation.

Ultimately, humans should be involved in every stage of AI development, from training the models to overseeing AI-powered moderation tools. So in addition to the AI companies that are hiring prompt engineers, “AI whisperers” ought to be a core component of every organization that uses generative AI, including for content moderation. And as AI evolves, so should these roles.