Consider a platform where a new image-generation tool goes viral over a weekend. By Monday, the platform’s classifiers have caught most of the obvious material, and the human queue is shorter than last month. It is also harder: every item is one the model couldn’t decide on, some of it AI-generated abusive imagery nobody on the team has seen before. That queue is where content moderator well-being is now decided.
Generative AI changes what people try to publish, and AI-first moderation changes which of it reaches a person’s screen. Together, these shifts make moderator well-being an operating risk for Trust & Safety, one that needs owners, controls, and metrics.
If you lead Trust & Safety, in-house or through a partner, this article will help you decide what to build into your moderation model, how to measure it alongside quality, and what to ask the partner who employs your moderators.
At a glance
- Generative AI raises the volume and severity of some harmful content. The Internet Watch Foundation found more than 260 times as many AI-generated child sexual abuse videos in 2025 as in 2024.
- AI screening can reduce how much harmful content people see overall, but it leaves them the most ambiguous and unfamiliar items, along with appeals and escalations.
- Moderator well-being affects decision consistency, continuity, and legal exposure, so each queue needs named owners and controls.
- Read well-being signals next to quality metrics, in aggregate, and never use them to score individuals.
- Ask your Trust & Safety partner for specifics you can verify: exposure limits, who provides support and for how long, and which well-being indicators they report.
What AI-generated content changes for the people who review it
Generative AI lets people produce harmful material faster, in more variations, and in forms classifiers haven’t yet learned to recognize, from realistic deepfake videos to new types of synthetic abuse imagery. For moderators, that changes the volume, variety, speed, and severity of what reaches review.
The clearest data comes from child protection. The Internet Watch Foundation’s 2025 Annual Data & Insights Report found 3,443 AI-generated child sexual abuse videos in 2025, a more than 260-fold increase on 2024. The material is also more severe: 65% of the AI-generated videos fell into Category A, the most severe classification, compared with 43% of videos involving real children. The figure covers one harm type, but it shows how quickly a pattern can move.
Variety and speed add their own pressure. One abusive image can be regenerated into many slightly different versions, so hash matching against known content may catch fewer of them. New harmful trends can follow new tool releases, and classifiers need examples before they learn a pattern. Until they do, people see the new pattern first.
Synthetic material also adds a task. A reviewer may need to judge whether an image or video is real or generated, because the answer can change the policy, the legal reporting route, or the urgency. Making that call means looking longer and more closely at exactly the content teams would prefer people looked at briefly. Our overview of AI content moderation trends covers how platforms are approaching detection.
Why AI screening concentrates the hardest work in the human queue
AI reduces how much harmful content people review in total, but whatever it can’t decide still goes to a person. The human queue becomes smaller and denser with difficult cases.
The mechanism is the confidence threshold. Content the model scores as a clear violation is actioned automatically, and content it scores as clearly safe goes through. The uncertain middle reaches reviewers, along with appeals, escalations, new types of harm the model hasn’t seen, and often the review and labeling of data used to train and evaluate the models themselves. That was the Monday queue in our example: fewer items, almost none of them easy.
Severity is only part of the strain. Ambiguity and time pressure weigh on reviewers too. Researchers at the University of Queensland found that moderators handle around 1,500 items in an eight-hour shift, or roughly 19 seconds per item. Time pressure pushed them toward removing content rather than down-ranking it, and toward relying too heavily on automated suggestions. When a queue concentrates the hardest calls, each call needs enough time to get right.
So as automation takes more routine volume, protection for the people handling the rest has to be designed in. It will not arrive as a side effect of a lighter workload. (See how AI content moderation works for the automated side.)
Why moderator well-being is a quality and risk issue
Platforms depend on human judgment for the decisions their models get wrong, and judgment under sustained strain tends to become slower, less consistent, and harder to keep on the team. That makes moderator well-being part of how a Trust & Safety leader manages decision quality and risk.
“AI can process far more images than a person, but it still misses nuance. I’ve seen a model flag a photo of a dad hugging his son as something criminal. It took a human to see what it really was. That judgment is what platforms are relying on, and it only holds up if the people making those calls are properly supported.”
The tension isn’t new. A 2021 review, The Psychological Well-Being of Content Moderators by Steiger and colleagues, quotes Facebook describing the difficulty of “finding the right balance between content reviewer well-being and resiliency, quality, and productivity.” More recent research suggests how widespread the strain is. In a Middlesex University study of 160 moderators, just over a quarter showed moderate to severe psychological distress, and a third reported low well-being.
The consequences reach the platform in three ways. Decisions on gray-area content become less consistent across reviewers and shifts. Specialized queues and languages lose experienced people, and with them policy knowledge that takes time to rebuild. And legal and reputational exposure grows. In December 2024, CNN reported that more than 140 former Facebook moderators in Kenya had been diagnosed with PTSD and other conditions, according to court filings.
Independent research that puts a number on the link between well-being and accuracy is still thin, so treat vendor figures with care. The mechanism, though, is one most Trust & Safety leaders will recognize in their own data. We cover the effects on moderators’ mental health in a separate article.
What a content moderator well-being model should include
A responsible model treats exposure to harmful content as a workplace risk, reduced by design, managed across the employee journey, and assigned to a named owner for each control.
Two benchmarks help. ISO 45003 gives guidelines for managing psychosocial risk as part of occupational health and safety. And in 2025, the Global Trade Union Alliance of Content Moderators, supported by UNI Global Union, published eight protocols. They include exposure limits, no productivity targets for the most egregious content, and mental health support for at least 24 months after a contract ends.
| Control | What it involves | What it protects |
| Selection and realistic preview | Honest preview of content types and a realistic discussion of fit. | Fit and early attrition |
| Training | Trauma-informed training for moderators and team leads, and a briefing before a new content type goes live | Consistency and early recognition of strain |
| Exposure controls | Blurring, greyscale, and muted audio by default, with reviewers able to reveal detail | Emotional impact |
| Protected workflows | Exposure caps, rotation, and no throughput targets on the most severe queues | Cumulative exposure and rushed decisions |
| Debriefs and check-ins | A structured debrief after distressing incidents, and one-to-ones that cover workload | Early signals of strain |
| Psychological support | Confidential 24/7 counseling in moderators’ languages, often via an Employee Assistance Program, continuing after they leave | Recovery and long-term health |
| Time off and recovery | Protected breaks that aren’t traded away during volume spikes | Fatigue and sustained quality |
| Manager accountability | Team leads own well-being indicators for their queue alongside quality | Follow-through when volume rises |
The usual objection to content moderator wellness tools is speed. Research presented at HCOMP 2020 found that interactive blurring can reduce emotional impact without sacrificing moderation accuracy or speed. Earlier Google research found that greyscale also changes emotional impact, while a full blur can make the reviewer’s job harder. So reduce detail by default, and let the reviewer reveal what the decision depends on.
These controls have a cost. Rotation and exposure caps reduce throughput on the most severe queues, so build them into capacity plans and keep them in place when volume peaks. Our guide covers how to recruit, train, and retain moderators.
How to measure well-being and moderation quality together
To see whether a content moderator wellness program is working, track a small set of aggregated well-being signals for each queue, and read them next to the quality and risk metrics you already report. A change in one can help explain a change in the other. The 2021 review by Steiger and colleagues recommends tracking absenteeism, low productivity, and turnover.
| Well-being signal (per queue) | What it may indicate | Read it next to | Typical response |
| Unplanned absence and short-notice leave | Cumulative strain on a queue | QA accuracy and appeal overturn rate on that queue | Review queue mix and rotation |
| Voluntary attrition and tenure, with the first 90 days tracked separately | Poor fit, weak onboarding, or exposure | Time to proficiency and QA scores for new starters | Adjust selection, preview, and onboarding |
| Handle-time variance on severe queues | Hesitation, avoidance, or unclear policy | Calibration agreement between reviewers | Clarify policy, recalibrate, check exposure |
| Escalation volume | Uncertainty or fatigue | Escalation acceptance rate | Targeted coaching, review new content types |
| A short, validated well-being survey, chosen with clinical advisers | How teams are coping over time | All of the above, as trends | Discuss in team-lead reviews, adjust controls |
| Counseling uptake, anonymous and aggregated | Rising need, or rising trust in the service | Absence and attrition trends | Never interpret on its own |
Three rules keep this fair: aggregate and anonymize everything, never use well-being data in an individual’s performance review, and report trends to the client, not personal information. Skip target percentages until you have your own baseline. The value is in spotting patterns early, such as calibration dropping on one queue in the same month its absence rate rises.
What to ask a Trust & Safety partner about moderator well-being
Ask for specifics you can verify. A wellness policy is easy to write, so the useful questions are about how support works on the busiest day of the quarter.
- What exposure limits apply to the most severe queues, and who can override them?
- Are throughput targets removed or adjusted for egregious content?
- Which exposure controls are on by default, and can reviewers adjust them?
- Who provides psychological support, are they licensed, and is it available 24/7 in moderators’ languages?
- Does support continue after someone leaves the queue or the company, and for how long?
- Which well-being indicators will you report to us, how often, and how are they anonymized?
- When a new type of harmful content appears, how are moderators briefed before volume arrives?
Vague answers, no exposure limits, or support that ends with the contract are all reasons to dig further. For the wider partner-selection process, see how to choose a content moderation partner.
How Conectys approaches moderator well-being
Most partners can describe a well-being program. The harder question is how it operates when volume rises.
At Conectys, Trust & Safety work starts with the question behind every Business Transformation Outsourcing (BTO) engagement: what should this process become? In moderation, that means deciding which content AI should handle, which decisions need people, and what those people need to make them well. AI-assisted review flags risk, and humans make the important calls. Because people stay accountable for the hardest decisions, their well-being is part of the operating design from the start.
We run a strong wellness program covering content moderators. It provides preventive and ongoing care through mental health resources, awareness training, and psychological support, including confidential counseling through an Employee Assistance Program. Rotation models and support systems are built into delivery, so moderator well-being is part of how each program runs day to day.
The questions in the previous section are a fair test for any partner, including us. Bring them to a conversation, and we’ll walk through how moderators would be supported on the queues you need covered. That includes exposure controls, rotation, psychological support, and what happens when a new type of harmful content appears. The principle behind it is simple: healthy teams make better decisions.
Where to start
Before you expand AI screening, map which content now reaches people and how that mix has changed. Set exposure controls and a named owner for the queues that carry the most severe and ambiguous material, and plan capacity for rotation before volume rises. Then add three or four well-being signals to the quality review you already run, and agree with your partner how they will be reported. The smaller Monday queue is the one that needs the most protection, because it is where your hardest decisions are made.
