Guide
Set up content moderation to meet Online Safety Act requirements
How to build a content moderation system that meets Online Safety Act 2023 duties. Covers automated detection tools, human moderation teams, user reporting mechanisms, content review workflows, removal timelines, record-keeping, and moderator wellbeing.
If you operate a user-to-user service or search service with users in the United Kingdom, the Online Safety Act 2023 requires you to have systems and processes for moderating content. This applies whether your platform is based in the UK or overseas — what matters is that UK users can access it.
Content moderation is not optional. Every regulated service must take proportionate steps to prevent users from encountering illegal content, and — for services likely to be accessed by children — content that is harmful to children. Category 1 services have additional duties around content harmful to adults.
The scale and sophistication of your moderation system should be proportionate to your service's size, risk profile, and the nature of content shared on it. A small community forum will need a different approach from a large social media platform, but both must demonstrate they have effective systems in place.
Before you start
You must complete an illegal content risk assessment before designing your moderation system. The risk assessment identifies the types of illegal content most likely to appear on your service and informs the measures you need to take. If your service is likely to be accessed by children, you also need a children's risk assessment. Your moderation system should directly address the risks identified in these assessments.
Understanding your obligations
Your content moderation duties depend on which categories of content your service must address. All regulated services must moderate illegal content. The Online Safety Act identifies over 130 priority offences in Schedule 7 that you must specifically consider when designing your systems.
How to set up your content moderation system
Follow these steps to build a moderation system that meets Ofcom's codes of practice. Each step should be proportionate to the size and risk profile of your service.
-
1. Map your content risks to moderation requirements
Use the findings from your illegal content risk assessment (and children's risk assessment if applicable) to identify exactly which content types your moderation system must detect and remove. Prioritise the highest-risk categories — for most services, child sexual exploitation and abuse (CSEA) material and terrorism content require the most robust automated measures.
-
2. Deploy automated detection tools for priority illegal content
Implement hash-matching technology (such as PhotoDNA for images or the GIFCT hash-sharing database for terrorism content) to automatically detect known CSEA and terrorism material. For other priority offences, consider AI-powered classifiers trained to flag content matching patterns of fraud, hate speech, or incitement to violence. Automated tools should flag content for review rather than remove it silently, unless you are dealing with verified CSEA hashes where immediate removal is required.
-
3. Establish a human moderation team
Recruit and train human moderators to review content flagged by automated systems and handle appeals. Moderators need clear guidelines on each content category, decision-making frameworks for borderline cases, and escalation procedures for complex decisions. Ensure moderators understand the legal definitions of priority offences — not just your platform's community standards. For smaller services, this may be a designated person rather than a full team, but someone must be accountable.
-
4. Build user reporting mechanisms
Create easy-to-use reporting tools that allow users to flag content they believe violates your terms of service or is illegal. Reporting must be accessible from every piece of content on your service (not buried in settings). Include category options so users can specify the type of concern — this helps route reports to the right moderation queue. Acknowledge receipt of reports and provide a reference number so users can track the outcome.
-
5. Design content review workflows with defined timelines
Create tiered review workflows based on severity. CSEA material and terrorism content should be reviewed and actioned within hours. Other illegal content should be reviewed within 24 hours of being flagged. Harmful content (for services with children's safety duties) should be reviewed within 48 hours. Document your target timelines and track actual performance against them. Ofcom will expect to see evidence that your processes work in practice, not just on paper.
-
6. Set up record-keeping and audit trails
Maintain records of all moderation decisions, including the content reviewed, the decision taken, who made the decision, the reasoning, and the timestamp. These records serve two purposes — they demonstrate compliance to Ofcom if you receive an information notice, and they help you identify patterns and improve your systems. Retain records for at least three years. Use structured data formats so you can produce reports on moderation volumes, decision types, and response times.
-
7. Implement moderator wellbeing protections
Content moderators are exposed to distressing material, particularly CSEA imagery, graphic violence, and self-harm content. You have a duty of care as an employer. Provide regular psychological support sessions (not just on request), limit continuous exposure time to harmful content (rotate moderators through different queues), offer pre-screening and opt-out for the most distressing categories, and conduct regular wellbeing check-ins. Document your wellbeing policy and review it at least annually. Ofcom's codes of practice explicitly reference moderator welfare as part of a proportionate moderation system.
-
8. Ensure proportionality across your measures
Ofcom assesses proportionality based on your service's size, resources, nature of content, and level of risk. A startup with 10,000 users is not expected to deploy the same technology as a platform with 10 million users. Document why you chose your specific measures and how they relate to your risk assessment findings. If you cannot afford enterprise-grade automated tools, demonstrate that you have alternative measures (such as more intensive human moderation or proactive content sampling) that achieve the same safety outcomes.
-
9. Test your systems before they go live
Run scenario testing using sample content across all priority offence categories. Verify that automated tools correctly flag target content, that flagged items reach human moderators within your target timelines, that user reports are properly routed and acknowledged, and that appeals are handled fairly. Document the results and address any gaps before your service launches or before your compliance deadline.
-
10. Establish ongoing review and improvement cycles
Content moderation is not a one-time setup. Review your systems quarterly — check detection accuracy, false positive rates, response times, user satisfaction with reporting tools, and moderator wellbeing metrics. Update your risk assessment annually or when your service changes significantly (new features, new content types, significant user growth). Ofcom expects continuous improvement, not static compliance.
What happens next
Once your moderation system is operational, you must keep your risk assessments up to date and ensure your terms of service accurately describe your moderation practices. Ofcom can issue information notices requiring you to demonstrate how your systems work, so maintain accessible documentation at all times.
If you receive a complaint that your moderation is inadequate, respond through your complaints procedure and document the outcome. Persistent failures to moderate effectively can lead to Ofcom enforcement action.