Unmasking Hidden Threats: A data-driven snapshot on how Facebook enhanced it’s proactive rate in detecting potentially violating content

A metric that proactively detects potentially violating content in FB

Facebook (FB) shares its content policies that link to Facebook’s Community Standards, which include the most comprehensive descriptions of these policies. To make progress on removing content that violates FB Community Standards, it introduced several metrics to measure the extent of violation and its impact. “Prevention is always better than cure” which means it’s better to stop something before it happens rather than fix it or deal with the consequences after it’s already happened. Facebook also doing the same by proactively detecting potentially violating content.

The proactive rate is a metric that shows the percentage of all content or accounts acted on, that Facebook found and flagged before users reported them to Facebook. Facebook uses this metric as an indicator of how effectively Facebook detects violations. The rate at which Facebook can proactively detect potentially violating content is high for some violations, meaning that Facebook finds and flags most content before users do. This is especially true when Facebook has been able to build machine learning technology that automatically identifies content that might violate Facebook’s standards.

Facebook calculates this percentage as the number of pieces of content acted on that Facebook found and flagged before people using Facebook reported them, divided by the total number of pieces of content that Facebook took action on. For fake accounts on Facebook, Facebook calculates this metric as the percentage of Facebook accounts disabled for being fake that Facebook found and flagged before users reported them. It’s calculated as the number of disabled accounts that Facebook found and flagged before users reported them, divided by the total number of accounts disabled for being fake.

Adult nudity and sexual activity

Facebook restricts the display of adult nudity and sexual activity. Facebook makes some exceptions when it is evident that the content is being shared in the context of a protest, for educational or medical reasons, or for similar purposes. However, Facebook defaults to removing sexual imagery to prevent the sharing of non-consensual or underage content. The proactive rate at which Facebook finds and takes action on violating content for adult nudity and sexual activity before users report it consistently remains around 90% over the years.

Bullying and Harassment

Facebook maintains a zero-tolerance policy towards bullying and harassment. Recognizing the particularly harmful impact on minors, Facebook’s policies afford heightened protections for them. The platform aims to foster open and crucial discussions about individuals in the news or those with significant public followings, thus permitting more critical discourse towards public figures compared to private individuals. Given the deeply personal nature of bullying and harassment, proactively detecting these behaviours using technology poses greater challenges than with other violations. Therefore, Facebook also relies on user reports to identify and remove such content. When measuring prevalence, the metric specifically targets bullying and harassment instances where a deeper understanding of context or meaning isn’t required to determine policy violations. Facebook remains committed to enhancing proactive detection technology to combat this issue and safeguard the community. The proactive rate for all content or accounts acted on for bullying and harassment saw a sixfold increase between 2018 and 2023. In the fourth quarter of 2023, Facebook found and flagged 87% of all content or accounts engaged in bullying and harassment before users reported them to the platform.

Child endangerment

Facebook strictly prohibits content that jeopardizes children, including material featuring nudity, physical abuse, or sexual exploitation on the platform. Upon identifying such violating content, Facebook promptly removes it, irrespective of context or the individual’s intent behind sharing it. Facebook reserves the right to disable the account of the content sharer unless there are indications that the sharing was not malicious, such as raising awareness of child exploitation. The proactive rate for all content or accounts engaged in child endangerment has consistently remained around 99%.

Dangerous Orgs

Facebook prohibits organizations or individuals advocating violence or actively engaging in violent behaviour from having a presence on the platform. Content that glorifies, endorses, or represents individuals or groups involved in terrorist activities or organized hate is also strictly forbidden. The proactive rate for all content or accounts associated with organized hate decreased between the third and fourth quarters of 2023, while for terrorism, it remained consistently high, ranging from 97% to 99%. In the fourth quarter of 2023, Facebook found and flagged between 93% and 99% of all content or accounts associated with dangerous organizations before users reported them to the platform.

Fake accounts

Facebook’s goal is to remove as many fake accounts as possible from the platform. These encompass accounts created with malicious intent to violate Facebook’s policies, as well as personal profiles established to represent businesses, organizations, or non-human entities, such as pets. Facebook prioritizes enforcement against fake accounts aiming to cause harm, many of which are utilized in spam campaigns driven by financial motives. Facebook anticipates fluctuations in the number of accounts it takes action against over time due to the unpredictable nature of adversarial account creation. The platform’s detection technology aids in blocking millions of attempts to create fake accounts daily and detecting millions more, often within minutes after their creation. Blocked attempts are not included in the metrics reported by Facebook. The proactive rate for all content or accounts engaged in addressing fake accounts has consistently remained at approximately 99%.

Hate speech

Facebook does not allow hate speech on their platform. Hate speech is defined as violent or dehumanizing speech, statements of inferiority, calls for exclusion or segregation based on protected characteristics, or slurs. These characteristics include race, ethnicity, national origin, religious affiliation, sexual orientation, caste, sex, gender, gender identity, and serious disability or disease. When the intent is clear, Facebook may allow people to share someone else’s hate speech content to raise awareness or discuss whether the speech is appropriate, to use slurs self-referentially in an effort to reclaim the term, or for other similar reasons. In the fourth quarter of 2023, Facebook found and flagged 95% of all content or accounts engaged in hate speech before users reported them to the platform. The proactive rate for all content or accounts involved in hate speech has increased by 1.6 times between 2018 and 2023.

Related goods

Facebook does not allow private individuals, manufacturers, or retailers to buy, sell, or trade non-medical drugs, pharmaceutical drugs, or marijuana on Facebook. Additionally, Facebook prohibits private individuals from engaging in the buying, selling, giving, exchanging, or transferring of firearms, including firearm parts or ammunition. While drugs and firearms are subject to different legal regulations globally, Facebook enforces these standards consistently across its platform due to the borderless nature of the community. Facebook found and flagged between 96% and 97% of all content or accounts engaged with regulated goods before users reported them to the platform in the fourth quarter of 2023. The proactive rate for all content or accounts involved with regulated goods has shown an upward trend over the years.

Spam

Facebook does not allow spam on its platform. Spam encompasses various deceptive and annoying content designed to mislead users into engaging with it. It can spread through automation, such as bots or scripts, or coordination, where individuals use multiple accounts to distribute deceptive content. Spammers typically seek to build audiences to boost the distribution and reach of their content, often for financial gain. The tactics employed by spammers, as well as Facebook’s ability to detect them, determine the volume of content actioned upon and Facebook’s proactive rate. Ninety-nine per cent of all content or accounts engaged in spam were identified and flagged by Facebook before users reported them to the platform in the fourth quarter of 2023. The proactive rate for all content or accounts involved in spam has consistently remained between 95% and 99% over the years.

Suicide and self-injury

Facebook removes content that encourages suicide or self-injury on its platform. Self-injury is defined as the intentional and direct injuring of the body, including self-mutilation and eating disorders. Facebook also removes content that identifies and negatively targets victims or survivors of self-injury or suicide. Facebook allows people to discuss suicide and self-injury because Facebook wants its platform to be a space where people can raise awareness about these issues and seek support. Ninety-nine per cent of all content or accounts engaged in Suicide and Self-Injury were identified and flagged by Facebook before users reported them to the platform in 2023 Q4 and the proactive rate for all such content or accounts has remained consistent over the years.

Violence and incitement / Violent & Graphic content

Facebook aims to prevent potential offline harm that may be related to content on the platform. While Facebook understands that people commonly express disdain or disagreement by threatening or calling for violence in non-serious ways, Facebook removes language that incites or facilitates serious violence. Facebook removes content, disables accounts, and collaborates with law enforcement when there is a genuine risk of physical harm or direct threats to public safety. Facebook also endeavours to consider the language and context to distinguish casual statements from content that constitutes a credible threat to public or personal safety. Facebook removes content that glorifies violence or celebrates the suffering or humiliation of others on Facebook. Facebook does allow people to share some graphic content to raise awareness about current events and issues. In these cases, Facebook may hide the content from people under 18 and cover it with a warning for those over 18, so people are aware it is graphic or violent before they choose to view it. Between 98% and 99% of all content or accounts involved in Violence – Incitement & Graphic Content were identified and flagged by Facebook before users reported them to the platform in 2023 Q4. The proactive rate for all such content or accounts has shown an increase over the years.

Facebook has made significant strides in proactively identifying potentially violating content, particularly in cases where machine learning technologies have been deployed to automatically flag such content before users do. However, while such technology holds promise, it remains a work in progress and is not yet universally effective across all types of violations. Contextual understanding and nuance, especially in text-based content, still present considerable challenges for proactive detection.

This metric, based on the volume of content actioned, has its limitations. It does not account for the time taken to detect violating content, the number of views before detection, or instances where violations went entirely undetected. Despite achieving proactive detection rates as high as 99% in certain categories, even the small remaining percentage can have a significant impact on users.

References

About Author:

Pankaj Chowdhury is a former Research Assistant at the International Economic Association. He holds a Master’s degree in Demography & Biostatistics from the International Institute for Population Sciences and a Bachelor’s degree in Statistics from Visva-Bharati University. His primary research interests focus on exploring new dimensions of in computational social science and digital demography.

Disclaimer: The views expressed in this article are those of the author and do not necessarily reflect the views of 360 Analytika.

Acknowledgement: The author extends his gratitude to the Facebook for providing data support.

This article is posted by Sahil Shekh, Editor-in-Chief at 360 Analytika.