synapti/nci-social-classifier-v3

NCI Social Media Classifier v3

A multi-label classifier for detecting propaganda techniques in social media content (tweets, posts, short-form text).

Model Description

This model is fine-tuned from vinai/bertweet-base for multi-label classification of 18 propaganda techniques defined in the SemEval-2020 Task 11.

Key improvements in v3:

• 4x better F1 Macro (0.55 vs 0.14 in v2) through synthetic data augmentation
• 15/18 techniques with good coverage (F1 ≥ 0.3), up from 6/18
• Better detection of rare techniques: Repetition, Flag-Waving, Slogans, Black-and-White Fallacy

Performance

Metric	Score
F1 Micro	0.796
F1 Macro	0.551
Techniques with F1 ≥ 0.3	15/18

Per-Technique Performance

Technique	F1	Support
Name_Calling,Labeling	0.960	724
Appeal_to_Authority	0.897	16
Red_Herring	0.880	14
Obfuscation,Intentional_Vagueness,Confusion	0.800	6
Loaded_Language	0.796	480
Whataboutism,Straw_Men,Red_Herring	0.700	26
Slogans	0.667	4
Appeal_to_fear-prejudice	0.662	246
Reductio_ad_hitlerum	0.612	68
Doubt	0.564	23
Exaggeration,Minimisation	0.524	146
Flag-Waving	0.467	23
Repetition	0.444	7
Bandwagon	0.400	8
Black-and-White_Fallacy	0.300	17

Intended Use

• Analyzing social media posts for propaganda techniques
• Content moderation assistance
• Media literacy tools
• Research on information manipulation

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

# Load model
tokenizer = AutoTokenizer.from_pretrained("vinai/bertweet-base", use_fast=False)
model = AutoModelForSequenceClassification.from_pretrained("synapti/nci-social-classifier-v3")

# Technique labels
TECHNIQUES = [
    "Loaded_Language", "Appeal_to_fear-prejudice", "Exaggeration,Minimisation",
    "Repetition", "Flag-Waving", "Name_Calling,Labeling", "Reductio_ad_hitlerum",
    "Black-and-White_Fallacy", "Causal_Oversimplification", "Whataboutism,Straw_Men,Red_Herring",
    "Straw_Man", "Red_Herring", "Doubt", "Appeal_to_Authority", "Thought-terminating_Cliches",
    "Bandwagon", "Slogans", "Obfuscation,Intentional_Vagueness,Confusion",
]

# Predict
text = "Build the wall! Build the wall! Build the wall! #MAGA"
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128)

with torch.no_grad():
    outputs = model(**inputs)
    probs = torch.sigmoid(outputs.logits)[0]

# Get detected techniques
detected = [(TECHNIQUES[i], float(probs[i])) for i in range(18) if probs[i] > 0.3]
print(detected)
# [('Slogans', 0.51), ('Repetition', 0.39), ...]

Training Data

Trained on synapti/nci-social-v3:

• 9,069 training samples
• 1,134 validation samples
• 1,134 test samples

Data sources:

• LLM-labeled HateXplain samples
• SemEval span_propaganda short samples
• Synthetic examples for underrepresented techniques
• Balanced negative samples

Training Procedure

• Base model: vinai/bertweet-base
• Loss: Focal Loss (γ=2.0) for class imbalance
• Optimizer: AdamW
• Learning rate: 2e-5
• Batch size: 32
• Epochs: 8
• Hardware: NVIDIA L4 GPU

Limitations

• Optimized for English social media content
• May not generalize well to formal news articles (use nci-technique-classifier for those)
• Some rare techniques (Straw_Man, Thought-terminating_Cliches) still have limited coverage

Citation

@misc{nci-social-classifier-v3,
  author = {Synapti},
  title = {NCI Social Media Classifier v3},
  year = {2024},
  publisher = {HuggingFace},
  url = {https://huggingface.co/synapti/nci-social-classifier-v3}
}

License

Apache 2.0