KidsNanny: A Two-Stage Multimodal Content Moderation Pipeline Integrating Visual Classification, Object Detection, OCR, and Contextual Reasoning for Child Safety

• URL: http://arxiv.org/abs/2603.16181v1
• Date: Tue, 17 Mar 2026 07:00:43 GMT
• Title: KidsNanny: A Two-Stage Multimodal Content Moderation Pipeline Integrating Visual Classification, Object Detection, OCR, and Contextual Reasoning for Child Safety
• Abstract summary: We present KidsNanny, a two-stage multimodal content moderation architecture for child safety.<n>Stage 1 combines a vision transformer (ViT) with an object detector for visual screening (11.7 ms)<n>Stage 2 applies OCR and a text based 7B language model for contextual reasoning (120 ms total pipeline)<n>We evaluate on the UnsafeBench Sexual category (1,054 images) under two regimes: vision-only, isolating Stage 1, and multimodal, evaluating the full Stage 1+2 pipeline.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present KidsNanny, a two-stage multimodal content moderation architecture for child safety. Stage 1 combines a vision transformer (ViT) with an object detector for visual screening (11.7 ms); outputs are routed as text not raw pixels to Stage 2, which applies OCR and a text based 7B language model for contextual reasoning (120 ms total pipeline). We evaluate on the UnsafeBench Sexual category (1,054 images) under two regimes: vision-only, isolating Stage 1, and multimodal, evaluating the full Stage 1+2 pipeline. Stage 1 achieves 80.27% accuracy and 85.39% F1 at 11.7 ms; vision-only baselines range from 59.01% to 77.04% accuracy. The full pipeline achieves 81.40% accuracy and 86.16% F1 at 120 ms, compared to ShieldGemma-2 (64.80% accuracy, 1,136 ms) and LlavaGuard (80.36% accuracy, 4,138 ms). To evaluate text-awareness, we filter two subsets: a text+visual subset (257 images) and a text-only subset (44 images where safety depends primarily on embedded text). On text-only images, KidsNanny achieves 100% recall (25/25 positives; small sample) and 75.76% precision; ShieldGemma-2 achieves 84% recall and 60% precision at 1,136 ms. Results suggest that dedicated OCR-based reasoning may offer recall-precision advantages on text-embedded threats at lower latency, though the small text-only subset limits generalizability. By documenting this architecture and evaluation methodology, we aim to contribute to the broader research effort on efficient multimodal content moderation for child safety.

Related papers

• When Seeing Overrides Knowing: Visual Dominance and Deferral-Based Method for Personalized Safety in VLMs [54.95635713318535]
  Vision-language models (VLMs) are increasingly deployed in high-stakes settings, where a response that is reasonable in general may still be unsafe for a particular user.<n>We study this problem of personalized safety in multimodal systems and introduce MPS-Bench, a benchmark of 5,181 scenarios from 584 real-world images.<n>We find that they almost always respond directly (86-99%) rather than seek missing context, and none exceeds 2.6/5 on personalized safety.<n>Motivated by this mechanism, we propose PRISM, a lightweight input monitor that uses bidirectional cross-modal modulation to predict when a query is likely to require
  arXiv  Detail & Related papers  (2026-09-03T03:58:56Z)
• T2LSC-Bench: Benchmarking Localized Semantic Control in Text-to-Image Generation [30.939965668317967]
  In product labeling, signage, and interface design, target text should be rendered within a designated text-bearing region without altering the predefined subject identity or surrounding scene semantics.<n>Existing visual-text benchmarks evaluate readability, spelling accuracy, and layout, leaving this form of semantic leakage largely unexamined.<n>We introduce T2LSC-Bench, a controlled diagnostic benchmark comprising 50 seed subjects and 1,200 prompt cases per model, yielding 7,160 evaluated images across six models.<n>A dual-branch protocol combines OCR-VLM text verification with structured VLM semantic judgments to measure Text-at-Anchor Accuracy (TAA), Semantic Subject Preservation (
  arXiv  Detail & Related papers  (2026-09-02T08:02:40Z)
• When Prompts Become Pixels: Prompt-Region Grounding for Multimodal Reasoning [73.56343820592399]
  We introduce Visualized Task Semantics (VTS), a controlled intervention that moves the question into the image while keeping the source problem and answer fixed.<n>Across six MLLMs and four benchmarks, accuracy drops in all 24 model-task pairs, by 17.8 points on average.<n>We present prompt-region grounding, whose core design aligns the question region with typed semantics and recovers its clean representation from a masked view.
  arXiv  Detail & Related papers  (2026-08-05T11:46:15Z)
• Lens: Rethinking Training Efficiency for Foundational Text-to-Image Models [88.94987156920575]
  We introduce Lens, a T2I model that achieves performance competitive with, and in several cases surpassing, state-of-the-art models with more than 6B parameters.<n>For example, Lens requires only about 19.3% of the training compute used by Z-Image.<n>Lens generates a 10242 image in 3.15 seconds on a single NVIDIA H100 GPU, while its distilled turbo version performs 4-step generation in 0.84 seconds.
  arXiv  Detail & Related papers  (2026-05-20T17:59:58Z)
• VisualLeakBench: Auditing the Fragility of Large Vision-Language Models against PII Leakage and Social Engineering [14.756677328512907]
  VisualLeakBench is an evaluation suite to audit LVLMs against OCR Injection and Contextual PII Leakage.<n>We use 1,000 synthetically generated adversarial images with 8 PII types, validated on 50 in-the-wild (IRL) real-world screenshots.<n>We release our dataset and code for reproducible robustness and safety evaluation of deployment-relevant vision-language systems.
  arXiv  Detail & Related papers  (2026-03-11T05:47:24Z)
• Malicious Image Analysis via Vision-Language Segmentation Fusion: Detection, Element, and Location in One-shot [18.80045630689047]
  We introduce a zero-shot pipeline that simultaneously detects if an image contains harmful content.<n>It identifies each critical element involved, and it localizes those elements with pixel-accurate masks.<n>The system processes an image in seconds, plugs seamlessly into existing VLM, and constitutes the first practical tool for fine-grained, explainable malicious-image moderation.
  arXiv  Detail & Related papers  (2025-12-04T09:18:14Z)
• VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safety [3.1109025622085693]
  We present Vision Language Safety Understanding, a comprehensive framework to evaluate multimodal safety.<n>Our evaluation of eleven state-of-the-art models reveals systematic joint understanding failures.<n>Our framework exposes weaknesses in joint image-text understanding and alignment gaps in current models.
  arXiv  Detail & Related papers  (2025-10-21T01:30:31Z)
• Probabilistic Language-Image Pre-Training [58.2451360061285]
  We introduce Probabilistic Language-Image Pre-training (ProLIP), the first probabilistic VLM pre-trained on a billion-scale image-text dataset.<n>ProLIP efficiently estimates uncertainty by an "uncertainty token" without extra parameters.<n>We also introduce a novel inclusion loss that enforces distributional inclusion relationships between image-text pairs and between original and masked inputs.
  arXiv  Detail & Related papers  (2024-10-24T15:42:25Z)
• TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokens [9.453667770656644]
  We present TextHawk2, a bilingual LVLM featuring efficient fine-grained perception and demonstrating cutting-edge performance across general-purpose, OCR, and grounding tasks with 16 times fewer image tokens. We enhance the visual encoder through LVLM co-training, unlocking its potential for previously unseen tasks like Chinese OCR and grounding. We assess TextHawk2 across multiple benchmarks, where it consistently delivers superior performance and outperforms closed-source models of similar scale.
  arXiv  Detail & Related papers  (2024-10-07T17:58:35Z)
• SafaRi:Adaptive Sequence Transformer for Weakly Supervised Referring Expression Segmentation [11.243400478302771]
  Referring Expression Consistency (RES) aims to provide a segmentation mask of the target object in an image referred to by the text. We propose a weakly-supervised bootstrapping architecture for RES with several new algorithmic innovations.
  arXiv  Detail & Related papers  (2024-07-02T16:02:25Z)
• Open-Vocabulary Segmentation with Unpaired Mask-Text Supervision [87.15580604023555]
  Unpair-Seg is a novel weakly-supervised open-vocabulary segmentation framework. It learns from unpaired image-mask and image-text pairs, which can be independently and efficiently collected. It achieves 14.6% and 19.5% mIoU on the ADE-847 and PASCAL Context-459 datasets.
  arXiv  Detail & Related papers  (2024-02-14T06:01:44Z)
• Turning a CLIP Model into a Scene Text Spotter [73.63953542526917]
  We exploit the potential of the large-scale Contrastive Language-Image Pretraining (CLIP) model to enhance scene text detection and spotting tasks. This backbone utilizes visual prompt learning and cross-attention in CLIP to extract image and text-based prior knowledge. FastTCM-CR50 introduces an instance-language matching process to enhance the synergy between image and text embeddings.
  arXiv  Detail & Related papers  (2023-08-21T01:25:48Z)
• Transferring Pre-trained Multimodal Representations with Cross-modal Similarity Matching [49.730741713652435]
  In this paper, we propose a method that can effectively transfer the representations of a large pre-trained multimodal model into a small target model. For unsupervised transfer, we introduce cross-modal similarity matching (CSM) that enables a student model to learn the representations of a teacher model. To better encode the text prompts, we design context-based prompt augmentation (CPA) that can alleviate the lexical ambiguity of input text prompts.
  arXiv  Detail & Related papers  (2023-01-07T17:24:11Z)
• BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers [117.79456335844439]
  We propose to use a semantic-rich visual tokenizer as the reconstruction target for masked prediction. We then pretrain vision Transformers by predicting the original visual tokens for the masked image patches. Experiments on image classification and semantic segmentation show that our approach outperforms all compared MIM methods.
  arXiv  Detail & Related papers  (2022-08-12T16:48:10Z)
• LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding [49.941806975280045]
  Pre-training of text and layout has proved effective in a variety of visually-rich document understanding tasks. We present text-bfLMv2 by pre-training text, layout and image in a multi-modal framework.
  arXiv  Detail & Related papers  (2020-12-29T13:01:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.