Now playing · Benchmarks
Announcing Ray Guardian: Visual Safety for Images and Video
Ray Guardian is Ray's unified visual safety system, benchmarked across 13 image-level safety categories and public-labeled video datasets.
Introducing Ray Guardian
Ray Guardian is Ray's visual safety system for images and video. It reviews visual media for sensitive content and returns category signals Ray can use for playback, creation, moderation, and family settings.
The important part is simple: Ray Guardian is one system. Still images and video clips enter through different media paths, but the product receives one Guardian decision.
For images, Ray Guardian evaluates the full safety taxonomy: nudity, sexual content, violence, gore, weapons, drugs, alcohol, smoking, disturbing content, kissing / intimacy, self-harm, animal cruelty, and medical gore.
For video, Ray Guardian adds time. A single frame can be ambiguous; a clip can show action, intent, and context. The video path combines frame-level safety recognition with clip-level temporal understanding so the final result reflects what appears on screen and what is happening over time.
That is the system we benchmarked.
13-category visual safety benchmark
Ray Guardian reached 100.000% accuracy on the Ray Guardian 13-category held-out image benchmark: 17,435 known labels, 0 errors.
This benchmark is closest to the product taxonomy itself: the same category definitions, held-out examples, and the full set of Ray Guardian visual safety categories.
| Benchmark | Scope | Items | Accuracy | Precision | Recall | F1 |
|---|---|---|---|---|---|---|
| Ray Guardian held-out image-level benchmark | All 13 Ray Guardian categories | 17,435 labels | 100.000% | 100.000% | 100.000% | 100.000% |
The benchmark includes categories that often overlap in real products: sexual vs nudity, gore vs medical gore, weapons vs violence, and disturbing content vs ordinary unpleasant context.
Public-labeled video benchmark suite
Video benchmarks are measured against the labels each dataset actually provides. A kiss-vs-hug dataset supports kissing / intimacy evidence. A violence dataset supports violence and disturbing-event evidence. A safe-action dataset tests whether ordinary action clips are allowed through cleanly.
That makes the results easier to read and harder to misuse: each row reports Ray Guardian performance on that dataset's native label scope.
| Dataset | Ray Guardian scope | Clips | Accuracy | Precision | Recall | F1 |
|---|---|---|---|---|---|---|
| HMDB51 Kiss vs Hug | Kissing / Intimacy | 182 | 98.901% | 99.020% | 99.020% | 99.020% |
| XD-Violence Public Labels | Violence / Disturbing | 248 | 100.000% | 100.000% | 100.000% | 100.000% |
| UCF-Crime Mapped Labels | Violence / Disturbing | 79 | 100.000% | 100.000% | 100.000% | 100.000% |
| UCF-Crime Broad Anomaly | Anomaly / Disturbing | 109 | 100.000% | 100.000% | 100.000% | 100.000% |
| UCF101 Safe-Action Holdout | Safe-action guardrail | 330 | 100.000% | n/a | n/a | n/a |
The UCF101 row is a negative-only guardrail: it measures whether ordinary safe actions are incorrectly blocked. In that test, Ray Guardian correctly allowed all 330 safe-action clips.
What the confusion matrices show
Accuracy is the headline number. A confusion matrix shows the shape of the decisions behind it.
Each matrix separates correct allows, false alarms, missed unsafe clips, and correctly flagged unsafe clips. That matters because a safety system has to protect people without interrupting ordinary media more than necessary.
Independent reference datasets
We also evaluate Ray Guardian on independent public and gated reference datasets. These datasets are useful because their labels were written by other teams, for other goals, and do not always line up perfectly with Ray Guardian categories.
The table below reports each result at the category level the dataset can support. When a dataset tests alcohol objects, we report alcohol behavior. When it tests medical wounds, we report medical gore or disturbing behavior.
| Reference dataset | Ray Guardian scope | Items | Accuracy | Recall | Notes |
|---|---|---|---|---|---|
| HoliSafe-Bench test split | Mapped image-safety categories | 32,248 labels | 94.266% | 75.779% | Independent gated safety benchmark |
| UnsafeBench train split | Mapped image-safety categories | 40,545 labels | 89.893% | 74.482% | Independent gated safety benchmark |
| SurgWound test split | Medical gore | 137 images | 86.861% | 100.000% | Medical wound sensitivity check |
| SurgWound test split | Disturbing | 137 images | 83.942% | 99.130% | Medical wound sensitivity check |
| Lower Limb and Feet Wound Dataset | Medical gore | 5,443 images | 65.185% | 98.958% | Wound-domain reference check |
| HOD Benchmark Dataset | Alcohol | 10,631 images | 97.667% | 87.095% | Research-use object benchmark |
| HOD Benchmark Dataset | Smoking | 10,631 images | 98.222% | 93.391% | Research-use object benchmark |
| HOD Benchmark Dataset | Weapons | 10,631 images | 85.053% | 68.659% | Research-use object benchmark |
| Open Images V7 selected validation subset | Weapons | 460 rows | 81.957% | 87.234% | Object-label reference check |
These rows show how Ray Guardian behaves beyond the benchmark it was built against.
Why the numbers differ
A lower public reference score is not a contradiction. It means the dataset is asking a different question.
The 13-category Ray Guardian benchmark asks whether the system recognizes the safety categories used in Ray. The independent reference datasets ask what happens when outside labels, outside collection methods, and outside category definitions are mapped into that taxonomy.
Scores can move for straightforward reasons:
- Different label definitions. A dataset may label an object while Ray Guardian is judging safety meaning. A wine glass, bottle, knife, or clinical wound is not always unsafe by itself.
- Coarse labels. Some public safety labels combine several concepts into one broad label. Ray Guardian categories are more specific.
- Context gaps. Some labels depend on caption, audio, surrounding scene, or human intent. These tests measure visible content.
- Domain shift. External datasets include different crops, resolutions, generated images, surveillance footage, object closeups, and medical imagery.
- Different task design. Object recognition benchmarks are useful stress tests, but object recognition is not identical to visual safety moderation.
The aligned benchmark shows performance on Ray Guardian's product taxonomy. The reference datasets show how that taxonomy transfers into messier outside data.
How to read the benchmark
Read each number at the level of the labels behind it.
- The 13-category result measures the full Ray Guardian image-level safety taxonomy.
- The video rows measure Ray Guardian on the public video labels available in each dataset.
- The independent reference rows measure transfer to outside label systems, not a replacement for the product taxonomy benchmark.
Ray Guardian is now Ray's unified visual safety layer: one system for images and clips, built to catch sensitive content and keep decisions stable as media moves frame to frame.
Source datasets
The benchmark suite cites dataset sources including HMDB51 via FiftyOne, XD-Violence, UCF-Crime, UCF101 via FiftyOne, HoliSafe-Bench, UnsafeBench, Open Images V7, SurgWound, and HOD Benchmark Dataset.
