Merge pull request #377 from roboflow/segmentation

Segmentation

Author: probicheaux

.gitignore

@@ -173,4 +173,10 @@ cython_debug/
 # PyPI configuration file
 .pypirc
 
-.idea/
+.idea/
+
+# model artifacts
+rf-detr*
+output/*
+
+train_test.py

README.md

@@ -1,4 +1,4 @@
-# RF-DETR: SOTA Real-Time Object Detection Model
+# RF-DETR: SOTA Real-Time Detection and Segmentation Model
 
 [![version](https://badge.fury.io/py/rfdetr.svg)](https://badge.fury.io/py/rfdetr)
 [![downloads](https://img.shields.io/pypi/dm/rfdetr)](https://pypistats.org/packages/rfdetr)
@@ -10,24 +10,31 @@
 [![roboflow](https://raw.githubusercontent.com/roboflow-ai/notebooks/main/assets/badges/roboflow-blogpost.svg)](https://blog.roboflow.com/rf-detr)
 [![discord](https://img.shields.io/discord/1159501506232451173?logo=discord&label=discord&labelColor=fff&color=5865f2&link=https%3A%2F%2Fdiscord.gg%2FGbfgXGJ8Bk)](https://discord.gg/GbfgXGJ8Bk)
 
-RF-DETR is a real-time, transformer-based object detection model developed by Roboflow and released under the Apache 2.0 license.
+RF-DETR is a real-time, transformer-based object detection and instance segmentation model architecture developed by Roboflow and released under the Apache 2.0 license.
 
-RF-DETR-N outperforms YOLO11-N by 10 mAP points on the [Microsoft COCO](https://cocodataset.org/#home) benchmark while running faster at inference. On [RF100-VL](https://github.com/roboflow/rf100-vl), RF-DETR achieves state-of-the-art results, with RF-DETR-M beating YOLO11-M by an average of 5 mAP points across aerial datasets including drone, satellite, and radar.
+RF-DETR is the first real-time model to exceed 60 AP on the [Microsoft COCO object detection benchmark](https://cocodataset.org/#home) alongside competitive performance at base sizes. It also achieves state-of-the-art performance on [RF100-VL](https://github.com/roboflow/rf100-vl), an object detection benchmark that measures model domain adaptability to real world problems. RF-DETR is fastest and most accurate for its size when compared current real-time objection models.
+
+On image segmentation, RF-DETR Seg (Preview) is 3x faster and more accurate than the largest YOLO when evaluated on the Microsoft COCO Segmentation benchmark, defining a new real-time state-of-the-art for the industry-standard benchmark in segmentation model evaluation.
 
 [![rf-detr-tutorial-banner](https://github.com/user-attachments/assets/555a45c3-96e8-4d8a-ad29-f23403c8edfd)](https://youtu.be/-OvpdLAElFA)
 
 ## News
 
-- `2025/09/02`: RF-DETR fine-tuning YouTube tutorial released. Learn step-by-step how to fine-tune RF-DETR on your custom dataset.
-- `2025/07/23`: Released three new checkpoints for RF-DETR: Nano, Small, and Medium.
-- `2025/05/16`: Added `optimize_for_inference` method, improving native PyTorch inference speed by up to 2x depending on platform.
-- `2025/04/03`: Introduced early stopping, gradient checkpointing, metric saving, training resume, TensorBoard, and W&B logging.
-- `2025/03/20`: Released RF-DETR real-time object detection model. Code and checkpoints for RF-DETR-Large and RF-DETR-Base are available.
+- `2025/10/02`: We release RF-DETR-Seg (Preview), a preview of our instance segmentation head for RF-DETR.
+- `2025/07/23`: We release three new checkpoints for RF-DETR: Nano, Small, and Medium.
+    - RF-DETR Base is now deprecated. We recommend using RF-DETR Medium which offers subtantially better accuracy at comparable latency.
+- `2025/03/20`: We release RF-DETR real-time object detection model. **Code and checkpoint for RF-DETR-large and RF-DETR-base are available.**
+- `2025/04/03`: We release early stopping, gradient checkpointing, metrics saving, training resume, TensorBoard and W&B logging support.
+- `2025/05/16`: We release an 'optimize_for_inference' method which speeds up native PyTorch by up to 2x, depending on platform.
 
 ## Results
 
 RF-DETR achieves state-of-the-art performance on both the Microsoft COCO and the RF100-VL benchmarks.
 
+The below tables shows how RF-DETR performs when validated on the Microsoft COCO benchmark for object detection and image segmentation.
+
+### Object Detection Benchmarks
+
 ![rf-detr-coco-rf100-vl-9](https://media.roboflow.com/rfdetr/pareto1.png)
 
 | Architecture | COCO AP<sub>50</sub> |  COCO AP<sub>50:95</sub>   |  RF100VL AP<sub>50</sub>   | RF100VL AP<sub>50:95</sub>  |  Latency (ms)   |   Params (M) |   Resolution  |
@@ -51,6 +58,28 @@ RF-DETR achieves state-of-the-art performance on both the Microsoft COCO and the
 
 _We are actively working on RF-DETR Large and X-Large models using the same techniques we used to achieve the strong accuracy that RF-DETR Medium attains. This is why RF-DETR Large and X-Large is not yet reported on our pareto charts and why we haven't benchmarked other models at similar sizes. Check back in the next few weeks for the launch of new RF-DETR Large and X-Large models._
 
+### Instance Segmentation Benchmarks
+
+![rf-detr-coco-rf100-vl-9](https://media.roboflow.com/rfdetr/pareto_segmentation.png)
+
+| Model Name              | Reported Latency | Reported mAP | Measured Latency | Measured mAP |
+|-------------------------|------------------|--------------|------------------|--------------|
+| RF-DETR Seg-Preview@312 |                  |              | 3.3              | 39.4         |
+| YOLO11n-Seg             | 1.8              | 32.0         | 3.6              | 30.0         |
+| YOLOv8n-Seg             |                  | 30.5         | 3.5              | 28.3         |
+| RF-DETR Seg-Preview@384 |                  |              | 4.5              | 42.7         |
+| YOLO11s-Seg             | 2.9              | 37.8         | 4.6              | 35.0         |
+| YOLOv8s-Seg             |                  | 36.8         | 4.2              | 34.0         |
+| RF-DETR Seg-Preview@432 |                  |              | 5.6              | 44.3         |
+| YOLO11m-Seg             | 6.3              | 41.5         | 6.9              | 38.5         |
+| YOLOv8m-Seg             |                  | 40.8         | 7.0              | 37.3         |
+| YOLO11l-Seg             | 7.8              | 42.9         | 8.3              | 39.5         |
+| YOLOv8l-Seg             |                  | 42.6         | 9.7              | 39.0         |
+| YOLO11x-Seg             | 15.8             | 43.8         | 13.7             | 40.1         |
+| YOLOv8x-Seg             |                  | 43.4         | 14.0             | 39.5         |
+
+For more information on measuring end-to-end latency for models, see our open source [Single Artifact Benchmarking tool](https://github.com/roboflow/single_artifact_benchmarking).
+
 ## Installation
 
 To install RF-DETR, install the `rfdetr` package in a [**Python>=3.9**](https://www.python.org/) environment with `pip`:
@@ -102,6 +131,8 @@ annotated_image = sv.BoxAnnotator(color=sv.ColorPalette.ROBOFLOW).annotate(annot
 annotated_image = sv.LabelAnnotator(color=sv.ColorPalette.ROBOFLOW).annotate(annotated_image, detections, labels)
 ```
 
+To use segmentation, use the `rfdetr-seg-preview` model ID. This model will return segmentation masks from a RF-DETR-Seg (Preview) model trained on the Microsoft COCO dataset.
+
 ## Predict
 
 You can also use the .predict method to perform inference during local development. The `.predict()` method accepts various input formats, including file paths, PIL images, NumPy arrays, and torch tensors. Please ensure inputs use RGB channel order. For `torch.Tensor` inputs specifically, they must have a shape of `(3, H, W)` with values normalized to the `[0..1)` range. If you don't plan to modify the image or batch size dynamically at runtime, you can also use `.optimize_for_inference()` to get up to 2x end-to-end speedup, depending on platform.
@@ -140,7 +171,7 @@ sv.plot_image(annotated_image)
 
 You can fine-tune an RF-DETR Nano, Small, Medium, and Base model with a custom dataset using the `rfdetr` Python package.
 
-[Read our training tutorial to get started](https://rfdetr.roboflow.com/learn/train/)
+[Learn how to train an RF-DETR model.](https://rfdetr.roboflow.com/learn/train/)
 
 ## Documentation
 

docs/index.md

@@ -1,27 +1,36 @@
 ---
 hide:
-- toc
 - navigation
 ---
 
-# RF-DETR: SOTA Real-Time Object Detection Model
+# RF-DETR: SOTA Real-Time Detection and Segmentation Model
 
 ## Introduction
 
-RF-DETR is a real-time, transformer-based object detection model architecture developed by Roboflow and released under the Apache 2.0 license.
+RF-DETR is a real-time, transformer-based object detection and instance segmentation model architecture developed by Roboflow and released under the Apache 2.0 license.
 
-RF-DETR is the first real-time model to exceed 60 AP on the [Microsoft COCO benchmark](https://cocodataset.org/#home) alongside competitive performance at base sizes. It also achieves state-of-the-art performance on [RF100-VL](https://github.com/roboflow/rf100-vl), an object detection benchmark that measures model domain adaptability to real world problems. RF-DETR is fastest and most accurate for its size when compared current real-time objection models.
+RF-DETR is the first real-time model to exceed 60 AP on the [Microsoft COCO object detection benchmark](https://cocodataset.org/#home) alongside competitive performance at base sizes. It also achieves state-of-the-art performance on [RF100-VL](https://github.com/roboflow/rf100-vl), an object detection benchmark that measures model domain adaptability to real world problems. RF-DETR is fastest and most accurate for its size when compared current real-time objection models.
+
+On image segmentation, RF-DETR Seg (Preview) is 3x faster and more accurate than the largest YOLO when evaluated on the Microsoft COCO Segmentation benchmark, defining a new real-time state-of-the-art for the industry-standard benchmark in segmentation model evaluation.
 
 RF-DETR is small enough to run on the edge using [Inference](https://github.com/roboflow/inference), making it an ideal model for deployments that need both strong accuracy and real-time performance.
 
-## Results
+## Benchmark Results
+
+### Object Detection
 
 We validated the performance of RF-DETR on both Microsoft COCO and the RF100-VL benchmarks.
 
 [See our full benchmarks.](learn/benchmarks/)
 
 <img src="https://media.roboflow.com/rfdetr/pareto1.png" style="max-height: 50rem" />
 
+### Instance Segmentation
+
+We benchmarked RF-DETR on the Microsoft COCO dataset for segmentation. Our results are below.
+
+![rf-detr-coco-rf100-vl-9](https://media.roboflow.com/rfdetr/pareto_seg.png)
+
 ## 💻 Install
 
 You can install and use `rfdetr` in a

docs/learn/deploy.md

@@ -6,17 +6,33 @@ Deploying to Roboflow allows you to create multi-step computer vision applicatio
 
 To deploy your model to Roboflow, run:
 
-```python
-from rfdetr import RFDETRNano
-
-x = RFDETRNano(pretrain_weights="<path/to/pretrain/weights/dir>")
-x.deploy_to_roboflow(
-  workspace="<your-workspace>",
-  project_id="<your-project-id>",
-  version=1,
-  api_key="<YOUR_API_KEY>"
-)
-```
+=== "Object Detection"
+
+    ```python
+    from rfdetr import RFDETRNano
+
+    x = RFDETRNano(pretrain_weights="<path/to/pretrain/weights/dir>")
+    x.deploy_to_roboflow(
+      workspace="<your-workspace>",
+      project_id="<your-project-id>",
+      version=1,
+      api_key="<YOUR_API_KEY>"
+    )
+    ```
+
+=== "Image Segmentation"
+
+    ```python
+    from rfdetr import RFDETRSegPreview
+
+    x = RFDETRSegPreview(pretrain_weights="<path/to/pretrain/weights/dir>")
+    x.deploy_to_roboflow(
+      workspace="<your-workspace>",
+      project_id="<your-project-id>",
+      version=1,
+      api_key="<YOUR_API_KEY>"
+    )
+    ```
 
 Above, set your Roboflow Workspace ID, the ID of the project to which you want to upload your model, and your Roboflow API key.
 
@@ -25,31 +41,62 @@ Above, set your Roboflow Workspace ID, the ID of the project to which you want t
 
 You can then run your model with Roboflow Inference:
 
-```python
-import os
-import supervision as sv
-from inference import get_model
-from PIL import Image
-from io import BytesIO
-import requests
 
-url = "https://media.roboflow.com/dog.jpeg"
-image = Image.open(BytesIO(requests.get(url).content))
+=== "Object Detection"
 
-model = get_model("rfdetr-base")  # replace with your Roboflow model ID
+    ```python
+    import os
+    import supervision as sv
+    from inference import get_model
+    from PIL import Image
+    from io import BytesIO
+    import requests
 
-predictions = model.infer(image, confidence=0.5)[0]
+    url = "https://media.roboflow.com/dog.jpeg"
+    image = Image.open(BytesIO(requests.get(url).content))
 
-detections = sv.Detections.from_inference(predictions)
+    model = get_model("rfdetr-base")  # replace with your Roboflow model ID
 
-labels = [prediction.class_name for prediction in predictions.predictions]
+    predictions = model.infer(image, confidence=0.5)[0]
 
-annotated_image = image.copy()
-annotated_image = sv.BoxAnnotator(color=sv.ColorPalette.ROBOFLOW).annotate(annotated_image, detections)
-annotated_image = sv.LabelAnnotator(color=sv.ColorPalette.ROBOFLOW).annotate(annotated_image, detections, labels)
+    detections = sv.Detections.from_inference(predictions)
 
-sv.plot_image(annotated_image)
-```
+    labels = [prediction.class_name for prediction in predictions.predictions]
+
+    annotated_image = image.copy()
+    annotated_image = sv.BoxAnnotator(color=sv.ColorPalette.ROBOFLOW).annotate(annotated_image, detections)
+    annotated_image = sv.LabelAnnotator(color=sv.ColorPalette.ROBOFLOW).annotate(annotated_image, detections, labels)
+
+    sv.plot_image(annotated_image)
+    ```
+
+=== "Image Segmentation"
+
+    ```python
+    import os
+    import supervision as sv
+    from inference import get_model
+    from PIL import Image
+    from io import BytesIO
+    import requests
+
+    url = "https://media.roboflow.com/dog.jpeg"
+    image = Image.open(BytesIO(requests.get(url).content))
+
+    model = get_model("rfdetr-seg-preview")  # replace with your Roboflow model ID
+
+    predictions = model.infer(image, confidence=0.5)[0]
+
+    detections = sv.Detections.from_inference(predictions)
+
+    labels = [prediction.class_name for prediction in predictions.predictions]
+
+    annotated_image = image.copy()
+    annotated_image = sv.MaskAnnotator(color=sv.ColorPalette.ROBOFLOW).annotate(annotated_image, detections)
+    annotated_image = sv.LabelAnnotator(color=sv.ColorPalette.ROBOFLOW).annotate(annotated_image, detections, labels)
+
+    sv.plot_image(annotated_image)
+    ```
 
 Above, replace `rfdetr-base` with the your Roboflow model ID. You can find this ID from the "Models" list in your Roboflow dashboard: