Supervisely
About SuperviselyEcosystemContact usSlack
  • 💻Supervisely Developer Portal
  • 🎉Getting Started
    • Installation
    • Basics of authentication
    • Intro to Python SDK
    • Environment variables
    • Supervisely annotation format
      • Project Structure
      • Project Meta: Classes, Tags, Settings
      • Objects
      • Tags
      • Image Annotation
      • Video Annotation
      • Point Clouds Annotation
      • Point Cloud Episode Annotation
      • Volumes Annotation
    • Python SDK tutorials
      • Images
        • Images
        • Image and object tags
        • Spatial labels on images
        • Keypoints (skeletons)
        • Multispectral images
        • Multiview images
        • Advanced: Optimized Import
        • Advanced: Export
      • Videos
        • Videos
        • Video and object tags
        • Spatial labels on videos
      • Point Clouds
        • Point Clouds (LiDAR)
        • Point Cloud Episodes and object tags
        • 3D point cloud object segmentation based on sensor fusion and 2D mask guidance
        • 3D segmentation masks projection on 2D photo context image
      • Volumes
        • Volumes (DICOM)
        • Spatial labels on volumes
      • Common
        • Iterate over a project
        • Iterate over a local project
        • Progress Bar tqdm
        • Cloning projects for development
    • Command Line Interface (CLI)
      • Enterprise CLI Tool
        • Instance administration
        • Workflow automation
      • Supervisely SDK CLI
    • Connect your computer
      • Linux
      • Windows WSL
      • Troubleshooting
  • 🔥App development
    • Basics
      • Create app from any py-script
      • Configuration file
        • config.json
        • Example 1. Headless
        • Example 2. App with GUI
        • v1 - Legacy
          • Example 1. v1 Modal Window
          • Example 2. v1 app with GUI
      • Add private app
      • Add public app
      • App Compatibility
    • Apps with GUI
      • Hello World!
      • App in the Image Labeling Tool
      • App in the Video Labeling Tool
      • In-browser app in the Labeling Tool
    • Custom import app
      • Overview
      • From template - simple
      • From scratch - simple
      • From scratch GUI - advanced
      • Finding directories with specific markers
    • Custom export app
      • Overview
      • From template - simple
      • From scratch - advanced
    • Neural Network integration
      • Overview
      • Serving App
        • Introduction
        • Instance segmentation
        • Object detection
        • Semantic segmentation
        • Pose estimation
        • Point tracking
        • Object tracking
        • Mask tracking
        • Image matting
        • How to customize model inference
        • Example: Custom model inference with probability maps
      • Serving App with GUI
        • Introduction
        • How to use default GUI template
        • Default GUI template customization
        • How to create custom user interface
      • Inference API
      • Training App
        • Overview
        • Tensorboard template
        • Object detection
      • High level scheme
      • Custom inference pipeline
      • Train and predict automation model pipeline
    • Advanced
      • Advanced debugging
      • How to make your own widget
      • Tutorial - App Engine v1
        • Chapter 1 Headless
          • Part 1 — Hello world! [From your Python script to Supervisely APP]
          • Part 2 — Errors handling [Catching all bugs]
          • Part 3 — Site Packages [Customize your app]
          • Part 4 — SDK Preview [Lemons counter app]
          • Part 5 — Integrate custom tracker into Videos Annotator tool [OpenCV Tracker]
        • Chapter 2 Modal Window
          • Part 1 — Modal window [What is it?]
          • Part 2 — States and Widgets [Customize modal window]
        • Chapter 3 UI
          • Part 1 — While True Script [It's all what you need]
          • Part 2 — UI Rendering [Simplest UI Application]
          • Part 3 — APP Handlers [Handle Events and Errors]
          • Part 4 — State and Data [Mutable Fields]
          • Part 5 — Styling your app [Customizing the UI]
        • Chapter 4 Additionals
          • Part 1 — Remote Developing with PyCharm [Docker SSH Server]
      • Custom Configuration
        • Fixing SSL Certificate Errors in Supervisely
        • Fixing 400 HTTP errors when using HTTP instead of HTTPS
      • Autostart
      • Coordinate System
      • MLOps Workflow integration
    • Widgets
      • Input
        • Input
        • InputNumber
        • InputTag
        • BindedInputNumber
        • DatePicker
        • DateTimePicker
        • ColorPicker
        • TimePicker
        • ClassesMapping
        • ClassesColorMapping
      • Controls
        • Button
        • Checkbox
        • RadioGroup
        • Switch
        • Slider
        • TrainValSplits
        • FileStorageUpload
        • Timeline
        • Pagination
      • Text Elements
        • Text
        • TextArea
        • Editor
        • Copy to Clipboard
        • Markdown
        • Tooltip
        • ElementTag
        • ElementTagsList
      • Media
        • Image
        • LabeledImage
        • GridGallery
        • Video
        • VideoPlayer
        • ImagePairSequence
        • Icons
        • ObjectClassView
        • ObjectClassesList
        • ImageSlider
        • Carousel
        • TagMetaView
        • TagMetasList
        • ImageAnnotationPreview
        • ClassesMappingPreview
        • ClassesListPreview
        • TagsListPreview
        • MembersListPreview
      • Selection
        • Select
        • SelectTeam
        • SelectWorkspace
        • SelectProject
        • SelectDataset
        • SelectItem
        • SelectTagMeta
        • SelectAppSession
        • SelectString
        • Transfer
        • DestinationProject
        • TeamFilesSelector
        • FileViewer
        • Dropdown
        • Cascader
        • ClassesListSelector
        • TagsListSelector
        • MembersListSelector
        • TreeSelect
        • SelectCudaDevice
      • Thumbnails
        • ProjectThumbnail
        • DatasetThumbnail
        • VideoThumbnail
        • FolderThumbnail
        • FileThumbnail
      • Status Elements
        • Progress
        • NotificationBox
        • DoneLabel
        • DialogMessage
        • TaskLogs
        • Badge
        • ModelInfo
        • Rate
        • CircleProgress
      • Layouts and Containers
        • Card
        • Container
        • Empty
        • Field
        • Flexbox
        • Grid
        • Menu
        • OneOf
        • Sidebar
        • Stepper
        • RadioTabs
        • Tabs
        • TabsDynamic
        • ReloadableArea
        • Collapse
        • Dialog
        • IFrame
      • Tables
        • Table
        • ClassicTable
        • RadioTable
        • ClassesTable
        • RandomSplitsTable
        • FastTable
      • Charts and Plots
        • LineChart
        • GridChart
        • HeatmapChart
        • ApexChart
        • ConfusionMatrix
        • LinePlot
        • GridPlot
        • ScatterChart
        • TreemapChart
        • PieChart
      • Compare Data
        • MatchDatasets
        • MatchTagMetas
        • MatchObjClasses
        • ClassBalance
        • CompareAnnotations
      • Widgets demos on github
  • 😎Advanced user guide
    • Objects binding
    • Automate with Python SDK & API
      • Start and stop app
      • User management
      • Labeling Jobs
  • 🖥️UI widgets
    • Element UI library
    • Supervisely UI widgets
    • Apexcharts - modern & interactive charts
    • Plotly graphing library
  • 📚API References
    • REST API Reference
    • Python SDK Reference
Powered by GitBook
On this page
  • Supported Shapes
  • Coordinate System
  • General Fields
  • Point
  • Rectangle
  • Polygon (without holes)
  • Polygon (with holes)
  • Polyline
  • Bitmap
  • Keypoint structure
  • Cuboids (2D annotation)
  • Mask3D (3D annotation)

Was this helpful?

Edit on GitHub
  1. Getting Started
  2. Supervisely annotation format

Objects

PreviousProject Meta: Classes, Tags, SettingsNextTags

Last updated 1 year ago

Was this helpful?

Supported Shapes

Supervisely Annotation Format supports the following figures:

Coordinate System

For two-dimensional mediums (images and videos) we use the following coordinate system (it's similar to a two-dimensional NumPy coordinate system):

All numerical values are provided in pixels.

General Fields

When generating JSON annotation files, we assign each figure a mix of general fields and fields unique for each geometric shape. Some general fields are optional: the system generates them automatically when the data is uploaded/first created. This means that these fields can be omitted during manual annotation.

Optional fields:

"id": 503051990,
"classId": 1693352,
"labelerLogin": "alexxx",
"createdAt": "2020-08-22T09:32:48.010Z",
"updatedAt": "2020-08-22T09:33:08.926Z".

Fields definitions:

  • id - unique identifier of the current object

  • classId - unique class identifier of the current object

  • labelerLogin - string - the name of user who created the current figure

  • createdAt - string - date and time of figure creation

  • updatedAt - string - date and time of the last figure update

Point

Example:

JSON format for this shape:

{
    "id": 503051990,
    "classId": 1693352,
    "labelerLogin": "alexxx",
    "createdAt": "2020-08-22T09:32:48.010Z",
    "updatedAt": "2020-08-22T09:33:08.926Z",
    "description": "",
    "geometryType": "point",
    "tags": [],
    "classTitle": "point",
    "points": {
        "exterior": [
            [1334, 907]
        ],
        "interior": []
    }
}

Fields definitions:

  • description - string - text description (optional)

  • geometryType: "point" - class shape

  • tags - list of tags assigned to the current object

  • classTitle - string - the title of the current class. It's used to identify the corresponding class shape from the meta.json file

  • points - object with two fields:

    • exterior - list of 2 values for coordinates (x and y in that order) for every figure

    • interior - always an empty field for this type of figure

Rectangle

Example:

JSON format for this figure:

{
  "id": 283051572,
  "classId": 1692857,
  "labelerLogin": "max",
  "createdAt": "2020-08-22T09:32:48.010Z",
  "updatedAt": "2020-08-22T09:33:08.926Z",
  "description": "",
  "geometryType": "rectangle",
  "tags": [],
  "classTitle": "person_bbox",
  "points": {
    "exterior": [
      [533, 63],
      [800, 830]
    ],
    "interior": []
  }
}

Fields definitions:

  • description - string - text description (optional)

  • geometryType: "rectangle" - class shape

  • tags - list of tags assigned to the current object

  • classTitle - string - the title of the current class. It's used to identify the corresponding class shape from the meta.json file

  • points - object with two fields:

  • exterior - list of two lists, each containing two coordinates (x and y in that order), with the following structure: [[left, top], [right, bottom]]

  • interior - always an empty list for this type of figure

Polygon (without holes)

Example:

{
    "id": 503004154,
    "classId": 1693021,
    "labelerLogin": "alexxx",
    "createdAt": "2020-08-21T15:15:28.092Z",
    "updatedAt": "2020-08-21T15:15:37.687Z",
    "description": "",
    "geometryType": "polygon",
    "tags": [],
    "classTitle": "triangle",
    "points": {
        "exterior": [
            [730, 2104],
            [2479 402],
            [3746, 1646]
        ],
        "interior": []
    }
}

Fields definitions:

  • description - string - text description (optional)

  • geometryType: "polygon" - class shape

  • tags - list of tags assigned to the current object

  • classTitle - string - the title of the current class. It's used to identify the corresponding class shape from the meta.json file

  • points - object with two fields:

  • exterior - list of points [point1, point2, point3, etc ...] where each point is a list of two numbers (coordinates) [col, row]

  • interior - list of elements with the same structure as the "exterior" field. In other words, this is the list of polygons that define object holes. For polygons without holes in them, this field is empty

Polygon (with holes)

Example:

{
    "id": 503004154,
    "classId": 1693021,
    "labelerLogin": "alexxx",
    "createdAt": "2020-08-21T15:15:28.092Z",
    "updatedAt": "2020-08-21T16:06:11.461Z",
    "description": "",
    "geometryType": "polygon",
    "tags": [],
    "classTitle": "triangle_hole",
    "points": {
        "exterior": [
            [730, 2104],
            [2479, 402],
            [3746, 1646]
        ],
        "interior": [
            [
                [1907, 1255],
                [2468, 875],
                [2679, 1577]
            ]
        ]
    }
}

Fields definitions:

  • description - string - text description (optional)

  • geometryType: "polygon" - class shape

  • tags - list of tags assigned to the current object

  • classTitle - string - the title of the current class. It's used to identify the corresponding class shape from the meta.json file

  • points - object with two fields:

  • exterior - list of points [point1, point2, point3, etc ...] where each point is a list of two numbers (coordinates) [col, row]

  • interior - list of elements with the same structure as the "exterior" field. In other words, this is the list of polygons that define object holes.

Polyline

Example:

{
    "id": 503049791,
    "classId": 1693340,
    "labelerLogin": "alexxx",
    "createdAt": "2020-08-22T08:39:29.386Z",
    "updatedAt": "2020-08-22T08:39:34.802Z",
    "description": "",
    "geometryType": "line",
    "tags": [],
    "classTitle": "line",
    "points": {
        "exterior": [
            [211, 2266],
            [1208, 1310],
            [369, 981]
        ],
        "interior": []
    }
}

Fields definitions:

  • description - string - text description (optional)

  • geometryType: "line" - class shape

  • tags - list of tags assigned to the current object

  • classTitle - string - the title of the current class. It's used to identify the corresponding class shape from the meta.json file

  • points - object with two fields:

  • exterior - list of points [point1, point2, point3, etc ...] where each point is a list of two numbers (coordinates) [col, row]

  • interior - always an empty list for this type of figure

Bitmap

Bitmap is a figure that is described by a point of "origin"(upper left corner), which defines the location of the bitmap within the image and a "data" - Boolean matrix encoded into a string, which defines each pixel of the bitmap.

Example:

{
    "id": 497489556,
    "classId": 1661459,
    "labelerLogin": "alexxx",
    "createdAt": "2020-07-24T07:30:39.202Z",
    "updatedAt": "2020-07-24T07:41:12.753Z",
    "description": "",
    "geometryType": "bitmap",
    "tags": [],
    "classTitle": "person",
    "bitmap": {
        "data": "eJwB ... kUnW",
        "origin": [535, 66]
    }
}

Fields description:

  • description - string - text description (optional)

  • geometryType: "bitmap" - class shape

  • tags - list of tags assigned to the current object

  • classTitle - string - the title of the current class. It's used to identify the corresponding class shape from the meta.json file

  • bitmap - object with two fields:

    • origin - points (x and y coordinates) of the top left corner of the bitmap, i.e. the position of the bitmap within the image

    • data - string - encoded representation of a string

A few words about bitmap → data. You can use these two python methods to convert a base64 encoded string to NumPy and vice versa.

def base64_2_mask(s):
    z = zlib.decompress(base64.b64decode(s))
    n = np.fromstring(z, np.uint8)
    mask = cv2.imdecode(n, cv2.IMREAD_UNCHANGED)[:, :, 3].astype(bool)
    return mask

def mask_2_base64(mask):
    img_pil = Image.fromarray(np.array(mask, dtype=np.uint8))
    img_pil.putpalette([0,0,0,255,255,255])
    bytes_io = io.BytesIO()
    img_pil.save(bytes_io, format='PNG', transparency=0, optimize=0)
    bytes = bytes_io.getvalue()
    return base64.b64encode(zlib.compress(bytes)).decode('utf-8')

Example:

import numpy as np
import cv2, zlib, base64, io
from PIL import Image

def base64_2_mask(s):
    z = zlib.decompress(base64.b64decode(s))
    n = np.fromstring(z, np.uint8)
    mask = cv2.imdecode(n, cv2.IMREAD_UNCHANGED)[:, :, 3].astype(bool)
    return mask

def mask_2_base64(mask):
    img_pil = Image.fromarray(np.array(mask, dtype=np.uint8))
    img_pil.putpalette([0,0,0,255,255,255])
    bytes_io = io.BytesIO()
    img_pil.save(bytes_io, format='PNG', transparency=0, optimize=0)
    bytes = bytes_io.getvalue()
    return base64.b64encode(zlib.compress(bytes)).decode('utf-8')

example_np_bool = np.ones((3, 3), dtype=bool)
example_np_bool[1][1] = False
example_np_bool[1][2] = False
print(example_np_bool)
encoded_string = mask_2_base64(example_np_bool)
print(encoded_string)
print(base64_2_mask(encoded_string))

Program output after executing the code:

[[ True  True  True]
 [ True False False]
 [ True  True  True]]

'eJzrDPBz5+WS4mJgYOD19HAJAtLMIMwIInOeqf8BUmwBPiGuQPr///9Lb86/C2QxlgT5BTM4PLuRBuTwebo4hlTMSa44cOHAB6DqY0yORgq8YkAZBk9XP5d1TglNANAFGzA='

[[ True  True  True]
 [ True False False]
 [ True  True  True]]

Keypoint structure

Keypoint structures consist of vertices (also called nodes or points) which are connected by edges (also called links or lines).

Example:

{
    "id": 503055304,
    "classId": 1693357,
    "description": "",
    "geometryType": "graph",
    "labelerLogin": "alexxx",
    "createdAt": "2020-08-22T10:50:28.336Z",
    "updatedAt": "2020-08-22T10:53:57.760Z",
    "tags": [],
    "classTitle": "graph",
    "nodes": {
        "8e20c830-ee86-450f-9d21-833eec53e3c5": {
            "loc": [1017, 1556]
        },
        "bf89e248-7b3b-4732-888a-99d3369fbb2f": {
            "loc": [1024, 394]
        },
        "66502c5b-8d98-492c-bb48-8ce7c4487038": {
            "loc": [1026, 738]
        },
        "56517c2a-6053-442a-9af2-bd6f29bae987": {
            "loc": [668, 574]
        },
        "7a40d5f7-bcc8-4e2f-bf3b-3e52d39c4206": {
            "loc": [1388, 549]
        }
    }
}

Fields definitions:

  • description - string - text description (optional)

  • geometryType: "graph" - class shape

  • tags - list of tags assigned to the current object

  • classTitle - string - the title of the current class. It's used to identify the corresponding class shape from the meta.json file

  • nodes - is a dictionary, where keys denote the names of the graph vertices and values in a dictionary, and where values denote location of a node on image

    • loc - list of single points (x and y coordinates) of vertices

Cuboids (2D annotation)

Example:

{
  "description": "",
  "tags": [],
  "classTitle": "Cuboid",
  "faces": [
    [0, 1, 2, 3],
    [0, 4, 5, 1],
    [1, 5, 6, 2]
  ],
  "points": [
    [277, 273],
    [840, 273],
    [840, 690],
    [277, 690],
    [688, 168],
    [1200, 168],
    [1200, 522]
  ]
}

Fields definitions:

  • description - string - text description (optional)

  • geometryType: "graph" - class shape

  • tags - list of tags assigned to the current object

  • classTitle - string - the title of the current class. It's used to identify the corresponding class shape from the meta.json file

  • points - an array of points that form the cuboid. There are always 7 points in a cuboid. Each Point is presented as an array of X and Y coordinates, i.e. [277, 690] means X is 277 and Y is 690, calculating from the top left corner of the image.

  • faces - an array of faces that indicates how points from the points array are connected. There are always 3 faces in a cuboid. In the example above, you can see that face number 3 that consists of points 1, 2, 5, 6 with coordinates [840, 273], [840, 690], [1200, 168], [1200, 522].

Mask3D (3D annotation)

Mask3D is a figure that is described by a 3D array corresponding to the dimensionality of the volume. It is used as an annotation type for volume projects. Geometry data stores in NRRD files and defines each pixel of the Mask3D. In the previous version whole data could be stored in JSON annotations as a base64 encoded string.

💡 It's strictly recommended to store whole data into NRRD files.

Below is an example of what the object looks like in the annotation file:

{
    "key": "daff638a423a4bcfa34eb12e42243a87",
    "objectKey": "6c1587f381bf419e9d5c2ebd5967e28f",
    "geometryType": "mask_3d",
    "geometry": {
        "mask_3d": {
            "data": "H4sIAGW9OmUC ... CYAE1Nj5QMACwC"
        },
        "shape": "mask_3d",
        "geometryType": "mask_3d"
    },    
    "labelerLogin": "username",
    "updatedAt": "2021-11-13T08:05:28.771Z",
    "createdAt": "2021-11-13T08:05:28.771Z"
}

Fields definitions:

  • key -

  • objectKey -

  • geometryType: "mask_3d" - class shape

  • geometry - describes geometry of the object, consist of:

    • mask_3d - object with one field:

      • data- string - encoded representation of a string.

    • shape: "mask_3d" - geometry name

    • geometryType": "mask_3d" geometry type

If the geometry data is stored in NRRD files, mask_3d → data will store an empty array represented as base64 encoded string.

Optional fields id, classId, labelerLogin, createdAt, updatedAt are described

Optional fields id, classId, labelerLogin, createdAt, updatedAt are described

Optional fields id, classId, labelerLogin, createdAt, updatedAt are described

Optional fields id, classId, labelerLogin, createdAt, updatedAt are described

Optional fields id, classId, labelerLogin, createdAt, updatedAt are described

Optional fields id, classId, labelerLogin, createdAt, updatedAt are described

Optional fields id, classId, labelerLogin, createdAt, updatedAt are described

Optional fields id, classId, labelerLogin, createdAt, updatedAt are described

👉 To learn how to create Mask3D from NRRD files using our SDK you can read .

🎉
this article
above
above
above
above
above
above
above
above
point
rectangle
polygon
line / polyline
bitmap
keypoint structures
cuboid
mask_3d
coordinate system
point example
rectangle example
polygon example
polygon example
polyline example
bitmap example
key point structure example
cuboid 2d example
mask3d example