Supervisely
About SuperviselyEcosystemContact usSlack
  • 💻Supervisely Developer Portal
  • 🎉Getting Started
    • Installation
    • Basics of authentication
    • Intro to Python SDK
    • Environment variables
    • Supervisely annotation format
      • Project Structure
      • Project Meta: Classes, Tags, Settings
      • Objects
      • Tags
      • Image Annotation
      • Video Annotation
      • Point Clouds Annotation
      • Point Cloud Episode Annotation
      • Volumes Annotation
    • Python SDK tutorials
      • Images
        • Images
        • Image and object tags
        • Spatial labels on images
        • Keypoints (skeletons)
        • Multispectral images
        • Multiview images
        • Advanced: Optimized Import
        • Advanced: Export
      • Videos
        • Videos
        • Video and object tags
        • Spatial labels on videos
      • Point Clouds
        • Point Clouds (LiDAR)
        • Point Cloud Episodes and object tags
        • 3D point cloud object segmentation based on sensor fusion and 2D mask guidance
        • 3D segmentation masks projection on 2D photo context image
      • Volumes
        • Volumes (DICOM)
        • Spatial labels on volumes
      • Common
        • Iterate over a project
        • Iterate over a local project
        • Progress Bar tqdm
        • Cloning projects for development
    • Command Line Interface (CLI)
      • Enterprise CLI Tool
        • Instance administration
        • Workflow automation
      • Supervisely SDK CLI
    • Connect your computer
      • Linux
      • Windows WSL
      • Troubleshooting
  • 🔥App development
    • Basics
      • Create app from any py-script
      • Configuration file
        • config.json
        • Example 1. Headless
        • Example 2. App with GUI
        • v1 - Legacy
          • Example 1. v1 Modal Window
          • Example 2. v1 app with GUI
      • Add private app
      • Add public app
      • App Compatibility
    • Apps with GUI
      • Hello World!
      • App in the Image Labeling Tool
      • App in the Video Labeling Tool
      • In-browser app in the Labeling Tool
    • Custom import app
      • Overview
      • From template - simple
      • From scratch - simple
      • From scratch GUI - advanced
      • Finding directories with specific markers
    • Custom export app
      • Overview
      • From template - simple
      • From scratch - advanced
    • Neural Network integration
      • Overview
      • Serving App
        • Introduction
        • Instance segmentation
        • Object detection
        • Semantic segmentation
        • Pose estimation
        • Point tracking
        • Object tracking
        • Mask tracking
        • Image matting
        • How to customize model inference
        • Example: Custom model inference with probability maps
      • Serving App with GUI
        • Introduction
        • How to use default GUI template
        • Default GUI template customization
        • How to create custom user interface
      • Inference API
      • Training App
        • Overview
        • Tensorboard template
        • Object detection
      • High level scheme
      • Custom inference pipeline
      • Train and predict automation model pipeline
    • Advanced
      • Advanced debugging
      • How to make your own widget
      • Tutorial - App Engine v1
        • Chapter 1 Headless
          • Part 1 — Hello world! [From your Python script to Supervisely APP]
          • Part 2 — Errors handling [Catching all bugs]
          • Part 3 — Site Packages [Customize your app]
          • Part 4 — SDK Preview [Lemons counter app]
          • Part 5 — Integrate custom tracker into Videos Annotator tool [OpenCV Tracker]
        • Chapter 2 Modal Window
          • Part 1 — Modal window [What is it?]
          • Part 2 — States and Widgets [Customize modal window]
        • Chapter 3 UI
          • Part 1 — While True Script [It's all what you need]
          • Part 2 — UI Rendering [Simplest UI Application]
          • Part 3 — APP Handlers [Handle Events and Errors]
          • Part 4 — State and Data [Mutable Fields]
          • Part 5 — Styling your app [Customizing the UI]
        • Chapter 4 Additionals
          • Part 1 — Remote Developing with PyCharm [Docker SSH Server]
      • Custom Configuration
        • Fixing SSL Certificate Errors in Supervisely
        • Fixing 400 HTTP errors when using HTTP instead of HTTPS
      • Autostart
      • Coordinate System
      • MLOps Workflow integration
    • Widgets
      • Input
        • Input
        • InputNumber
        • InputTag
        • BindedInputNumber
        • DatePicker
        • DateTimePicker
        • ColorPicker
        • TimePicker
        • ClassesMapping
        • ClassesColorMapping
      • Controls
        • Button
        • Checkbox
        • RadioGroup
        • Switch
        • Slider
        • TrainValSplits
        • FileStorageUpload
        • Timeline
        • Pagination
      • Text Elements
        • Text
        • TextArea
        • Editor
        • Copy to Clipboard
        • Markdown
        • Tooltip
        • ElementTag
        • ElementTagsList
      • Media
        • Image
        • LabeledImage
        • GridGallery
        • Video
        • VideoPlayer
        • ImagePairSequence
        • Icons
        • ObjectClassView
        • ObjectClassesList
        • ImageSlider
        • Carousel
        • TagMetaView
        • TagMetasList
        • ImageAnnotationPreview
        • ClassesMappingPreview
        • ClassesListPreview
        • TagsListPreview
        • MembersListPreview
      • Selection
        • Select
        • SelectTeam
        • SelectWorkspace
        • SelectProject
        • SelectDataset
        • SelectItem
        • SelectTagMeta
        • SelectAppSession
        • SelectString
        • Transfer
        • DestinationProject
        • TeamFilesSelector
        • FileViewer
        • Dropdown
        • Cascader
        • ClassesListSelector
        • TagsListSelector
        • MembersListSelector
        • TreeSelect
        • SelectCudaDevice
      • Thumbnails
        • ProjectThumbnail
        • DatasetThumbnail
        • VideoThumbnail
        • FolderThumbnail
        • FileThumbnail
      • Status Elements
        • Progress
        • NotificationBox
        • DoneLabel
        • DialogMessage
        • TaskLogs
        • Badge
        • ModelInfo
        • Rate
        • CircleProgress
      • Layouts and Containers
        • Card
        • Container
        • Empty
        • Field
        • Flexbox
        • Grid
        • Menu
        • OneOf
        • Sidebar
        • Stepper
        • RadioTabs
        • Tabs
        • TabsDynamic
        • ReloadableArea
        • Collapse
        • Dialog
        • IFrame
      • Tables
        • Table
        • ClassicTable
        • RadioTable
        • ClassesTable
        • RandomSplitsTable
        • FastTable
      • Charts and Plots
        • LineChart
        • GridChart
        • HeatmapChart
        • ApexChart
        • ConfusionMatrix
        • LinePlot
        • GridPlot
        • ScatterChart
        • TreemapChart
        • PieChart
      • Compare Data
        • MatchDatasets
        • MatchTagMetas
        • MatchObjClasses
        • ClassBalance
        • CompareAnnotations
      • Widgets demos on github
  • 😎Advanced user guide
    • Objects binding
    • Automate with Python SDK & API
      • Start and stop app
      • User management
      • Labeling Jobs
  • 🖥️UI widgets
    • Element UI library
    • Supervisely UI widgets
    • Apexcharts - modern & interactive charts
    • Plotly graphing library
  • 📚API References
    • REST API Reference
    • Python SDK Reference
Powered by GitBook
On this page
  • Project Structure Example
  • Main concepts
  • Cuboid direction vector
  • Key id map file
  • Format of frame_pointcloud_map.json
  • Photo context image annotation file
  • Related apps
  • Example projects

Was this helpful?

Edit on GitHub
  1. Getting Started
  2. Supervisely annotation format

Point Cloud Episode Annotation

PreviousPoint Clouds AnnotationNextVolumes Annotation

Last updated 1 year ago

Was this helpful?

Project Structure Example

<PROJECT_NAME>
├── key_id_map.json (optional)                
├── meta.json     
├── <EPISODE_NAME_1>                       
│ ├── annotation.json           
│ ├── frame_pointcloud_map.json     
│ ├── pointcloud                    
│ │   ├── scene_1.pcd           
│ │   ├── scene_2.pcd   
│ │   └── ...                
│ └── related_images (optional)        
│     ├── scene_1_pcd               
│     │ ├── scene_1_cam0.png       
│     │ ├── scene_1_cam0.png.json  
│     │ ├── scene_1_cam1.png       
│     │ ├── scene_1_cam1.png.json  
│     │ └── ... 
│     ├── scene_2_pcd               
│     │ ├── scene_2_cam0.png       
│     │ ├── scene_2_cam0.png.json  
│     │ ├── scene_2_cam1.png       
│     │ ├── scene_2_cam1.png.json  
│     │ └── ... 
│     └── ...      
├── <EPISODE_NAME_2>                       
│ ├── annotation.json               
│ ├── frame_pointcloud_map.json     
│ ├── pointcloud                    
│ │   ├── scene_1.pcd                 
│ │   └── ...               
│ └── related_images (optional)          
│     ├── scene_1_pcd               
│     │ ├── scene_1_cam0.png       
│     │ ├── scene_1_cam0.png.json    
│     │ └── ... 
│     ├── scene_2_pcd               
│     │ ├── scene_2_cam0.png       
│     │ ├── scene_2_cam0.png.json  
│     │ └── ... 
│     └── ...                    
└── <EPISODE_NAME_...>                       

Main concepts

Point cloud Episodes Project

Point cloud Episodes (PCE) Project consists of one or many sequences of frames. Each sequence is called an episode/dataset in Supervisely.

PCE also includes Sensor fusion feature that supports video camera sensor in the Labeling Tool UI.

Project Meta (meta.json)

Project Meta contains the essential information about the project - Classes and Tags. These are defined project-wide and can be used for labeling in every episode inside the current project.

Episodes/Datasets (<EPISODE_NAME_1>, <EPISODE_NAME_2>, ...)

Episodes are the second level folders inside the project, they host a sequence of frames (point clouds), related photo context (images) and annotations.

Items/Point clouds (pointcloud)

Every .pcd file in a sequence has to be stored inside a pointcloud folder of episodes.

Key
Value

x

The x coordinate of the point.

y

The y coordinate of the point.

z

The z coordinate of the point.

r

The red color channel component. An 8-bit value (0-255).

g

The green color channel component. An 8-bit value (0-255)

b

The blue color channel component. An 8-bit value (0-255)

All the positional coordinates (x, y, z) are in meters. Supervisely supports all PCD encoding: ASCII, binary, binary_compressed.

Items Annotation (annotation.json)

Point cloud Episode Annotation contains the information for the entire episode including labels on all point clouds (frames) in the episode and objects. The mapping between frame numbers and point cloud names is specified in the file frame_pointcloud_map.json which guarantees the order.

An episode contains a list of objects that are used to track labels between frames. The list of objects is defined for the entire episode

Figures represent individual labels on frames. Label contains information about the geometry, frame number and object that it belongs to.

[
    {
    "description": "",
    "key": "e9f0a3ae21be41d08eec166d454562be",
    "tags": [],
    "objects": [
        {
            "key": "6663ca1d20c74bea83bd48c24568989d",
            "classTitle": "car",
            "tags": []
        }],
    "framesCount": 48,
    "frames": [
         {
            "index": 0,
            "figures": [
               {
                "key": "cb8e067dadfc423aa8575a0c4e62de33",
                "objectKey": "6663ca1d20c74bea83bd48c24568989d",
                "geometryType": "cuboid_3d",
                "geometry": {
                    "position": {
                        "x": -10.863547325134277,
                        "y": -93.57706451416016,
                        "z": -4.598618030548096
                    },
                    "rotation": {
                        "x": 0,
                        "y": 0,
                        "z": 3.250733629393711
                    },
                    "dimensions": {
                        "x": 1.978,
                        "y": 4.607,
                        "z": 1.552
                        }
                      }
                    }
            ]
         },
         {
            "index": 1,
            "figures": [               
               {
                "key": "71e0fe52dc4f4f6aaf059ad095f43c1f",
                "objectKey": "6663ca1d20c74bea83bd48c24568989d",
                "labelerLogin": "username",
                "updatedAt": "2021-11-11T17:19:11.448Z",
                "createdAt": "2021-11-11T16:53:03.670Z",
                "geometryType": "cuboid_3d",
                "geometry": {
                    "position": {
                        "x": -11.10418701171875,
                        "y": -91.33098602294922,
                        "z": -4.5446248054504395
                    },
                    "rotation": {
                        "x": 0,
                        "y": 0,
                        "z": 3.24780199600921
                    },
                    "dimensions": {
                        "x": 1.978,
                        "y": 4.607,
                        "z": 1.552
                    }
                }
              }
            ]
         }
    ]
    }
]

Optional fields and loading These fields are optional and are not needed when loading the project. The server can automatically fill in these fields while project is loading.

  • id - unique identifier of the current object

  • classId - unique class identifier of the current object

  • labelerLogin - string - the name of user who created the current figure

  • createdAt - string - date and time of figure creation

  • updatedAt - string - date and time of the last figure update

Fields definitions:

  • description - string - (optional) - this field is used to store the text to assign to the sequence.

  • key - string, unique key for a given sequence (used in key_id_map.json to get the sequence ID)

  • tags - list of strings that will be interpreted as episode tags

  • objects - list of objects that may be present on the episode

  • frames - list of frames of which the episode consists. List contains only frames with an object from the 'objects' field

    • index - integer - number of the current frame

    • figures - list of figures in the current frame.

  • framesCount - integer - total number of frames in the episode

  • geometryType - "cuboid_3d" or other 3D geometry - class shape

Fields definitions for objects field:

  • key - string - unique key for a given object (used in key_id_map.json)

  • classTitle - string - the title of a class. It's used to identify the class shape from the meta.json file

  • tags - list of strings that will be interpreted as object tags (can be empty)

Fields description for figures field:

  • key - string - unique key for a given figure (used in key_id_map.json)

  • objectKey - string - unique key to link figure to object (used in key_id_map.json)

  • geometryType - "cuboid_3d" or other 3D geometry -class shape

  • geometry - geometry of the object

Description for geometry field (cuboid_3d):

  • position 3D vector of box center coordinates:

    • x - forward in the direction of the object

    • y - left

    • z - up

  • dimensions is a 3D vector that scales a cuboid from its local center along x,y,z:

    • x - width

    • y - length

    • z - height

  • rotation is a 3D Vector that rotates a cuboid along an axis in world space:

    • x - pitch

    • y - roll

    • z - yaw (direction)

Cuboid direction vector

Rotation values bound inside [-pi ; pi] When yaw = 0 box direction will be strict +y

Key id map file

The basic idea behind key-id-map is that it maps the unique identifiers of entities from Supervisely to local entities keys. It is needed for such local data manipulations as cloning entities and reassigning relations between them. Examples of entities in key_id_map.json: datasets (videos/episodes), tags, objects, figures.

{
    "tags": {},
    "objects": {
        "198f727d40c749eebcacc4aed299b39a": 20520
    },
    "figures": {
        "65f21690780e43b49863c3cbd07eab3a": 503130811
    },
    "videos": {
        "e9f0a3ae21be41d08eec166d454562be": 42656
    }
}
  • objects - dictionary, where the key is a unique string, generated inside Supervisely environment to set mapping of current object in annotation, and values are unique integer ID related to the current object

  • figures - dictionary, where the key is a unique string, generated inside Supervisely environment to set mapping of object on current frame in annotation, and values are unique integer ID related to the current frame

  • videos - dictionary, where the key is unique string, generated inside Supervisely environment to set mapping of episode (dataset) in annotation, and value is a unique integer ID related to the current sequence

  • tags - dictionary, where the keys are unique strings, generated inside Supervisely environment to set mapping of tag on current frame in annotation, and values are a unique integer ID related to the current tag

  • Value - returned by server integer identifier while uploading object / figure / sequence / tag

Format of frame_pointcloud_map.json

This file stores mapping between point cloud files and annotation frames in the correct order.

{
    "0" : "frame1.pcd",
    "1" : "frame2.pcd",
    "2" : "frame3.pcd" 
}

Keys - frame order number Values - point cloud name (with extension)

Photo context image annotation file

    {
        "name": "host-a005_cam4_1231201437716091006.jpeg",
        "entityId": 2359620,
        "meta": {
            "deviceId": "CAM_BACK_LEFT",
            "timestamp": "2019-01-11T03:23:57.802Z",
            "sensorsData": {
                "extrinsicMatrix": [
                    -0.8448329028461443,
                    -0.5350302199120708,
                    0.00017334762588639086,
                    -0.012363736761232369,
                    -0.0035124448582330757,
                    0.005222293412494302,
                    -0.9999801949951969,
                    -0.16621728572112304,
                    0.5350187183638307,
                    -0.8448167798004226,
                    -0.006291229448121315,
                    -0.3527897896721229
                ],
                "intrinsicMatrix": [
                    882.42699274,
                    0,
                    602.047851885,
                    0,
                    882.42699274,
                    527.99972239,
                    0,
                    0,
                    1
                ]
            }
        }
    }

Fields description:

  • name - string - Name of image file

  • entityId (OPTIONAL) - integer >= 1 ID of the Point Cloud in the system, that photo attached to. Doesn't require while uploading.

  • deviceId - string - Device ID or name.

  • timestamp - (OPTIONAL) - string - Time when the frame occurred in ISO 8601 format

Related apps

Example projects

The PCD file format description can be found

Main idea of key fields and id you can see below in section.

Key - . The unique string. All key values and id's should be unique inside single project and can not be shared between frames\sequences.

sensorsData - Sensors data such as Pinhole camera model parameters. See wiki: and .

intrinsicMatrix - Array of number - 3x3 flatten matrix (dropped last zeros column) of intrinsic parameters in row-major order, also called camera matrix. It's used to denote camera calibration parameters. See .

extrinsicMatrix - Array of number - 4x3 flatten matrix (dropped last zeros column) of extrinsic parameters in row-major order, also called joint rotation-translation matrix. It's used to denote the coordinate system transformations from 3D world coordinates to 3D camera coordinates. See .

app.

app.

- demo sample from dataset with labels.

- demo sample from dataset without labels.

- demo sample from dataset with labels.

- demo sample from dataset without labels.

🎉
here
generated by python3 function uuid.uuid4().hex
Pinhole camera model
OpenCV docs for 3D reconstruction
Intrinsic parameters
Extrinsic_parameters
Import Supervisely point cloud episodes
Export Supervisely point cloud episodes
Demo LYFT 3D dataset annotated
Lyft
Demo LYFT 3D dataset
Lyft
Demo KITTI point cloud episodes annotated
KITTI 3D
Demo KITTI point cloud episodes
KITTI 3D
Key id map file
3D Episodes labeling interface