Supervisely
About SuperviselyEcosystemContact usSlack
  • 💻Supervisely Developer Portal
  • 🎉Getting Started
    • Installation
    • Basics of authentication
    • Intro to Python SDK
    • Environment variables
    • Supervisely annotation format
      • Project Structure
      • Project Meta: Classes, Tags, Settings
      • Objects
      • Tags
      • Image Annotation
      • Video Annotation
      • Point Clouds Annotation
      • Point Cloud Episode Annotation
      • Volumes Annotation
    • Python SDK tutorials
      • Images
        • Images
        • Image and object tags
        • Spatial labels on images
        • Keypoints (skeletons)
        • Multispectral images
        • Multiview images
        • Advanced: Optimized Import
        • Advanced: Export
      • Videos
        • Videos
        • Video and object tags
        • Spatial labels on videos
      • Point Clouds
        • Point Clouds (LiDAR)
        • Point Cloud Episodes and object tags
        • 3D point cloud object segmentation based on sensor fusion and 2D mask guidance
        • 3D segmentation masks projection on 2D photo context image
      • Volumes
        • Volumes (DICOM)
        • Spatial labels on volumes
      • Common
        • Iterate over a project
        • Iterate over a local project
        • Progress Bar tqdm
        • Cloning projects for development
    • Command Line Interface (CLI)
      • Enterprise CLI Tool
        • Instance administration
        • Workflow automation
      • Supervisely SDK CLI
    • Connect your computer
      • Linux
      • Windows WSL
      • Troubleshooting
  • 🔥App development
    • Basics
      • Create app from any py-script
      • Configuration file
        • config.json
        • Example 1. Headless
        • Example 2. App with GUI
        • v1 - Legacy
          • Example 1. v1 Modal Window
          • Example 2. v1 app with GUI
      • Add private app
      • Add public app
      • App Compatibility
    • Apps with GUI
      • Hello World!
      • App in the Image Labeling Tool
      • App in the Video Labeling Tool
      • In-browser app in the Labeling Tool
    • Custom import app
      • Overview
      • From template - simple
      • From scratch - simple
      • From scratch GUI - advanced
      • Finding directories with specific markers
    • Custom export app
      • Overview
      • From template - simple
      • From scratch - advanced
    • Neural Network integration
      • Overview
      • Serving App
        • Introduction
        • Instance segmentation
        • Object detection
        • Semantic segmentation
        • Pose estimation
        • Point tracking
        • Object tracking
        • Mask tracking
        • Image matting
        • How to customize model inference
        • Example: Custom model inference with probability maps
      • Serving App with GUI
        • Introduction
        • How to use default GUI template
        • Default GUI template customization
        • How to create custom user interface
      • Inference API
      • Training App
        • Overview
        • Tensorboard template
        • Object detection
      • High level scheme
      • Custom inference pipeline
      • Train and predict automation model pipeline
    • Advanced
      • Advanced debugging
      • How to make your own widget
      • Tutorial - App Engine v1
        • Chapter 1 Headless
          • Part 1 — Hello world! [From your Python script to Supervisely APP]
          • Part 2 — Errors handling [Catching all bugs]
          • Part 3 — Site Packages [Customize your app]
          • Part 4 — SDK Preview [Lemons counter app]
          • Part 5 — Integrate custom tracker into Videos Annotator tool [OpenCV Tracker]
        • Chapter 2 Modal Window
          • Part 1 — Modal window [What is it?]
          • Part 2 — States and Widgets [Customize modal window]
        • Chapter 3 UI
          • Part 1 — While True Script [It's all what you need]
          • Part 2 — UI Rendering [Simplest UI Application]
          • Part 3 — APP Handlers [Handle Events and Errors]
          • Part 4 — State and Data [Mutable Fields]
          • Part 5 — Styling your app [Customizing the UI]
        • Chapter 4 Additionals
          • Part 1 — Remote Developing with PyCharm [Docker SSH Server]
      • Custom Configuration
        • Fixing SSL Certificate Errors in Supervisely
        • Fixing 400 HTTP errors when using HTTP instead of HTTPS
      • Autostart
      • Coordinate System
      • MLOps Workflow integration
    • Widgets
      • Input
        • Input
        • InputNumber
        • InputTag
        • BindedInputNumber
        • DatePicker
        • DateTimePicker
        • ColorPicker
        • TimePicker
        • ClassesMapping
        • ClassesColorMapping
      • Controls
        • Button
        • Checkbox
        • RadioGroup
        • Switch
        • Slider
        • TrainValSplits
        • FileStorageUpload
        • Timeline
        • Pagination
      • Text Elements
        • Text
        • TextArea
        • Editor
        • Copy to Clipboard
        • Markdown
        • Tooltip
        • ElementTag
        • ElementTagsList
      • Media
        • Image
        • LabeledImage
        • GridGallery
        • Video
        • VideoPlayer
        • ImagePairSequence
        • Icons
        • ObjectClassView
        • ObjectClassesList
        • ImageSlider
        • Carousel
        • TagMetaView
        • TagMetasList
        • ImageAnnotationPreview
        • ClassesMappingPreview
        • ClassesListPreview
        • TagsListPreview
        • MembersListPreview
      • Selection
        • Select
        • SelectTeam
        • SelectWorkspace
        • SelectProject
        • SelectDataset
        • SelectItem
        • SelectTagMeta
        • SelectAppSession
        • SelectString
        • Transfer
        • DestinationProject
        • TeamFilesSelector
        • FileViewer
        • Dropdown
        • Cascader
        • ClassesListSelector
        • TagsListSelector
        • MembersListSelector
        • TreeSelect
        • SelectCudaDevice
      • Thumbnails
        • ProjectThumbnail
        • DatasetThumbnail
        • VideoThumbnail
        • FolderThumbnail
        • FileThumbnail
      • Status Elements
        • Progress
        • NotificationBox
        • DoneLabel
        • DialogMessage
        • TaskLogs
        • Badge
        • ModelInfo
        • Rate
        • CircleProgress
      • Layouts and Containers
        • Card
        • Container
        • Empty
        • Field
        • Flexbox
        • Grid
        • Menu
        • OneOf
        • Sidebar
        • Stepper
        • RadioTabs
        • Tabs
        • TabsDynamic
        • ReloadableArea
        • Collapse
        • Dialog
        • IFrame
      • Tables
        • Table
        • ClassicTable
        • RadioTable
        • ClassesTable
        • RandomSplitsTable
        • FastTable
      • Charts and Plots
        • LineChart
        • GridChart
        • HeatmapChart
        • ApexChart
        • ConfusionMatrix
        • LinePlot
        • GridPlot
        • ScatterChart
        • TreemapChart
        • PieChart
      • Compare Data
        • MatchDatasets
        • MatchTagMetas
        • MatchObjClasses
        • ClassBalance
        • CompareAnnotations
      • Widgets demos on github
  • 😎Advanced user guide
    • Objects binding
    • Automate with Python SDK & API
      • Start and stop app
      • User management
      • Labeling Jobs
  • 🖥️UI widgets
    • Element UI library
    • Supervisely UI widgets
    • Apexcharts - modern & interactive charts
    • Plotly graphing library
  • 📚API References
    • REST API Reference
    • Python SDK Reference
Powered by GitBook
On this page
  • Project Structure System
  • Downloaded Project Structure
  • Project Structure Eample
  • Extended Project Structure
  • Understanding Blob Files and Offsets for Optimized Project Handling

Was this helpful?

Edit on GitHub
  1. Getting Started
  2. Supervisely annotation format

Project Structure

PreviousSupervisely annotation formatNextProject Meta: Classes, Tags, Settings

Last updated 1 month ago

Was this helpful?

In Supervisely, all data and annotations are stored inside individual projects which consist of datasets containing files and Project Meta - a collection of classes and tags.

When downloaded, each project is converted into a folder structure that includes a meta.json file containing Project Meta, and dataset folders with individual annotation files (and optionally the original data files). This organization enables seamless data transfer between Supervisely and local storage using the Supervisely Format import plugin when needed.

This structure remains the same for every type of project in Supervisely.

Project Structure System

Project Folder

On the top level we have Project folders, these are the elements visible on the main Supervisely dashboard. Inside them, they can contain only Datasets and Project Meta information, all other data has to be stored a level below in a Dataset. All datasets within a project have to contain content of the same category.

Project Meta

Project Meta contains essential information about the project, including Classes and Tags, which are defined project-wide and can be used for labeling in any dataset within the current project. It also includes the Project Type and Settings, which configure the labeling interface.

Datasets

Datasets are the second level folders inside the project, they host the individual data files and their annotations.

Items

Every data file in the project has to be stored inside a dataset. Each file as its own set of annotations.

Downloaded Project Structure

All projects downloaded from Supervisely maintain the same basic structure, with the contents varying based on which download option you chose.

Download Archive

When you select one of the download option, the system automatically creates an archive with the following name structure: project_name.tar

Downloaded Project

All projects downloaded from Supervisely have the following structure:

📂 Root folder for the project named project name:

  • 📄 meta.json file

  • 📂 Dataset folders, each named dataset_name, which contains:

    • 📂 ann folder, contains annotation files, each named source_media_file_name.json for the corresponding file

    • 📂 img (video or pointcloud) folder, contains source media

    • 📂 img_info folder, contains JSON files with representation of ImageInfo downloaded from instance

    • 📂 meta optional folder, contains corresponding JSON files with metadata for images

Project Structure Eample

The following structure is an example of a project with 2 datasets, each containing 2 images with annotations, and also meta directory with metadata for each image.

📦 project-name
 ┣ 📂 dataset-name-001
 ┃ ┣ 📂 ann
 ┃ ┃ ┣ 📄 pexels-photo-101063.png.json
 ┃ ┃ ┗ 📄 pexels-photo-103127.png.json
 ┃ ┣ 📂 img
 ┃ ┃ ┣ 🏞️ pexels-photo-101063.png
 ┃ ┃ ┗ 🏞️ pexels-photo-103127.png
 ┃ ┣ 📂 meta
 ┃ ┃ ┣ 📄 pexels-photo-101063.png.json
 ┃ ┃ ┗ 📄 pexels-photo-103127.png.json
 ┃ ┣ 📂 img_info
 ┃ ┃ ┣ 📄 pexels-photo-101063.png.json
 ┃ ┃ ┗ 📄 pexels-photo-103127.png.json
 ┣ 📂 dataset-name-002
 ┃ ┣ 📂 ann
 ┃ ┃ ┣ 📄 pexels-photo-100583.png.json
 ┃ ┃ ┗ 📄 pexels-photo-106118.png.json
 ┃ ┣ 📂 img
 ┃ ┃ ┣ 🏞️ pexels-photo-100583.png
 ┃ ┃ ┗ 🏞️ pexels-photo-106118.png
 ┃ ┗ 📂 meta
 ┃ ┃ ┣ 📄 pexels-photo-100583.png.json
 ┃ ┃ ┗ 📄 pexels-photo-106118.png.json
 ┃ ┣ 📂 img_info
 ┃ ┃ ┣ 📄 pexels-photo-100583.png.json
 ┃ ┃ ┗ 📄 pexels-photo-106118.png.json
 ┗ 📄 meta.json

Extended Project Structure

A project directory may contain the following folders or files:

  • 📂 blob optional folder, contains blob files that are used for optimized uploads of projects. These blob files are TAR archives with hundreds of thousands of small images.

  • 📄 key_id_map.json - optional file, created when annotating inside the Supervisely interface. Establishes correspondence between unique identifiers (keys and IDs) of items, objects, and frames where objects are located. The project file system stores these identifiers and keys on disk, which is necessary for navigation and for using the high-level API and applications.

    A dataset directory may contain the following folders or files:

    • 📄 blob_1_offsets.pkl optional pickle files, contain batches (lists) of BlobImageInfo objects, which represent file names and their offsets inside blob files. These files are used to add images to the project dataset based on their offsets.

Understanding Blob Files and Offsets for Optimized Project Handling

Supervisely provides a powerful optimization for projects containing a large number of small image files through its blob file system. Instead of handling thousands of individual files (which can lead to significant overhead in network transfers and filesystem operations), blob files consolidate many images into a single large binary file. This approach dramatically improves upload and download speeds, especially when dealing with datasets containing tens or hundreds of thousands of images.

Complementing the blob files are offset files with the suffix _offsets.pkl, which store metadata about each image's location within the blob. These files contain BlobImageInfo objects that define the byte range representing each image in the binary.

📂 project-name
 ┣ 📂 blob
 ┃  ┗ 📦 small_images.tar
 ┣ 📂 dataset-name-001
 ┃  ┣ 📄 small_images_offsets.pkl
 ┃  ┣ 📂 ann
 ┃  ┃  ┣ 📄 pexels-photo-101063.png.json
 ┃  ┃  ┣ 📄 small-image-0000001.png.json
 ┃  ┃  ┣ ...
 ┃  ┃  ┗ 📄 small-image-0999999.png.json
 ┃  ┗ 📂 img
 ┃     ┗ 🏞️ pexels-photo-101063.png
 ┗ 📄 meta.json

Related:

📄 obj_class_to_machine_color.json - optional file for image annotation projects. Mapping between machine colors and classes in machine mask. Could be generated by applications such as

📂 masks_human optional folder for image annotation projects, contains .png files with RGB semantic segmentation masks where every pixel has the color of the corresponding class. Could be generated by applications such as

📂 masks_machine optional folder for image annotation projects, contains .png files with semantic segmentation masks (machine annotations). This files should have the same name as the original images (may have a different extension). Could be generated by applications such as

📂 masks_instances optional folder contains BW instance segmentation masks for every object on the image. Could be generated by applications such as

To learn more about the offsets file format and how to prepare it, refer to this article:

To export extended Supervisely format with the blob files and offsets, use the application. ☝️ However, other applications export projects in the Supervisely format using the traditional method, downloading each image separately.

Importing the extended Supervisely format happens automatically in applications that previously imported projects in the Supervisely format without blobs. Such as application or tool.

🎉
Export As Masks
Export As Masks
Export As Masks
Export As Masks
Optimized Import of Small Images
Export to Supervisely format: Blob
Import Images in Supervisely Format
Auto Import
Project structure system
Project structure system