Labeling Jobs

Guide explains how to manage labeling jobs using Supervisely SDK and

Introduction

In this tutorial you will learn how to manage Labeling Jobs using Supervisely SDK and API.

📗 Everything you need to reproduce this tutorial is on GitHub: source code and demo data.

How to debug this tutorial

Step 1. Prepare ~/supervisely.env file with credentials. Learn more here.

Step 2. Clone repository with source code and demo data and create Virtual Environment.

git clone https://github.com/supervisely-ecosystem/automation-with-python-sdk-and-api
cd automation-with-python-sdk-and-api
./create_venv.sh

Step 3. Open repository directory in Visual Studio Code.

code -r .

Step 4. change ✅ IDs ✅ in local.env file by copying the IDs from Supervisely instance.

• Change Team ID in local.env file by copying the ID from the context menu of the team.

  TEAM_ID=8                 # ⬅️ change it

  
• Get Lemons (Test) project from ecosystem. Lemons (Test) is an example project with 6 images of lemons and kiwi fruits.

Change project id in local.env file by copying the ID from the context menu of the project.

PROJECT_ID=5555                 # ⬅️ change it

• Change User ID and user login in local.env to your own from Team members page.

  USER_ID=7                 # ⬅️ change it
  USER_LOGIN="my_username"  # ⬅️ change it

  

Step 5. Start debugging examples/labeling-jobs-automation/main.py

Labeling Jobs automation

Import libraries

import os
from dotenv import load_dotenv
import supervisely as sly

Init API client

​Init API for communicating with Supervisely Instance. First, we load environment variables with credentials:

if sly.is_development():
    load_dotenv("local.env")
    load_dotenv(os.path.expanduser("~/supervisely.env"))
api = sly.Api.from_env()

Get your IDs and username from environment

TEAM_ID = sly.env.team_id()
PROJECT_ID = sly.env.project_id()
USER_ID = sly.env.user_id()
USER_LOGIN = sly.env.user_login()

Prepare project for Labeling Job

Function will populate project meta with classes: "kiwi", "lemon", and tags: "size", "origin".

prepare_project(api=api, id=PROJECT_ID)

Labeling jobs automation

Step 1. Create and add annotators to the team, before creating Labeling Job

Create accounts for annotators with restrictions.

Note: Creating users requires admin permission.

labeler_1 = api.user.get_info_by_login(login='labeler_1')
if labeler_1 is None:
    labeler_1 = api.user.create(login='labeler_1', password='11111abc', is_restricted=True)

labeler_2 = api.user.get_info_by_login(login='labeler_2')
if labeler_2 is None:
    labeler_2 = api.user.create(login='labeler_2', password='22222abc', is_restricted=True)

Labelers will be able to login only after being added to at least one team

Note: Adding users to the Team requires admin permission.

if api.user.get_team_role(labeler_1.id, TEAM_ID) is None:
    api.user.add_to_team(labeler_1.id, TEAM_ID, api.role.DefaultRole.ANNOTATOR)
if api.user.get_team_role(labeler_2.id, TEAM_ID) is None:
    api.user.add_to_team(labeler_2.id, TEAM_ID, api.role.DefaultRole.ANNOTATOR)

Step 2. Define project and datasets for labeling job

project_meta_json = api.project.get_meta(PROJECT_ID)
project_meta = sly.ProjectMeta.from_json(project_meta_json)
print(project_meta)

Output:

ProjectMeta:
Object Classes
+-------+--------+----------------+--------+
|  Name | Shape  |     Color      | Hotkey |
+-------+--------+----------------+--------+
|  kiwi | Bitmap |  [208, 2, 27]  |        |
| lemon | Bitmap | [80, 227, 194] |        |
+-------+--------+----------------+--------+
Tags
+--------+--------------+------------------------------+--------+---------------+--------------------+
|  Name  |  Value type  |       Possible values        | Hotkey | Applicable to | Applicable classes |
+--------+--------------+------------------------------+--------+---------------+--------------------+
| origin |  any_string  |             None             |        |  objectsOnly  | ['kiwi', 'lemon']  |
|  size  | oneof_string | ['small', 'medium', 'large'] |        |  objectsOnly  | ['kiwi', 'lemon']  |
+--------+--------------+------------------------------+--------+---------------+--------------------+

datasets = api.dataset.get_list(project.id)
print(datasets)

Output:

[
  DatasetInfo(
    id=10555, 
    name='ds1',
    description='', 
    size='1277440',
    project_id=5555,
    images_count=6, 
    created_at='2022-10-18T15:39:57.377Z',
    updated_at='2022-10-18T15:39:57.377Z'
  )
]

Step 3. Create Labeling Jobs

Create labeling job for labeler 1, and assign class lemon to label.

created_jobs = api.labeling_job.create(name='labeler1_lemons_task',
                                       dataset_id=datasets[0].id,
                                       user_ids=[labeler_1.id],
                                       readme='annotation manual for fruits in markdown format here (optional)',
                                       description='short description is here (optional)',
                                       classes_to_label=["lemon"])
print(created_jobs)

Output:

[
  LabelingJobInfo(
      id=1, 
      name='labeler1_lemons_task', 
      readme='annotation manual for fruits in markdown format here (optional)',
      description='short description is here (optional)', 
      team_id=8,
      workspace_id=349,
      workspace_name='Testing Workspace', 
      project_id=5555,
      project_name='Lemons (Test)', 
      dataset_id=10555,
      dataset_name='ds1', 
      created_by_id=7,
      created_by_login='my_username', 
      assigned_to_id=101, 
      assigned_to_login='labeler_1',
      created_at='2022-10-05T08:42:30.588Z', 
      started_at=None, 
      finished_at=None,
      status='pending',
      disabled=False, 
      images_count=6, 
      finished_images_count=0, 
      rejected_images_count=0,
      accepted_images_count=0,
      classes_to_label=[],
      tags_to_label=[], 
      images_range=(None, None), 
      objects_limit_per_image=None, 
      tags_limit_per_image=None,
      filter_images_by_tags=[], 
      include_images_with_tags=[],
      exclude_images_with_tags=[],
      entities=[
        {
          'reviewStatus': 'none', 
          'id': 3882702, 
          'name': 'IMG_8144.jpeg'
        }, 
        {
          'reviewStatus': 'none', 
          'id': 3882697, 
          'name': 'IMG_4451.jpeg'
        },
        {
          'reviewStatus': 'none',
          'id': 3882698,
          'name': 'IMG_3861.jpeg'
        }, 
        {
          'reviewStatus': 'none', 
          'id': 3882700, 
          'name': 'IMG_2084.jpeg'
        }, 
        {
          'reviewStatus': 'none',
          'id': 3882701, 
          'name': 'IMG_1836.jpeg'
        }, 
        {
          'reviewStatus': 'none',
          'id': 3882699,
          'name': 'IMG_0748.jpeg'
        }
      ]
  )
]

You can stop Labeling Job if you need. Job will become unavailable for labeler.

api.labeling_job.stop(created_jobs[0].id)

Create labeling job for labeler 2, and assign class kiwi to label, and also tags "size" and "origin", with objects and tags limit. You can also specify which images to label by providing images ids.

List all images in dataset and get their IDs. As an example we will select only half of images in the dataset.

dataset_images_infos = api.image.get_list(dataset_id=datasets[0].id)
dataset_images_ids = [image_info.id for image_info in dataset_images_infos]
selected_images_ids = dataset_images_ids[:len(dataset_images_ids) // 2]

created_jobs = api.labeling_job.create(
                                name='labeler2_kiwi_task_with_complex_settings',
                                dataset_id=datasets[0].id,
                                user_ids=[labeler_2.id],
                                readme='annotation manual for fruits in markdown format here (optional)',
                                description='short description is here (optional)',
                                classes_to_label=["kiwi"],
                                objects_limit_per_image=10,
                                tags_to_label=["size", "origin"],
                                tags_limit_per_image=20,
                                images_ids=selected_images_ids
                                )
print(created_jobs)

Output:

[
  LabelingJobInfo(
      id=2, 
      name='labeler2_kiwi_task_with_complex_settings', 
      readme='annotation manual for fruits in markdown format here (optional)',
      description='short description is here (optional)', 
      team_id=8,
      workspace_id=349,
      workspace_name='Testing Workspace', 
      project_id=5555,
      project_name='Lemons (Test)', 
      dataset_id=10555,
      dataset_name='ds1', 
      created_by_id=100,
      created_by_login='my_username', 
      assigned_to_id=102, 
      assigned_to_login='labeler_2',
      created_at='2022-10-05T08:42:30.588Z', 
      started_at=None, 
      finished_at=None,
      status='pending',
      disabled=False, 
      images_count=6, 
      finished_images_count=0, 
      rejected_images_count=0,
      accepted_images_count=0,
      classes_to_label=["kiwi"],
      tags_to_label=["size", "origin"], 
      images_range=(None, None), 
      objects_limit_per_image=10, 
      tags_limit_per_image=20,
      filter_images_by_tags=[], 
      include_images_with_tags=[],
      exclude_images_with_tags=[],
      entities=[
          {
            'reviewStatus': 'none', 
            'id': 3882697, 
            'name': 'IMG_4451.jpeg'
          }, 
          {
            'reviewStatus': 'none', 
            'id': 3882698, 
            'name': 'IMG_3861.jpeg'
          }, 
          {
            'reviewStatus': 'none', 
            'id': 3882699, 
            'name': 'IMG_0748.jpeg'
          }
      ]
  )
]

Created Labeling Jobs

Get all labeling jobs in a team

jobs = api.labeling_job.get_list(TEAM_ID)
print(jobs)

Output:

[
  LabelingJobInfo(
      id=1, 
      name='labeler1_lemons_task', 
      readme='annotation manual for fruits in markdown format here (optional)',
      description='short description is here (optional)', 
      team_id=8,
      workspace_id=349,
      workspace_name='Testing Workspace', 
      project_id=5555,
      project_name='Lemons (Test)', 
      dataset_id=10555,
      dataset_name='ds1', 
      created_by_id=7,
      created_by_login='my_username', 
      assigned_to_id=101, 
      assigned_to_login='labeler_1',
      created_at='2022-10-05T08:42:30.588Z', 
      started_at=None, 
      finished_at=None,
      status='pending',
      disabled=False, 
      images_count=6, 
      finished_images_count=0, 
      rejected_images_count=0,
      accepted_images_count=0,
      classes_to_label=[],
      tags_to_label=[], 
      images_range=(None, None), 
      objects_limit_per_image=None, 
      tags_limit_per_image=None,
      filter_images_by_tags=[], 
      include_images_with_tags=[],
      exclude_images_with_tags=[],
      entities=[
        {
          'reviewStatus': 'none', 
          'id': 3882702, 
          'name': 'IMG_8144.jpeg'
        }, 
        {
          'reviewStatus': 'none', 
          'id': 3882697, 
          'name': 'IMG_4451.jpeg'
        },
        {
          'reviewStatus': 'none',
          'id': 3882698,
          'name': 'IMG_3861.jpeg'
        }, 
        {
          'reviewStatus': 'none', 
          'id': 3882700, 
          'name': 'IMG_2084.jpeg'
        }, 
        {
          'reviewStatus': 'none',
          'id': 3882701, 
          'name': 'IMG_1836.jpeg'
        }, 
        {
          'reviewStatus': 'none',
          'id': 3882699,
          'name': 'IMG_0748.jpeg'
        }
      ]
  ),
  LabelingJobInfo(
      id=2, 
      name='labeler2_kiwi_task_with_complex_settings', 
      readme='annotation manual for fruits in markdown format here (optional)',
      description='short description is here (optional)', 
      team_id=8,
      workspace_id=349,
      workspace_name='Testing Workspace', 
      project_id=5555,
      project_name='Lemons (Test)', 
      dataset_id=10555,
      dataset_name='ds1', 
      created_by_id=100,
      created_by_login='my_username', 
      assigned_to_id=102, 
      assigned_to_login='labeler_2',
      created_at='2022-10-05T08:42:30.588Z', 
      started_at=None, 
      finished_at=None,
      status='pending',
      disabled=False, 
      images_count=6, 
      finished_images_count=0, 
      rejected_images_count=0,
      accepted_images_count=0,
      classes_to_label=["kiwi"],
      tags_to_label=["size", "origin"], 
      images_range=(None, None), 
      objects_limit_per_image=10, 
      tags_limit_per_image=20,
      filter_images_by_tags=[], 
      include_images_with_tags=[],
      exclude_images_with_tags=[],
      entities=[
          {
            'reviewStatus': 'none', 
            'id': 3882697, 
            'name': 'IMG_4451.jpeg'
          }, 
          {
            'reviewStatus': 'none', 
            'id': 3882698, 
            'name': 'IMG_3861.jpeg'
          }, 
          {
            'reviewStatus': 'none', 
            'id': 3882699, 
            'name': 'IMG_0748.jpeg'
          }
      ]
  )
]

Labeling Jobs filtering

List of available filters:

• created_by_id

• assigned_to_id

• project_id

• dataset_id

Note: filters can be used in various combinations

Get all labeling jobs that were created by user 'my_username'

Note: Getting UserInfo by login requires admin permission.

user = api.user.get_info_by_login(USER_LOGIN)
jobs = api.labeling_job.get_list(TEAM_ID, created_by_id=user.id)
print(jobs)

Output

[
  LabelingJobInfo(
      id=1, 
      name='labeler1_lemons_task', 
      readme='annotation manual for fruits in markdown format here (optional)',
      description='short description is here (optional)', 
      team_id=8,
      workspace_id=349,
      workspace_name='Testing Workspace', 
      project_id=5555,
      project_name='Lemons (Test)', 
      dataset_id=10555,
      dataset_name='ds1', 
      created_by_id=7,
      created_by_login='my_username', 
      assigned_to_id=101, 
      assigned_to_login='labeler_1',
      created_at='2022-10-05T08:42:30.588Z', 
      started_at=None, 
      finished_at=None,
      status='pending',
      disabled=False, 
      images_count=6, 
      finished_images_count=0, 
      rejected_images_count=0,
      accepted_images_count=0,
      classes_to_label=[],
      tags_to_label=[], 
      images_range=(None, None), 
      objects_limit_per_image=None, 
      tags_limit_per_image=None,
      filter_images_by_tags=[], 
      include_images_with_tags=[],
      exclude_images_with_tags=[],
      entities=[
        {
          'reviewStatus': 'none', 
          'id': 3882702, 
          'name': 'IMG_8144.jpeg'
        }, 
        {
          'reviewStatus': 'none', 
          'id': 3882697, 
          'name': 'IMG_4451.jpeg'
        },
        {
          'reviewStatus': 'none',
          'id': 3882698,
          'name': 'IMG_3861.jpeg'
        }, 
        {
          'reviewStatus': 'none', 
          'id': 3882700, 
          'name': 'IMG_2084.jpeg'
        }, 
        {
          'reviewStatus': 'none',
          'id': 3882701, 
          'name': 'IMG_1836.jpeg'
        }, 
        {
          'reviewStatus': 'none',
          'id': 3882699,
          'name': 'IMG_0748.jpeg'
        }
      ]
  ),
  LabelingJobInfo(
      id=2, 
      name='labeler2_kiwi_task_with_complex_settings', 
      readme='annotation manual for fruits in markdown format here (optional)',
      description='short description is here (optional)', 
      team_id=8,
      workspace_id=349,
      workspace_name='Testing Workspace', 
      project_id=5555,
      project_name='Lemons (Test)', 
      dataset_id=10555,
      dataset_name='ds1', 
      created_by_id=100,
      created_by_login='my_username', 
      assigned_to_id=102, 
      assigned_to_login='labeler_2',
      created_at='2022-10-05T08:42:30.588Z', 
      started_at=None, 
      finished_at=None,
      status='pending',
      disabled=False, 
      images_count=6, 
      finished_images_count=0, 
      rejected_images_count=0,
      accepted_images_count=0,
      classes_to_label=["kiwi"],
      tags_to_label=["size", "origin"], 
      images_range=(None, None), 
      objects_limit_per_image=10, 
      tags_limit_per_image=20,
      filter_images_by_tags=[], 
      include_images_with_tags=[],
      exclude_images_with_tags=[],
      entities=[
          {
            'reviewStatus': 'none', 
            'id': 3882697, 
            'name': 'IMG_4451.jpeg'
          }, 
          {
            'reviewStatus': 'none', 
            'id': 3882698, 
            'name': 'IMG_3861.jpeg'
          }, 
          {
            'reviewStatus': 'none', 
            'id': 3882699, 
            'name': 'IMG_0748.jpeg'
          }
      ]
  )
]

Get all labeling jobs that were created by user "my_username" and were assigned to labeler 2

jobs = api.labeling_job.get_list(TEAM_ID, created_by_id=user.id, assigned_to_id=labeler_2.id)
print(jobs)

Output:

[
  LabelingJobInfo(
      id=2, 
      name='labeler2_kiwi_task_with_complex_settings', 
      readme='annotation manual for fruits in markdown format here (optional)',
      description='short description is here (optional)', 
      team_id=8,
      workspace_id=349,
      workspace_name='Testing Workspace', 
      project_id=5555,
      project_name='Lemons (Test)', 
      dataset_id=10555,
      dataset_name='ds1', 
      created_by_id=100,
      created_by_login='my_username', 
      assigned_to_id=102, 
      assigned_to_login='labeler_2',
      created_at='2022-10-05T08:42:30.588Z', 
      started_at=None, 
      finished_at=None,
      status='pending',
      disabled=False, 
      images_count=6, 
      finished_images_count=0, 
      rejected_images_count=0,
      accepted_images_count=0,
      classes_to_label=["kiwi"],
      tags_to_label=["size", "origin"], 
      images_range=(None, None), 
      objects_limit_per_image=10, 
      tags_limit_per_image=20,
      filter_images_by_tags=[], 
      include_images_with_tags=[],
      exclude_images_with_tags=[],
      entities=[
          {
            'reviewStatus': 'none', 
            'id': 3882697, 
            'name': 'IMG_4451.jpeg'
          }, 
          {
            'reviewStatus': 'none', 
            'id': 3882698, 
            'name': 'IMG_3861.jpeg'
          }, 
          {
            'reviewStatus': 'none', 
            'id': 3882699, 
            'name': 'IMG_0748.jpeg'
          }
      ]
  )
]

Get all active labeling jobs in a team

jobs = api.labeling_job.get_list(TEAM_ID)
print(jobs)

Output:

[
  LabelingJobInfo(
      id=1, 
      name='labeler1_lemons_task', 
      readme='annotation manual for fruits in markdown format here (optional)',
      description='short description is here (optional)', 
      team_id=8,
      workspace_id=349,
      workspace_name='Testing Workspace', 
      project_id=5555,
      project_name='Lemons (Test)', 
      dataset_id=10555,
      dataset_name='ds1', 
      created_by_id=7,
      created_by_login='my_username', 
      assigned_to_id=101, 
      assigned_to_login='labeler_1',
      created_at='2022-10-05T08:42:30.588Z', 
      started_at=None, 
      finished_at=None,
      status='pending',
      disabled=False, 
      images_count=6, 
      finished_images_count=0, 
      rejected_images_count=0,
      accepted_images_count=0,
      classes_to_label=[],
      tags_to_label=[], 
      images_range=(None, None), 
      objects_limit_per_image=None, 
      tags_limit_per_image=None,
      filter_images_by_tags=[], 
      include_images_with_tags=[],
      exclude_images_with_tags=[],
      entities=[
        {
          'reviewStatus': 'none', 
          'id': 3882702, 
          'name': 'IMG_8144.jpeg'
        }, 
        {
          'reviewStatus': 'none', 
          'id': 3882697, 
          'name': 'IMG_4451.jpeg'
        },
        {
          'reviewStatus': 'none',
          'id': 3882698,
          'name': 'IMG_3861.jpeg'
        }, 
        {
          'reviewStatus': 'none', 
          'id': 3882700, 
          'name': 'IMG_2084.jpeg'
        }, 
        {
          'reviewStatus': 'none',
          'id': 3882701, 
          'name': 'IMG_1836.jpeg'
        }, 
        {
          'reviewStatus': 'none',
          'id': 3882699,
          'name': 'IMG_0748.jpeg'
        }
      ]
  ),
  LabelingJobInfo(
      id=2, 
      name='labeler2_kiwi_task_with_complex_settings', 
      readme='annotation manual for fruits in markdown format here (optional)',
      description='short description is here (optional)', 
      team_id=8,
      workspace_id=349,
      workspace_name='Testing Workspace', 
      project_id=5555,
      project_name='Lemons (Test)', 
      dataset_id=10555,
      dataset_name='ds1', 
      created_by_id=100,
      created_by_login='my_username', 
      assigned_to_id=102, 
      assigned_to_login='labeler_2',
      created_at='2022-10-05T08:42:30.588Z', 
      started_at=None, 
      finished_at=None,
      status='pending',
      disabled=False, 
      images_count=6, 
      finished_images_count=0, 
      rejected_images_count=0,
      accepted_images_count=0,
      classes_to_label=["kiwi"],
      tags_to_label=["size", "origin"], 
      images_range=(None, None), 
      objects_limit_per_image=10, 
      tags_limit_per_image=20,
      filter_images_by_tags=[], 
      include_images_with_tags=[],
      exclude_images_with_tags=[],
      entities=[
          {
            'reviewStatus': 'none', 
            'id': 3882697, 
            'name': 'IMG_4451.jpeg'
          }, 
          {
            'reviewStatus': 'none', 
            'id': 3882698, 
            'name': 'IMG_3861.jpeg'
          }, 
          {
            'reviewStatus': 'none', 
            'id': 3882699, 
            'name': 'IMG_0748.jpeg'
          }
      ]
  )
]

Labeling Jobs Statuses

api.labeling_job.Status.PENDING - labeling job is created, labeler still has not started api.labeling_job.Status.IN_PROGRESS - labeler started, but not finished api.labeling_job.Status.ON_REVIEW - labeler finished his job, reviewer is in progress api.labeling_job.Status.COMPLETED - reviewer completed job api.labeling_job.Status.STOPPED - job was stopped at some stage

job_id = jobs[-2].id
api.labeling_job.get_status(job_id)

Output

<Status.STOPPED: 'stopped'>

job_id = jobs[-1].id
api.labeling_job.get_status(job_id)

Output:

<Status.PENDING: 'pending'>

If you want to change Labeling Job status you can use api.labeling_job.set_status() method

job_id = jobs[-1].id
api.labeling_job.set_status(id=job_id, status="completed")

Output:

<Status.COMPLETED: 'completed'>

The following methods will wait until labeling job will change status to the given expected status:

• Labeler has finished annotating

  api.labeling_job.wait(job_id, target_status=api.labeling_job.Status.ON_REVIEW)

• Reviewer has finished his annotation review

  api.labeling_job.wait(job_id, target_status=api.labeling_job.Status.COMPLETED) 

Archive Labeling Job

Archive Labeling job by ID. Data can be retrieved after archiving.

api.labeling_job.archive(jobs[0].id)

Archived Labelling Job

Remove Labeling Jobs

Labeling job will be removed permanently.

Remove single Labeling Job

Remove Labeling job by ID.

job_id = 278
api.labeling_job.remove(job_id)

Remove list of Labeling Jobs

Remove Labeling jobs by IDs.

job_ids = [279, 280, 281]
api.labeling_job.remove_batch(job_ids)