Labeling Jobs
Guide explains how to manage labeling jobs using Supervisely SDK and
Introduction
In this tutorial you will learn how to manage Labeling Jobs using Supervisely SDK and API.
📗 Everything you need to reproduce this tutorial is on GitHub: source code and demo data.
How to debug this tutorial
Step 1. Prepare ~/supervisely.env file with credentials. Learn more here.
Step 2. Clone repository with source code and demo data and create Virtual Environment.
git clone https://github.com/supervisely-ecosystem/automation-with-python-sdk-and-api
cd automation-with-python-sdk-and-api
./create_venv.shStep 3. Open repository directory in Visual Studio Code.
code -r .Step 4. change ✅ IDs ✅ in local.env file by copying the IDs from Supervisely instance.
Change Team ID in
local.envfile by copying the ID from the context menu of the team.TEAM_ID=8 # ⬅️ change it
Get Lemons (Test) project from ecosystem. Lemons (Test) is an example project with 6 images of lemons and kiwi fruits.
Change project id in local.env file by copying the ID from the context menu of the project.
PROJECT_ID=5555 # ⬅️ change it
Change User ID and user login in
local.envto your own from Team members page.USER_ID=7 # ⬅️ change it USER_LOGIN="my_username" # ⬅️ change it
Step 5. Start debugging examples/labeling-jobs-automation/main.py
Labeling Jobs automation
Import libraries
import os
from dotenv import load_dotenv
import supervisely as slyInit API client
Init API for communicating with Supervisely Instance. First, we load environment variables with credentials:
if sly.is_development():
load_dotenv("local.env")
load_dotenv(os.path.expanduser("~/supervisely.env"))
api = sly.Api.from_env()Get your IDs and username from environment
TEAM_ID = sly.env.team_id()
PROJECT_ID = sly.env.project_id()
USER_ID = sly.env.user_id()
USER_LOGIN = sly.env.user_login()Prepare project for Labeling Job
Function will populate project meta with classes: "kiwi", "lemon", and tags: "size", "origin".
prepare_project(api=api, id=PROJECT_ID)
Labeling jobs automation
Step 1. Create and add annotators to the team, before creating Labeling Job
Create accounts for annotators with restrictions.
Note: Creating users requires admin permission.
labeler_1 = api.user.get_info_by_login(login='labeler_1')
if labeler_1 is None:
labeler_1 = api.user.create(login='labeler_1', password='11111abc', is_restricted=True)
labeler_2 = api.user.get_info_by_login(login='labeler_2')
if labeler_2 is None:
labeler_2 = api.user.create(login='labeler_2', password='22222abc', is_restricted=True)Labelers will be able to login only after being added to at least one team
Note: Adding users to the Team requires admin permission.
if api.user.get_team_role(labeler_1.id, TEAM_ID) is None:
api.user.add_to_team(labeler_1.id, TEAM_ID, api.role.DefaultRole.ANNOTATOR)
if api.user.get_team_role(labeler_2.id, TEAM_ID) is None:
api.user.add_to_team(labeler_2.id, TEAM_ID, api.role.DefaultRole.ANNOTATOR)
Step 2. Define project and datasets for labeling job
project_meta_json = api.project.get_meta(PROJECT_ID)
project_meta = sly.ProjectMeta.from_json(project_meta_json)
print(project_meta)Output:
ProjectMeta:
Object Classes
+-------+--------+----------------+--------+
| Name | Shape | Color | Hotkey |
+-------+--------+----------------+--------+
| kiwi | Bitmap | [208, 2, 27] | |
| lemon | Bitmap | [80, 227, 194] | |
+-------+--------+----------------+--------+
Tags
+--------+--------------+------------------------------+--------+---------------+--------------------+
| Name | Value type | Possible values | Hotkey | Applicable to | Applicable classes |
+--------+--------------+------------------------------+--------+---------------+--------------------+
| origin | any_string | None | | objectsOnly | ['kiwi', 'lemon'] |
| size | oneof_string | ['small', 'medium', 'large'] | | objectsOnly | ['kiwi', 'lemon'] |
+--------+--------------+------------------------------+--------+---------------+--------------------+datasets = api.dataset.get_list(project.id)
print(datasets)Output:
[
DatasetInfo(
id=10555,
name='ds1',
description='',
size='1277440',
project_id=5555,
images_count=6,
created_at='2022-10-18T15:39:57.377Z',
updated_at='2022-10-18T15:39:57.377Z'
)
]Step 3. Create Labeling Jobs
Create labeling job for labeler 1, and assign class lemon to label.
created_jobs = api.labeling_job.create(name='labeler1_lemons_task',
dataset_id=datasets[0].id,
user_ids=[labeler_1.id],
readme='annotation manual for fruits in markdown format here (optional)',
description='short description is here (optional)',
classes_to_label=["lemon"])
print(created_jobs)Output:
[
LabelingJobInfo(
id=1,
name='labeler1_lemons_task',
readme='annotation manual for fruits in markdown format here (optional)',
description='short description is here (optional)',
team_id=8,
workspace_id=349,
workspace_name='Testing Workspace',
project_id=5555,
project_name='Lemons (Test)',
dataset_id=10555,
dataset_name='ds1',
created_by_id=7,
created_by_login='my_username',
assigned_to_id=101,
assigned_to_login='labeler_1',
created_at='2022-10-05T08:42:30.588Z',
started_at=None,
finished_at=None,
status='pending',
disabled=False,
images_count=6,
finished_images_count=0,
rejected_images_count=0,
accepted_images_count=0,
classes_to_label=[],
tags_to_label=[],
images_range=(None, None),
objects_limit_per_image=None,
tags_limit_per_image=None,
filter_images_by_tags=[],
include_images_with_tags=[],
exclude_images_with_tags=[],
entities=[
{
'reviewStatus': 'none',
'id': 3882702,
'name': 'IMG_8144.jpeg'
},
{
'reviewStatus': 'none',
'id': 3882697,
'name': 'IMG_4451.jpeg'
},
{
'reviewStatus': 'none',
'id': 3882698,
'name': 'IMG_3861.jpeg'
},
{
'reviewStatus': 'none',
'id': 3882700,
'name': 'IMG_2084.jpeg'
},
{
'reviewStatus': 'none',
'id': 3882701,
'name': 'IMG_1836.jpeg'
},
{
'reviewStatus': 'none',
'id': 3882699,
'name': 'IMG_0748.jpeg'
}
]
)
]You can stop Labeling Job if you need. Job will become unavailable for labeler.
api.labeling_job.stop(created_jobs[0].id)Create labeling job for labeler 2, and assign class kiwi to label, and also tags "size" and "origin", with objects and tags limit. You can also specify which images to label by providing images ids.
List all images in dataset and get their IDs. As an example we will select only half of images in the dataset.
dataset_images_infos = api.image.get_list(dataset_id=datasets[0].id)
dataset_images_ids = [image_info.id for image_info in dataset_images_infos]
selected_images_ids = dataset_images_ids[:len(dataset_images_ids) // 2]created_jobs = api.labeling_job.create(
name='labeler2_kiwi_task_with_complex_settings',
dataset_id=datasets[0].id,
user_ids=[labeler_2.id],
readme='annotation manual for fruits in markdown format here (optional)',
description='short description is here (optional)',
classes_to_label=["kiwi"],
objects_limit_per_image=10,
tags_to_label=["size", "origin"],
tags_limit_per_image=20,
images_ids=selected_images_ids
)
print(created_jobs)Output:
[
LabelingJobInfo(
id=2,
name='labeler2_kiwi_task_with_complex_settings',
readme='annotation manual for fruits in markdown format here (optional)',
description='short description is here (optional)',
team_id=8,
workspace_id=349,
workspace_name='Testing Workspace',
project_id=5555,
project_name='Lemons (Test)',
dataset_id=10555,
dataset_name='ds1',
created_by_id=100,
created_by_login='my_username',
assigned_to_id=102,
assigned_to_login='labeler_2',
created_at='2022-10-05T08:42:30.588Z',
started_at=None,
finished_at=None,
status='pending',
disabled=False,
images_count=6,
finished_images_count=0,
rejected_images_count=0,
accepted_images_count=0,
classes_to_label=["kiwi"],
tags_to_label=["size", "origin"],
images_range=(None, None),
objects_limit_per_image=10,
tags_limit_per_image=20,
filter_images_by_tags=[],
include_images_with_tags=[],
exclude_images_with_tags=[],
entities=[
{
'reviewStatus': 'none',
'id': 3882697,
'name': 'IMG_4451.jpeg'
},
{
'reviewStatus': 'none',
'id': 3882698,
'name': 'IMG_3861.jpeg'
},
{
'reviewStatus': 'none',
'id': 3882699,
'name': 'IMG_0748.jpeg'
}
]
)
]
Get all labeling jobs in a team
jobs = api.labeling_job.get_list(TEAM_ID)
print(jobs)Output:
[
LabelingJobInfo(
id=1,
name='labeler1_lemons_task',
readme='annotation manual for fruits in markdown format here (optional)',
description='short description is here (optional)',
team_id=8,
workspace_id=349,
workspace_name='Testing Workspace',
project_id=5555,
project_name='Lemons (Test)',
dataset_id=10555,
dataset_name='ds1',
created_by_id=7,
created_by_login='my_username',
assigned_to_id=101,
assigned_to_login='labeler_1',
created_at='2022-10-05T08:42:30.588Z',
started_at=None,
finished_at=None,
status='pending',
disabled=False,
images_count=6,
finished_images_count=0,
rejected_images_count=0,
accepted_images_count=0,
classes_to_label=[],
tags_to_label=[],
images_range=(None, None),
objects_limit_per_image=None,
tags_limit_per_image=None,
filter_images_by_tags=[],
include_images_with_tags=[],
exclude_images_with_tags=[],
entities=[
{
'reviewStatus': 'none',
'id': 3882702,
'name': 'IMG_8144.jpeg'
},
{
'reviewStatus': 'none',
'id': 3882697,
'name': 'IMG_4451.jpeg'
},
{
'reviewStatus': 'none',
'id': 3882698,
'name': 'IMG_3861.jpeg'
},
{
'reviewStatus': 'none',
'id': 3882700,
'name': 'IMG_2084.jpeg'
},
{
'reviewStatus': 'none',
'id': 3882701,
'name': 'IMG_1836.jpeg'
},
{
'reviewStatus': 'none',
'id': 3882699,
'name': 'IMG_0748.jpeg'
}
]
),
LabelingJobInfo(
id=2,
name='labeler2_kiwi_task_with_complex_settings',
readme='annotation manual for fruits in markdown format here (optional)',
description='short description is here (optional)',
team_id=8,
workspace_id=349,
workspace_name='Testing Workspace',
project_id=5555,
project_name='Lemons (Test)',
dataset_id=10555,
dataset_name='ds1',
created_by_id=100,
created_by_login='my_username',
assigned_to_id=102,
assigned_to_login='labeler_2',
created_at='2022-10-05T08:42:30.588Z',
started_at=None,
finished_at=None,
status='pending',
disabled=False,
images_count=6,
finished_images_count=0,
rejected_images_count=0,
accepted_images_count=0,
classes_to_label=["kiwi"],
tags_to_label=["size", "origin"],
images_range=(None, None),
objects_limit_per_image=10,
tags_limit_per_image=20,
filter_images_by_tags=[],
include_images_with_tags=[],
exclude_images_with_tags=[],
entities=[
{
'reviewStatus': 'none',
'id': 3882697,
'name': 'IMG_4451.jpeg'
},
{
'reviewStatus': 'none',
'id': 3882698,
'name': 'IMG_3861.jpeg'
},
{
'reviewStatus': 'none',
'id': 3882699,
'name': 'IMG_0748.jpeg'
}
]
)
]Labeling Jobs filtering
List of available filters:
created_by_id
assigned_to_id
project_id
dataset_id
Note: filters can be used in various combinations
Get all labeling jobs that were created by user 'my_username'
Note: Getting UserInfo by login requires admin permission.
user = api.user.get_info_by_login(USER_LOGIN)
jobs = api.labeling_job.get_list(TEAM_ID, created_by_id=user.id)
print(jobs)Output
[
LabelingJobInfo(
id=1,
name='labeler1_lemons_task',
readme='annotation manual for fruits in markdown format here (optional)',
description='short description is here (optional)',
team_id=8,
workspace_id=349,
workspace_name='Testing Workspace',
project_id=5555,
project_name='Lemons (Test)',
dataset_id=10555,
dataset_name='ds1',
created_by_id=7,
created_by_login='my_username',
assigned_to_id=101,
assigned_to_login='labeler_1',
created_at='2022-10-05T08:42:30.588Z',
started_at=None,
finished_at=None,
status='pending',
disabled=False,
images_count=6,
finished_images_count=0,
rejected_images_count=0,
accepted_images_count=0,
classes_to_label=[],
tags_to_label=[],
images_range=(None, None),
objects_limit_per_image=None,
tags_limit_per_image=None,
filter_images_by_tags=[],
include_images_with_tags=[],
exclude_images_with_tags=[],
entities=[
{
'reviewStatus': 'none',
'id': 3882702,
'name': 'IMG_8144.jpeg'
},
{
'reviewStatus': 'none',
'id': 3882697,
'name': 'IMG_4451.jpeg'
},
{
'reviewStatus': 'none',
'id': 3882698,
'name': 'IMG_3861.jpeg'
},
{
'reviewStatus': 'none',
'id': 3882700,
'name': 'IMG_2084.jpeg'
},
{
'reviewStatus': 'none',
'id': 3882701,
'name': 'IMG_1836.jpeg'
},
{
'reviewStatus': 'none',
'id': 3882699,
'name': 'IMG_0748.jpeg'
}
]
),
LabelingJobInfo(
id=2,
name='labeler2_kiwi_task_with_complex_settings',
readme='annotation manual for fruits in markdown format here (optional)',
description='short description is here (optional)',
team_id=8,
workspace_id=349,
workspace_name='Testing Workspace',
project_id=5555,
project_name='Lemons (Test)',
dataset_id=10555,
dataset_name='ds1',
created_by_id=100,
created_by_login='my_username',
assigned_to_id=102,
assigned_to_login='labeler_2',
created_at='2022-10-05T08:42:30.588Z',
started_at=None,
finished_at=None,
status='pending',
disabled=False,
images_count=6,
finished_images_count=0,
rejected_images_count=0,
accepted_images_count=0,
classes_to_label=["kiwi"],
tags_to_label=["size", "origin"],
images_range=(None, None),
objects_limit_per_image=10,
tags_limit_per_image=20,
filter_images_by_tags=[],
include_images_with_tags=[],
exclude_images_with_tags=[],
entities=[
{
'reviewStatus': 'none',
'id': 3882697,
'name': 'IMG_4451.jpeg'
},
{
'reviewStatus': 'none',
'id': 3882698,
'name': 'IMG_3861.jpeg'
},
{
'reviewStatus': 'none',
'id': 3882699,
'name': 'IMG_0748.jpeg'
}
]
)
]Get all labeling jobs that were created by user "my_username" and were assigned to labeler 2
jobs = api.labeling_job.get_list(TEAM_ID, created_by_id=user.id, assigned_to_id=labeler_2.id)
print(jobs)Output:
[
LabelingJobInfo(
id=2,
name='labeler2_kiwi_task_with_complex_settings',
readme='annotation manual for fruits in markdown format here (optional)',
description='short description is here (optional)',
team_id=8,
workspace_id=349,
workspace_name='Testing Workspace',
project_id=5555,
project_name='Lemons (Test)',
dataset_id=10555,
dataset_name='ds1',
created_by_id=100,
created_by_login='my_username',
assigned_to_id=102,
assigned_to_login='labeler_2',
created_at='2022-10-05T08:42:30.588Z',
started_at=None,
finished_at=None,
status='pending',
disabled=False,
images_count=6,
finished_images_count=0,
rejected_images_count=0,
accepted_images_count=0,
classes_to_label=["kiwi"],
tags_to_label=["size", "origin"],
images_range=(None, None),
objects_limit_per_image=10,
tags_limit_per_image=20,
filter_images_by_tags=[],
include_images_with_tags=[],
exclude_images_with_tags=[],
entities=[
{
'reviewStatus': 'none',
'id': 3882697,
'name': 'IMG_4451.jpeg'
},
{
'reviewStatus': 'none',
'id': 3882698,
'name': 'IMG_3861.jpeg'
},
{
'reviewStatus': 'none',
'id': 3882699,
'name': 'IMG_0748.jpeg'
}
]
)
]Get all active labeling jobs in a team
jobs = api.labeling_job.get_list(TEAM_ID)
print(jobs)Output:
[
LabelingJobInfo(
id=1,
name='labeler1_lemons_task',
readme='annotation manual for fruits in markdown format here (optional)',
description='short description is here (optional)',
team_id=8,
workspace_id=349,
workspace_name='Testing Workspace',
project_id=5555,
project_name='Lemons (Test)',
dataset_id=10555,
dataset_name='ds1',
created_by_id=7,
created_by_login='my_username',
assigned_to_id=101,
assigned_to_login='labeler_1',
created_at='2022-10-05T08:42:30.588Z',
started_at=None,
finished_at=None,
status='pending',
disabled=False,
images_count=6,
finished_images_count=0,
rejected_images_count=0,
accepted_images_count=0,
classes_to_label=[],
tags_to_label=[],
images_range=(None, None),
objects_limit_per_image=None,
tags_limit_per_image=None,
filter_images_by_tags=[],
include_images_with_tags=[],
exclude_images_with_tags=[],
entities=[
{
'reviewStatus': 'none',
'id': 3882702,
'name': 'IMG_8144.jpeg'
},
{
'reviewStatus': 'none',
'id': 3882697,
'name': 'IMG_4451.jpeg'
},
{
'reviewStatus': 'none',
'id': 3882698,
'name': 'IMG_3861.jpeg'
},
{
'reviewStatus': 'none',
'id': 3882700,
'name': 'IMG_2084.jpeg'
},
{
'reviewStatus': 'none',
'id': 3882701,
'name': 'IMG_1836.jpeg'
},
{
'reviewStatus': 'none',
'id': 3882699,
'name': 'IMG_0748.jpeg'
}
]
),
LabelingJobInfo(
id=2,
name='labeler2_kiwi_task_with_complex_settings',
readme='annotation manual for fruits in markdown format here (optional)',
description='short description is here (optional)',
team_id=8,
workspace_id=349,
workspace_name='Testing Workspace',
project_id=5555,
project_name='Lemons (Test)',
dataset_id=10555,
dataset_name='ds1',
created_by_id=100,
created_by_login='my_username',
assigned_to_id=102,
assigned_to_login='labeler_2',
created_at='2022-10-05T08:42:30.588Z',
started_at=None,
finished_at=None,
status='pending',
disabled=False,
images_count=6,
finished_images_count=0,
rejected_images_count=0,
accepted_images_count=0,
classes_to_label=["kiwi"],
tags_to_label=["size", "origin"],
images_range=(None, None),
objects_limit_per_image=10,
tags_limit_per_image=20,
filter_images_by_tags=[],
include_images_with_tags=[],
exclude_images_with_tags=[],
entities=[
{
'reviewStatus': 'none',
'id': 3882697,
'name': 'IMG_4451.jpeg'
},
{
'reviewStatus': 'none',
'id': 3882698,
'name': 'IMG_3861.jpeg'
},
{
'reviewStatus': 'none',
'id': 3882699,
'name': 'IMG_0748.jpeg'
}
]
)
]Labeling Jobs Statuses
api.labeling_job.Status.PENDING - labeling job is created, labeler still has not started api.labeling_job.Status.IN_PROGRESS - labeler started, but not finished api.labeling_job.Status.ON_REVIEW - labeler finished his job, reviewer is in progress api.labeling_job.Status.COMPLETED - reviewer completed job api.labeling_job.Status.STOPPED - job was stopped at some stage
job_id = jobs[-2].id
api.labeling_job.get_status(job_id)Output
<Status.STOPPED: 'stopped'>job_id = jobs[-1].id
api.labeling_job.get_status(job_id)Output:
<Status.PENDING: 'pending'>If you want to change Labeling Job status you can use api.labeling_job.set_status() method
job_id = jobs[-1].id
api.labeling_job.set_status(id=job_id, status="completed")Output:
<Status.COMPLETED: 'completed'>The following methods will wait until labeling job will change status to the given expected status:
Labeler has finished annotating
api.labeling_job.wait(job_id, target_status=api.labeling_job.Status.ON_REVIEW)Reviewer has finished his annotation review
api.labeling_job.wait(job_id, target_status=api.labeling_job.Status.COMPLETED)
Archive Labeling Job
Archive Labeling job by ID. Data can be retrieved after archiving.
api.labeling_job.archive(jobs[0].id)
Remove Labeling Jobs
Labeling job will be removed permanently.
Remove single Labeling Job
Remove Labeling job by ID.
job_id = 278
api.labeling_job.remove(job_id)Remove list of Labeling Jobs
Remove Labeling jobs by IDs.
job_ids = [279, 280, 281]
api.labeling_job.remove_batch(job_ids)Last updated
Was this helpful?