Image and object tags

How to tag image and object using Supervisely SDK

Introduction

In this tutorial we will be focusing on working with tags using Supervisely SDK. We'll go through complete cycle from creating TagMeta in project to assigning Tags to images and objects directly.

You will learn:

1. how to create tags for different tasks and scenarios with various parameters.

2. how to create tags (sly.TagMeta) in project

3. how to assign tags (sly.Tag) to images and objects

📗 Everything you need to reproduce this tutorial is on GitHub: source code and demo data.

How to debug this tutorial

Step 1. Prepare ~/supervisely.env file with credentials. Learn more here.

Step 2. Clone repository with source code and demo data and create Virtual Environment.

git clone https://github.com/supervisely-ecosystem/tutorial-working-with-tags
cd tutorial-working-with-tags
./create_venv.sh

Step 3. Open repository directory in Visual Studio Code.

code -r .

Step 4. Get Lemons (Annotated) project from ecosystem. Lemons (Annotated) is an example project with annotated lemons and kiwi fruits, with 6 images in it.

Step 5. change ✅ project ID ✅ in local.env file by copying the ID from the context menu of the project.

PROJECT_ID=111 # ⬅️ change value

Step 6. Start debugging src/main.py

Part 1. Tag Meta

Import libraries

import os
from dotenv import load_dotenv
import supervisely as sly

Init API client

Init API for communicating with Supervisely Instance. First, we load environment variables with credentials and project ID:

load_dotenv("local.env")
load_dotenv(os.path.expanduser("~/supervisely.env"))
api = sly.Api()

Create Tag Meta

TagMeta object contains general information about Tag. In order to create Tag itself you must create TagMeta object (information about tags that we will create and assign to images or objects) with parameters such as:

• name (required) - name of the Tag.

• value_type (required)- restricts Tag to have a certain value type. Available value types:

  • NONE = "none"

  • ANY_STRING = "any_string"

  • ANY_NUMBER = "any_number"

  • ONEOF_STRING = "oneof_string"

• possible_values (required if value type is "oneof_string") - list of possible Tag values.

• color (optional) - color of the Tag, must be an RGB value, if not specified, random color will be generated.

• applicable_to (optional) - defines if Tag can be assigned to only images, to only objects or both. By default tag can be assigned to both images and objects.

• applicable_classes (optional) - defines applicability of Tag only to certain classes. List of strings (class names).

Let's start with creating a simple TagMeta for showcasing, it can be applied to both images and objects, and also to any class. We won't use it for our project later. Tags with value type "none" can be used as "train" and "val" tags for example.

tag_meta = sly.TagMeta(
    name="simple_tag", 
    value_type=sly.TagValueType.NONE
    )
print(tag_meta)
# Name: simple_tag
# Value type: none
# Possible values: None
# Hotkey
# Applicable to all
# Applicable classes []

Let's make this TagMeta applicable only to images. We can recreate TagMeta with additional parameters. Most supervisely classes are immutable, so you have to assign or reassign them to variables.

tag_meta = sly.TagMeta(
    name="simple_tag", 
    value_type=sly.TagValueType.NONE,
    applicable_to=sly.TagApplicableTo.IMAGES_ONLY
    )

print(tag_meta)
# Name: simple_tag
# Value type: none
# Possible values: None
# Hotkey
# Applicable imagesOnly
# Applicable classes []

Now let's create a few TagMetas with different value types that we will apply to our project.

We can start with creating fruit name TagMeta with "any_string" value type. This Tag can be assigned only to objects of classes "lemon" and "kiwi".

fruit_name_tag_meta = sly.TagMeta(
    name="name",
    applicable_to=sly.TagApplicableTo.OBJECTS_ONLY,
    value_type=sly.TagValueType.ANY_STRING,
    applicable_classes=["lemon", "kiwi"]
)
print(fruit_name_tag_meta)
# Name: name
# Value type: any_string
# Possible values: None
# Hotkey
# Applicable to objectsOnly
# Applicable classes ["lemon", "kiwi"]

Create fruit size TagMeta with "oneof_string" value type. This Tag can be assigned only to objects of any classes and has possible values.

fruit_size_tag_meta = sly.TagMeta(
    name="size",
    applicable_to=sly.TagApplicableTo.OBJECTS_ONLY,
    value_type=sly.TagValueType.ONEOF_STRING,
    possible_values=["small", "medium", "big"]
)
print(fruit_size_tag_meta)
# Name: size
# Value type: oneof_string
# Possible values: ["small", "medium", "big"]
# Hotkey
# Applicable to objectsOnly
# Applicable classes []

Now we create a TagMeta with "any_string" value type to enter the origin of the fruit into it. This Tag can be assigned only to objects of classes "lemon" and "kiwi".

fruit_origin_tag_meta = sly.TagMeta(
    name="imported_from", 
    value_type=sly.TagValueType.ANY_STRING, 
    applicable_to=sly.TagApplicableTo.OBJECTS_ONLY,
    applicable_classes=["lemon", "kiwi"]
    )
print(fruit_origin_tag_meta)
# Name: imported_from
# Value type: any_string
# Possible values: None
# Hotkey
# Applicable to objectsOnly
# Applicable classes ["lemon", "kiwi"]

And one more TagMeta with "any_number" value type for counting total fruits on image. This Tag is applicable only to images.

fruits_count_tag_meta = sly.TagMeta(
    name="fruits_count",
    value_type=sly.TagValueType.ANY_NUMBER,
    applicable_to=sly.TagApplicableTo.IMAGES_ONLY
)
print(fruits_count_tag_meta)
# Name: fruits_count
# Value type: any_number
# Possible values: None
# Hotkey
# Applicable to imagesOnly
# Applicable classes []

Bring all created TagMetas together in a list

tag_metas = [fruit_name_tag_meta, fruit_size_tag_meta, fruit_origin_tag_meta, fruits_count_tag_meta]

Part 2. Add TagMetas to project

Get project meta from server

project_id = sly.env.project_id()
project_meta_json = api.project.get_meta(id=project_id)
project_meta = sly.ProjectMeta.from_json(data=project_meta_json)

Check that created Tag Metas don't already exist in project meta, and if not, add them to project meta.

for tag_meta in tag_metas:
    if tag_meta not in project_meta.tag_metas:
        project_meta = project_meta.add_tag_meta(new_tag_meta=tag_meta)

Update project meta on Supervisely instance after adding Tag Metas to project meta.

api.project.update_meta(id=project_id, meta=project_meta)

Part 3. Create Tags and update annotation on server

# get list of datasets in our project
datasets = api.dataset.get_list(project_id)
dataset_ids = [dataset.id for dataset in datasets]
# iterate over all images in project datasets
for dataset_id in dataset_ids:
    # get list of images in dataset
    images_infos = api.image.get_list(dataset_id=dataset_id)
    for image_info in images_infos:
        # get image id from image info
        image_id = image_info.id

        # download annotation
        ann_json = api.annotation.download_json(image_id=image_id)
        ann = sly.Annotation.from_json(data=ann_json, project_meta=project_meta)
        
        # create and assign Tag to image
        fruits_count_tag = sly.Tag(meta=fruits_count_tag_meta, value=len(ann.labels))
        ann = ann.add_tag(fruits_count_tag)

        # iterate over objects in annotation and assign appropriate tag
        new_labels = []
        for label in ann.labels:
            new_label = None
            if label.obj_class.name == "lemon":
                name_tag = sly.Tag(meta=fruit_name_tag_meta, value="lemon")
                size_tag = sly.Tag(meta=fruit_size_tag_meta, value="medium")
                origin_tag = sly.Tag(meta=fruit_origin_tag_meta, value="Spain")
                new_label = label.add_tags([name_tag, size_tag, origin_tag])
            elif label.obj_class.name == "kiwi":
                name_tag = sly.Tag(meta=fruit_name_tag_meta, value="kiwi")
                size_tag = sly.Tag(meta=fruit_size_tag_meta, value="small")
                origin_tag = sly.Tag(meta=fruit_origin_tag_meta, value="Italy")
                new_label = label.add_tags([name_tag, size_tag, origin_tag])
            if new_label:
                new_labels.append(new_label)

        # update and upload ann to Supervisely instance
        ann = ann.clone(labels=new_labels)
        api.annotation.upload_ann(img_id=image_id, ann=ann)

Retrieve images with object tags of interest

Sometimes, you may need to filter your images to retrieve only those that feature specific tags or meet certain criteria.

The code snippet below demonstrates how to get images featuring objects with multiple tags attached:

target_imageids = []
# Iterate over each dataset
for dataset in api.dataset.get_list(project_id):
    # Get only images that contain object tags
    filters = [{"type": "objects_tag", "data": {"include": True}}]
    images = api.image.get_filtered_list(dataset.id, filters)
    # Iterate over images in batches
    for batch_infos in sly.batched(images, batch_size=500):
        batch_ids = [info.id for info in batch_infos]
        # Download annotations for batch images
        ann_infos = api.annotation.download_batch(dataset.id, batch_ids)
        # Extract image ID if any object on an image contains more than 1 tag
        for ann_info in ann_infos:
            ann = sly.Annotation.from_json(ann_info.annotation, project_meta)
            if any([len(label.tags) > 1 for label in ann.labels]):
                target_imageids.append(ann_info.image_id)
print(target_imageids)
# [31927, 31933, 31968, 32009, 32155, 32366]

Advanced API

Advanced API allows user to manage tags directly on images or objects without downloading annotation data from server.

Get project meta again after updating it with new tags.

project_meta_json = api.project.get_meta(id=project_id)
project_meta = sly.ProjectMeta.from_json(data=project_meta_json)

Create TagCollection from image Tags without downloading annotation

Get image id from image info (see part 3)

image_id = image_info.id

Get image tags

image_tags = image_info.tags
print(f"{image_info.name} tags: {image_tags}")
# IMG_1836.jpeg tags: 
# [
#   {
#       'entityId': 3315606, 
#       'tagId': 369190,
#       'id': 2298259,
#       'value': 3,
#       'labelerLogin': 'cxnt',
#       'createdAt': '2022-10-04T15:43:12.155Z',
#       'updatedAt': '2022-10-04T15:43:12.155Z'
#   }
# ]

Create TagCollection from image tags

tag_collection = sly.TagCollection().from_api_response(
    data=image_tags, tag_meta_collection=project_meta.tag_metas
)
print(tag_collection)
# +--------------+------------+-------+
# |     Name     | Value type | Value |
# +--------------+------------+-------+
# | fruits_count | any_number |   3   |
# +--------------+------------+-------+

Add Tag directly to image

Get image tag ID from project meta

fruits_count_tag_meta = project_meta.get_tag_meta("fruits_count")

Add Tag to image using Tag supervisely ID from project meta

api.image.add_tag(image_id=image_id, tag_id=fruits_count_tag_meta.sly_id, value=3)

Add Tags to objects directly

ann_json = api.annotation.download_json(image_id=image_id)
ann = sly.Annotation.from_json(data=ann_json, project_meta=project_meta)
# iterate over objects in annotation and add appropriate tag
for label in ann.labels:
    # get figure sly id
    figure_id = label.geometry.sly_id
    # get tag sly id
    tag_meta = project_meta.get_tag_meta("imported_from")
    if label.obj_class.name == "lemon":
        api.advanced.add_tag_to_object(tag_meta_id=tag_meta.sly_id, figure_id=figure_id, value="Spain")
    elif label.obj_class.name == "kiwi":
        api.advanced.add_tag_to_object(tag_meta_id=tag_meta.sly_id, figure_id=figure_id, value="Italy")

Add Tag to set of images

With api.image.add_tag_batch() method you can add a tag to a list of images without need to update annotation of each image one by one.

# get tag meta from project meta
tag_meta = project_meta.get_tag_meta("fruits")

# create a list of images ids from images infos
image_ids = [image_info.id for image_info in images_infos]

# get tag meta id
tag_meta_id = tag_meta.sly_id

# update tags in batches.
api.image.add_tag_batch(image_ids, tag_meta_id, value=None, tag_meta=tag_meta)