Keypoints (skeletons)

How to create keypoints annotation in Python using Supervisely

Introduction

In this tutorial we will show you how to use sly.GraphNodes class to create data annotation for pose estimation / keypoints detection task. The tutorial illustrates basic upload-download scenario:

create project and dataset on server
upload image
programmatically create annotation and upload it to image
download image and annotation

ℹ️ Everything you need to reproduce this tutorial is on GitHub: source code, Visual Studio Code configuration, and a shell script for creating virtual env.

How to debug this tutorial

Step 1. Prepare ~/supervisely.env file with credentials. Learn more here.

Step 2. Clone repository with source code and demo data and create Virtual Environment.

git clone https://github.com/supervisely-ecosystem/keypoints-labeling-example
cd keypoints-labeling-example
./create_venv.sh

Step 3. Open repository directory in Visual Studio Code

code -r .

Step 4. Start debugging src/main.py

Python Code

Importing Necessary Libraries

Import necessary libraries:

import supervisely as sly
from supervisely.geometry.graph import Node, KeypointsTemplate
import os
import json
from dotenv import load_dotenv

Before we will start creating our project, let's learn how to create keypoints template - we are going to use it in our project.

Working With Keypoints Template

We will need an image to create and visualize our keypoints template.

Image for building keypoints template:

Create keypoints template:

# initialize template
template = KeypointsTemplate()
# add nodes
template.add_point(label="nose", row=635, col=427)
template.add_point(label="left_eye", row=597, col=404)
template.add_point(label="right_eye", row=685, col=401)
template.add_point(label="left_ear", row=575, col=431)
template.add_point(label="right_ear", row=723, col=425)
template.add_point(label="left_shoulder", row=502, col=614)
template.add_point(label="right_shoulder", row=794, col=621)
template.add_point(label="left_elbow", row=456, col=867)
template.add_point(label="right_elbow", row=837, col=874)
template.add_point(label="left_wrist", row=446, col=1066)
template.add_point(label="right_wrist", row=845, col=1073)
template.add_point(label="left_hip", row=557, col=1035)
template.add_point(label="right_hip", row=743, col=1043)
template.add_point(label="left_knee", row=541, col=1406)
template.add_point(label="right_knee", row=751, col=1421)
template.add_point(label="left_ankle", row=501, col=1760)
template.add_point(label="right_ankle", row=774, col=1765)
# add edges
template.add_edge(src="left_ankle", dst="left_knee")
template.add_edge(src="left_knee", dst="left_hip")
template.add_edge(src="right_ankle", dst="right_knee")
template.add_edge(src="right_knee", dst="right_hip")
template.add_edge(src="left_hip", dst="right_hip")
template.add_edge(src="left_shoulder", dst="left_hip")
template.add_edge(src="right_shoulder", dst="right_hip")
template.add_edge(src="left_shoulder", dst="right_shoulder")
template.add_edge(src="left_shoulder", dst="left_elbow")
template.add_edge(src="right_shoulder", dst="right_elbow")
template.add_edge(src="left_elbow", dst="left_wrist")
template.add_edge(src="right_elbow", dst="right_wrist")
template.add_edge(src="left_eye", dst="right_eye")
template.add_edge(src="nose", dst="left_eye")
template.add_edge(src="nose", dst="right_eye")
template.add_edge(src="left_eye", dst="left_ear")
template.add_edge(src="right_eye", dst="right_ear")
template.add_edge(src="left_ear", dst="left_shoulder")
template.add_edge(src="right_ear", dst="right_shoulder")

Visualize your keypoints template:

template_img = sly.image.read("images/girl.jpg")
template.draw(image=template_img, thickness=7)
sly.image.write("images/template.jpg", template_img)

Explore Keypoints Template in JSON Format

You can also transfer your template to json:

template_json = template.to_json()

Click to see the example of template in json format

{
  "nodes": {
    "nose": {
      "label": "nose",
      "loc": [635, 427],
      "color": "#0000FF"
    },
    "left_eye": {
      "label": "left_eye",
      "loc": [597, 404],
      "color": "#0000FF"
    },
    "right_eye": {
      "label": "right_eye",
      "loc": [685, 401],
      "color": "#0000FF"
    },
    "left_ear": {
      "label": "left_ear",
      "loc": [575, 431],
      "color": "#0000FF"
    },
    "right_ear": {
      "label": "right_ear",
      "loc": [723, 425],
      "color": "#0000FF"
    },
    "left_shoulder": {
      "label": "left_shoulder",
      "loc": [502, 614],
      "color": "#0000FF"
    },
    "right_shoulder": {
      "label": "right_shoulder",
      "loc": [794, 621],
      "color": "#0000FF"
    },
    "left_elbow": {
      "label": "left_elbow",
      "loc": [456, 867],
      "color": "#0000FF"
    },
    "right_elbow": {
      "label": "right_elbow",
      "loc": [837, 874],
      "color": "#0000FF"
    },
    "left_wrist": {
      "label": "left_wrist",
      "loc": [446, 1066],
      "color": "#0000FF"
    },
    "right_wrist": {
      "label": "right_wrist",
      "loc": [845, 1073],
      "color": "#0000FF"
    },
    "left_hip": {
      "label": "left_hip",
      "loc": [557, 1035],
      "color": "#0000FF"
    },
    "right_hip": {
      "label": "right_hip",
      "loc": [743, 1043],
      "color": "#0000FF"
    },
    "left_knee": {
      "label": "left_knee",
      "loc": [541, 1406],
      "color": "#0000FF"
    },
    "right_knee": {
      "label": "right_knee",
      "loc": [751, 1421],
      "color": "#0000FF"
    },
    "left_ankle": {
      "label": "left_ankle",
      "loc": [501, 1760],
      "color": "#0000FF"
    },
    "right_ankle": {
      "label": "right_ankle",
      "loc": [774, 1765],
      "color": "#0000FF"
    }
  },
  "edges": [
    {
      "src": "left_ankle",
      "dst": "left_knee",
      "color": "#00FF00"
    },
    {
      "src": "left_knee",
      "dst": "left_hip",
      "color": "#00FF00"
    },
    {
      "src": "right_ankle",
      "dst": "right_knee",
      "color": "#00FF00"
    },
    {
      "src": "right_knee",
      "dst": "right_hip",
      "color": "#00FF00"
    },
    {
      "src": "left_hip",
      "dst": "right_hip",
      "color": "#00FF00"
    },
    {
      "src": "left_shoulder",
      "dst": "left_hip",
      "color": "#00FF00"
    },
    {
      "src": "right_shoulder",
      "dst": "right_hip",
      "color": "#00FF00"
    },
    {
      "src": "left_shoulder",
      "dst": "right_shoulder",
      "color": "#00FF00"
    },
    {
      "src": "left_shoulder",
      "dst": "left_elbow",
      "color": "#00FF00"
    },
    {
      "src": "right_shoulder",
      "dst": "right_elbow",
      "color": "#00FF00"
    },
    {
      "src": "left_elbow",
      "dst": "left_wrist",
      "color": "#00FF00"
    },
    {
      "src": "right_elbow",
      "dst": "right_wrist",
      "color": "#00FF00"
    },
    {
      "src": "left_eye",
      "dst": "right_eye",
      "color": "#00FF00"
    },
    {
      "src": "nose",
      "dst": "left_eye",
      "color": "#00FF00"
    },
    {
      "src": "nose",
      "dst": "right_eye",
      "color": "#00FF00"
    },
    {
      "src": "left_eye",
      "dst": "left_ear",
      "color": "#00FF00"
    },
    {
      "src": "right_eye",
      "dst": "right_ear",
      "color": "#00FF00"
    },
    {
      "src": "left_ear",
      "dst": "left_shoulder",
      "color": "#00FF00"
    },
    {
      "src": "right_ear",
      "dst": "right_shoulder",
      "color": "#00FF00"
    }
  ]
}

Now, when we have successfully created keypoints template, we can start creating keypoints annotation for our project.

Programmatically Create Keypoints Annotation

Authenticate (learn more here):

load_dotenv(os.path.expanduser('~/supervisely.env'))
api = sly.Api.from_env()
my_teams = api.team.get_list()
team = my_teams[0]
workspace = api.workspace.get_list(team.id)[0]

Input image:

Create project and dataset:

project = api.project.create(workspace.id, "Human Pose Estimation", change_name_if_conflict=True)
dataset = api.dataset.create(project.id, "Person with dog", change_name_if_conflict=True)
print(f"Project {project.id} with dataset {dataset.id} are created")

Now let's create annotation class using our keypoints template as a geometry config (unlike other supervisely geometry classes, sly.GraphNodes requires geometry config to be passed - it is necessary for object class initialization):

person = sly.ObjClass("person", geometry_type=sly.GraphNodes, geometry_config=template)
project_meta = sly.ProjectMeta(obj_classes=[person])
api.project.update_meta(project.id, project_meta.to_json())

You can also go to Supervisely platform and check that class with shape "Keypoints" was successfully added to your project:

Upload image:

image_info = api.image.upload_path(
    dataset.id, name="person_with_dog.jpg", path="images/person_with_dog.jpg"
)

Build keypoints graph:

nodes = [
    sly.Node(label="nose", row=146, col=670),
    sly.Node(label="left_eye", row=130, col=644),
    sly.Node(label="right_eye", row=135, col=701),
    sly.Node(label="left_ear", row=137, col=642),
    sly.Node(label="right_ear", row=142, col=705),
    sly.Node(label="left_shoulder", row=221, col=595),
    sly.Node(label="right_shoulder", row=226, col=738),
    sly.Node(label="left_elbow", row=335, col=564),
    sly.Node(label="right_elbow", row=342, col=765),
    sly.Node(label="left_wrist", row=429, col=555),
    sly.Node(label="right_wrist", row=438, col=784),
    sly.Node(label="left_hip", row=448, col=620),
    sly.Node(label="right_hip", row=451, col=713),
    sly.Node(label="left_knee", row=598, col=591),
    sly.Node(label="right_knee", row=602, col=715),
    sly.Node(label="left_ankle", row=761, col=573),
    sly.Node(label="right_ankle", row=766, col=709),
]

Label the image:

input_image = sly.image.read("images/person_with_dog.jpg")
img_height, img_width = input_image.shape[:2]
label = sly.Label(sly.GraphNodes(nodes), person)
ann = sly.Annotation(img_size=[img_height, img_width], labels=[label])
api.annotation.upload_ann(image_info.id, ann)

You can check that keypoints annotation was successfully created in Annotation Tool:

Download data:

image = api.image.download_np(image_info.id)
ann_json = api.annotation.download_json(image_info.id)

Draw annotation:

ann = sly.Annotation.from_json(ann_json, project_meta)
output_path = "images/person_with_dog_labelled.jpg"
ann.draw_pretty(image, output_path=output_path, thickness=7)

Result:

PreviousSpatial labels on images NextMultispectral images

Last updated 1 year ago