Ctrlk
About SuperviselyEcosystemContact usSlack
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Inference API

Inference API

Introduction

There are two ways how you can infer your models:

In this tutorial, you'll learn how to infer deployed models from your code with the sly.nn.inference.Session class. This class is a convenient wrapper for a low-level API. It under the hood is just a communication with the serving app via requests.

Before starting you have to deploy your model with a Serving App (e.g. Serve YOLOv5)

Try with Colab: Open In Colab

Table of Contents:

Let's start with a quick example of how you can connect and make inference of your model!

Quick overview

(for detailed tutorial go to the next section)

Example usage: visualize prediction

Image with predictions of the YOLOv5 model

List of all inference methods

Image inference methods:

Video inference methods:

A Complete Tutorial

1. Initialize sly.nn.inference.Session

First serve the model you want (e.g. Serve YOLOv5) and copy the task_id from the App sessions section in the Supervisely platform:

Copy the Task ID here

Init your sly.Api:

(for more info see Basics of authentication tutorial)

Create an Inference Session, a connection to the model:

(Optional) You can pass the inference settings in init:

Or with a YAML file:

2. Get the model info

Session info

Each app with a deployed model has its own unique task_id (or session_id which is the same), model_name, pretrained_dataset and other useful info that can be obtained with the get_session_info() method.

Model Meta. Classes and tags

The model may be pretrained on various datasets, like a COCO, ImageNet or even your custom data. Datasets are different in classes/tags they have. Therefore each dataset has its own meta information called project_meta in Supervisely. The model also contains this information and it's called model_meta. You can get the model_meta with method get_model_meta():

The model_meta will be used later, when we will visualize model predictions.

Inference settings

Each model has its own inference settings, like a conf_thres, iou_thres and others. You can get the full list of supported settings with get_default_inference_settings():

Set the inference settings

There are 3 ways to set the inference settings:

  • update_inference_settings(**kwargs)

  • set_inference_settings(dict)

  • set_inference_settings(YAML)

Also you can pass it earlier at creating the Session.

a) Update only the parameters you need:

Output:

b) Set parameters with a dict:

Output:

c) Set parameters with a YAML file:

Output:

3. Image Inference

There are several ways how to infer an image:

  • by Supervisely ID

  • by local path

  • by URL from the web

And you can also infer a batch of images:

Inspecting the model prediction

The prediction is a sly.Annotation object. It contains all labels and tags for an image and can be uploaded directly to the Supervisely platform.

(see more in SDK reference)

Visualize model prediction

sly.Annotation has a draw_pretty() method for convenient visualization routines:

Image with predictions of the YOLOv5 model

Upload prediction to the Supervisely platform

Now you can upload the image with predictions to the Supervisely platform:

Note: when you update a project_meta with api.project.update_meta() the server generates ids for the classes and tags that have pushed for the first time and you have to update the model_meta too for the further uploading a prediction. This is where api.project.pull_meta_ids() method is helpful. It assigns the ids directly to the model_meta object. Because of all predictions have a reference to the model_meta, without this step we can't upload the predictions to the platform as predictions' ProjectMeta will not have the ids.

Result on the Supervisely platform:

Result in the Supervisely Labeling Tool

4. Video Inference

Method 1. Inferring video with iterator

The video inference is simple too.

The first way is to infer the video with inference_video_id_async method. It returns an iterator, which can be useful in processing predictions frame by frame. As soon as the model done with a one frame it will be yielded by the iterator:

There are some parameters can be passed to the video inference:

  • start_frame_index: the first frame to start

  • frames_count: total frames to infer

  • frames_direction: video playback direction, either "forward" or "backward"

Getting more information about the inference process:

Stop video inference

If you need to stop the inference, use session.stop_async_inference():

Method 2. Inferring video without iterator

If you don't need to iterate every frame, you can use the inference_video_id method:

Note: it is recommended to use this method for very small videos, because the code will wait until the whole video has been inferred and you even can't to track the progress.

5. Project Inference

Method 1. Inferring project with iterator

There is extra parameter that can be passed to the project inference:

  • dest_project_id: destination project id. If not passed, iterator will return annotation infos. If dest_project_id is equal to project_id, iterator will upload annotations to images in the project. If it is different from project_id, iterator will copy images and upload annotations to the new project.

Method 2. Inferring project without iterator

If you don't need to iterate every image, you can use the inference_project_id method:

Advanced. Working with raw JSON output

SessionJSON

There is a sly.nn.inference.SessionJSON class which is useful when it needed to work with raw json outputs.

The class has all the same methods as Session, it just returns a raw JSONs.

The prediction is a dict with the following fields:

  • "annotation": contains a predicted annotation, that can be easily converted to sly.Annotation.

  • "data": additional metadata of the prediction. In most cases you won't need this.

PreviousHow to create custom user interfaceNextTraining App

Last updated