In this tutorial, you'll learn how to infer deployed models from your code with the sly.nn.inference.Session class. This class is a convenient wrapper for a low-level API. It under the hood is just a communication with the serving app via requests.
Before starting you have to deploy your model with a Serving App (e.g.Serve YOLOv5)
# Inference image_idimage_id =19386161prediction = session.inference_image_id(image_id)# prediction is a `sly.Annotation` object# Download and load the image that was inferredsave_path ="demo_image.jpg"api.image.download_path(image_id, path=save_path)image_np = sly.image.read(save_path)# Draw the annotation and save it to the disksave_path_predicted ="demo_image_pred.jpg"predicted_annotation.draw_pretty(bitmap=image_np, output_path=save_path_predicted, fill_rectangles=False, thickness=7)
# Showfrom matplotlib import pyplot as pltimage_pred = sly.image.read(save_path_predicted)plt.imshow(image_pred)plt.axis('off');
List of all inference methods
Image inference methods:
# Infer single image by local pathpred = session.inference_image_path("image_01.jpg")# Infer batch of images by local pathspred = session.inference_image_paths(["image_01.jpg", "image_02.jpg"])# Infer image by IDpred = session.inference_image_id(17551748)# Infer batch of images by IDspred = session.inference_image_ids([17551748, 17551750])# Infer image by urlurl = "https://images.unsplash.com/photo-1674552791148-c756b0899dba?ixlib=rb-4.0.3&ixid=MnwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8&auto=format&fit=crop&w=387&q=80"
pred = session.inference_image_url(url)
Video inference methods:
from tqdm import tqdmvideo_id =18635803# Infer video getting each frame as soon as it's readyfor frame_pred intqdm(session.inference_video_id_async(video_id)):print(frame_pred)# Infer video without iteratorpred = session.inference_video_id(video_id)
A Complete Tutorial
1. Initialize sly.nn.inference.Session
First serve the model you want (e.g. Serve YOLOv5) and copy the task_id from the App sessions section in the Supervisely platform:
Create an Inference Session, a connection to the model:
# Get your Serving App's task_id from the Supervisely platformtask_id =27209# create sessionsession = sly.nn.inference.Session(api, task_id=task_id)
(Optional) You can pass the inference settings in init:
# pass settings by dictinference_settings ={"conf_thres":0.45}session = sly.nn.inference.Session(api, task_id=task_id, inference_settings=inference_settings)
Or with a YAML file:
# pass settings by YAMLinference_settings_yaml ="settings.yml"session = sly.nn.inference.Session(api, task_id=task_id, inference_settings=inference_settings_yaml)
2. Get the model info
Session info
Each app with a deployed model has its own unique task_id (or session_id which is the same), model_name, pretrained_dataset and other useful info that can be obtained with the get_session_info() method.
The model may be pretrained on various datasets, like a COCO, ImageNet or even your custom data. Datasets are different in classes/tags they have. Therefore each dataset has its own meta information called project_meta in Supervisely. The model also contains this information and it's called model_meta. You can get the model_meta with method get_model_meta():
model_meta = session.get_model_meta()print("The first 10 classes of the model_meta:")[cls.name for cls in model_meta.obj_classes][:10]
The first 10 classes of the model_meta:
['person',
'bicycle',
'car',
'motorcycle',
'airplane',
'bus',
'train',
'truck',
'boat',
'traffic light']
The model_meta will be used later, when we will visualize model predictions.
Inference settings
Each model has its own inference settings, like a conf_thres, iou_thres and others. You can get the full list of supported settings with get_default_inference_settings():
# Infer image by local pathpred = session.inference_image_path("image_01.jpg")# Infer image by IDpred = session.inference_image_id(image_id=17551748)# Infer image by urlurl = "https://images.unsplash.com/photo-1674552791148-c756b0899dba?ixlib=rb-4.0.3&ixid=MnwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8&auto=format&fit=crop&w=387&q=80"
pred = session.inference_image_url(url)
And you can also infer a batch of images:
# Infer batch of images by local pathspred = session.inference_image_paths(["image_01.jpg", "image_02.jpg"])# Infer batch of images by IDspred = session.inference_image_ids([17551748, 17551750])
Inspecting the model prediction
The prediction is a sly.Annotation object. It contains all labels and tags for an image and can be uploaded directly to the Supervisely platform.
sly.Annotation has a draw_pretty() method for convenient visualization routines:
# Draw the annotation and save it to disksave_path_predicted ="demo_image_pred.jpg"prediction.draw_pretty(bitmap=image_np, output_path=save_path_predicted, fill_rectangles=False, thickness=7)# Showfrom matplotlib import pyplot as pltimage_pred = sly.image.read(save_path_predicted)plt.imshow(image_pred)plt.axis('off');
Upload prediction to the Supervisely platform
Now you can upload the image with predictions to the Supervisely platform:
workspace_id =662# Create new project and datasetproject_info = api.project.create(workspace_id, "My model predictions", change_name_if_conflict=True)dataset_info = api.dataset.create(project_info.id, "First dataset")# Update project meta with model's classesapi.project.update_meta(project_info.id, model_meta)api.project.pull_meta_ids(project_info.id, model_meta)# Upload the imageimage_name = os.path.basename(image_path)img_info = api.image.upload_path(dataset_info.id, name=image_name, path=image_path)# Upload model predictions to Superviselyapi.annotation.upload_ann(img_info.id, prediction)
Note: when you update a project_meta with api.project.update_meta() the server generates ids for the classes and tags that have pushed for the first time and you have to update the model_meta too for the further uploading a prediction. This is where api.project.pull_meta_ids() method is helpful. It assigns the ids directly to the model_meta object. Because of all predictions have a reference to the model_meta, without this step we can't upload the predictions to the platform as predictions' ProjectMeta will not have the ids.
Result on the Supervisely platform:
4. Video Inference
Method 1. Inferring video with iterator
The video inference is simple too.
The first way is to infer the video with inference_video_id_async method. It returns an iterator, which can be useful in processing predictions frame by frame. As soon as the model done with a one frame it will be yielded by the iterator:
from tqdm import tqdmvideo_id =18635803pred_frames = []for frame_ann intqdm(session.inference_video_id_async(video_id)): pred_frames.append(frame_ann)
There are some parameters can be passed to the video inference:
start_frame_index: the first frame to start
frames_count: total frames to infer
frames_direction: video playback direction, either "forward" or "backward"
Getting more information about the inference process:
video_id =18635803video_info = api.video.get_info_by_id(video_id)frame_iterator = session.inference_video_id_async(video_id)total_frames = video_info.frames_countfor i, frame_ann inenumerate(frame_iterator): labels = frame_ann.labels predicted_classes = [x.obj_class.name for x in labels]print(f"Frame {i+1}/{total_frames} done. Predicted classes = {predicted_classes}")
If you need to stop the inference, use session.stop_async_inference():
from tqdm import tqdmvideo_id =18635803for i, frame_ann inenumerate(tqdm(session.inference_video_id_async(video_id))):if i ==2: session.stop_async_inference()
{"message": "The video is preparing on the server, this may take a while...", "timestamp": "2023-02-09T23:15:47.232Z", "level": "info"}
{"message": "Inference has started:", "progress": {"current": 0, "total": 10}, "is_inferring": true, "cancel_inference": false, "result": null, "pending_results": [], "timestamp": "2023-02-09T23:15:55.878Z", "level": "info"}
20%|██ | 2/10 [00:03<00:13, 1.63s/it]{"message": "Inference will be stopped on the server", "timestamp": "2023-02-09T23:16:01.559Z", "level": "info"}
30%|███ | 3/10 [00:05<00:13, 1.88s/it]
Method 2. Inferring video without iterator
If you don't need to iterate every frame, you can use the inference_video_id method:
Note: it is recommended to use this method for very small videos, because the code will wait until the whole video has been inferred and you even can't to track the progress.
5. Project Inference
Method 1. Inferring project with iterator
from tqdm import tqdmproject_id =18635pred_ann_infos = []for ann_info intqdm(session.inference_project_id_async(project_id)): pred_ann_infos.append(ann_info)
There is extra parameter that can be passed to the project inference:
dest_project_id: destination project id. If not passed, iterator will return annotation infos. If dest_project_id is equal to project_id, iterator will upload annotations to images in the project. If it is different from project_id, iterator will copy images and upload annotations to the new project.
Method 2. Inferring project without iterator
If you don't need to iterate every image, you can use the inference_project_id method:
