Tensorboard template
Step-by-step tutorial explains how to use custom training script and log results in Tensorboard
Last updated
Was this helpful?
Step-by-step tutorial explains how to use custom training script and log results in Tensorboard
Last updated
Was this helpful?
This tutorial will teach you how to integrate your custom training script into Supervisely Ecosystem. The following procedure can be used with any Neural Network architecture and for any Computer Vision task.
It is the simplest integration with of NN training with Supervisely, that do not require any special modifications of your source codes.
📗 GitHub source code of tensorboard training template can be found .
The high level overview of the procedure is the following:
Take input directory (--input-dir
) with training data in Supervisely format (see python3 src/train.py
command at run.sh
).
Transform labeled data (in Supervisely format) to any format you need
Train your model (use your training script almost without modifications).
Save artifacts (checkpoints and tensorboard metrics) to the output directory (--output-dir
).
After the training all artefacts will be automatically uploaded to Supervisely platform to Team Files.
Full code of training tensorboard template can be found .
Note, that you can always load your previous logs just by simply specifying HISTORY_DIR
in run.sh
. Here how it will look like in the tensorboard interface:
Step 3. Open the repository directory in Visual Studio Code.
Step 5. Check self-explanatory run.sh
script to get the idea how app works. You can modify it the way you need. The resulted directory with output artefacts data will have the following path: "/my-training/$PROJECT_ID-$PROJECT_NAME/$TASK_ID"
. Note that you can always change the DST_DIR
in the run.sh
to suit your needs in any way.
You should also note that in case if you do not have any history logs. (i.e. *.tfevents.*
files), the script will automatically ignore non-existence of the history folder (HISTORY_DIR
). It means that you do not need to bother about additional run.sh
customization!
Step 6. Configure your training script
Modify src/train.py
with your own training loop:
Step 7. Start debugging.
Step 8. Watch tensorboard while training.
Step 9. Open output artefacts in Team files.
Choose your folder in Team Files containing tensorboard logs or the file itself
Right-click on the object and click on three-dot menu. Then, choose Run App -> Tensorboard
. Click Run
.
After running, the tensorboard server will be available with Open
button in workspace. Click on it.
That's it! Now you can view your tensorboard logs.
Step 10. Release your private app
Step 11. Run app on your Supervisely instance
Choose your project and click on three-dot menu. Then, choose Run App -> Training tensorboard template
and, if you need, specify selected Advanced Settings
. Click Run
.
Step 12. Open Tensorboard while training
Wait until your project will be downloaded and your tensorboard logging server will start. You can open it in Workspace Tasks
interface with clicking Open
button.
Step 13. Open link with output artefacts in Team files
Step 1. Prepare ~/supervisely.env
file with credentials.
Step 2. Clone with source code and create .
Step 4. Change variables in local.env
to your values. PROJECT_ID - id of the project with training data, TEAM_ID - id of the team where the project is located.
Tensorboard is available in browser using address
You can always examine your logs by simply using . To do that, just follow this steps (learn more in app readme):
Just run the following command in the root directory of you app. Learn more in .
After successful task ending, Tensorboard server stops and there will be a direct link to a Team files folder. You can always examine your logs by simply using .