Segment Anything 2

These instructions are based on the detailed instructions on the Meta github: https://github.com/facebookresearch/segment-anything-2.

Download and install miniconda from https://docs.anaconda.com/miniconda/. Select the “Miniconda3 Windows 64-bit” version for the Windows platform.

This will install the Anaconda Prompt (minoconda3) App. Run this App which will open a command line interface.

Create a conda environment and give it a name like “sam2” by typing:

conda create --name sam2 python=3.10

Then activate this environment by typing:

activate sam2

Please follow the instructions to install both PyTorch and TorchVision dependencies here: https://pytorch.org/get-started/locally/
Select all the correct elements in the table and it will provide a conda or pip install command to install PyTorch and TorchVision such as:

conda install pytorch torchvision torchaudio pytorch-cuda=12.4 -c pytorch -c nvidia

If you don’t have cuda installed yet you can download cuda 12.4 here: https://developer.nvidia.com/cuda-12-4-0-download-archive

Visit https://github.com/facebookresearch/segment-anything-2 and download the ZIP file using the green button.
Create a directory like C:/SAM2 and unzip this file into this directory.

You can then install all the requirements of segment anything 2 by going into the segment-anything-2-main folder and typing:

pip install -e . (do not forget the space and .)

Download the checkpoints and place them in the checkpoints folder:
sam2_hiera_tiny.pt
sam2_hiera_small.pt
sam2_hiera_base_plus.pt
sam2_hiera_large.pt

Now download the sam2_server.py file and place it in the segment-anything-2-main folder. You can also find this file in the associated GitHub: https://github.com/TristanWhitmarsh/segment-anything-2-server/tree/main

Install Flask (https://pypi.org/project/Flask/), which handles the server calls by typing:

pip install flask

Load the image and adjust the contrast and brightness by dragging the mouse while holding the right button to optimize the visibility for the structures you want to segment.

The Segment Anything section can be found in the “Segment” tab. Leave the URL as it is unless you have manually changed it in the sam2_server.py code. Select the model in the dropdown menu. The tiny model is the fastest and seems to be sufficient for most tasks.

Optionally you can constrain the segmentation to a region of interest by pressing the “Show box” button on the right of the model selection dropdown menu and dragging the box to define the region. Keep in mind that segment anything 2 will resize every image to 1024 x 1024 pixels. So in some cases, such as with CT scans, this would not matter, while in others, like pathology images, it will have an effect on the level of detail of the resulting segmentation.

Press the “Initialize Segment Anything” button next to the “Show box” button. After a few seconds you will be able to start the segmentation.

The first button allows you to select a region by dragging a box with the left mouse button pressed. A right mouse press will remove the box.

The second button lets you set points. With the left mouse button you add points that define the segmentation, while with the right you set points in areas that should be outside of the segmentation.

You can change the label value in the top left corner of the “Segment” tab to add different regions.

With the third button you can add the current mask to Segment Anything with every label that is visible in the mask. For this you can also use the normal paint tool to draw a segmentation that can then be tracked through other sections or timepoints.

The fourth button will apply the annotations to the entire volume and timpoints. After this process you can no longer add new labels or define regions. You can, however, correct segmentation in other slices or timepoints using the point tool and even re-run this function to apply the corrections throughout.

The fifth button will reset everything.

Press the “Initialize Segment Anything” button again to stop Segment Anything 2.