Skip to content

Selfie Segmentation

Overview

Selfie segmentation allows developers to easily separate the background from users within a scene and focus on what matters. Adding cool effects to selfies or inserting your users into interesting background environments has never been easier.

Demo

Install SDK

You can install our SDK for your device by following the guide at: Getting Started.

Downloading the Model

Download the Deep Lab or you can use any segmentation model from the EdgeStore.

Download Model

Running Inference

Place the edgem file in your project or assets and run the following code:

Swift

let modelPath:String = Bundle.main.path(forResource: "deeplabv3-mobilenetv2-semantic-segmentation", ofType: "edgem")!
let model =  EdgeModel(modelPath: modelPath)

// Preparing data for model
let image = // your input image

// Run the model on prepared inputs and get recognitions as output
let results: Results = model.run([image])

// Draw the results on your Image or UI 
let drawnImage = results.draw(on:image)

// Use the results in your app logic:

// access segmentation mask and utils
let segmentation = results.smanticSegmentationMask;

for detection in results.recognitions{
    let label:EEMLabel = detection.label!
    // the name of segmentation mask
    let name:String  = label.name!
}
NSString *modelPath = [[NSBundle mainBundle] pathForResource:@"model" ofType:@"edgem"];
EdgeModel* model = [[EdgeModel alloc] initWithModelPath:modelPath];

// Preparing data for model        
CVPixelBufferRef image = // your input image

// Run the model on prepared inputs and get results as output
Results* results = [model run:@[ (__bridge id)image]];

// Draw the results on your Image or UI 
[results drawOnPixelBuffer:image ];

// Use the results in your app logic:
// access segmentation mask and utils
Segmentation* segmentation = [results smanticSegmentationMask];


for (Recognition* detection in results.recognitions) {
    // name of detected object
    NSString *name = detection.label.name;

}
val model = EdgeModel.fromAsset(context, "deeplabv3-mobilenetv2-semantic-segmentation.edgem")

// Preparing data for model
val image = // your input image

// Run the model on prepared inputs and get recognitions as output
val recognitions: Recognitions = model.run(listOf(image))

// Draw the recognitions on your Image or UI
val drawing: Bitmap = recognitions.drawOnBitmap(image)

// Use the recognitions in your app:
for (detection in recognitions) {
    val id = detection.id!!
    val displayName = detection.displayName!!
    val color = detection.color!!
    val segmentation_mask = detection.segmentation_mask!!
}
EdgeModel model = EdgeModel.fromAsset(context, "deeplabv3-mobilenetv2-semantic-segmentation.edgem");

// Preparing data for model
Bitmap image = // your input image

// Run the model on prepared inputs and get recognitions as output
Recognitions recognitions = model.run(Arrays.asList(image));

// Draw the recognitions on your Image or UI
Bitmap drawing = recognitions.drawOnBitmap(image);

// Use the recognitions in your app:
for (Recognition detection : recognitions) {
    var id = detection.id;
    var displayName = detection.displayName;
    var color = detection.color;
    var segmentation_mask = detection.segmentation_mask;
}
from edgeengine import EdgeModel
import matplotlib.pyplot as plt

model = EdgeModel("deeplabv3-mobilenetv2-semantic-segmentation.edgem")
image = #your input np.ndarray image
results = model.run([image])

# draw results on input image
visualization = results.visualize(image)

# show results
plt.imshow(visualization)
plt.show()

# Full semantic segmentation mask
full_segmentation_mask = results.semantic_segmentation_mask

# Use results
for detection in results.recognitions:
    class_id = detection.id
    display_name = detection.display_name
    color = detection.color
    # per class segmentation mask
    segmentation_mask = detection.segmentation

Supported Image Formats

We support following input formats for images:

  • CvPixelBuffer (Recommended, iOS)

  • UIImage (Slow, iOS)

  • Bitmap (Android)

Interpreting and Using Results

The output results variable has type Recognitions that is a type of List. All the useful

information the model provides is compacted into this variable. Not matter your use case the output of model will always be Recognitions object. You can use segmentation (inside the for loop) data to get mask of each object.

Coordinate System

Our coordinate system is same as used in UIKit. i.e. our origin is at top left corner of image, with x axis to the right and y axis to the bottom.

Coordinates Demo

The recognitions object returned after model execution provides coordinates in image domain. i.e. 0 < x < Input Image Width and 0 < y < Input Image Height.

You can directly apply transformations on Recognitions object by using the Recognitions.map function in Android or Recogntitions.applying in iOS