OpenVINO Inference

Once we have the trained model, all we have to do is build a pipeline that can take a frame as input and return another image with the plotted result.
We can directly use the TensorFlow models, but we have used the OpenVINO runtime for inference because of the range of tools available that can help us improve inference and make deployment easier.
To use OpenVINO inference, we will need our model to be serialized in OpenVINO Intermediate representation, i.e., an XML file, which is a standardized file format for representing neural network models optimised for use with Intel hardware. This OpenVINO-IR file makes the solution cross-platform, as it can be used on any system with the OpenVINO runtime.

After getting OpenVINO-IR we can use the models in our pipeline to achieve desired results.

def pipeline(frame):
    '''
    Complete pipeline
    Image --> Detection --> Cropping --> Semantic-Segmentation --> Reading --> Plotting

    Parameters
        frame (np.ndarray) : image

    Retursn
        plotted_image (np.ndarray) : image with detection box and reading
    '''


    # Getting Detection BBOXs
    frame = cv2.resize(frame,(1920,1080),cv2.INTER_AREA)
    prediction = detector(frame.reshape(1,1080,1920,3))
    result_ratio = prediction['detection_boxes'][0]
    scores = prediction['detection_scores'][0]
    selected_rr = []

    for i in range(len(scores)):
        if scores[i]>0.5:
            selected_rr.append(result_ratio[i])
        else:
            break
    try:
        results = np.multiply(selected_rr,[1082,1920,1082,1920]).astype(np.int64)
    except:
        return(frame)

    # Cropping Meters
    roi_imgs,loc = roi_crop(frame,results,1,1)

    # Preprocess uneven cropped imgs to 256,256
    crop_img = []
    for roi_img in roi_imgs:
        resized = cv2.resize(roi_img,(256,256),cv2.INTER_AREA)
        crop_img.append(resized)

    if len(crop_img) > 0:
        # Getting Segmentation Maps
        pred_seg = segmentor(np.array(crop_img))[segmentor.output(0)]
        processed_seg = []
        for i in pred_seg:
            processed_seg.append(cv2.resize(i,(512,512),cv2.INTER_AREA))
        pred = np.argmax(processed_seg,axis=3)

        # Getting Reading from predicted Maps
        pred = erode(pred,2)
        rectangle_meters = circle_to_rectangle(pred)
        line_scales, line_pointers = rectangle_to_line(rectangle_meters)
        binaried_scales = mean_binarization(line_scales)
        binaried_pointers = mean_binarization(line_pointers)
        scale_locations = locate_scale(binaried_scales)
        pointer_locations = locate_pointer(binaried_pointers)
        pointed_scales = get_relative_location(scale_locations, pointer_locations)
        meter_readings = calculate_reading(pointed_scales)

        # Plotting reading and BBOXs on image
        plotted_img = plot_result(frame,meter_readings,results)

    else:
        plotted_img = frame
    return plotted_img

Pipeline will first preprocess the image for detection, then crop the detected part and preprocess it for segmentation, then use the predicted segmentation map for calculating readings, and finally plot the results. For more details on the code and inference process, visit here.