recognition error improvement

[keides2]

see #18 (comment)

[szaza]

Thanks for your pull request, now your proposal is clear. During the review I was thinking about your solution and finally, it seems for me like a workaround for the real problem. In my opinion the main problem is the accuracy of the trained model. With this solution we only put a threshold above the recognized classes in order to hide the possibly wrong values. Tiny-yolov2 is not so accurate, by using another model we can avoid this workaround.
Thank you very much for your contribution!

[szaza]

We could create a new type to wrap these values. I'm not sure that the best solution is to declare several helper variables at the beginning of the file. Should all of them be class member?

[keides2]

lastRecognizedClass, matchCount, lastResult, and nowResult were made class members because it is necessary to refer to the previously held value each time the run () function is called in the background.

tts can be declared in private functions.

Since masg1 and msg2 need two to display in two lines and I can not think of a way to return two as the return value of the makeTts () function, I made them class members.

[szaza]

nowRecognizedClass looks strange, probably currentRecognizedClass would be better. Probably, lastRecognizedClass and currentRecognizedClass could be stored in a single structure or array.

[keides2]

It turns out that the currentRecognizedClass is better.
By the way, I reviewed the logic of continuous match.
In the last time, even if it was judged as continuous match, the display was not updated and the utterance was not made if it was the same as the previous display result, but this time, even if it is the same as the previous display result, the display is updated every time it is judged as continuous match I would like to make the smartphone speak.

The run () function is now as follows: I reduced the if statement.

    private void run () {
        final long startTime = SystemClock.uptimeMillis ();
        final List <Recognition> results = recognizer.recognizeImage (croppedBitmap);
        // lastProcessingTimeMs = SystemClock.uptimeMillis ()-startTime;
        overlayView.setResults (results);

        // shimatani
        Log.d (LOGGING_TAG, "Debug: runInBackground ()");

        if (! (results.isEmpty ())) {
            nowResult = results.get (0) .getTitle ();
            Log.d (LOGGING_TAG, String.format ("Find:% s", nowResult));

            if (lastResult.equals (nowResult)) {
                matchCount ++;
                if (matchCount> 3) {
                    // match 4 times
                    matchCount = 0;

                    Log.d (LOGGING_TAG, "Match 4 times:" + nowResult);
                    lastRecognizedClass = nowResult;

                    String tts = makeTts (nowResult);
                    Log.d (LOGGING_TAG, "makeTts ():" + tts);
                    if (! (tts.equals (""))) {
                        speak2 (results, tts);
                        tts = "";
                    }
                } else {
                    // nothing to do
                }
            } else {
                matchCount = 0;
            }
            lastResult = nowResult;
            Log.d (LOGGING_TAG, String.format ("matchCount:% d", matchCount));
        }

        requestRender ();
        computing = false;
    }

[szaza]

Please convert kanji characters to English ones to become readable for everybody.

[keides2]

Display the trash bin number and talk the displayed contents.

            case "C-type remote control": // 0
                msgInf1 = "C-type remote control case gabage number is 13,";
                msgInf2 = "The printed circuit board is number 19";
                break;
            case "Hands and leather glove": // 1
                msgInf1 = "Garmour, cut resistant, skinned";
                msgInf2 = "No. 7";
                break;
            case "marker pen ": // 2
                msgInf1 = "marker, fluorescence, water, oil, correction pen,";
                msgInf2 = "A mechanical pencil, a ballpoint pen is number 13.";
                break;
            case "G-type remote control": // 3
                msgInf1 = "G-type remote control case garbage number is 13,";
                msgInf2 = "The printed circuit board is number 19";
                break;
            case "E-type remote control": // 4
                msgInf1 = "E-type remote control case garbage number is 13,";
                msgInf2 = "The printed circuit board is number 19";
                break;
            case "Button battery": // 5
                msgInf1 = "The button battery that does not contain mercury is number 23.";
                msgInf2 = "The battery is also number 23";
                break;
            case "dry cell": // 6
                msgInf1 = "The dry cell is number 23";
                msgInf2 = "The button battery that does not contain mercury is also number 23.";
                break;
            case "tie wrap": // 7
                msgInf1 = "White tie wrap is 10";
                msgInf2 = "A non-white tie wrap is number 12.";
                break;
            case "PP band": // 8
                msgInf1 = "PP band is number 9";
                msgInf2 = "";
                break;
            default:
                msgInf1 = "";
                msgInf2 = "";
                break;

[keides2]

It depends on very individual circumstances.

[szaza]

What kind of information is represented in the msgInf1 and msgInf2 variables?

[keides2]

It is because the descriptive text has become long.
The second line in the image below is msgInf1 and the third line is msgInf2.
https://github.com/keides2/android-yolo-v2/blob/master/sample/%E2%98%85%E7%B2%98%E7%9D%80%E3%83%86%E3%83%BC%E3%83%97%E3%81%A8%E3%83%9C%E3%82%BF%E3%83%B3%E9%9B%BB%E6%B1%A02.png

[szaza]

nowResult looks strange, currentResult would be better name. Probably, lastResult and currentResult could be stored in the same structure (class, array,...)

[keides2]

It is fine as you say.

[szaza]

Why do we need msgInf1 and msgInf2?

[keides2]

It is because the descriptive text has become long.
The second line of the display in the lower screen is msg1, the third line is msg2.

img

[szaza]

If the default value for tts is null then you don't need to check its value with equals.

[keides2]

Depending on the specifications of the makeTts()function, the return value may be null. In this case, I made it so as not to talk. (I made sure that the speak2() function does not get an error)

[szaza]

speak2 method name looks very strange. Is there speak1, speak2, speak3...? What is the difference between the speak methods?

[keides2]

I have prepared speak2 () function to speak two lines of message in Japanese.
I thought it would be better to change the name of the function.

TextToSpeechActivity.java

    protected void speak(List<Recognition> results) {
        if (!(results.isEmpty() || lastRecognizedClass.equals(results.get(0).getTitle()))) {
            lastRecognizedClass = results.get(0).getTitle();
            if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.LOLLIPOP) {
                textToSpeech.speak(lastRecognizedClass, TextToSpeech.QUEUE_FLUSH, null, null);
            } else {
                textToSpeech.speak(lastRecognizedClass, TextToSpeech.QUEUE_FLUSH, null);
            }
            Log.d(LOGGING_TAG, "speak: " + lastRecognizedClass);    // shimatani
        }
    }

    protected void speak2(List<Recognition> results, String tts) {
        this.tts = tts;
        Log.d(LOGGING_TAG, "speak2 entry: " + this.tts);
        if (!(results.isEmpty() || this.tts.equals(results.get(0).getTitle()))) {
            if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.LOLLIPOP) {
                textToSpeech.speak(this.tts, TextToSpeech.QUEUE_FLUSH, null, null);
            } else {
                textToSpeech.speak(this.tts, TextToSpeech.QUEUE_FLUSH, null);
            }
            Log.d(LOGGING_TAG, "speak2: " + this.tts);    // shimatani
        }
    }

[szaza]

So you do not even have to leave here this else clause.

[keides2]

I followed the old Japanese C language coding standards. I also think that it is unnecessary now.

[szaza]

It looks for me more like a workaround to have a threshold above the recognized classes. The main problem is either the model is not well trained thus it often gives you bad results or the threshold is not properly set in your configuration file.
Another question is why you chose the 3 magic number as comparison value?
Unfortunately, I cannot consider these changes as a good solution for the real problem.

[keides2]

As you say, I think this is not an essential improvement.

I think that it is originally to make a model with high accuracy, but I have taken easier improvement measures than improving the accuracy of the model.

During operation I noticed that this application was immediately judging if it detected an object even before holding it over the object.
Therefore, I tried to judge it as an object (speaking) when it matched four times (the value for comparison is 3).

I will reselect the images for learning and try to improve the accuracy of the model. Please let me know if you know any other effective method.

[keides2]

Hi,
Excuse me for a personal topic. I am currently traveling to Central Europe and will go to Budapest tomorrow. By the way, the name of the bus driver is zoltan!

[szaza]

Yes, that easily can happen because Zoltan is a very common hungarian person name. There are many Zoltans also in Romania because Transilvania was part of Hungary before the Second World War, so there are many hungarian people in Transilvania as well. Enjoy your trip in Budapest, that is a very beautiful city. BTW. what are you doing in Budapest?