Home
This is an impressive project. The breadth is enormous, an almost complete system for interpreting and making available human activities. I have been fiddling with the Activity Recorder and Sensor Logger application on and off for a few months and only now just realised the more complete Context Analyser application exists. This is because Context Analyser is not available in the Android Market as it doesn’t by itself produce any output. You can get Context Analyser from this repository and the applications that use it i.e Places and Contexthome (which doesn’t work on a Nexus One Froyo). To understand the project you need to read the excellent report.
I hope it will continue and further depth will be added. The activity recorder is not very useful by itself but allows investigation of activity classification. It is a background application that must be started but can be left running continuously without obviously draining the battery. Following on from this review there is some notes on what to do next.
To generate the classification algorithms and train them classification data was acquired using the Sensor Logger application. The first 3 graphs are accelerometer data and the next three are compass data and the accelerometer graphs look like this.
You can see test data I have acquired but this data is only available graphically. Orientation is the same as Android uses. . Chris has provided the training data in an anonymised format.
Pages 20-29 in Chris’s final report deal with activity classification, and 35-38 deal a bit with the implementation.
Chris says he was able to “classify activities with an accuracy of in excess of 95%” (pg 26) using an algorithm that was later simplified to one in which “a similar accuracy was achieved using only four features” (pg 27). Chris states “this is an (sic) very useful result as the features are extremely cheap to calculate, and almost trivial to implement on the device.” Unfortunately I don’t get anywhere near 95% classification accuracy, I haven’t tried to quantify accuracy but I reckon its less than 25%. It doesn’t mean Chris’s 95% was wrong but it depends on what activities are classified and the definition of accuracy. I don’t think for practical purposes it is accurate enough.
For example, here is some data from today. I switched on the application, left the phone for a few minutes, put it in my pocket, got in the car and travelled to work, walked to the office, placed the phone on the desk and did stuff.
| Activity Recorded | Actual Activity | Comments |
|---|---|---|
| Unknown for <1 min | nothing | Always generates a spurious activity when first started. Bug. |
| Sitting Down for 2 mins | Phone on table. | There is an assumption that phone is always carried but phones are frequently left on a surface. So whatever classification is done in this case is independent of the persons activity. |
| Unknown for <1 min | I’m not sure | Less than 1 minute is too short for me to know unless I watch the screen continuously. |
| Walking for 2 min | Probably right | |
| Unknown for <1 min | I’m not sure | Still walking around |
| Sitting down < 1 min | Not sitting down | Still walking around packing stuff etc |
| Unknown for <1 min | Not sure | Still walking around packing stuff etc |
| Unknown for <1 min | Not sure | |
| Standing still for 8 mins | Driving car | The activity is wrong but at it has recognised I’m undertaking the same activity for 8 mins |
| Unknown for <1 min | Not sure | |
| Walking for 6 mins | Walking | Good |
| Unknown for <1 min | Not sure | |
| Various things mostly sitting down for 2 hours 30mins | Phone on desk | Occasionally I’ve picked it up but as I’m not carrying the phone my activities are unrelated to those detected. |
Two walking events were correctly classified but everything else was wrong or unknown including walking events otherwise classified. It was successful at detecting the same event was continuing for the driving event (wrongly classified) and one walking event.
There needs to be an activity of phone not carried. There is an assumption that phone is always carried but phones are frequently left on a surface and for women this occurs more often as phones are usually carried in a hand bag. There needs to be a classification for when a phone is not being carried. This is a major source of error as whatever classification is made it is independent of the persons actual activity. This classification is in a different category to others and ought to be determined differently prior to other attempts at classification. Currently 128 samples are taken at 20Hz which gives a sampling interval of 6.5 secs. If the acceleration on all three axes is constant for 6.5 secs it is likely the phone is not being carried. If the phone is on a surface all of the acceleration will be on the blue axis (Z axis) in the image above. If it is in a hand bag it could be in any direction but whatever direction it is it will be in the same direction when the accelerometers are next sampled. This constant direction could be used to determine periods of “phone not carried”.
Classification is done using two accelerometers considered independently. The important axes for a person are up and down (i.e. in the direction of gravity), forward and back, and sideways. I’ve sometimes felt that classification worked much better than other times and I think this explains why. If I carry the phone in one orientation the data that is classified is completely different to if I carry it in another. In my case this is hip pocket vs breast pocket (rotation about the vertical/green/Y axis) or sometimes carried in hand (hip pocket but rotated about blue/Z axis and swings when walking). For someone using a handbag orientation would be almost random. Chris uses phones Y and Z axes (see picture above). If the signal that is classified is independent of phone orientation then the patterns associated with any activity are hugely reduced. Without considering orientation you could get high classification success with your training data but as soon as someone else carries the phone in a different orientation, classification will be poor. So acceleration needs to be either a single vector or vectors in the direction of up/down forward/backwards sideways (maybe sideways doesn’t matter much).
Pedometers achieve this by designing the device so it always attached to a person in the same direction but phones aren’t like that.
Gravity will generally dominate the acceleration signal. If the orientation of the phone is not changed over the sampling interval (sometimes it will) it can be mostly removed by averaging all three signals and subtracting the mean from each value. The remaining signal could then be resolved into vertical acceleration and an acceleration in an unknown direction in the horizontal plane. This will eliminate phone orientation as a variable and reduce the cases to be classified hugely. Curiously, two of the four signals Chris used for classification were the absolute mean (absolute means the same value would be measured if the phone was inverted) of two accelerometer signals. This is a measure of the phones orientation in two axes, which if the phone is always attached to the person in the same way is also the persons orientation in two axes. So orientation of the person looks to be a big factor in determining their activity. If the phone was in a hip pocket it would indeed be excellent for detecting sitting vs standing but would be useless with the phone in the breast pocket.
I never travel on a bus but I’m often classified to be doing so. Intuitively I suspect that travelling on a bus will have pretty similar accelerations to travelling by car so it seems unlikely it can be distinguished. Similarly sitting vs standing seems problematical. Walking up/down stairs seems too detailed as well though I guess my accelerations are quite different to normal walking when stair climbing or descending. If it was nearly always right that level of detail would be impressive but it is probably not necessary and is too often wrong.
One thing I’ve noticed is that once it classifies an activity wrongly it tends to keep getting it wrong. I was once classified as travelling in a bus for two hours while working in an office environment. This can also been seen above where driving a car was classified as standing up for 8 minutes. It is good that it recognised it as the same activity but would have been better if it was classified correctly. Activity aggregation involves weighting the last activity so that the current activity is more likely to be classified the same. The algorithm increases the likelihood score of each component in a classication by a constant amount, and reduces all other components by a fixed ratio.
I think this means that when an activity is classified wrongly the wrong classification is used to increase the likelihood next time so an activity that without weighting would be classified as walking becomes classified as standing up because of the previous classifications. Then once classified as standing up it increases the likelihood that the next walking classification will also be weighted to standing up so that you get a series of walking classifications all changed by weighting to standing up.
Chris provided some useful feedback on the above and that has prompted further testing and analysis. In summary the results are that if you use the system in the correct way and define accuracy appropriately, it is amazingly accurate. Different definitions of accuracy give different results and some errors types are much more important to user perceptions of accuracy than others.
It is planned to use activity recorder for getting data on flat water kayaking training. First thing we need to know is; are we paddling? Once we know we are kayaking we want to measure acceleration data, compass data, data from elbow angles (if available) and location data. Then we want to summarise it to get stroke rate, quality of stroke, speed etc. then upload data to website. We should also upload some full data for representative strokes. Then we need to produce training reports like runkeeper, stroke coach and google maps
Justin (Jungoo) Lee has been developing this software as part of his Masters course. Project plans, presentations and the report is available from the course page . Look for the report from Justin (Jungoo) Lee.