Similarity Module (trajectroy distances)

examples/example_similarity/TI_similarity_introduction.ipynb

@@ -0,0 +1,228 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# trackintel Similarity Module \n",
+    "demonstration notebook. run the following cells to get an overview what the similarity module provides."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### import of framework and data"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import trackintel as ti\n",
+    "\n",
+    "pfs = ti.io.file.read_positionfixes_csv('testtplset.csv')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### preprocessing of the trajectories\n",
+    "When starting from raw tracking data, the following steps have to be performed:\n",
+    "- extract staypoints \n",
+    "- extract triplegs\n",
+    "\n",
+    "the test data are already preprocessed positionfixes. To calculate some similarities between trajectories you always need trajectories as positionfixes. The positionfixes have tripleg_ids to distinguish the trajectories. You can access a single trajectory using command as:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pfs[pfs['tripleg_id']==22]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Let's store two trajectories out of the pfs data frame. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ta = pfs[pfs['tripleg_id']==22]\n",
+    "tb = pfs[pfs['tripleg_id']==33]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "A trajectory distance between these two can be calculated using the methods available in ti.similarity.measures. These are Dynamic Time Warping (DTW) and Edit Distance on Real Sequences (EDR). An algorithm called Start End Distance is also available, but works a bit different. This one is explained later.\n",
+    "\n",
+    "The DTW distance of two trajectories can be calculated like this:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ti.similarity.e_dtw(ta,tb)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "How can this distance be interpreted? The DTW distance uses a euclidian distance function. So the distance is dependent on the coordinates of the positionfixes. To see this information you can call:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(pfs.crs)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The crs is empty. To set the initial projection, you can write the EPSG id of the coordinate system in the pfs GeoDataFrame. In this case this would be WGS84 with EPSG id 4326."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pfs.crs='EPSG:4326'"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "To reproject the data set you could call pfs.to_crs(epsg=1234). To avoid changes in the positionfixes, it is recommended to reproject a copy or to reproject directly when calculating the similarity matrix of the data set. In this example the data is reprojected to CH1903+. To calculate a distance matrix with the DTW method, the following code can be executed."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "distmatrix = ti.similarity.similarity_matrix(pfs.to_crs(epsg=2056), 'dtw', dist=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The similarity values are stored in 'simmatrix'. To access a value, normal python matrix syntax can be used, the row and column indices correspond to the tripleg_ids."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "distmatrix[22,33]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The dtw distance value, compared to the value calculated above, is now in meters. "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The method similarity matrix can also be executed on a positionfixes object. By not setting the dist parameter to True, the trajectory distances will be inverted. This is recommended for large data sets (or data sets with high tripleg ids), as the matrix will not store zero values and be more performant."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "simmatrix = pfs.to_crs(epsg=2056).as_positionfixes.similarity_matrix('dtw')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "simmatrix[22,33]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Comparing the two matrices, the distance matrix has at each position a stored value, the similarity matrix stores only the relevant values:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(distmatrix)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(simmatrix)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

examples/example_similarity/testtplset.csv

@@ -0,0 +1,32 @@
+tracked_at,tripleg_id,user_id,longitude,latitude
+2020-03-23 13:05:01+00:00,22,22,7.571296692000001,47.53789097
+2020-03-23 13:05:08+00:00,22,22,7.571897507,47.53702179
+2020-03-23 13:05:15+00:00,22,22,7.571940422000001,47.53631196
+2020-03-23 13:05:22+00:00,22,22,7.5717473029999995,47.53573973
+2020-03-23 13:05:29+00:00,22,22,7.57158637,47.53536308
+2020-03-23 13:05:36+00:00,22,22,7.571178675,47.534732899999995
+2020-03-23 13:05:43+00:00,22,22,7.570695877,47.53392886
+2020-03-23 13:05:50+00:00,22,22,7.57029891,47.53326969
+2020-03-23 13:05:57+00:00,22,22,7.569547892,47.53195132
+2020-03-23 13:06:04+00:00,22,22,7.56919384,47.53111102
+2020-03-23 13:06:11+00:00,22,22,7.569032907,47.53044457
+2020-03-23 13:06:18+00:00,22,22,7.568678856,47.5294159
+2020-03-23 13:06:25+00:00,22,22,7.568421364,47.52805397
+2020-03-23 13:06:32+00:00,22,22,7.568228245,47.52686587
+2020-03-23 13:06:39+00:00,22,22,7.567884922,47.52590957
+2020-03-23 13:05:57+00:00,33,33,7.575137615,47.53776059
+2020-03-23 13:06:04+00:00,33,33,7.575330734,47.53728255
+2020-03-23 13:06:11+00:00,33,33,7.575502396,47.53673207
+2020-03-23 13:06:18+00:00,33,33,7.575073242,47.53593531
+2020-03-23 13:06:25+00:00,33,33,7.57481575,47.53518199
+2020-03-23 13:06:32+00:00,33,33,7.574365139,47.53458803
+2020-03-23 13:06:39+00:00,33,33,7.573978901,47.53367534
+2020-03-23 13:06:46+00:00,33,33,7.57352829,47.53297994
+2020-03-23 13:06:53+00:00,33,33,7.573442459,47.53228454
+2020-03-23 13:07:00+00:00,33,33,7.573335171,47.53138629
+2020-03-23 13:07:07+00:00,33,33,7.573292255,47.53057496
+2020-03-23 13:07:14+00:00,33,33,7.573356627999999,47.52912613
+2020-03-23 13:07:21+00:00,33,33,7.573378086,47.52809743
+2020-03-23 13:07:28+00:00,33,33,7.573335171,47.52683689
+2020-03-23 13:07:35+00:00,33,33,7.573120594,47.52605447
+2020-03-23 13:07:42+00:00,33,33,7.572691441,47.52541693

requirements.txt

@@ -9,4 +9,6 @@ geoalchemy2
 osmnx 
 psycopg2
 imageio
-simplification
+simplification
+console_progressbar
+scipy

setup.py

@@ -33,7 +33,9 @@
     'geoalchemy2',
     'osmnx',
     'scikit-learn',
-    'simplification'
+    'simplification',
+    'console_progressbar',
+    'scipy'
 ]
 
 # What packages are optional?

tests/data/sim1.csv

@@ -0,0 +1,7 @@
+user_id;tracked_at;latitude;longitude;elevation;accuracy
+1;2015-11-27T08:00:00Z;47.37651505;8.548779488;456;1
+1;2015-11-27T08:20:22Z;47.39935;8.5277;524;1
+1;2015-11-27T12:39:28Z;47.40854;8.51169;537;1
+1;2015-11-27T13:57:20Z;47.41144;8.54472;447;1
+1;2015-11-28T12:39:28Z;43.40854;8.51169;537;1
+

tests/data/sim2.csv

@@ -0,0 +1,7 @@
+user_id;tracked_at;latitude;longitude;elevation;accuracy
+1;2015-11-27T08:00:00Z;47.37651505;8.548779488;456;1
+1;2015-11-27T08:20:22Z;47.39935;8.5277;524;1
+1;2015-11-27T12:39:28Z;47.40854;8.51169;537;1
+1;2015-11-27T13:57:20Z;47.41144;8.54472;447;1
+1;2015-11-28T12:39:28Z;43.40854;8.51169;537;1
+1;2015-11-28T13:57:20Z;43.41144;8.54472;447;1

tests/data/test_edr_no.csv

@@ -0,0 +1,22 @@
+tracked_at,longitude,latitude,user_id
+2020-03-23T10:01:11Z,7.739524841,47.53366085,1
+2020-03-23T10:01:23Z,7.740460932,47.5340683,1
+2020-03-23T10:01:47Z,7.741346061,47.5343997,1
+2020-03-23T10:01:59Z,7.741574049,47.53449748,1
+2020-03-23T10:02:11Z,7.742451131,47.53482706,1
+2020-03-23T10:02:23Z,7.743835151,47.53535221,1
+2020-03-23T10:02:35Z,7.744918764,47.53575965,1
+2020-03-23T10:02:47Z,7.74631083,47.53629023,1
+2020-03-23T10:02:59Z,7.750661373,47.53794891,1
+2020-03-23T10:03:11Z,7.753686905,47.5389919,1
+2020-03-23T10:03:23Z,7.760617733,47.54087503,1
+2020-03-23T10:03:35Z,7.772955894,47.54323608,1
+2020-03-23T10:03:47Z,7.783727646,47.54471349,1
+2020-03-23T10:03:59Z,7.79705286,47.54455416,1
+2020-03-23T10:04:11Z,7.800765038,47.54506111,1
+2020-03-23T10:04:23Z,7.802739143,47.54533631,1
+2020-03-23T10:04:35Z,7.799928188,47.54342438,1
+2020-03-23T10:04:47Z,7.803447247,47.53803583,1
+2020-03-23T10:04:59Z,7.80600071,47.53419688,1
+2020-03-23T10:05:11Z,7.808768749,47.53005339,1
+2020-03-23T10:05:23Z,7.812395096,47.52772072,1

tests/data/test_edr_outlier.csv

@@ -0,0 +1,22 @@
+tracked_at,user_id,longitude,latitude
+2020-03-23T10:01:11Z,2,7.739524841,47.53366085
+2020-03-23T10:01:23Z,2,7.740460932,47.5340683
+2020-03-23T10:01:47Z,2,7.741346061,47.5343997
+2020-03-23T10:01:59Z,2,7.741574049,47.53449748
+2020-03-23T10:02:11Z,2,7.742451131,47.53482706
+2020-03-23T10:02:23Z,2,7.743835151,47.53535221
+2020-03-23T10:02:35Z,2,7.744918764,47.53575965
+2020-03-23T10:02:47Z,2,7.74631083,47.53629023
+2020-03-23T10:02:59Z,2,7.750661373,47.53794891
+2020-03-23T10:03:11Z,2,7.753686905,47.5389919
+2020-03-23T10:03:23Z,2,7.760617733,47.54087503
+2020-03-23T10:03:35Z,2,7.755661010742187,47.56413326214536
+2020-03-23T10:03:47Z,2,7.783727646,47.54471349
+2020-03-23T10:03:59Z,2,7.79705286,47.54455416
+2020-03-23T10:04:11Z,2,7.800765038,47.54506111
+2020-03-23T10:04:23Z,2,7.802739143,47.54533631
+2020-03-23T10:04:35Z,2,7.799928188,47.54342438
+2020-03-23T10:04:47Z,2,7.803447247,47.53803583
+2020-03-23T10:04:59Z,2,7.80600071,47.53419688
+2020-03-23T10:05:11Z,2,7.808768749,47.53005339
+2020-03-23T10:05:23Z,2,7.812395096,47.52772072