StatisticalTestSelection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"top\"></a>\n",
    "\n",
    "---\n",
    "\n",
    "# Choosing the Correct Statistical Test\n",
    "\n",
    "References:<br>\n",
    "http://www.ats.ucla.edu/stat/mult_pkg/whatstat/<br>\n",
    "http://www-users.cs.umn.edu/~ludford/stat_guide.html<br>\n",
    "http://wise.cgu.edu/\n",
    "\n",
    "## Contents\n",
    "\n",
    "[0 independent variables](#zero)\n",
    "- One sample t-test\n",
    "- One sample median test\n",
    "- Binomial test\n",
    "- Chi square goodness of fit (Pearson's)\n",
    "\n",
    "[1 independent binary categorical variable](#binary)\n",
    "- Two sample independent t-test\n",
    "- Paired t-test\n",
    "- Mann-Whitney U test\n",
    "- Wilcoxon signed rank test\n",
    "- Chi square test for independence\n",
    "- Fisher's exact test\n",
    "\n",
    "[1 independent multivalued categorical variable](#one)\n",
    "- One-way ANOVA\n",
    "- One-way repeated measures ANOVA\n",
    "- Kruskal-Wallis H test\n",
    "\n",
    "[2 independent multivalued categorical variables](#two)\n",
    "- Two-way ANOVA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "\n",
    "# Quick reference\n",
    "\n",
    "  &nbsp; | Categorical Dependent Variable | Continuous Dependent Variable\n",
    "  ------------- | -------------\n",
    "  **Categorical<br>Independent Variable** | Chi square | t-test or ANOVA\n",
    "  **Continuous<br>Independent Variable** | LDA or QDA | Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<a id=\"zero\"></a>\n",
    "\n",
    "<br>\n",
    "[Return to top](#top)\n",
    "___\n",
    "# 0 Independent Variables\n",
    "___\n",
    "\n",
    "Comparing a dataset against a hypothesized value (e.g., average age of people is 30 years old).\n",
    "\n",
    "# 1 Continuous Dependent Variable with Normal Distribution\n",
    "\n",
    "(*Technically it is the residuals that must be normally distributed*)\n",
    "\n",
    "## One sample t-test\n",
    "\n",
    "Null hypothesis is that the mean value of the dataset is equal to the test/hypothesized value ($\\mu_0$).\n",
    "$$ H_o: \\mu = \\mu_0 $$\n",
    "$$ H_A: \\mu < \\ or > \\ or \\ \\ne \\mu_0 $$\n",
    "\n",
    "The test determines if there is a statistically significant difference between $\\mu$ and $\\mu_0$. \n",
    "\n",
    "This requires calculation of the mean of the dataset ($\\bar{x}$),  the degrees of freedom dataset ($df$), the corresponding t-value and the p-value for the predetermined significance level (most commonly $\\alpha=0.05$). \n",
    "\n",
    "If the p-value calculated from the t-distribution at the given t-value is smaller than the selected signficance level (e.g., $p<0.05$) then the null hypothesis can be rejected and the alternative hypothesis is supported.\n",
    "\n",
    "Important calculations,\n",
    "\n",
    "$$ \\bar{x} = \\frac{1}{n} \\sum_{i=1}^{n} x_i \\ , \\quad df = n-1 \\ , \\quad t = \\frac{\\bar{x}-\\mu_0}{s/\\sqrt{n}} \\ , \\quad s = \\sqrt{\\frac{1}{n-1}\\sum_{i=1}^{n}(x_i-\\bar{x})^2}$$\n",
    "\n",
    "Typically, $p$ is found for a given t-distribution in a table or computer software as it requires integrating the t-distribution to find the probability from the area under the curve.\n",
    "\n",
    "If the alternative hypothesis includes $\\ne$, then a ***two-tail test*** is used where the significance level is split at the two extremes of the probability density curve. Otherwise for $<$ or $>$, a ***one-tail test*** is used where the significance level is all at the left or right side of the curve, respectively.\n",
    "\n",
    "***This test is not appropriate if the number of data points is below 30 ($n<30$) and the data is not normally distributed.*** Larger values of $n$ may be more robust against this requirement.\n",
    "\n",
    "### Implementation\n",
    "\n",
    "In Python using [scipy](http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_1samp.html),\n",
    "```\n",
    "scipy.stats.ttest_1samp(data, popmean, axis=0, nan_policy='propagate')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1 Continuous Dependent Variable with Non-normal Distribution\n",
    "\n",
    "## One sample median test\n",
    "\n",
    "This test is non-parameteric (it is not fit to a specific distribution) so it doesn't have the same requirements as the t-test. Typically a ***one-sample Wilcoxon Signed Rank Test*** is used instead."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1 Binary Categorical Dependent Variable\n",
    "\n",
    "## Binomial test\n",
    "\n",
    "For $n$ trials of success/failure, returns the probability $p$ that the number of successes is not equal, greater than, or less than $k$.\n",
    "$$ H_0: n_{success} = k $$\n",
    "$$ H_A: n_{success} \\ne k $$\n",
    "\n",
    "### Example\n",
    "For example, probability of returning $k$ heads given $n$ coin flips. Of course for a coin there are only two probabilities and each are likely to appear ($p=0.5$ for heads). The binomial test will determine if $k$ heads are likely to occur. If the null hypothesis is rejected for the pre-determined significance level, this can be interpreted as evidence that the coin is not fair.\n",
    "\n",
    "### Implementation\n",
    "\n",
    "In Python using [scipy](http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.binom_test.html),\n",
    "```\n",
    "scipy.stats.binom_test(k, n, p, alternative=['two-sided','greater','less'])\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1 Categorical Dependent Variable\n",
    "\n",
    "## Chi square goodness of fit test (Pearson's)\n",
    "\n",
    "Similar to binomial test however there are more than two categories in the dependent variable. ***This test will determine if the observed proportions of each category differ significantly from the hypothesized proportions.***\n",
    "\n",
    "For example, testing the number of people over age 65 in a jury vs the proportion in the local population. This example includes 2 groups (below and 65 or above) it could also be used on multiple intervals (less than 35, 35-45, above 45, etc).\n",
    "\n",
    "In other words, it tests a null hypothesis stating that the frequency distribution of certain events observed in a sample is consistent with a particular theoretical distribution. The events (or categories) must be mutually exclusive and have total probability 1. \n",
    "\n",
    "This assumes the categories are independent and identically distributed (*iid*) with the normal distribution, i.e., unpaired.\n",
    "\n",
    "Here the test statistic $\\chi^2$ follows the chi-squared distribution (*from wikipedia*):\n",
    "<img src=\"https://upload.wikimedia.org/wikipedia/commons/8/8e/Chi-square_distributionCDF-English.png\" width=\"350\">\n",
    "\n",
    "### Procedure\n",
    "1. Calculate the test statistic $\\chi^2$\n",
    "$$ \\chi^2 = \\sum_{i=1}^{n} \\frac{(O_i - E_i)^2}{E_i} $$\n",
    "1. Determine the degrees of freedom $df = n-p$ where $n$ is the number of outcomes/categories and $p=s+1$ where $s$ is the number of parameters in the distribution (s=2 for normal, mean and standard deviation; s=0 for discrete uniform, no parameters)\n",
    "1. Select a significance level $\\alpha$ ($\\alpha = 0.05$ typically)\n",
    "1. Compare calculated $\\chi^2$ with its critical value from the chi-squared distribution with the appropriate $df$ (one sided since only check if $\\chi^2$ is greater than critical)\n",
    "1. Accept or reject the null hypothesis that the observed frequency distribution is different from the theoretical distribution based on whether the test statistic exceeds the critical value of $\\chi^2$ ***or*** report the corresponding p-value and compare with pre-determined significance.\n",
    "\n",
    "### Example\n",
    "\n",
    "Another simple **example** is testing the outcome from a six-sided die to determine if the die is fair (all 6 outcomes equally likely to occur). If all 6 results are equally likely, the expected frequency is 1/6 from each. this is compared against the measured frequency of occurence for each of the 6 possible values for a large number of die rolls (e.g., 60 rolls). The $\\chi^2$ statistic is calculated and compared against the critical value for the given significance level and the null hypothesis is rejected or not rejected.\n",
    "\n",
    "### Implementation\n",
    "\n",
    "In python using [scipy](http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.chisquare.html),\n",
    "```\n",
    "scipy.stats.chisquare(f_obs, f_exp=None, ddof=0, axis=0)\n",
    "```\n",
    "This returns the calculated chi-square and the corresponding p-value which can be compared with the desired significance. The default value for this test assumes all expected probabilities are equally likely (come from a discrete uniform distribution).\n",
    "\n",
    "A rule of thumb states that there must be at least 5 values for each category for this test to be valid.\n",
    "\n",
    "A good example can be found at: https://en.wikipedia.org/wiki/Pearson%27s_chi-squared_test#Examples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"binary\"></a>\n",
    "<br>\n",
    "[Return to top](#top)\n",
    "___\n",
    "# 1 Independent binary categorical variable\n",
    "___\n",
    "Comparing two samples such as in an experiment, control vs. treatment groups or male vs. female.\n",
    "\n",
    "# 1 Continuous Dependent Variable with Normal Distribution\n",
    "\n",
    "## Two sample independent t-test\n",
    "\n",
    "This t-test assesses whether the means of two groups/categories of data are statistically different.\n",
    "\n",
    "It is very similar to the one-sample t-test with a few exceptions:\n",
    "1. The t-statistic is calculed from,\n",
    "$$ t = \\frac{\\bar{x}_1-\\bar{x}_2}{\\sqrt{\\frac{s_1^2}{n_1} + \\frac{s_2^2}{n_2}}} $$\n",
    "\n",
    "1. And the degrees of freedom are,\n",
    "$$ df = \\frac{(s_1^2/n_1 + s_2^2/n_2)^2}{(s_1^2/n_1)^2/(n_1-1) + (s_2^2/n_2)^2/(n_2-1)} $$\n",
    "\n",
    "The calculation for the t-statistic is appropriate for equal or unequal sample sizes and unequal variances between the two independent categories. This variation is known as ***Welch's t-test***.\n",
    "\n",
    "To determine if two groups/samples have equal variance, an [F-test](https://en.wikipedia.org/wiki/F-test_of_equality_of_variances) can be used (requires normality in both groups, there are other tests for non-normal distributions).\n",
    "\n",
    "### Example\n",
    "\n",
    "Test if the mean call length for two call centers is the same or significantly different. A random of sample of 30 calls is collected from each, this process is repeated 10 times.\n",
    "\n",
    "Another example would be comparing the fuel economy (miles per gallon) between a sample of US cars and Japanese cars. The null hypothesis would be no difference between the mean fuel economies of each group and the alternative hypothesis would be a significant difference between means.\n",
    "\n",
    "### Implementation\n",
    "\n",
    "In Python using [scipy](http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_ind.html),\n",
    "```\n",
    "scipy.stats.ttest_ind(a, b, axis=0, equal_var=True, nan_policy='propagate')\n",
    "```\n",
    "\n",
    "Or if only the descriptive statistics (or parameters of the normal distribution) are known, use [scipy](http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_ind_from_stats.html) function,\n",
    "```\n",
    "scipy.stats.ttest_ind_from_stats(mean1, std1, nobs1, mean2, std2, nobs2, equal_var=True)\n",
    "```\n",
    "\n",
    "\n",
    "## Paired t-test\n",
    "\n",
    "For ***paired samples*** (e.g., experiments looking at before/after treatment values to find a statistically significant difference), the t-statistic is calculated from,\n",
    "$$ t = \\frac{\\bar{x}_D - \\mu_0}{s_D / \\sqrt{n}} $$\n",
    "\n",
    "where the $D$ subscript represents the difference between pairs, so the mean and standard deviation are calculated from the paired differences and not the measured values. Here the degrees of freedom is the same as the one-sample t-test $df = n-1$.\n",
    "\n",
    "The paired test can help reduce the influence of confounding variables. For an explanation see wikipedia: https://en.wikipedia.org/wiki/Paired_difference_test#Use_in_reducing_confounding.\n",
    "\n",
    "When there are more than two independent categories, the *repeated measures ANOVA* test can be applied (see below).\n",
    "\n",
    "### Implementation\n",
    "\n",
    "In Python using [scipy](http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_rel.html),\n",
    "```\n",
    "scipy.stats.ttest_rel(a, b, axis=0, nan_policy='propagate')\n",
    "```\n",
    "where `a` and `b` are the two equal-length variables before calculating the paired difference."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1 Continuous Dependent Variable with Non-normal Distribution\n",
    "\n",
    "\n",
    "## Mann-Whitney U test\n",
    "\n",
    "This is a nonparametric test of the null hypothesis that it is equally like that a randomly selected value from one sample will be less than or greater than a randomly selected value from a second sample. Similar in application to a t-test.\n",
    "\n",
    "### Assumptions\n",
    "1. All the observations from both groups are independent of each other\n",
    "1. The responses are ordinal (i.e., one can at least say, of any two observations, which is the greater)\n",
    "\n",
    "The null hypothesis $H_0$ is interpreted as the distribution of the two samples are equivalent.\n",
    "The alternative hypothesis $H_A$ is interpreted as the distribution of the two samples are different.\n",
    "\n",
    "For large sample sizes, the U-statistic is approximately normally distributed.\n",
    "\n",
    "### Advantages over t-test\n",
    "\n",
    "1. **Ordinal data**: The Mann–Whitney U test remains the logical choice when the data are ordinal but not interval scaled, so that the spacing between adjacent values cannot be assumed to be constant.\n",
    "1. **Robustness**: As it compares the sums of ranks, the Mann–Whitney U test is less likely than the t-test to spuriously indicate significance because of the presence of outliers i.e., the Mann–Whitney U test may be more robust.\n",
    "1. **Efficiency**: this test is more effective for sample distributes far from normal.\n",
    "\n",
    "\n",
    "### Example\n",
    "\n",
    "https://secure.brightstat.com/index.php?p=c&d=1&c=2&i=5\n",
    "\n",
    "A physician is interested in the effect of an anaesthetic on reaction times. Two groups are compared, one with (A) and one without (B) taking the anaesthetic. Subjects had to react on a simple visual stimulus. Reaction times are not normally distributed in this experiment, so data is analysed with the Mann-Whitney U-Test for ordinal scaled measurements.\n",
    "\n",
    "The ranks for each measurement in each group are calculated along with the U statistic. This value can be examined with a U-statistical table for the appropriate significance and the null hypothesis is rejected or not.\n",
    "\n",
    "### Implementation\n",
    "\n",
    "In python using [scipy](http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.mannwhitneyu.html),\n",
    "```\n",
    "scipy.stats.mannwhitneyu(x, y, use_continuity=True, alternative=None)\n",
    "```\n",
    "Use only when the number of observation in each sample is > 20 and you have 2 independent samples of ranks. Mann-Whitney U is significant if the u-obtained is LESS THAN or equal to the critical value of U.\n",
    "\n",
    "This test corrects for ties and by default uses a continuity correction.\n",
    "\n",
    "\n",
    "## Wilcoxon signed ranks test (paired samples)\n",
    "\n",
    "This is a non-parametric type of paired difference test that is appropriate for non-normal population distributions and small sample sizes.\n",
    "\n",
    "It requires that the data are paired and each pair is chosen randomly and independently.\n",
    "\n",
    "### Procedure\n",
    "\n",
    "Let $N$ be the number of pairs and $2N$ the number of data points, $x_{1,i}$ and $x_{2,i}$ for $i=1,...,N$.\n",
    "\n",
    "The hypotheses are,\n",
    "\n",
    "$H_0:$ *difference between pairs follows a symmetric distribution about zero*<br>\n",
    "$H_A:$ *difference between pairs does not follow a symmetric distribution about zero*\n",
    "\n",
    "1. For all N pairs, calculate $|x_{2,i}-x_{1,i}|$ and $sgn(x_{2,i}-x_{1,i})$ where $sgn$ is the sign function ($ x = sgn(x)\\cdot|x|$)\n",
    "1. Exclude pairs where the difference is zero, let $N_r$ be the reduced pair sample size\n",
    "1. Ordered the remaining $N_r$ pairs in ascending order by $|x_{2,i}-x_{1,i}|$\n",
    "1. Rank the pairs $R_i$ with the smallest as 1. Ties receive a rank equal to the average of the ranks they span.\n",
    "1. Calculate the test statistic $$W = \\sum{i=1}{N_r} \\ [sgn(x_{2,i}-x_{1,i})\\cdot R_i]$$\n",
    "1. The distribution for $W$ is not easily described, lookup critical value in a reference table for pre-determined significance level. For the two-tail test, reject $H_0$ if $|W| \\ge W_{critical, N_r}$.\n",
    "1. As $N_r$ increases, the sampling distribution of $W$ approaches a normal distribution so a typical z score can be computed where,\n",
    "$$ z = \\frac{W}{\\sigma_W} \\ , \\quad \\sigma_W = \\sqrt{\\frac{N_r(N_r+1)(2N_r+1)}{6}} $$\n",
    "Then compare $z$ with $z_{critical}$ for the appropriate significance level at one or two tail test or calculate the p-value.\n",
    "\n",
    "An ***effect size*** (quantitative measure of the strength of a phenomenon, similar to coefficient of determination) can be calculated from the ***rank correlation***, $r=W/S$, where $S$ is the sum of the ranks for the $N_r$ data pairs.\n",
    "\n",
    "### Example\n",
    "\n",
    "One **example** for using this test would be analyzing survey results from a multiple-point likert scale (interval scale) to questions such as \"I am comfortable using this product\".\n",
    "\n",
    "Another would be comparing blood pressures before and after administration of some treatment. Here it is not necessary for the blood pressures to be normally distributed in the 'before' and 'after' groups.\n",
    "\n",
    "\n",
    "\n",
    "### Implementation\n",
    "\n",
    "In Python using [scipy](http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.wilcoxon.html),\n",
    "```\n",
    "scipy.stats.wilcoxon(x, y=None, zero_method='wilcox', correction=False)\n",
    "```\n",
    "For this application, the $W$ distribution is assumed to be close to normal, so the rule of thumb is to require the number of pairs to be > 20."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1 Categorical Dependent Variable\n",
    "\n",
    "## Chi square test for independence\n",
    "\n",
    "This technique ***tests the null hypothesis that there is no statistical difference between the effect of categories in the independent variable on the frequency of occurence of the categories in the dependent variable***. The alternative hypothesis is that the independent and dependent variables do have a relationship.\n",
    "\n",
    "For example, if we measured the percentage of left-handed men and compared that with the percentage of left-handed women. I would expect the null hypothesis to *not* be rejected and infer that gender does not have an effect on left-handedness.\n",
    "\n",
    "This test can also be applied when there are ***more than two possible categories in the independent variable***. However it requires each category to have > 5 samples.\n",
    "\n",
    "When applying this test, it is also possible to use a *two sample independent t-test*, where the categorical dependent variable is expressed in proportions (50% in category A, 50% in category B, etc).\n",
    "\n",
    "If there are more than two categories in the independent variable, another appropriate choice would be to apply *ANOVA* and again express the data in terms of proportions.\n",
    "\n",
    "### Example\n",
    "An from wikipedia that tests for independence (equal occurence of independent variable categories in each dependent variable category): https://en.wikipedia.org/wiki/Chi-squared_test#Example_chi-squared_test_for_categorical_data. \n",
    "\n",
    "In this example, in a city of 1 million residents with four neighborhoods: A, B, C, and D, a random sample of 650 residents of the city is taken and their occupation is recorded as \"white collar\", \"blue collar\", or \"no collar\". The null hypothesis is that each person's neighborhood of residence (I.V.) is independent of the person's occupational classification (D.V.).\n",
    "\n",
    "||A|B|C|D|Total|\n",
    "|---|---|---|---|---|\n",
    "|White collar|90|60|104|95|349|\n",
    "|Blue collar|30|50|51|20|151|\n",
    "|No collar|30|40|45|35|150|\n",
    "|Total|150|150|200|150|650|\n",
    "\n",
    "For the first I.V. (neighborhood A), we expect the number of white collar workers to be $150 \\times 349/650 \\approx 80.54$. Calculate the $\\chi^2$ value for the other IVs (neighbors B,C,D) and sum them.\n",
    "\n",
    "If the $\\chi^2$ value exceeds the critical value for the pre-determined level of significance or the p-value is below the critical level, then the null hypothesis of no relationship or independence can be rejected.\n",
    "\n",
    "### Implementation\n",
    "\n",
    "In Python with scipy,\n",
    "```\n",
    "scipy.stats.chi2_contingency(observed, correction=True, lambda_=None)\n",
    "```\n",
    "Here the `observed` argument is an RxC (simplest case is 2x2) **contingency table** containing observed frequencies.\n",
    "\n",
    "\n",
    "## Fisher's exact test\n",
    "\n",
    "This test is valid for all sample sizes include small ones, where a category has < 5 entries. It is exact and does not rely on the approximation that the test becomes exact as the sample size limit approaches infinity.\n",
    "\n",
    "There is no test statistic here, simply determine the appropriate degrees of freedom then calculate the p-value and compare with the predetermined significance level (typically $\\alpha = 0.05$). The probability of obtaining any set of values (a,b,c,d) for two-level independent and dependent variables is given by the ***hypergeometric distribution***, this is what allows exact determination of the p-value.\n",
    "\n",
    "### Implementation\n",
    "\n",
    "In Python using [scipy](http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.fisher_exact.html) a 2x2 contingency table can be analyzed,\n",
    "```\n",
    "scipy.stats.fisher_exact(table, alternative='two-sided')\n",
    "```\n",
    "\n",
    "See wikipedia for an example: https://en.wikipedia.org/wiki/Fisher%27s_exact_test#Example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"one\"></a>\n",
    "\n",
    "<br>\n",
    "[Return to top](#top)\n",
    "___\n",
    "# 1 Independent multi-valued categorical variable\n",
    "\n",
    "___\n",
    "\n",
    "# 1 Continuous Dependent Variable with Normal Distribution\n",
    "\n",
    "## One-way ANOVA\n",
    "\n",
    "The null hypothesis ($H_0$) tested by one-way ANOVA is that two or more population\n",
    "means are equal. A statistically significant test indicates that observed data sampled from each of the populations would be unlikely if the null hypothesis were true.\n",
    "\n",
    "### Assumptions\n",
    "\n",
    "The data from each independent category must be:\n",
    "1. Random and independent\n",
    "1. Normally distributed\n",
    "1. Equal variances\n",
    "\n",
    "The last two assumptions are relaxed for large samples.\n",
    "\n",
    "### Implementation\n",
    "\n",
    "In Python using [scipy](http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.f_oneway.html),\n",
    "```\n",
    "scipy.stats.f_oneway(*args)\n",
    "```\n",
    "\n",
    "## One-way repeated measures ANOVA (rANOVA)\n",
    "\n",
    "Here, the independent variable represents measurements collected under a variety of conditions or at several different time points from the same group. \n",
    "\n",
    "One example would be collecting blood pressure at a variety of time points from the same group of people, or perhaps in a taste test for different types of cake being sampled by the same group of people.\n",
    "\n",
    "There is no easy Python implementation for repeated measures ANOVA. One possibility is to use [rpy2](http://rpy2.bitbucket.org/) to call the R-package for this method."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1 Continuous Dependent Variable with Non-normal Distribution\n",
    "\n",
    "## Kruskal-Wallis H test\n",
    "\n",
    "This test makes use of measurement ranks and is a non-parametric method for testing whether multiple samples originate from the same distribution. Its parameteric equivalent is one-way ANOVA and its two-sample equivalent is the Mann-Whitney U test.\n",
    "\n",
    "Null hypothesis $H_0$: the medians of all groups are equal.<br>\n",
    "Alternative hypothesis $H_A$: at least one population median of one group is different from the population median of at least one other group.\n",
    "\n",
    "The measurements of each group are ranked and a H-statistic is calculated. Tie corrections are applied if necessary. The p-value for the H-statistic is calculated where $H$ is assumed to have a chi-square distribution.\n",
    "\n",
    "### Assumptions\n",
    "\n",
    "1. All groups/samples must be identically shaped and scaled except for any difference in medians.\n",
    "\n",
    "### Implementation\n",
    "\n",
    "In python using [scipy](http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.kruskal.html),\n",
    "```\n",
    "scipy.stats.kruskal(*args, **kwargs)\n",
    "```\n",
    "\n",
    "Due to the assumption that H has a chi square distribution, the number of samples in each group must not be too small. A typical rule is that each sample must have at least 5 measurements."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"two\"></a>\n",
    "<br>\n",
    "[Return to top](#top)\n",
    "___\n",
    "# 2 Independent multi-valued categorical variable\n",
    "___\n",
    "\n",
    "<br>\n",
    "# 1 Continuous Dependent Variable with Normal Distribution\n",
    "\n",
    "## Two-way ANOVA\n",
    "\n",
    "This test will assess the ***main effect*** of each independent variable (its effect on the DV averaging across the levels of other IVs) and ***interaction*** between independent variables. An interaction between three variables indicates that considering the effect of two on the third is not strictly additive, there is an interaction (or simultaneous influence) between the two that influences the value of the third.\n",
    "\n",
    "### Implementation\n",
    "\n",
    "There is no easy Python implementation for this test. See this site for more information: http://www.marsja.se/three-ways-to-carry-out-2-way-anova-with-python/."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}