[W2D5] T2: equation for greedy policy

[willwayaka]

In the equation for greedy policy:

\begin{align}
P (a_{t} = a) =
\begin{cases}
1 - \epsilon & \quad \text{if } a_{t} = \text{argmax}{a} ; q{t} (a) \
\epsilon/N & \quad \text{else}
\end{cases}
\end{align}

(between sec2.1 and ex1) what does N stand for? There is no mention of it in the text.

[rschween]

As of slack discussion with @marcelomattar, N stands for the number of actions, but there are inconsistencies both within equation and in the text below regarding whether the argmax-option is among the options to choose in the non-greedy case. What the solution code does is it chooses the greedy option with probability (1-epsilon) and between all options (including the greedy), otherwise.

Suggested fix (in line with Sutton & Barto)

• change first row of equation (probabiligy for argmax-option) to 1-epsilon + epsilon/N
• define N below equation
• adapt text below first bar graph accordingly

Alternatively (appears to also be used in practice; makes equation look slightly less complex):

• change second row of equation (probability for non-argmax options to epsilon/(N-1)
• define N below equation
• adapt text below equation accordingly
• change solution code to only select among non-argmax option in the "else:" condition.

[willwayaka]

After going through the slack discussion, I propose the following fix:

$\epsilon$-greedy policy:
\begin{align}
P (a_t = a_k) = 
        \begin{cases}
        1 - \epsilon    & \quad \text{then } a_{k} = \arg\max_{a} q_{t} (a) \\
        \epsilon/N       & \quad \text{then } a_{k} \in \{a\}
        \end{cases} 
\end{align}
where $\epsilon \in [0,1]$, and $N$ is number of actions in $\{a\}$ available to the agent (including the optimal action, so that $1 - \epsilon + \epsilon/N \times N = 1$).

[JesseLivezey]

@willwayaka IMO, the first case should be 1 - \epsilon + 1/N, then the second condition should be an "else". I think it confusing to have the actions contained in the "if" and the "else" be overlapping.