farmers-report.Rmd

---
title: "Farmers report"
subtitle: for the week prior `r Sys.Date()`
output:
  html_document: default
params:
  config_path: "config.yaml"
  bom: "Human Genetics"
  informatics_teams:
  - "hgi"
  - "mercury-grp"
  static: FALSE
---

This is a report for farm efficiency among `r params$bom` teams for the last week.

```{r setup, include=FALSE}
library(dplyr)
library(ggplot2)
library(logging)

source('src/table_helpers.R')
source('src/elastic_helpers.R')
source('src/plot_helpers.R')
source('src/stat_helpers.R')
source("src/constants.R")
source('src/config.R')
source('src/logging.R')

config <- read_config(params$config_path, section = 'proxy')

knitr::opts_chunk$set(echo = FALSE)
removeHandler('basic.stdout')
```

```{r connect, include=FALSE}
library(elastic)
elastic_con <- connect(
  host = config$elastic$host,
  path = "",
  user = config$elastic$username,
  pwd = config$elastic$password,
  port = config$elastic$port,
  transport_schema = "http"
)

index <- config$elastic$index
attr(elastic_con, 'index') <- config$elastic$index
table_view_opts <- ifelse(params$static, 't', 'ftp')
```

## Number of failed jobs
```{r pie chart, echo=FALSE, fig.height=3, fig.width=3, include=!params$static}
b <- build_agg_query("Job", query = build_humgen_query(filters = build_humgen_filters(BOM = params$bom)))

res <- Search(elastic_con, index = index, body = b, asdf = T)

df <- parse_elastic_agg(res, b) %>%
  mutate_for_piechart()

piechart(df, count_field = 'doc_count', key_field = 'job_status', legend_name = 'Job status')
```

### per team
```{r failed stats, echo=FALSE, fig.height=4, fig.width=9}
b <- build_terms_query(fields = c("ACCOUNTING_NAME", "Job"), query = build_humgen_query(filters = build_humgen_filters(BOM = params$bom)))

res <- Search(elastic_con, index = index, body = b, asdf = T)

parse_elastic_agg(res, b) %>%
  rename_group_column() -> df

make_per_team_job_plot(df, static = params$static)
```

```{r echo=FALSE, include=!params$static}
df %>%
  tidyr::pivot_wider(id_cols = 'accounting_name', names_from = 'job_status', values_from = 'doc_count', values_fill = 0) %>%
  mutate(fail_rate = Failed / (Failed + Success)) %>%
  arrange(desc(Failed)) -> dt
make_dt(dt, table_view_opts = table_view_opts)
```

## Wasted resources
Resource is considered to be wasted if it is not used or its job failed.
For memory we assume that memory consumption under the peak value is not wasted.
```{r elastic agg query, echo=FALSE}
b <- build_bom_aggregation(
  query = build_humgen_query(filters = build_humgen_filters(BOM = params$bom))
)

res <- Search(elastic_con, index = index, body = b, asdf = T)

df <- parse_elastic_agg(res, b) %>%
  select(-doc_count)
```

```{r echo=FALSE, include=!params$static}
df %>%
  group_by(accounting_name) %>%
  generate_efficiency_stats() %>%
  select(accounting_name, cpu_wasted_hrs, cpu_wasted_frac, mem_wasted_gb_hrs, mem_wasted_frac, wasted_cost) %>%
  rename_group_column() -> dt

make_dt(dt, table_view_opts = table_view_opts)
```

```{r adjustments explanation, results='asis', echo=FALSE}
cat(adjustments_explanation)
```

```{r echo=FALSE, fig.width=10}
dt <- generate_bom_statistics(df)

dt %>%
  tidyr::pivot_longer(cols = c('cpu_wasted_hrs', 'mem_wasted_gb_hrs')) %>%
  make_wastage_plot(renamer = column_rename)
```
<div style="page-break-before: always;" />

```{r awesomeness explanation, results='asis', echo=FALSE}
cat(awesomeness_explanation)
```

```{r awesomeness formula, results='asis', eval=!params$static, echo=FALSE}
cat(awesomeness_formula)
```


```{r echo=FALSE}
if(params$static) {
  dt <- select(dt, -mem_wasted_gb_hrs, -cpu_wasted_hrs)
}

make_dt(dt, all_rows = params$static, table_view_opts = table_view_opts)
```

```{r get leading team, echo=FALSE}
dt %>%
  arrange(desc(wasted_cost)) %>%
  pull(accounting_name) %>%
  setdiff(params$informatics_teams) %>%
  first() -> leading_team
```

Let's see per user statistics for the leading team other than the informatics team(s) (`r params$informatics_teams`): `r leading_team`
```{r per user stat, echo=FALSE}

if(leading_team %in% team_map$team_name){
  leading_team <- filter(team_map, team_name == leading_team)[['team_code']]
}

q <- build_humgen_query(
  filters = build_humgen_filters(
    BOM = params$bom,
    accounting_name = leading_team
  )
)

dt <- get_team_statistics(elastic_con, query = q)

if(params$static){
  dt <- select(dt, -cpu_wasted_hrs, -mem_wasted_gb_hrs)
}

make_dt(dt, table_view_opts = table_view_opts)
```