integrate changes from StatsModels 0.7 (upcoming release)

[kleinschmidt]

WIP. The major thing that needs to be addressed is how FunctionTerms are represented (JuliaStats/StatsModels.jl#183).

Don't merge until

• StatsModels 0.7 released
  • fix bug in hasintercept StatsModels.jl#275
  • Support unprotected mode for specials with FullRank schemas as well StatsModels.jl#274
• GLM release with StatsModels 0.7 compat released add compat for statsmodels 0.7 GLM.jl#512

closes #672

[kleinschmidt]

I've got most of the easy stuff sorted out now. The remaining issue is how implicit intercept/full-rank promotion is handled in the ranef terms. Our decision to :just: copy/paste the intercept handling stuff from statsmodels is coming back to bite now: the lhs of a ranef term like (0+f | g) is still a FunciontTerm{+} when we see it in apply_schema(::RanefTerm), so we can't do the normal "has intercept" checks. the mechanism that promotes it to a TupleTerm is...apply_schema, so we're in a kind of chicken/egg situation: we need to handle the implicit intercept behavior before applying schema to the rest, but we need a TupleTerm before we can handle the intercept behavior without some seriously bad and ugly hacks.

My current best idea is to use some kinda wrapper context like WithIntercept that we can use for dispatch. Then apply_schema with this context will carry it through until it gets to a TupleTerm, at which point it'll do the intercept detction/correction behavior and then continue with the original context. But I need to pay around a bit more to see whether that's workable and a good idea. If it is a good idea we can consider upstreaming it to statsmodels.

The other (hackier) thing to do would be to would be to use a "dummy schema" like Schema(Dict(t => t for t in terms(lhs))) to "unprotect" the lhs, then we can proceed with the intercept wrangling as usual.

[codecov]

Codecov Report

Patch coverage: 100.00% and no project coverage change.

Comparison is base (658aea1) 96.27% compared to head (03ce750) 96.28%.

❗ Current head 03ce750 differs from pull request most recent head ced6def. Consider uploading reports for the commit ced6def to get more accurate results

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #664   +/-   ##
=======================================
  Coverage   96.27%   96.28%           
=======================================
  Files          29       29           
  Lines        2740     2747    +7     
=======================================
+ Hits         2638     2645    +7     
  Misses        102      102           

Impacted Files	Coverage Δ
src/randomeffectsterm.jl	96.55% <100.00%> (+0.30%)	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[kleinschmidt]

this is ready for review with the caveat that I have no idea why the documenter build failed and haven't looked into it other than looking at the logs

[palday]

@kleinschmidt This is what's failing (it's a non sense model but it was chosen to show off the formula syntax on something quick to fit)

using MixedModels
sleepstudy = MixedModels.dataset(:sleepstudy)
fit(MixedModel, @formula(reaction ~ 1 + days + (1|subj) + zerocorr(days|subj)), sleepstudy,
    contrasts = Dict(:days => DummyCoding()))

[kleinschmidt]

@kleinschmidt This is what's failing (it's a non sense model but it was chosen to show off the formula syntax on something quick to fit)

using MixedModels
sleepstudy = MixedModels.dataset(:sleepstudy)
fit(MixedModel, @formula(reaction ~ 1 + days + (1|subj) + zerocorr(days|subj)), sleepstudy,
    contrasts = Dict(:days => DummyCoding()))

hmmmmmmmm that makes me think there may be an actual error there, it looks like (Intercept) is showing up twice when it should only be once...

[kleinschmidt]

AH I think I know what's happened here 😞

the hacky FunctionTerm{typeof(|)}-has-infinite-degree thing is causing the "bare" ranef to get sorted last:

julia> ff = @formula(reaction ~ 1 + days + (1|subj) + zerocorr(days|subj))
FormulaTerm
Response:
  reaction(unknown)
Predictors:
  1
  days(unknown)
  (days,subj)->zerocorr(days | subj)
  (subj)->1 | subj

I think we can work around this by adding a method for degree(::FunctionTerm{typeof(zerocorr)}) buuuut I don't like this precedent...