better cv resamples

[dajmcdon]

As best I can tell, the rsample package has rolling_origin sampling and group_vfold_cv. It would be nice to get the rolling origin version to work with an epi_df appropriately.

[dajmcdon]

Update: In rsample there is sliding_index() which is almost the right thing. To get it to work for an epi_df, we should either:

1. Wrap it with more informative arguments and to correctly handle possibly differently-sized panels (if geos / keys enter or leave).
2. Write our own function based off it.

The first is probably easier and avoids using ::: funs in rsample but makes us have to deal with possible breaking changes.

library(rsample)

tib <- tibble::tibble(
  tv = rep(1:20, each = 2), 
  val = rnorm(40), 
  gv = rep(letters[1:2], times = 20)
)

r1 <- rolling_origin(tib)
r1$splits[[1]]$in_id
#> [1] 1 2 3 4 5
r1$splits[[1]]$out_id
#> [1] 6
# ignores everything, simply imagines that there are evenly spaced,
# sorted time values with a single obs per time value
 
r2 <- sliding_index(tib, tv)
# seems close, use some args to modify

r3 <- sliding_index(tib, tv, lookback = 3, assess_start = -3)
r3$splits[[1]]$in_id
#> [1] 1 2 3 4 5 6 7 8
r3$splits[[3]]$in_id
#> [1]  3  4  5  6  7  8  9 10
# result has the training set with both geo_vals of size 4 time values
# (`lookback + 1`)
r3$splits[[1]]$out_id
#>  [1]  1  2  3  4  5  6  7  8  9 10
r3$splits[[3]]$out_id
#>  [1]  3  4  5  6  7  8  9 10 11 12
# the negative `assess_start` includes all training data in the validation
# set. So, we sould calibrate this based on the number of necessary lags.
# 
# Problem 1: the splits are repeated
all(r3$splits[[2]]$in_id == r3$splits[[1]]$in_id)
#> [1] TRUE
all(r3$splits[[2]]$out_id == r3$splits[[1]]$out_id)
#> [1] TRUE

# Fix with `step`
r4 <- sliding_index(tib, tv, lookback = 3, assess_start = -3, step = 2)
r4$splits[[1]]$in_id
#> [1] 1 2 3 4 5 6 7 8
r4$splits[[1]]$out_id
#>  [1]  1  2  3  4  5  6  7  8  9 10
r4$splits[[2]]$in_id
#> [1]  3  4  5  6  7  8  9 10
r4$splits[[2]]$out_id
#>  [1]  3  4  5  6  7  8  9 10 11 12
# This is exactly what we want for this data (modulo the right inclusion
# of training data in the validation set to create lags)
#
# * However, if the number of geos varies over time, then we have a problem
# 
# Docs: Index can't contain missing values. Does this mean NA or gaps?
# 
# `complete` determines whether we initial, growing, training sets
# The `step` arg is potentially useful if we want to do weekly or 
# * Can't seem to increment by anything other than 1 directly.
#   We could multiply `step` if wanted fewer folds
#   `sliding_period()` has `every` which could solve this, but disallows
#   negative `assess_start`

^{Created on 2022-10-14 with reprex v2.0.2}

[dajmcdon]

But see this Gist from Davis Vaughan to make sure we're not leaking data.