cvms

R package for model evaluation and comparison.

  • Cross-validate one or multiple regression or classification models with relevant evaluation metrics in a tidy format.
  • Validate the best model on a test set and compare it to a baseline evaluation.
  • Perform hyperparameter tuning with grid search.
  • Evaluate predictions from an external model.
  • Extract the observations that were the most challenging to predict.

Currently supports regression ('gaussian'), binary classification ('binomial'), and (some functions only) multiclass classification ('multinomial'). Many of the functions allow parallelization, e.g. through the doParallel package.

GitHub repository: https://github.com/LudvigOlsen/cvms

CRAN index: https://cran.r-project.org/package=cvms

Installation

CRAN

install.packages("cvms")

Development version

install.packages("devtools")

devtools::install_github("LudvigOlsen/groupdata2")

devtools::install_github("LudvigOlsen/cvms")

Main functions

Function Description
cross_validate() Cross-validate linear models with lm()/lmer()/glm()/glmer()
cross_validate_fn() Cross-validate a custom model function
validate() Validate linear models with (lm/lmer/glm/glmer)
validate_fn() Validate a custom model function
evaluate() Evaluate predictions with a large set of metrics
baseline()
baseline_gaussian()
baseline_binomial()
baseline_multinomial()
Perform baseline evaluations of a dataset

Evaluation utilities

Function Description
confusion_matrix() Create a confusion matrix from predictions and targets
evaluate_residuals() Evaluate residuals from a regression task
most_challenging() Find the observations that were the most challenging to predict
summarize_metrics() Summarize numeric columns with a set of descriptors

Formula utilities

Function Description
combine_predictors() Generate model formulas from a list of predictors
reconstruct_formulas() Extract formulas from output tibble
simplify_formula() Remove inline functions with more from a formula object

Plotting utilities

Function Description
plot_confusion_matrix() Plot a confusion matrix
plot_metric_density() Create a density plot for a metric column
font() Set font settings for plotting functions (currently only plot_confusion_matrix())

Custom functions

Function Description
model_functions() Example model functions for cross_validate_fn()
predict_functions() Example predict functions for cross_validate_fn()
preprocess_functions() Example preprocess functions for cross_validate_fn()
update_hyperparameters() Manage hyperparameters in custom model functions

Other utilities

Function Description
select_metrics() Select the metric columns from the output
select_definitions() Select the model-defining columns from the output
gaussian_metrics()
binomial_metrics()
multinomial_metrics()
Create list of metrics for the common metrics argument
multiclass_probability_tibble() Generate a multiclass probability tibble

Datasets

Name Description
participant.scores Made-up experiment data with 10 participants and two diagnoses
wines A list of wine varieties in an approximately Zipfian distribution
musicians Made-up data on 60 musicians in 4 groups for multiclass classification
predicted.musicians Predictions by 3 classifiers of the 4 classes in the musicians dataset
precomputed.formulas Fixed effect combinations for model formulas with/without two- and three-way interactions
compatible.formula.terms 162,660 pairs of compatible terms for building model formulas with up to 15 fixed effects