wyoung

WYOUNG: control the family-wise error rate when performing multiple hypothesis tests

Overview

wyoung is a Stata command that controls the family-wise error rate using the free step-down resampling methodology of Westfall and Young (1993). This method leverages resampling techniques, such as bootstrapping (sampling with replacement) or permutation (shuffling), to adjust the standard p-values obtained from model estimation. The command also computes the Bonferroni-Holm and Sidak-Holm adjusted p-values. Detailed documentation, including simulation results, is available here. Syntax and usage instructions can be accessed directly in Stata by typing help wyoung at the command prompt.

This command was developed as part of the Illinois Workplace Wellness Study.

Installation

Type which wyoung at the Stata prompt to determine your current version number. To install the most recent version, copy and paste the following line of code:

net install wyoung, from("https://raw.githubusercontent.com/reifjulian/wyoung/master") replace

To install the latest version that was uploaded to SSC, copy and paste the following line of code:

ssc install wyoung, replace

After installing, type help wyoung to learn the syntax.

Examples

Example 1. Estimate a model separately for three outcomes (mpg, headroom, and turn) and calculated adjusted p-value for displacement (3 hypotheses).

sysuse auto.dta, clear
set seed 20
wyoung mpg headroom turn, cmd(regress OUTCOMEVAR displacement length) familyp(displacement) reps(100)

Example 1 For each regression, the output provides both unadjusted and adjusted p-values for testing the null hypothesis that the coefficient on the variable displacement equals 0. For example, in the regression regress turn displacment length, the unadjusted p-value is 0.09, while the Westfall-Young adjusted p-value is 0.14. The reps(100) option specifies 100 bootstrap replications, which is the default setting and is omitted in the examples below for simplicity.

Example 2. Estimate a model separately for three outcomes and for two subgroups defined by foreign (3 X 2 = 6 hypotheses).

sysuse auto.dta, clear
set seed 20
local yvars "mpg headroom turn"
wyoung `yvars', cmd(reg OUTCOMEVAR displacement length) familyp(displacement) subgroup(foreign)

Example 2

Example 3. Estimate a model for three outcomes, for two subgroups defined by foreign, and calculate adjusted p-values for both displacement and length (3 X 2 X 2 = 12 hypotheses).

sysuse auto.dta, clear
set seed 20
local yvars "mpg headroom turn"
wyoung `yvars', cmd(reg OUTCOMEVAR displacement length) familyp(displacement length) subgroup(foreign)

Example 3

Example 4. Estimate a model for three outcomes and test the linear restriction _b[length] + 50*_b[displacement] = 0 (3 hypotheses).

sysuse auto.dta, clear
set seed 20
local yvars "mpg headroom turn"
wyoung `yvars', cmd(reg OUTCOMEVAR displacement length) familyp(length+50*displacement) familypexp

Example 4

Permutation

By default, wyoung uses bootstrapping to resample the data. Alternatively, the permute() can be used to perform permutation-based resampling. As with bootstrapping, permutation-based resampling can incorporate stratification and clustering via the strata() and cluster() options (see Example 5 below). For more complex treatment assignment schemes, users can specify a custom program using the permuteprogram() option (see Example 6). In this case, the contents of permute() are passed as the first argument to the custom program. Refer to the Stata help file (help wyoung) for additional details.

Example 5. Perform the Westfall-Young adjustment using permutation with a stratified random sample (3 hypotheses).

sysuse auto.dta, clear
set seed 20
gen stratum = floor(mpg/11)
gen treat = foreign
local yvars "mpg headroom turn"
wyoung `yvars', cmd(regress OUTCOMEVAR treat) familyp(treat) permute(treat) strata(stratum)

Example 5

Example 6. Perform the Westfall-Young adjustment using permutation with a customized assignment program (3 hypotheses).

program define myshuffle
	syntax varname [, *]
	tempvar n_init randsort shuffled 

	gen long `n_init' = _n
	gen double `randsort' = uniform()
	sort `randsort', stable
	gen `shuffled' = `varlist'[`n_init']

	drop `varlist'
	ren `shuffled' `varlist'
end

sysuse auto, clear
set seed 20 
gen treat = foreign
local yvars "price headroom mpg"
wyoung `yvars', cmd(regress OUTCOMEVAR treat) familyp(treat) permute(treat) permuteprogram(myshuffle)

Example 6

Update History

Citation

wyoung is not an official Stata command. It is a free contribution to the research community. You may cite it as:

Jones, D., D. Molitor, and J. Reif. “What Do Workplace Wellness Programs Do? Evidence from the Illinois Workplace Wellness Study.” Quarterly Journal of Economics, November 2019, 134(4): 1747-1791.

This site is open source. Improve this page.