RRFSS Sample Design and Analysis

RRFSS is a series of ongoing monthly telephone surveys designed to monitor community trends in risk factors within the service area of participating health units.  The sample has been designed to represent the adult population 18 years and over, who speak either English or French and who reside in private households.  Note that occupied households without telephones are not included in the sample population, but according to Statistics Canada, these households are only about 3% of all Ontario households.

Choosing individuals in RRFSS basically follows a two-stage probability selection process.  The first stage involves the selection of households by randomly selecting residential telephone numbers.  A Random Digit Dialing (RDD) approach is used to select the phone numbers by randomly selecting from commercially available list of telephone numbers as well as using telephone numbers of either side of the listed numbers (to cater for numbers that might be unlisted or new).  The second stage, which is the disproportionate selection of an adult from a cluster of adult respondents in the household, is made by choosing the person with the most recent birthday.  Overall RRFSS can be considered as a disproportionate sample design within each public health unit.

Survey Analysis in RRFSS
Unweighted data in RRFSS are the actual responses of each participant. Unweighted data represent results before any adjustment is made either for variation in respondents' probability of selection, for disproportionate selection of population groups or subgroups relative to the overall population distribution, or for nonresponse. Weighted RRFSS data represent results that have been adjusted to compensate for such differences. 

As long as the sample weight has been computed, generating a point estimate (a single number that is the best estimate of the indicator) becomes a simple process of applying the appropriate sample design formula to the data.  Statistics can also be computed to assess the precision levels of the estimates, including the standard error (also referred to by ‘se’ and defined as the square root of sample variance), confidence intervals (range of values that describes the uncertainty around a point estimate) and the coefficient of variation (which measures the relative variability around a point estimate and is defined as the standard error divided by the point estimate).

Although standard data analysis software can be used to compute RRFSS point estimates, unless the survey design is a simple random sample (SRS), accurate generation of precision statistics requires appropriate survey analysis software that can accommodate complex survey designs.  In the absence of the survey analysis tools analysts can assume a simpler survey design or compute proxy measures for estimating the precision statistics.

When RRFSS commenced in 2001, the majority of data analysts did not have access to the survey analysis tools to compute accurate precision measures associated with the point estimates.  By assuming that the sample of respondents was representative of a random sample of 18+ individuals in the PHU, analysts applied standard formula for estimating the variance and other precision statistics.  These estimates are presented in the RQ System under Applying Simple Random Sampling.

More recently most of the popular statistical analysis programs, including SPSS which is the standard analysis tool for RRFSS analysts, have incorporated functionality for handling complex survey designs.  As a result some analysts have begun to apply this methodology for generating point estimates.  The RQ allows RRFSS analysts to access these estimates under Appling Complex Survey Sampling.

Methodology for Applying Simple Random Sampling

Household weights

This method of computing household level estimates is based on unweighted data so no household weights were generated.

Person weights

Formulation of person weights with this method is based on the principle that the number of observations in the unweighted data set must equal the observations in the weighted dataset.  The weights are calculated as follows:

·          Inclusion Probability
If we let:
n_h º # sample phone numbers selected from the h-th PHU
A_i º number of adults at each residence for the i^th respondent, then

Prob(include individual in sample) = ( n_h A_i / S n_h A_i )

·          Person Weight  w_ri = 1 / Prob(include individual in sample) 
w_ri = (S n_h A_i / n_h A_i)

Point Estimates

The table below outlines the formula for computing point estimates for the weighted percentage and weighted total.

Sample Variance

For a simple random sample, computation of the sample variance is given by (1-n_h/N_h) s²/n where n_h is the size of the sample, N_h is the size of the target population and s² =p(1-p)/n_h , if p is  the proportion as computed from the above formula.  The (1-n_h/N_h) component of the sample variance formula is referred to as the finite population correction factor which can be omitted if n_h is small relative to N_h.  There the sample variance formula for Applying Simple Random Sampling is estimated as s²/n .

Methodology for Applying Complex Survey Sampling

The steps involved in formulating of the sample weights for the Complex Survey Sampling method are outlined below for both households and persons.  In addition to formulating the basic sample weight, adjustments have been made for differential seasonal effects and for representation of the overall number of households or persons in the population (post-stratification).  The adjustments for seasonal effects have been integrated to ensure that each month (or survey wave) is represented by one twelfth (or the reciprocal of all available months) of the total sample.

·          Inclusion Probability
If we let:
n_h º # sample phone numbers selected from the h-th PHU
N_h º # phone numbers on sample frame in the h-th PHU              
[approximated by number of households in last census]

Prob(include individual’s residence) = ( n_h / N_h )

If A_i º number of adults at each residence for the i^th respondent, then
Prob(include individual within the individual’s residence) = 1 / A_{i ,} and
Prob(include individual within h-th PHU)  = ( n_h / N_h ) * ( 1 / A_i )

·          Initial Household Weight  w_hi = 1 / Prob(include residence within h-th PHU)  
w_hi = ( N_h / n_h )

Initial Person Weight  w_hi = 1 / Prob(include individual within h-th PHU) 
w_pi = ( N_h / n_h ) * ( A_i )

Note that for the Ottawa PHU households were divided into a French and non-French strata.

·         Weight adjustment for seasonal effect – household: w_hi* , person: w_pi*
If  s  represents one of twelve seasonal periods (survey wave), then

·         Post stratified weights

Household weight w_hi**
If N_w º Sum of seasonally adjusted household weights (S w_hi*) for the PHU,  then

Note: Total occupied private dwellings from 2001 census was used to estimate total households within PHU for 2001-2005 and total occupied private dwellings from 2006 census was used to estimate total households within PHU for 2006-2007.

Post stratified person weight w_pi** for each sex, age group: 18-44, 45-64, 65+, and for Ottawa French and non-French speaking strata.

If M_p º Population estimate for the PHU, sex and age group of i-th respondent,  and
M_p º Sum of seasonally adjusted weights (S w_pi*) for the PHU, sex and age group of i-th respondent,  then

Estimate of variance

Variance estimates of RRFSS indicators have been computed using Taylor’s Series Linearization.  The approximate formula for the variance of the mean (ignoring the finite population correction factor) is given below:

where for the h-th PHU, n-th household, i-th respondent, total of H PHU’s, total sample of n_h households in the h-th PHU, and ..

RRFSS General Guidelines for Analysis

These guidelines are based on the analysis of the 1999 RRFSS pilot project, the 2001 RRFSS data and the general knowledge, experience and technical expertise of the RRFSS Analysis Group. 

1.        Unweighted denominator data – cell sizes less than 30 should be suppressed.

2.        Unweighted numerator data – cell sizes less than 5 should be suppressed.

3.        The following categories for Coefficient of variation (CV) determine the reliability of the estimates:

a.        CV between 0 and 16.5% are deemed to be acceptable for reporting

b.        CV between 16.6% and 33.3% are to be ‘Interpreted with caution’

c.        CV greater than 33.3% should be suppressed.

4.        95% confidence intervals (CI) should accompany all point estimates.  If we define the percentage of a particular characteristic as p, then the 95% CI can be computed as
p ± 1.96 * se .

5.        If weighted cell sizes of the ‘Don’t know’ or ‘Refusal’ responses are 5% or greater then these responses should be included in the analysis and separately reported.