Sequential Multiple Assignment Randomized Trials (SMART)

Summary of the Previous Session

In the previous session, we explored the Multiphase Optimization Strategy (MOST). The focus was on designing, optimizing, and evaluating an experiment before testing it through a randomized controlled trial (RCT). MOST is an excellent approach for developing experiments that are efficient, affordable, scalable, and effective. However, its primary limitation is that once the experiment is designed and optimized, it remains fixed—it is essentially a “one-shot” experiment that cannot be adjusted at least in the short term. While MOST provides insights on the optimal combination of components, it does not allow us to examine the optimal sequence of treatment components.

This rigidity can be a drawback in situations where interventions might benefit from adaptation over time. Since participants may respond differently to an intervention, it can be advantageous to modify the treatment based on early feedback. For instance, if a participant responds well, the same treatment can be continued; if not, switching to an alternative might yield better outcomes.

To address this need for flexibility, we will be covering the SMART (Sequential Multiple Assignment Randomized Trial) methodology in this session. SMART offers a structured way to adapt interventions over time, tailoring treatments to individual responses and potentially enhancing overall effectiveness.

What is SMART?

SMART is an experimental design used to develop and evaluate adaptive interventions—strategies that change based on how people respond over time. Unlike a traditional RCT, which tests one fixed approach against another in a single go, SMART involves multiple stages of randomization. At each stage, you assess how participants are doing (via intermediate outcomes) and use pre-set decision rules to adapt the intervention for the next step. It’s perfect for situations where one-size-fits-all doesn’t work—like in business, where customer or employee responses can vary widely.

SMART Implementation of the Stress Reduction App at University of Miami

SMART builds on optimization frameworks like MOST. While MOST enables us to determine which components—such as physical activity, social support, and mindfulness—are most effective, SMART takes the next step by testing how to sequence and adapt these components based on individual responses during the intervention. Following our example from the previous two sessions, in our stress reduction app, this means we can identify the optimal adaptive path to reduce student stress, ensuring that each participant receives the most effective intervention according to their unique needs. Let’s take the stress reduction app as an example to demonstrate how we can implement SMART methodology.

Step 1: Define the Problem and Goal

Problem: University of Miami students experience varying levels of stress, and a single fixed intervention (e.g., just mindfulness) may not work for all—some need different support as they progress.

Goal: Develop an adaptive stress reduction app that adjusts its features over 8 weeks to minimize self-reported stress scores (e.g., on a 0–10 scale).

Components (from MOST): We’ll use the three optimized components. Here we can assume that the following three components were determined to be the most effective ones based on the prior factorial analysis in MOST.

• Physical Activity: Daily step goals or exercise prompts.

• Social Support: Group chats or buddy check-ins.

• Mindfulness Exercises: Guided meditation sessions.

Step 2: Set Up the First Stage (Initial Randomization)

Intervention Options: Start with two initial app versions. The selection of these two components is based on the assumption that our MOST optimization results indicate that these are top two most effectuve components.

• Option A: Physical Activity (e.g., “Walk 5,000 steps daily”).

• Option B: Mindfulness Exercises (e.g., “10-minute meditation daily”).

Randomization: Randomly assign 200 students between the two treatment options A and B. Run this stage for 4 weeks. Here, we’re testing two promising components from MOST, but we expect some students won’t respond well initially and need adaptation. (Social support will come into play later.)

We have a synthetic dataset that represents the data collected through a SMART design. The following SAS code shows the number of observations in each option along with their average level of baseline stress. We also run a simple t-test to examine the equality of means between the two groups.

/* Tabulate stress_base and stress_week4 by stage1 using PROC TABULATE */
PROC TABULATE DATA=stress_app_synthetic;
    CLASS stage1;
    VAR stress_base stress_week4;
    TABLE stage1 ALL, 
          N='Sample Size' 
          stress_base * (MEAN='Mean' STD='Std Dev') 
          stress_week4 * (MEAN='Mean' STD='Std Dev') / RTS=20;
    FORMAT stress_base stress_week4 6.2;
    
RUN;

/* Independent t-test to compare baseline stress between stage1 groups */
PROC TTEST DATA=stress_app_synthetic;
    CLASS stage1;
    VAR stress_base;
    TITLE "T-Test: Baseline Stress by Stage 1 Treatment";
RUN;

In the synthetic data, we can observe that 88 people were assigned to option A, and the remaining 112 people were assigned to option B. The randomization is implemented correctly, because we can see that there is no statistically significant difference between the groups when it comes to baseline stress.

Step 3: Measure an Intermediate Outcome

What to Measure: Stress reduction after 4 weeks, via self-reported stress scores (0–10). We are going to define a threshold based on the level of improvement in the stress. If an individual has shown significant improvement, we call them responder, otherwise, they are non-responder because they have not responded well to our treatment.

• Responders: Stress reduction $\geq 2$ points (e.g., from 8 to 6—they’re improving).

• Non-Responders: Stress reduction $< 2$ points (e.g., from 8 to 7 or no change—they need a tweak).

In the synthetic data, we have the following

• Option A (Physical Activity): 65 responders (stress dropped 2+), 23 non-responders.

• Option B (Mindfulness): 70 responders, 42 non-responders.

/* Calculate stress reduction and summarize by path */
DATA stress_paths;
    SET stress_app_synthetic;
    stress_reduction = stress_base - stress_week4; /* Positive = reduction */
RUN;
    PROC TABULATE DATA=stress_paths;
    CLASS path;
    VAR stress_reduction;
    TABLE path ALL, 
          N='Sample Size' 
          stress_reduction * (MEAN='Mean Reduction' );
    TITLE "SMART Paths: Stress Reduction by Path";
RUN;

Step 4: Define Decision Rules for Adaptation

Here we should either rely on theory, analytical optimization models, results from prior studies, or pragmatic considerations to define a set of rules that determine what we should do based on the response levels of participants to our initial intervention components. Here is an example:

• If Responder: Keep their current intervention (A or B) for weeks 5–8—it’s working.

• If Non-Responder: Re-randomize to a new option incorporating the third component (social support) or a combination:

  • Option C: Social Support (e.g., “Join a stress buddy group”).

  • Option D: Combo of their Stage 1 option + Social Support (e.g., Physical + Social or Mindfulness + Social).

Step 5: Second Stage (Adaptive Randomization)

Responders (135 total):

• 65 from Option A stay on Physical Activity.

• 70 from Option B stay on Mindfulness Exercises.

Non-Responders (65 total):

• 23 from Option A:

  • Randomize 9 to Option C (Social Support).

  • Randomize 14 to Option D (Physical + Social).

• 42 from Option B:

  • Randomize 27 to Option C (Social Support).

  • Randomize 15 to Option D (Mindfulness + Social).

Duration: Run this stage for 4 more weeks (weeks 5–8).

Step 6: Measure the Final Outcome

What to Measure: Stress score at week 8 (with the goal of achieving the lowest possible score).

• A $\rightarrow$ A (Physical kept): Average stress = 2.30.

• B $\rightarrow$ B (Mindfulness kept): Average stress = 2.20.

• A $\rightarrow$ C (Physical to Social): Average stress = 4.80.

• A $\rightarrow$ D (Physical to Physical + Social): Average stress = 5.13.

• B $\rightarrow$ C (Mindfulness to Social): Average stress = 5.34.

• B $\rightarrow$ D (Mindfulness to Mindfulness + Social): Average stress = 4.85.

The following SAS code calculates the descriptive statistics per each path.

PROC MEANS DATA=stress_paths MEAN STD MIN MAX N;
    CLASS path;
    VAR stress_week8;
    TITLE "Descriptive Statistics for Stress at Week 8 by Path";
RUN;

Step 7: Interpret the Adaptive Paths

Key Findings:

• Physical Responders: Best to continue with Physical Activity (Avg. stress = 2.30).

• Mindfulness Responders: Best to continue with Mindfulness Exercises (Avg. stress = 2.20).

• Physical Non-Responders: Benefit most from switching to Social Support (Avg. stress = 4.8).

• Mindfulness Non-Responders: Benefit most from switching to a combination of Mindfulness and Social Support (Avg. stress = 4.85).

Empirical Methods for Analyzing SMART Data

Analyzing data from a SMART design involves methods that account for its multi-stage, adaptive structure:

1. Descriptive Analysis: Summarize the outcomes for each adaptive path using measures such as means, standard deviations, and visualizations (e.g., bar charts). This is what we have done so far already.

2. Regression-Based Methods: Model the final stress scores as a function of baseline stress and treatment path. For example:

   $$\begin{equation} \text{stress}_{\text{week8}, i} = \beta_0 + \sum_{j=1}^{5} \beta_j \text{Path}_{j,i} + \beta_6 \text{stress}_{\text{base}, i} + \epsilon_i \end{equation}$$ where $\text{stress}_{\text{week8}, i}$ is the outcome for individual $i$, $\beta_0$ is the intercept (path A_A at zero baseline stress), $\text{Path}_{j,i}$ are dummies for paths (A_A as reference), $\beta_j$ are their coefficients, $\beta_6$ is the baseline stress effect, and $\epsilon_i \sim N(0, \sigma^2)$.

   
   proc glm data=stress_paths;
       class path (ref="A_A");
       model stress_week8 = path stress_base /  solution;
   
       /* Contrast A-based vs. B-based paths */
       contrast "A vs. B Paths" path 1 1 1 -1 -1 -1; /* A_A, A_C, A_D vs. B_B, B_C, B_D */
       /* Contrast Responders vs. Non-Responders */
       contrast "Responders vs. Non-Responders" path 1 0 0 1 -1 -1; /* A_A, B_B vs. B_C, B_D */
       
       lsmeans path / adjust=tukey pdiff;  /* Adjusted means and pairwise comparisons */
       title "GLM Analysis of Stress at Week 8 by Treatment Path, Adjusted for Baseline Stress";
   run;

   Listing 6.4. SAS Code to run the GLM model

   l S[table-format=-1.3] S[table-format=1.3] S[table-format=-2.2] S[table-format=1.4] Parameter & Estimate & Standard Error & $t$ Value & $Pr > |t|$
   Intercept & -4.213 & 0.235 & -17.89 & <0.0001
   path A_C & 3.024 & 0.242 & 12.49 & <0.0001
   path A_D & 2.520 & 0.200 & 12.59 & <0.0001
   path B_B & -0.049 & 0.117 & -0.42 & 0.676
   path B_C & 3.079 & 0.155 & 19.81 & <0.0001
   path B_D & 2.443 & 0.194 & 12.57 & <0.0001
   path A_A & 0.000 & . & . & .
   stress_base & 0.949 & 0.032 & 29.64 & <0.0001
   

   The intercept ($-4.213$, $p < 0.0001$) in this GLM model represents the predicted stress at week 8 for path A_A when baseline stress is zero. Adjusted for baseline stress ($0.949$ per unit, $p < 0.0001$), A_A and B_B ($-0.049$, $p = 0.6755$) are the most effective paths, with significantly lower stress than A_C ($3.024$, $p < 0.0001$), A_D ($2.520$, $p < 0.0001$), B_C ($3.079$, $p < 0.0001$), and B_D ($2.443$, $p < 0.0001$), highlighting the superiority of keeping responders on their initial treatment.

   Table 6.5. Contrast Results for GLM Analysis of Stress at Week 8

   Contrast	DF	Contrast SS	Mean Square	F Value	Pr $>$ F
   A vs. B Paths	1	0.00232	0.00232	0.01	0.9435
   Responders vs. Non-Responders	1	57.87872	57.87872	125.65	$<$0.0001

   The GLM contrast analysis reveals no significant difference between A-based (A_A, A_C, A_D) and B-based (B_B, B_C, B_D) paths in terms of stress at week 8 ($F = 0.01$, $p = 0.9435$), suggesting that the initial treatment choice (A or B) does not substantially affect outcomes when averaged across all adaptations. However, a highly significant difference exists between responders (A_A, B_B) and non-responders switched from B to C or D (B_C, B_D) ($F = 125.65$, $p < 0.0001$), with responders exhibiting markedly lower stress, underscoring the effectiveness of maintaining responders on their initial treatment. This contrast excludes non-responders from A (A_C, A_D), limiting its scope, but aligns with prior findings that A_A and B_B outperform most non-responder paths.

   Understanding LSMeans vs. Model Estimates in PROC GLM

   /* 3. Summarize LSMEANS for Optimal Strategy */
   proc glm data=stress_paths;
       class path (ref="A_A");
       model stress_week8 = path stress_base;
       lsmeans path / adjust=tukey stderr; /* Get LSMEANS with standard errors */
       ods output lsmeans=lsmeans_table; /* Save LSMEANS to dataset */
       title "LSMEANS for Ranking Treatment Paths";
   run;
   proc sort data=lsmeans_table;
       by lsmean; /* Sort by adjusted mean stress */
   run;
   proc print data=lsmeans_table;
       var path lsmean stderr;
       title "Ranked LSMEANS of Stress at Week 8 by Path";
   run;

   Listing 6.5. SAS Code to summerize the optimal strategy

   Table 6.6. LSMeans Estimates for Treatment Paths from PROC GLM

   Path	LSMean	Standard Error	Pr $> |t|$	LSMean Number
   A_C	5.3087	0.2269	$<0.0001$	1
   A_D	4.8049	0.1817	$<0.0001$	2
   B_B	2.2358	0.0811	$<0.0001$	3
   B_C	5.3635	0.1306	$<0.0001$	4
   B_D	4.7281	0.1753	$<0.0001$	5
   A_A	2.2849	0.0842	$<0.0001$	6

   LSMeans (Least Squares Means) are the adjusted average stress levels at week 8 for each path, factoring in baseline stress (stress_base). They tell us what stress we’d expect if everyone started with the same baseline, making it a breeze to compare paths directly. For ranking strategies—like finding the best path to reduce stress—LSMeans are our go-to numbers.

   Here’s what we get from our proc glm run:

   Table 6.7. LSMeans Estimates for Treatment Paths from PROC GLM

   Path	LSMean	Standard Error	Pr > |t|	LSMean Number
   A_C	5.3087	0.2269	$<0.0001$	1
   A_D	4.8049	0.1817	$<0.0001$	2
   B_B	2.2358	0.0811	$<0.0001$	3
   B_C	5.3635	0.1306	$<0.0001$	4
   B_D	4.7281	0.1753	$<0.0001$	5
   A_A	2.2849	0.0842	$<0.0001$	6

   Look at these: B_B (2.2358) and A_A (2.2849) show the lowest stress—our top picks! Meanwhile, A_C (5.3087) and B_C (5.3635) are the highest, suggesting they’re less effective. The small standard errors (e.g., 0.0811 for B_B) mean these numbers are precise, and the p-values ($<0.0001$) confirm they’re real effects—though we care more about comparing them to each other than to zero.

   How They Differ from Model Estimate Coefficients

   Model estimate coefficients show how each path’s stress shifts compared to our reference, A_A. They’re like the building blocks of the model—slopes telling us the difference from A_A’s baseline stress, adjusted for stress_base. For example, if A_C’s coefficient is 3.0239, it means A_C is 3.0239 units higher than A_A’s 2.2849, landing close to 5.3088—matching its LSMean.

   So why pick LSMeans over coefficients?

   • Ready-to-Use: LSMeans are the final averages—like 5.3087 for A_C—no extra math needed. Coefficients (e.g., 3.0239 for A_C) require you to add them to A_A’s value to get the full story.

   • Big Picture: LSMeans give us each path’s stress level outright, perfect for ranking. Coefficients are just the “shift” from A_A, less intuitive for quick comparisons.

   • Extra Perks: LSMeans come with standard errors and can pair with Tukey-adjusted differences (e.g., B_B vs. A_C), showing which paths win significantly—coefficients don’t do that alone.

   LSMeans are our shortcut to the answers we want: which path reduces stress best? They skip the behind-the-scenes details of coefficients and deliver clear, adjusted averages.

3. Marginal Structural Models (MSM) to Address the Confounding Issue:

   In a SMART design, Stage 1 randomly assigns A or B, but Stage 2 depends on week 4 response: responders stay on None (e.g., A_A, B_B), while non-responders get C or D (e.g., A_C, A_D). This adaptive process causes confounding because intermediate (stress_week4) and baseline (stress_base) factors affect both the path and outcome (stress_week8), biasing the GLM estimates. For example, A_D might seem worse due to higher stress in non-responders, not D itself.

   Picture this: in our SMART experiment, some people end up on $A\_D$ because they’re struggling at week 4—high stress, non-responders. Meanwhile, $A\_A$ gets the “easier” crowd—low stress, responders. When we compare their week 8 stress, $A\_D$ looks worse, but is it really $D$’s fault? Or is it just that $A\_D$ got stuck with tougher cases? That’s the confounding mess we need to fix.

   Inverse Probability Weighting (IPW) is like a magic scale adjuster. It looks at each person and asks, “How likely were you to land on your path, given your starting stress and week 4 stress?”

   • If someone had high week 4 stress and was almost guaranteed to get $A\_D$, IPW says, “You’re too common in this group—let’s shrink your influence a bit” and gives them a small weight (like 0.5).

   • If someone had low stress and still ended up on $A\_A$ when they could’ve switched, IPW says, “You’re rare—let’s boost your voice” and gives them a big weight (like 2).

   By tweaking everyone’s “volume” this way, IPW creates a pretend world where paths aren’t stacked with certain types of people. It’s like giving everyone an equal chance to be on any path, no matter their stress levels. Now, when we compare $A\_A$ and $A\_D$, we’re seeing what each path actually does to stress—not just who they got assigned to.

   Step 1: Figure Out the Odds (Assigning Probabilities)

   We start by figuring out how likely each person is to end up on their actual path—like A_D or B_B—based on their situation. For Stage 1, everyone gets A or B, and since it’s random, we assume a 50% chance for each. In Stage 2, it depends: responders stick with None (e.g., A_A), so their probability is 100%. Non-responders switch to C or D (e.g., A_C, A_D), and that chance hinges on their starting stress (stress_base), week 4 stress (stress_week4), and initial treatment (stage1).
   It’s like guessing someone’s next move in a game—knowing their odds helps us spot who’s typical or surprising on their path.

   /* Stage 1: Probability of A vs. B (assuming equal randomization) */
   data stress_paths;
       set stress_paths;
       if stage1 = "A" then prob_stage1 = 0.5;
       else if stage1 = "B" then prob_stage1 = 0.5;
   run;
   
   
   

   Listing 6.6. SAS Code to calculate and assign probabilities to each observationl

   Step 2: Let Logistic Regression Do the Guesswork

   For non-responders, we use proc logistic to predict their chance of getting D over C. We feed it stress_base, stress_week4, and stage1, asking: “Given this person’s vibe, how likely are they to land on D?”

   /* Stage 2: Probability of treatment conditional on responder status */
   proc logistic data=stress_paths;
       where responder = 0; /* Non-responders only */
       class stage2 (ref="C") stage1 (ref="A"); 
       model stage2 (event="D") = stress_base stress_week4 stage1;
       output out=stage2_probs pred=prob_stage2;
   run;
   
   proc sort data=stress_paths;
       by id;
   run;
   
   proc sort data=stage2_probs;
       by id;
   run;
   
   /* Now merge the sorted datasets */
   data stress_paths_weighted;
       merge stress_paths stage2_probs (keep=id prob_stage2);
       by id;
       if responder = 1 then prob_stage2 = 1; /* Assign 1 for responders */
       if prob_stage2 = . then prob_stage2 = 1; /* Fill missing values */
   run;

   Listing 6.7. SAS Code to use logistic regression to calculate probabilities

   Step 3: Flip the Odds (Inverse Probabilities)

   Next, we flip those probabilities upside down to make weights. If someone’s got a 70% chance (0.7) of D, their weight is $1 / 0.7 \approx 1.43$. For their full path (Stage 1 $\times$ Stage 2), say 0.5 $\times$ 0.7 = 0.35, the weight becomes $1 / 0.35 \approx 2.86$. Responders? Their Stage 2 probability is 1, so their weight is just $1 / 0.5 = 2$.
   This is the fairness trick—big weights boost rare cases, small weights quiet down the usual suspects, leveling the field. In the code below, stabilized IPW is used to ensure that the weights do not get too large.

   data stress_paths_weighted;
       set stress_paths_weighted;
       ipw = 1 / (prob_stage1 * prob_stage2);
       if stage1 = "A" then stab_ipw = 0.5 * ipw;
       else if stage1 = "B" then stab_ipw = 0.5 * ipw;
   run;
   

   Listing 6.8. SAS Code to calculate stabilized IPWs

   Step 4: Use the Weights to Rebalance Everything

   We plug these weights into proc genmod with weight stab_ipw, tweaking how much each person’s stress score counts. A rare A_A person (weight = 3) counts triple, while a common A_D person (weight = 0.5) counts half.
   Why it matters: It’s like remixing a playlist—weights turn down overplayed cases and crank up the rare ones, so every path gets a fair shot in our analysis.

   
   proc genmod data=stress_paths_weighted;
       class path (ref="A_A");
       model stress_week8 = path stress_base / dist=normal link=identity;
       weight stab_ipw;
       lsmeans path / diff adjust=tukey;
       ods output lsmeans=msm_lsmeans;
       title "MSM Estimates of Path Effects with IPW";
   run;

   Listing 6.9. SAS code to estimate the smart coefficients using stabilized weights

   Table 6.8. Analysis of Maximum Likelihood Parameter Estimates

   Parameter	DF	Estimate	Standard Error	Wald 95% Confidence Limits	Wald Chi-Square	Pr $>$ ChiSq
   Intercept	1	-4.2918	0.2378	[-4.7580, -3.8257]	325.62	$<0.0001$
   path A_C	1	3.0851	0.2190	[2.6558, 3.5144]	198.36	$<0.0001$
   path A_D	1	2.5802	0.1968	[2.1945, 2.9659]	171.90	$<0.0001$
   path B_B	1	-0.0484	0.1391	[-0.3211, 0.2243]	0.12	0.7280
   path B_C	1	3.2477	0.1304	[2.9922, 3.5033]	620.40	$<0.0001$
   path B_D	1	2.5763	0.1604	[2.2619, 2.8907]	257.98	$<0.0001$
   path A_A	0	0.0000	0.0000	[0.0000, 0.0000]	–	–
   stress_base	1	0.9604	0.0314	[0.8989, 1.0220]	934.94	$<0.0001$
   Scale	1	0.8077	0.0404	[0.7323, 0.8908]	–	–

   Table 6.9. Path Least Squares Means

   Path	Estimate	Standard Error	z Value	Pr $>|z|$
   A_C	5.3697	0.1947	27.58	$<0.0001$
   A_D	4.8648	0.1694	28.72	$<0.0001$
   B_B	2.2362	0.0965	23.16	$<0.0001$
   B_C	5.5323	0.0834	66.36	$<0.0001$
   B_D	4.8609	0.1253	38.80	$<0.0001$
   A_A	2.2846	0.1002	22.80	$<0.0001$

   The proc genmod LSMeans, adjusted with IPW (e.g., B_B = 2.2362, B_C = 5.5323), show each path’s true stress_week8, fixing confounding that proc glm (e.g., B_B = 2.2358, B_C = 5.3635) didn’t catch. Both rank B_B and A_A best (around 2.23–2.28) and A_C and B_C worst (5.3–5.5), but IPW nudges C and D paths up (e.g., B_C +0.1688), showing they’re less effective than proc glm hinted, due to biased assignments. Parameter estimates (e.g., A_C = 3.0851) shift from A_A like in proc glm, but IPW refines them.

4. Power Analysis:

   Simulating power for an MSM in a SMART.

   For MSM with IPW in our SMART—comparing paths like A_A vs. B_B—simulation beats formulas due to adaptive splits and weights. We assume A_A stress = 2.5, B_B = 2.0 ($\Delta = 0.5$), $\sigma = 2$, and $N = 200$ (100 A, 100 B; 70% responders, 30% split 1:2 C:D). Steps: (1) Generate stress_base ($N(6.8, 2)$), assign stage1 (50/50), model stress_week4 ($0.8 \times \text{stress\_base} + \epsilon$), set responder (70% via logit), assign stage2, and simulate stress_week8. (2) Apply IPW (e.g., $1/(0.5 \times 0.7)$ for A_A), fit proc genmod, test A_A vs. B_B. (3) Repeat 1000 times—power is the % of p < 0.05. Our 200 might give 60% power (p = 0.1193 aligns), needing  400 for 80%. Simulation captures our design’s quirks—no simple $n$ formula fits!

5. Q-Learning: It is a reinforcement learning trick that’s like teaching a coach to pick the best plays in a game. For our SMART experiment, it figures out the optimal treatment path—starting with A or B at Stage 1, then choosing C, D, or None at Stage 2—to minimize stress at week 8 (stress_week8). It works backward: first, it predicts Stage 2 outcomes based on week 4 stress (stress_week4) and responder status, then uses those to pick Stage 1 based on baseline stress (stress_base). The “Q” stands for quality—how good each choice is, tailored to where a patient’s at. Unlike fixed rankings, it builds a flexible rule, adapting to each individual person’s journey.

   The following SAS code implements Q-learning to optimize treatment in our SMART data, minimizing stress_week8 across two stages.

   • Stage 2 Q-Function: proc glm fits stress_week8 for non-responders (responder = 0) using stage2 (C, D), stress_week4, stress_base, and stage1 as predictors. The / solution option details coefficients, and output out=stage2_preds predicted=q2_pred saves predicted stress as q2_pred.

   • Assign Q2: The data step merges predictions with the full dataset, setting q2 to observed stress_week8 for responders (responder = 1), or q2_pred for non-responders, with a fallback to stress_week8 if predictions are missing.

   • Stage 2 LSMeans: proc glm re-models stress_week8 for non-responders, computing LSMeans for stage2 at mean stress_week4 and stress_base. adjust=tukey adjusts p-values, and ods output saves results to stage2_lsmeans.

   • Stage 1 Q-Function: proc glm models q2 (optimal Stage 2 stress) with stage1 (A, B) and stress_base, outputting predictions as q1 to stage1_preds.

   • Stage 1 LSMeans: Another proc glm calculates LSMeans for stage1 at mean stress_base, using adjust=tukey for significance, saving to stage1_lsmeans.

   • Output: proc print displays stage2_lsmeans and stage1_lsmeans, showing predicted stress for C vs. D and A vs. B, guiding our policy.

   This backward process—Stage 2 to Stage 1—builds a dynamic treatment rule based on Q-values, leveraging regression to predict and compare outcomes at each decision point.

   
   /* Stage 2 Q-Function for Non-Responders */
   proc glm data=stress_paths;
       where responder = 0;
       class stage2 stage1;
       model stress_week8 = stage2 stress_week4 stress_base stage1 / solution;
       output out=stage2_preds predicted=q2_pred;
   run;
   
   /* Assign Q2 values */
   data stress_paths_q2;
       merge stress_paths stage2_preds (keep=id q2_pred);
       by id;
       if responder = 1 then q2 = stress_week8; /* Responders: Q2 is observed */
       else if q2_pred = . then q2 = stress_week8; /* Fallback for missing preds */
       else q2 = q2_pred; /* Non-responders: Q2 is predicted */
   run;
   
   /* Stage 2 LSMeans for Non-Responders  */
   proc glm data=stress_paths_q2;
       where responder = 0;
       class stage2 stage1;
       model stress_week8 = stage2 stress_week4 stress_base stage1 / solution;
       lsmeans stage2 / adjust=tukey; /* No 'at', uses mean values */
       ods output lsmeans=stage2_lsmeans;
   run;
   
   /* Stage 1 Q-Function */
   proc glm data=stress_paths_q2;
       class stage1;
       model q2 = stage1 stress_base / solution;
       output out=stage1_preds predicted=q1;
   run;
   
   /* Stage 1 LSMeans  */
   proc glm data=stress_paths_q2;
       class stage1;
       model q2 = stage1 stress_base / solution;
       lsmeans stage1 / adjust=tukey; /* No 'at', uses mean stress_base */
       ods output lsmeans=stage1_lsmeans;
   run;
   
   /* Output Results for Policy */
   proc print data=stage2_lsmeans;
       title "Stage 2 LSMeans for Non-Responders (C vs D)";
   run;
   
   proc print data=stage1_lsmeans;
       title "Stage 1 LSMeans (A vs B)";
   run;

   Listing 6.10. SAS code to implement Q-Learning

   Here’s the output:

   Table 6.10. Stage 2 LSMeans for Non-Responders (C vs. D)

   Stage 2	LSMean	Std Err	Pr $>|t|$
   C	5.2660	0.15	0.0033
   D	4.8963	0.14	–

   Table 6.11. Stage 1 LSMeans (A vs. B)

   Stage 1	LSMean	Std Err	Pr $>|t|$
   A	2.9950	0.10	0.1193
   B	3.3236	0.09	–

   At Stage 2, for non-responders, switching to D predicts a lower stress (4.8963) than C (5.2660), with a significant difference (p = 0.0033)—so D wins. At Stage 1, starting with A gives a lower expected stress (2.9950) than B (3.3236), though it’s not a slam dunk (p = 0.1193). Our policy: “Start with A; if a responder at week 4, stay (A_A); if a non-responder, switch to D (A_D).” This averages over our data’s mean stress_base (around 6.8), so it’s a broad rule.

   Our IPW-adjusted proc genmod gave fixed LSMeans: A_A = 2.2846, B_B = 2.2362, A_D = 4.8648, B_D = 4.8609, A_C = 5.3697, B_C = 5.5323. It ranked B_B and A_A as best, with C paths lagging. Q-learning aligns somewhat: A_A fits its Stage 1 pick of A (2.9950) for responders, and D over C matches A_D (4.8648) beating A_C (5.3697). But Q-learning flips Stage 1, favoring A over B, unlike IPW’s slight edge to B_B. Why? IPW averages whole paths, while Q-learning optimizes step-by-step, suggesting A sets up better flexibility. The real difference: IPW ranks static paths; Q-learning crafts a dynamic “if-then” plan—perfect for personalizing treatment!

   Class Activity Take a few minutes to review "An adaptive behavioral intervention for weight loss management: a randomized clinical trial." JAMA, 332(1):21–30, 2024. Given what you have learned in class, present the study and draw parallels between the concepts discussed in class and what has been implemented in the paper.

Required readings