Ams and Objectves Meta-Analyss D. Andy Feld When and Why? Theoy Fxed Effects Models Random Effects Models Can meta-analyss analyss be tusted? Pactcal poblems Choce of methods Andy Feld Slde When and Why? To assmlate nfomaton fom ndependent studes Dscusve Lteatue Revews Selectve ncluson of studes Subjectve weghtng of studes Based ntepetaton of evdence Meta-Analyss Objectve assmlaton of studes? Use of Meta-Analyss (98-0) Numbe of Publcatons 0 0 00 900 0 700 600 00 0 0 0 0 0 In Socal Scence Jounals In Scence Jounals 8 8 83 84 8 86 87 88 89 90 9 9 93 94 9 96 97 98 99 00 0 Yea Andy Feld Slde 3 Andy Feld Slde 4 What do scentsts do? Inteested n nowng a tue effect sze n ou populaton of nteest. Use sams to estmate ths tue effect Expess the effect: Glass s Cohen s d Hedge s g Peason s Odds atos/s ates Dscove that othes have also ted to fnd ths effect Assmlate these dffeent studes. Meta-Analyss: Bascs Calculate effect szes fo each study Convet to a common metc Weght each effect sze by the samplng pecson sze (o some functon of t) Calculate the mean effect sze acoss studes Expess ths mean as a Z-scoeZ Calculate sgnfcance/confdence ntevals Modeato Analyss? Andy Feld Slde Andy Feld Slde 6
Fxed Effects Models 0. Random Effects Models Supepopulaton 0. 0. 0. 0.7 0.36 0.9 0. 0.4 0. Andy Feld Slde 7 0. 0.9 0. 0.4 Andy Feld Slde 8 Hedges et al. s Fxed Method Sgnfcance of Mean Effect Sze: Mean Effect Sze: z + ( ) log e ( e z ) ( z e ) + SE ( ) SE( ) w ( n 3) wz w w v v n 3 w n 3 ( n 3) z ( n 3) Z z SE ( z ) Andy Feld Slde 9 Andy Feld Slde Homogenety of Effect Szes: Hedges et al. s Random Method Mean Effect Sze: Q ( n 3)( z z ) w z w w v w v +τ n 3 n 3 + τ + τ z n 3 + τ n 3 Andy Feld Slde Andy Feld Slde
Estmatng Between-Study Vaance Hunte-Schmdt Method c w τ ( w ) w ( ) Q c c ( n 3) ( n 3) ( n 3) Mean Effect Sze: n n Andy Feld Slde 3 Andy Feld Slde 4 Sgnfcance of Mean Effect Sze: Homogenety of Effect Szes: SD n ( ) n Z SE SE SD χ ( n )( ) ( ) Andy Feld Slde Andy Feld Slde 6 Can meta-analyss analyss be tusted? Publcaton bas: the fle dawe poblem Sgnfcant fndngs moe often publshed 97% of publshed psychology atcles epot sgnfcant esults (Stelng, 99) Meta-analyss analyss wll oveestmate the mean effect sze Atefacts Is all eseach equally good? Whch method: Fxed o Random? Real data ae lely to have heteogeneous effect szes: Howeve, Fxed methods ae moe commonly apd: Hunte & Schmdt (0) No andom effects meta-analyses analyses n Psychologcal Bulletn Feld (03): Bas when fxed effects methods ae used on heteogeneous effect szes. Andy Feld Slde 7 Andy Feld Slde 8
Feld (03) 0. 0.7 0.70 0.6 0.60 0. 0.0 0.4 0. 0.3 0. 0. 0. 0. 0. 0.0 Sze Fxed-Effects Models Apd to Heteogeneous Effect Szes Whch Method: Hedges o Hunte-Schmdt? Tansfomng -to-z Coects fo bas n samplng dstbuton of Tansfomng s a good thng (Slve & Dunlap (987) Tansfomng maes no dffeence (Stube,, 988; Hunte et al., 996) Rosenthal-Rubn Andy Feld Slde 9 Andy Feld Slde Estmates of Standad Eo Hedges & Vevea (998) Hunte-Schmdt use suboptmal weghts Hence, when between study vaance exsts H-S H wll undeestmate SE and oveestmate Z (Type I eos). Feld (0): Hedges & Vevea s method uses tuncated estmates of between-study vaance. Hence based when the numbe of studes n the meta-analyss analyss ae small. Andy Feld Slde Compang Methods Johnson et al. (99) Compaed Hedges (fxed) method wth H-S H S by manpulatng a sngle data set. H-S S povded consevatve estmates of sgnfcance Feld (0): Johnson et al. s study s pants: Results could be due to the data set used They got the H-S H S equatons wong Used Hedges d method and conveted to Only looed at fxed-effect effect data Andy Feld Slde Monte Calo Study Feld (0) Effect szes samd fom a populaton wth a fxed o vaable mean effect sze. Fo each: 0,000 epettons Vaables: Populaton effect sze (0,.,.3,.,.8) Mean sam sze of study (,,, ) Numbe of studes n meta-analyss analyss (,,,,, ) Vaablty n populaton 0.6 Andy Feld Slde 3 Hedges Effect Sze Estmates (Fxed) 0.0 0. 0. 0.8 Andy Feld Slde 4 0.3-0.0-0.00 +0.00 +0.0
HunteHunte-Schmdt Effect Sze Estmates (Fxed) 0. Hedges Effect Sze Estmates (Random) 0.0 0.3 0. 0. +0.00 +0.0 0.0 0. 0.8-0. -0. -0. -0.0 0.8-0. -0. -0. -0.0 +0.0 +0. +0. Andy Feld Slde 6 Heteogeneous Case Hunte-Schmdt 0.7 0.7 0. 0. 0. 0.0 0.07 0.0 0.07 0.00 Sze +0.0 +0. +0. 0. 0.00 Sze yss Meta-Anal Studes n yss Meta-Anal Studes n Andy Feld Slde 7 Andy Feld Populaton vaance.04 Populaton vaance.08 Hunte-Schmdt Vaables: Vaables: 0.7 0.7 0. 0. Effect Effect szes szes samd samd fom fom aa populaton populaton wth wth aa vaable vaable mean mean effect effect sze. sze. Fo Fo each: each: 0,000 0,000 epettons epettons Monte Monte Calo Calo Study Study 0. 0.0 0.07 0.00 Populaton Populaton effect effect sze sze (0, (0,.,.,.3,.3,.,.,.8).8) Mean sam sze of study (,, Mean sam sze of study (,,,, ) ) 0. 0.0 0.07 0.00 Sze S ze Meta-Analyss Studes n Numbe -analyss (, meta Numbe of of studes studes n n metameta-analyss (,,,,,,,,, ) ) Vaablty Vaablty n n populaton populaton effect effect szes szes 0.04, 0.04, 0.08, 0.08, 0.6, 0.6, 0.3, 0.3, 0.64 0.64 Andy Feld Slde 8 Feld (Submtted) Feld (Submtted) Homogeneous Case Effect Szes: Type I Eo 0.3 HunteHunte-Schmdt Effect Sze Estmates (Random) 0.0 Slde Andy Feld 0. -0.0-0.00 0.8 0.3 0.0 Slde 9 Meta-Analyss Studes n Andy Feld Slde
Feld (Submtted) Populaton vaance.64 Populaton vaance.3 0.7 0.7 0. 0. 0.7 0.0 0.07 0.00 0. 0. 0. 0.0 0.07 0.00 Feld (Submtted) Hunte-Schmdt Populaton vaance.6 Sze Sze Meta-Analyss Studes n Meta-Analyss Studes n 0. 0.0 0.07 0.00 Sze ss Meta-Analy Studes n Andy Feld Slde 3 Andy Feld Slde 3 Conclusons -analyss be meta Can Can metameta-analyss be tusted? tusted? Not Not ff fxed fxed effects effects methods methods ae ae used used when when populaton populaton effect effect szes szes vay vay (sgnfcance (sgnfcance tests tests of of most most of of the the ones ones you ead ae lely to be type I eos) you ead ae lely to be type I eos) Hunte-Schmdt geneally Hunte Hunte-Schmdt geneally gves gves bette bette estmates estmates Hedges Hedges geneally geneally contols contols the the Type Type II eo eo bette bette than than HH-S, -S, but but both both methods methods OK OK when when combnng combnng o o moe moe studes studes But But Ae Ae we we eve eve nteested nteested n n sgnfcance sgnfcance tests tests anyway? anyway? Ae Ae modeato modeato vaables vaables moe moe nteestng? nteestng? Andy Feld Slde 33 6