Research Design

  • PSU Methodology Center Resources

Sample size planning and power

  • Power analysis working group

  • Power calculator

  • Power calculator for APIM models

  • Power calculator for EMA/ESM studies

  • Power calculator for multilevel models

  • Power for causal evaluation designs

  • Power in multilevel relationships research

  • Powering your interaction

  • Research with sample size constraints

  • Sample size in grants

  • Computing power resources

  • Randomization tests for small samples

  • Randomization tests for small samples

  • Practical guide to power analysis in simple experimental designs (Perugini et al.)

  • G*Power: free software that computes power for a variety of designs

  • Declaredesign: Interactive R program that computes power based on your design (using simulations).

  • Pangea: A web applet that computes power for General Anova designs (Jake Westfall)

  • Computing power for interactions involving one continuous and one dichotomous variable

  • Computing power when you have more than one within subject factor (D’Amico et al.)

  • Calculating power for a 3-way ANOVA in G*Power

  • Online calculator for power estimation in mixed models.

  • Powerlmm: An R package for computing power in multilevel models (Kristofer Magnusson)

  • Estimating sample size in multilevel models (in relationship research; Lane & Hennes)

  • Sequential data analysis: Maximizing power and minimizing sample size at the same time. (Daniël Lakens)

  • How many participants do I need to test a moderation of the effect I found in my first study? Post on Data Colada

  • Determining sample size in social psychology (in French).

  • Sample size planning adjusting for publication bias and uncertainty (Anderson et al.)

  • Sample size planning for cognition and perception with repeated measured (Jeff Rouder)

EFFECT SIZES

THEORY AND RESEARCH QUESTIONS

  • Answering developmental questions using secondary data

  • Theory mapping

  • Creative hypothesis generation in psychology (Bill McGuire)

  • Classic paper on construct validity (Cronbach & Meehl)

  • Appraising theories in psychology (Meehl)

MEASUREMENT

  • Measurement myths

  • Measurement reading list

  • Measurement Schmeasurement symposium summary

  • New developments in validity

  • Resources for measurement training

  • Stress/health methods and measures

  • Scale development (Clark and Watson, 1995)

  • Chapter on question wording and item formulation (Malte Elson)

  • Guidelines on designing rating scales

  • Online measurement resources (Elko Fried & Jessica Flake)

EXPERIMENTAL DESIGN

  • How to design studies with mediation in mind (Pirlott & McKinnon)

  • Why designing experiments is preferable to mediational analysis in examining causal processes (Spencer, Zanna & Fong)

  • Time series experiments: a solution when randomized controlled trials are too costly or difficult (Biglan et al.)

  • Example of combining qualitative and quantitative methods to tackle a research question (Maykel Verkuyten)

  • Example of combining qualitative and quantitative methods to tackle a research question (Anouk Manassen)

ETHICS

  • Tips on getting parent consent

  • APA’s guidelines on Responsible Conduct of Research

  • Using deception ethically (Antonio Pascual Leone et al.)

CAUSAL INFERENCE

  • Introduction to endogeneity: when a "causal" variable is correlated with the error term of the DV (usually in non-experimental designs)

  • The C-Word: Scientific Euphemisms Do Not Improve Causal Inference From Observational Data (Hernan) How to move from statistical association to causation? (Genetian et al.)

  • Guide to instrumental variables in the social sciences (Bollen)

  • Applying causal modeling to epidemiological data (Petersen & Van der Laan)

  • Thinking Clearly About Correlations and Causation: Graphical Causal Models for Observational Data (Julia Rohrer).

  • How to properly assess causality using linear models using path diagrams (Judea Pearl)

  • Are randomized controlled trials the panacea for establishing causality? (Deaton and Cartwright)

  • Thinking about causality in complex interventions (Rickles, 2009) 

Sampling

  • Participant databases

  • The use of Western, Educated, Industrialized, Rich and Democratic (WEIRD) samples in psychology (Henrich et al.)

  • OSF resource on sampling (workshop @ SIPS 2019).

Offline software platforms

  • Psychopy: open source software for conducting psychology experiments offline

  • Psyscope: software to design experiments using Mac OS.

ONLINE SURVEY PLATFORMS

  • Psytoolkit: free to use software to conduct experiments online.

  • Gorilla: user friendly software for conducting experiments online (the researcher pays per subject)

  • EMA tools

  • Experience sampler app

  • Lab.js: free online study builder

  • Guide for linking Qualtrics to MTurk

  • Mobile apps for text reminders

  • Recommendations for EMA study platforms

  • Introduction to crowdsourcing platforms in cognitive science (Neil Stewart et al.)

  • Strength and weaknesses of Amazon Mechanical Turk Samples (Goodman et al.)

  • Evidence that Mechanical Turk provides high quality data (Buhrmeister et al.)

  • A list of crowdsourcing platforms provided by Gorilla Science.

  • A list of resources on online studies (Gabriele Paolacci).

  • Validity problems with attrition in online surveys and how to deal with them (Zhou & Fishbach)

  • Identifying careless responses in survey data (Meade et al.)

  • Seriousness checks to improve the reliability of online surveys (Aust et al.)

  • Detecting and deterring insufficient efforts in responding to surveys (Huang et al.)

  • Comparing methods for detecting bots responding to online surveys (Meyer et al.)

  • Script to detect bots (Matt Motyl)

  • Another script to detect bots (Andy Wood)

  • Paper on low quality answers from humans using VPNs and how to detect them (Dennis et al.)

  • Randomly assigning people to different conditions in limesurvey.

  • Videos covering issues in conducting online research: ensuring data quality, technical issues, recruiting participants.

  • Reddit page where Mturk workers share their qualms.

  • OpenMTur: open tool for managing MTurk studies.

Materials

Stimuli creation

  • Image/flier design

  • Twitter thread on resources for designing Vignette studies

  • Image/flier design

Stimuli sets

  • Experience sampling item repository

  • Questionnaire instrument compendium

  • Diverse creative commons image resources