Experimental design

http://www.nature.com/nm/journal/v12/n11/full/nm1491.html

http://arxiv.org/pdf/1010.1092.pdf

http://nsaunders.wordpress.com/2012/07/23/we-really-dont-care-what-statistical-method-you-used/

https://github.com/

http://figshare.com/

https://github.com/jtleek/datasharing

Question: Does changing the text on your website improve donations?

Experiment:

1. Randomly show visitors one version or the other
2. Measure how much they donate
3. Determine which is better

http://www.wired.com/business/2012/04/ff_abtesting

http://www.gs.washington.edu/academics/courses/akey/56008/lecture/lecture2.pdf

http://www.nejm.org/doi/full/10.1056/NEJMon1211064

Sometimes called spurious correlation*

• If you can (and want to) fix a variable
  • Website always says Obama 2014 on it
• If you don't fix a variable, stratify it
  • If you are testing sign up phrases and have two website colors, use both phrases equally on both.
• If you can't fix a variable, randomize it

http://www.gs.washington.edu/academics/courses/akey/56008/lecture/lecture1.pdf

http://www.biostat.jhsph.edu/~iruczins/teaching/140.615/

http://www.biostat.jhsph.edu/~iruczins/teaching/140.615/

http://xkcd.com/882/

http://xkcd.com/882/

http://xkcd.com/882/

• Good experiments
  • Have replication
  • Meaure variability
  • Generalize to the problem you care about
  • Are transparent
• Prediction is not inference
  • Both can be important
• Beware data dredging