Informed Design of Experiments?Martin Modrák2018/06/111 / 76

Simulations!

photo by Maurizio Pesce, CC-BY

2 / 76

Why & What3 / 76

Why & WhatDesign of experiments
4 / 76

Why & What

Design of experiments
- No. of replicates, comparison groups, ...

5 / 76

Why & What

Design of experiments
- No. of replicates, comparison groups, ...
Understanding the methods you use

6 / 76

Why & What

Design of experiments
- No. of replicates, comparison groups, ...
Understanding the methods you use
Case Studies

7 / 76

Why & What

Design of experiments
- No. of replicates, comparison groups, ...
Understanding the methods you use
Case Studies
- t-test

8 / 76

Why & What

Design of experiments
- No. of replicates, comparison groups, ...
Understanding the methods you use
Case Studies
- t-test
- DESeq2

9 / 76

Power Analysis10 / 76

Power AnalysisSimulations:
11 / 76

Power Analysis

Simulations:
- Easier

12 / 76

Power Analysis

Simulations:
- Easier
- Test the whole process

13 / 76

Power Analysis

Simulations:
- Easier
- Test the whole process
- More assumptions

14 / 76

photo: U.S. government work

15 / 76

photo: U.S. government work

Case Study 1

Two sample t-test

16 / 76

A Hypothetical Experiment17 / 76

A Hypothetical ExperimentCell culture
18 / 76

A Hypothetical Experiment

Cell culture
Does unoptanium increase midichlorian production?

19 / 76

A Hypothetical Experiment

Cell culture
Does unoptanium increase midichlorian production?
5 replicates

20 / 76

A Hypothetical Experiment

Cell culture
Does unoptanium increase midichlorian production?
5 replicates
Analyze with t-test, significant if $p < 0.05$

21 / 76

A Hypothetical Experiment

Cell culture
Does unoptanium increase midichlorian production?
5 replicates
Analyze with t-test, significant if $p < 0.05$
Simulation assumptions
- Unoptanium helps ( $+ 2 μ g$ on average)

22 / 76

A Hypothetical Experiment

Cell culture
Does unoptanium increase midichlorian production?
5 replicates
Analyze with t-test, significant if $p < 0.05$
Simulation assumptions
- Unoptanium helps ( $+ 2 μ g$ on average)
- $s d = 8 μ g$

23 / 76

What do we care about?24 / 76

What do we care about?Observed effect size
25 / 76

What do we care about?

Observed effect size
How frequently will we claim significance

26 / 76

What do we care about?

Observed effect size
How frequently will we claim significance
- a.k.a. power

27 / 76

What do we care about?

Observed effect size
How frequently will we claim significance
- a.k.a. power
- But there's more!

28 / 76

What do we care about?

Observed effect size
How frequently will we claim significance
- a.k.a. power
- But there's more!
Let's simulate 10000 datasets

29 / 76

photo: U.S. government work

30 / 76

What We Observe

31 / 76

Filter for Significance

32 / 76

Filter for Significance

Power:

## p < 0.05 in 0.0561 cases

33 / 76

A Closer Look

34 / 76

A Closer Look

Type S Error (wrong Sign)

35 / 76

A Closer Look

Type S Error (wrong Sign)

Type S error	95% CI excludes true
16.9%	36.4%

36 / 76

A Closer Look

Type M Error (wrong Magnitude)

37 / 76

A Closer Look

Type M Error (wrong Magnitude)

Mean exaggeration	Min. exaggeration
5.5	2.1

38 / 76

Significance is Not a Savior!39 / 76

Impact on the Literature40 / 76

Impact on the LiteraturePublished effects are exaggerated
41 / 76

Impact on the Literature

Published effects are exaggerated
- Exaggeration depends on amount of noise

42 / 76

Impact on the Literature

Published effects are exaggerated
- Exaggeration depends on amount of noise
- Negligible in high-powered studies

43 / 76

Impact on the Literature

Published effects are exaggerated
- Exaggeration depends on amount of noise
- Negligible in high-powered studies
If a results looks too good given the noise

44 / 76

Impact on the Literature

Published effects are exaggerated
- Exaggeration depends on amount of noise
- Negligible in high-powered studies
If a results looks too good given the noise it probably is.

45 / 76

photo by Llann Wé, CC-BY

46 / 76

photo by Llann Wé, CC-BY

Case Study 2

Differential Expression (DESeq2)

47 / 76

Less Hypothetical Experiment48 / 76

Less Hypothetical ExperimentDifferential expression upon unoptanium stress
49 / 76

Less Hypothetical Experiment

Differential expression upon unoptanium stress
Control, treatment, 3 replicates each

50 / 76

Less Hypothetical Experiment

Differential expression upon unoptanium stress
Control, treatment, 3 replicates each
1000 genes

51 / 76

Less Hypothetical Experiment

Differential expression upon unoptanium stress
Control, treatment, 3 replicates each
1000 genes
We use DESeq2 to test for effect = $| l o g_{2} (f c) | > 1$

52 / 76

Simulating DESeq253 / 76

Simulating DESeq2Where do the read counts come from?
54 / 76

Simulating DESeq2

Where do the read counts come from?
- From a previous experiment

55 / 76

Simulating DESeq2

Where do the read counts come from?
- From a previous experiment
How to set $l o g_{2} (f c)$ ?

56 / 76

Simulating DESeq2

Where do the read counts come from?
- From a previous experiment
How to set $l o g_{2} (f c)$ ?
- 80% genes have $l o g_{2} (f c) = 0$

57 / 76

Simulating DESeq2

Where do the read counts come from?
- From a previous experiment
How to set $l o g_{2} (f c)$ ?
- 80% genes have $l o g_{2} (f c) = 0$
- 0, 2, 4 and 6 for the other 20%

58 / 76

Simulating DESeq2

Where do the read counts come from?
- From a previous experiment
How to set $l o g_{2} (f c)$ ?
- 80% genes have $l o g_{2} (f c) = 0$
- 0, 2, 4 and 6 for the other 20%
100 simulations each

59 / 76

Some results

log_fc	True Pos.	False Pos.	Type S error	Mean exaggeration	Mean shrunk exaggeration
0	0.0	1.8	0.0	NaN	NaN
2	2.8	2.0	0.1	3.1	1.9
4	76.3	5.0	0.1	1.3	1.0
6	161.3	6.4	0.0	1.0	0.9

We tested for $| l o g_{2} (f c) | > 1$

60 / 76

Replicating DeSeq2 results61 / 76

Replicating DeSeq2 resultsExact experiment replication (3 replicates each)
62 / 76

Replicating DeSeq2 results

Exact experiment replication (3 replicates each)
Replicated = significant in both

63 / 76

Replication results
 
    log_fc 
    Significant 1st experiment 
    Replicated 
    Smaller effect - significant 
  



4 
3 
9 
  


9 
6 
7 
  


2 
4 
6 
  

/ 76

log_fc	Significant 1st experiment	Replicated	Smaller effect - significant
2	4.4	0.3	0.9
4	79.9	38.6	0.7
6	169.2	141.4	0.6

DESeq2 Summary65 / 76

DESeq2 SummaryDE experiments have low power
66 / 76

DESeq2 Summary

DE experiments have low power
DESeq2 rocks!

67 / 76

DESeq2 Summary

DE experiments have low power
DESeq2 rocks!
DESeq2 avoids false positives at all costs

68 / 76

DESeq2 Summary

DE experiments have low power
DESeq2 rocks!
DESeq2 avoids false positives at all costs -> high false negatives

69 / 76

Take Home70 / 76

Take HomeWorry about Type S & M errors
71 / 76

Take Home

Worry about Type S & M errors
Simulate experiments before investing money

72 / 76

Take Home

Worry about Type S & M errors
Simulate experiments before investing money
Simulate to understand published research

73 / 76

Take Home

Worry about Type S & M errors
Simulate experiments before investing money
Simulate to understand published research
Code available at https://github.com/cas-bioinf/statistical-simulations

74 / 76

Take Home

Worry about Type S & M errors
Simulate experiments before investing money
Simulate to understand published research
Code available at https://github.com/cas-bioinf/statistical-simulations

Thanks for your attention!

75 / 76

What about 6 replicates?

log_fc	True Pos.	False Pos.	Type S error	Mean exaggeration	Mean shrunk exaggeration
0	0.0	0.7	0.0	NaN	NaN
2	8.1	0.8	0.0	1.8	1.4
4	150.9	2.4	0.1	1.1	1.0
6	184.1	3.4	0.0	1.0	0.9

We tested for $| l o g_{2} (f c) | > 1$

76 / 76

Help

Keyboard shortcuts

↑, ←, Pg Up, k

Go to previous slide

↓, →, Pg Dn, Space, j

Go to next slide

Home

Go to first slide

End

Go to last slide

Number + Return

Go to specific slide

b / m / f

Toggle blackout / mirrored / fullscreen mode

Clone slideshow

Toggle presenter mode

Restart the presentation timer

?, h

Toggle this help