Estimated Read Time: 4 min
TL;DR: I finally got around to making my first Shiny app! In this post, I’ll (1) describe what Shiny is; (2) link some resources on power and the replication crisis; and (3) describe how to use my app.
I’ve had “learn Shiny” on my to-do list for months, but it wasn’t until a couple weeks ago when classes finally wrapped up that I was able to start building my first app: an app which runs power analysis simulations using user uploaded data and parameters.
What is Shiny?
Shiny is an R package that makes it super simple to build web apps directly in R. These apps can then be freely hosted on https://www.shinyapps.io/ and shared with collaborators or the general public. Shiny apps can vary in their complexity, both what is displayed to the user and how reactive the app is to user interaction. For example, you could build an interactive app that allows users to select a sample size and then simulates and plots data from a normal distribution to demonstrate the Central Limit Theorem. Or you could build a static dashboard that pulls data from online sources and generates a series of plots and metrics. You can explore examples and get inspiration here.
Some resources on power and reproducibility
I’m not going to try to describe what statistical power is, why it’s important, or how it relates to the replication crisis in the social sciences in this post, but here are some resources I’ve found helpful on these topics for you to explore if you haven’t already:
- This is a great introductory post from EGAP on “10 things you need to know about power”
- Andrew Gelman has several great blog posts about power
- Here is an article on replication of economic lab experiments
- And here’s a paper that discusses power and exaggeration bias in economic experiments.
You can find my Shiny app here.
Based on quick search, I found several Shiny apps related to power and sample size (including a Shiny app embedded in EGAP’s post on power which I mentioned above), but none of the apps I saw allowed for simulations based on user uploaded data.
Analytical formulas exist to calculate power of simple null hypothesis significance tests. However, social scientists typically use more complex experimental designs (e.g. clustered random assignment and different statistical models). Further, analytical power calculations make distributional assumptions about the data. To get a more precise estimate of statistical power, simulations can be used to estimate power: using baseline data, the experiment is simulated thousands of times. The percentage of statistically significant results estimates the power of the design.
Earlier this year I created an R-markdown document to run some power simulations for a field experiment I am working on with my PhD advisor and another collaborator. The document includes some summary stats and then a dozen power analysis simulations using various covariates and outcome variables. To run these repeated simulations, I wrote a function in R to simulate power based on the OLS model I was testing. And then I had the idea to turn this function into a Shiny app.
How to use the app
My thought is that this app will not replace the kind of power analysis I wrote in R-markdown, which can be written to incorporate experiment-specific details. Instead, I see the app serving as a precursor to a full fledged power analysis. Users can upload a baseline data set, input a few parameters and get an estimate of necessary sample size or minimum detectable effect in seconds – without having to write any code.
Again, the first version of the app can be found here: https://marisalyn.shinyapps.io/powerAnalysisApp_v1/
To use the app, just upload your (cleaned) data in CSV format, select the sample size(s) and effect size(s) you want to test, and fill in the other parameters. After you hit “Run Power Simulation”, the power results will be simulated. The simulation is based on a simple OLS model, described in more detail here. The power results will be displayed in a table and as a plot in the main area of the app.
Fair warning: there are probably some bugs in the app right now since this is my first ever app!
I’ve uploaded the files for the app on Github here. There’s a more detailed description of the simulate power function I wrote in the README file here too.
Another warning: this is also my first time posting anything on Github so I’m not sure I’m doing it all correctly!
As I mentioned above, this app doesn’t replace power simulations written for a specific experiment design. However, I am considering building on this app to make it more flexible and allow for other common experimental design elements like cluster randomized designs, clustered standard errors, or other statistical models like logistic regression.
If I build on the app, what kind of flexibility would you want to see?