Papers Experimental Research summarized.
This doesn't include stat videos nor lectures.
But I did upload those seperately on my account, and also all exam materials uploaded as 1 document (including lectures, papers and stat videos).
Experimental Research – papers
Inhoud
Week 1 ......................................................................................................................................... 2
Paper - Thomke & Manzi (2014) | The discipline of business experimentation ............................... 2
Paper - Kohavi & Thomke (2017) | The surprising power of online experiments ............................. 3
Week 2 ......................................................................................................................................... 4
Paper – Spiller et. al. (2013) | SPotlights, floodlights, and the magic number zero: simple effects
tests in moderated regression .................................................................................................... 4
Paper – Matz et. al.(2017) | Targeting ......................................................................................... 9
Eckles’ reply .........................................................................................................................10
Matz’ reply to Eckles .............................................................................................................11
Week 5 ........................................................................................................................................11
Paper – Charness et. al. | Within experiments ............................................................................11
Paper – Gordon et. al. | Facebook experiments for ad effectiveness ............................................13
Paper – Konings et. al................................................................................................................14
Week 6 ........................................................................................................................................15
Paper – Simmons et. al. | Researcher degrees of freedom ..........................................................15
Paper – Brysbaert | Power rules of thumb .................................................................................16
Week 8 ........................................................................................................................................17
Paper – Mislavsky et. al. | Ethics of experiments.........................................................................17
Paper – Sunstein (2015) | THe ethics of nudging.........................................................................17
,WEEK 1
PAPER - THOMKE & MANZI (2014) | THE DISCIPLINE OF BUSINESS EXPERIMENTATION
We lack data to inform decisions on innovation. Big data only provides information about past
behavior, not how to react to changes. And managers often rely on their experience or intuition, but
truly innovative ideas often go against them. Managers can do a rigorous test, to discover if a new
product will succeed.
Most organizations don’t test, because they’re reluctant to fund experiments and find it difficult to
execute them. Majority of business is conducted through more complex distribution systems like
store networks and fast-food franchises, and not just testing a direct channel like the internet. The
sample sizes are often too small to get valid results.
In an ideal experiment the IV is manipulated to study changes in the DV, holding all other potential
causes constant.
Questions to ask to know if business experimentation is worth the expense and effort:
- Does the experiment have a clear purpose?
o Only do an experiment if they’re the only way to answer specific questions
o First figure out what you want to learn, then choose the best testing approach, and
finally the scope of the experiment.
o Companies often lack discipline to hone their hypothesis (their hypotheses are
weak), making experiments inefficient
o Also think about side effects: positively (selling more of current products) and
negatively (cannibalizing sales from other profitable items).
- Have stakeholders made a commitment to abide by the results?
o Before testing, stakeholders must agree how to proceed once they have the results.
o They must be willing to walk away from a project if it’s not supported by the data
o A process is needed to ensure test results aren’t ignored, even if they’re
contradictory to executives’ assumptions
o If projects make the cut, first develop test designs, and later they’re conducted
- Is the experiment doable?
o The high causable density of the environment (complexity of variables) can make it
hard to determine cause-and-effect relationship
▪ To deal with this, firms should consider if it’s feasible to use a sample large
enough to average out the effects of all variables, except those in the study.
o Environments constantly change, thus outcomes are often uncertain and unknown
o The required sample size depends on the magnitude of the expected effect
▪ The smaller the expected effect, the more observations needed to detect it
from the surrounding noise, with statistical confidence
- How can we ensure reliable results?
o Companies should trade-off reliability, costs and time.
o 3 methods to reduce trade-offs, thus increase reliability of the results:
▪ Randomized field trials:
• Randomly divide people in 2 groups, and give on the treatment
• Prevents systematic bias and evenly spreads remaining, unknown,
potential causes of the outcome between test and control groups.
▪ Blind tests:
• Minimizes biases and increases reliability further
, • Prevents Hawthorne effect: tendency of participants to modify their
behavior, (un)consciously, when they’re aware they’re part of an
experiment
• Double-blind tests: neither experimenters, nor test subjects are
aware of which participants are in the test group, and which are in
the control group.
▪ Big data: utilizing specialized algorithms with multiple sets of big data, can
help by:
• Filtering statistical noise and identify cause-and-effect relationships.
Experiments ideally employ many samples, using big data can help.
• With small samples, we should match test subjects to control
subjects, which depends on identifying many variables to
characterize test subjects, with which big data can help. Then we
build control groups with all elements of the test group, except for
what’s tested, to determine the influencing factors.
• Targeting the best opportunities (highest ROI’s, what investments to
avoid)
• Tailoring the program: characterizing program components that are
more or less effective
o Repeatability is important: others conducting the same test, should get familiar
results.
- Have we gotten the most value out of the experiment?
o It’s not what works, but what works where?
▪ Roll out the program only in stores most similar to test stores with the best
results
▪ Value engineering: implement components with a high ROI
o Just the experiment is not enough, the value comes from analyzing and exploiting
the data
PAPER - KOHAVI & THOMKE (2017) | THE SURPRISING POWER OF ONLINE
EXPERIMENTS
Lessons on how to design and execute A/B tests:
- Appreciate the value of A/B tests
o Using online A/B tests, companies have access to large customer samples (if you
have many website visitors). Allowing to iterate rapidly, fail fast, and pivot.
o Managers need to understand:
▪ Tiny changes can have big impact
• Online success means getting many small changes right, rather than
1 big investment
▪ Experiments can guide investment decisions
• Help decide how much investment in a potential improvement is
optimal
• But can the value of improvement be quantified? (F.e. how many
people should be working on a task)
• They help make trade-offs
o F.e. improve Bing’s results, but they slow the response time
- Build a large-scale capability
o Majority of ideas fail in experiments, even experts (10-20% lead to positive results)
o We need to experiment with everything, to assure changes aren’t degrading nor
have no effects
Les avantages d'acheter des résumés chez Stuvia:
Qualité garantie par les avis des clients
Les clients de Stuvia ont évalués plus de 700 000 résumés. C'est comme ça que vous savez que vous achetez les meilleurs documents.
L’achat facile et rapide
Vous pouvez payer rapidement avec iDeal, carte de crédit ou Stuvia-crédit pour les résumés. Il n'y a pas d'adhésion nécessaire.
Focus sur l’essentiel
Vos camarades écrivent eux-mêmes les notes d’étude, c’est pourquoi les documents sont toujours fiables et à jour. Cela garantit que vous arrivez rapidement au coeur du matériel.
Foire aux questions
Qu'est-ce que j'obtiens en achetant ce document ?
Vous obtenez un PDF, disponible immédiatement après votre achat. Le document acheté est accessible à tout moment, n'importe où et indéfiniment via votre profil.
Garantie de remboursement : comment ça marche ?
Notre garantie de satisfaction garantit que vous trouverez toujours un document d'étude qui vous convient. Vous remplissez un formulaire et notre équipe du service client s'occupe du reste.
Auprès de qui est-ce que j'achète ce résumé ?
Stuvia est une place de marché. Alors, vous n'achetez donc pas ce document chez nous, mais auprès du vendeur mandyvervoort. Stuvia facilite les paiements au vendeur.
Est-ce que j'aurai un abonnement?
Non, vous n'achetez ce résumé que pour €3,99. Vous n'êtes lié à rien après votre achat.
