Types of validity
Quantitative
Internal validity
- Causality
- Cross sectional (bad)
- Longitudinal study (good)
- Rejection of all other explanations of causality
- Selection bias, exclusion bias, response bias, test bias, social desirable answer bias
- Common method bias (same sample, same instrument, same scale, same time)
- Confounding
- Control variables, control groups, randomization
- Cross sectional / longitudinal
- Social desirable answers if no anonymous answers
- Control variables = used to rule out some alternative explanations for the causality or
confounding variables
- Common method bias: it’s not because there is a sample with different people in it
that there are multiple samples used, it’s not because the data was acquired over 3
months that the time wasn’t the same that people filled in the survey
- Use more control variables not only demographic ones but also variables that
might cause confounding
- Exclusion bias bcs the survey was only sent to people on the short list and not the
rejected people, even though it would be more interesting to target the people who
were rejeceted
Construct validity
- Does the instrument really measure what it says it measures?
- Fit between the measures/items and the construct they measure
- Are all the dimensions of the construct measured?
- Do they measure one construct or more?
- Are subject matter experts used for validation?
- Pilot fill out by people in the population?
- Are cronbachs a higher than 70%?
- Are validated rating scales used from prior research?
- It’s informed by: that’s not enough validation
- Is there a factor analysis?
- If there is only one item: not good. There can’t even be a alfa calculated
- If there is data being self-reported eg actual delay = not validated
- A lot of items were dropped but is this scale then still valid and does it still contain all
the construct’s aspects?
- The items are in the appendix
- Are there enough items?
, - If there are convergent validity and discrimination validity test = good
Statistical validity
- P < 0,05,…
- Sample size
- Sobel test,…
- What kind of analysis
- Not too many variables but enough
External validity
- Representative: are the results generalizable across time, settings and individuals?
- Relevant sample? look at the variables, is the sample relevant for the variables
eg sample: students and the variable is: used leadership styles. But look at the sample:
maybe there are some working students
- Convenience sample or random?
- One country, one university, one sample? generalizable?
- If you look for leadership, don’t look at a sample of student = relevance
- Random or convenience sample?
- Is there info about the response rate? = high response rate = high representativeness
- Is there a large sample size = representativeness
- Is there a comparison with non-responders or with the characteristics of the
population? Is there an explanation why the responder group is representative
Qualitative
Internal & construct validity
= what did the authors do to ensure a correct answer on the research question?
- Use an interview a a format
- Pilot test
- Look for a neutral environment
- Transcribe, record the interview and make notes
- Word codes, word trees, word clouds
- Multiple researchers on all the levels in the analysis of the interview
- Good code process
= was there bias?
- Selection bias
- Social desirable answer bias
, - …
= was there a good reliability
- Inter rater reliability
- Member checks
External validity
- Representativeness (generalizable eg: not to all ages)
- Purpose sampling
- Sampling until data saturation?
- Descripition of data saturation?
- Relevant? (students >< leadershipstyles)
- Is the sample size big enough?
Grounding hypotheses
How to evaluate the grounding of hypotheses?
- Explanation of the relationship between the proposed variables (moderation,
mediation, direct effect)
- Explain theories we build on = are the variables in line with what the theory says?
Eg if we use the SDT, we use the variables ECM and no other variables
- Explain empirical evidence
- Explain the relationship with prior research eg: in prior research a certain variables
has been used a mediator, we don’t we do this now? Or in prior research the direct
effects of a certain variable was researched, why not now?
- Explain the reasoning for certain relationships we proposed in our hypothesis, so
why do we think that crisis respons is better than no repsons?
- Explain why we use these variables = is there a clear overarching story or research
question to explain the use of the variables?
Introduction
Who cares:
- Case, trend, quote
, - Catch attention = also by explaining the meaning of a word in Latin etc
- Theoretical relevance
- Practical relevance
What do we know, what don’t we know and so what?
- Explain the theories and empirical evidence for what we already know
- What don’t we know
- Establishing the field and problematizing the field
- So what is it explained why we are problematizing the field, eg: there is little know
suggestion incompleteness, but if this is the reason for your paper, then you don’t
problematize the literature enough, there is not always a first mover advantage and
sometimes gaps shouldn’t be fulfilled. But is you say: so what why should they
research gap be fulfilled, then it’s good, then we have problematized the field and
addressed the incompleteness.
What will we learn?
- The research question is described.
- Make clear what is new in comparison with prior research?
- But how are we changing or enhancing to the understanding of people?
- What is the contribution to the consensus?
- Filling a gap and solving a practical issue is not a good contribution we need a
theoretical contribution.
- How can you explain whether there is contribution to the existing literature?
Think about: is this already been examined in another context? Or is this the first time
these issues are studied in this context? And why would the relationships be different
in this specific context? think about the theoretical contribution, not about: yes
we will learn more about this topic.
Are the hypotheses well formulated?
- If there is moderation, say if it makes the relationship weaker or more strong
- No null hypotheses (there is no effect)
- Always include a direction
- Viscious circles (if wellbeing increases, the score on my test for wellbeing will
increase)
- Not too vague eg what kind of motivation, what is better?
What are the independent, moderating, and dependent
variables?
