Research Design and Analysis - Semester 2
Topics include measurement and psychometric principles; quantitative design principles and features; evaluation of designs; and factorial designs.
1.1 SOURCING AND SELECTING INSTRUMENTS
Measurement consists of the assignment, according to fixed rules, of numbers to indicate
differences in the magnitude of some attribute of persons or objects. (Anything can be
turned into a numerical, code.)
Quantified (numerical) data produced by measure.
Instruments determine how you measure which determines how you analyse etc.
Instruments often used in quantitative: formal observation, standardised interviews,
questionnaires, and surveys.
Quantitative instruments = an “objective” standardised measure of a sample of
behaviour. However, it can never be fully objective. The diagnostic or predictive value of
a (quantitative instrument) depends on which it indicates a relatively broad and
significant area of behaviour. Research can be standardised for specific contexts, not all
contexts. Can’t be truly objective for multiple reasons.
Core features of quantitative instruments = (1) it is standardised, (2) produces
scores/numerical data and (3) captures a reasonable sample of behaviour related to
constructs being measured.
If we have instruments, we need psychometrics. Psychometrics focuses on evaluating
the theory and practice of already existing psychological measurements and the
development of quantitative instruments.
Questions to ask when selecting a quantitative instrument:
• What construct/variable/phenomenon does the instrument measure?
• What is the instrument response format (self-report or other; open-ended or
closed-ended; response format required)?
• How are respondents likely to react to the items/ questions (content and format)?
• In what form is the data obtained (e.g. online vs. paper) and how is it scored?
o Online: can access a ton of people but it will only be people who have
access to the internet.
, o Pen and Paper: can access specific groups like people in rural areas but
must provide paper, pens and maybe food.
• What is the nature of the output generated and how will this link to the proposed
analysis?
• What is the availability of the instrument and how appropriate (practical) is it to
use? Commented [VM1]: Something can be reliable but
invalid. But nothing can be unreliable and valid.
Sourcing existing instruments
Some instruments are licensed and copyrighted. These instruments can only be obtained
through test distributors like JVR Africa Group, PsyTech South Africa.
Unlicenced instruments can be accessed in journal articles, research dissertations,
online search engines and test databases.
Advantages: cost-effective Disadvantage: not as established and validated.
Adapting and modifying existing instrument
One might need to adapt an instrument for already existing phenomena to a specific
context. You can adapt it by removing inappropriate sections or changing the language
or phrasing. Changing and modifying the test can impact the reliability and validity.
It is important to carefully consider the reason behind making the changes, the
willingness to pay for psychometric evaluation, the potential impact modification may
have etc.
Types of modifications: (a) changes in vocabulary, (b) translation, (c) change of response
format, (d) changes in scoring/interpretation (e) eliminating/adding items, (f)
administering a subscale/ short form (gives an idea, not full picture), (g) combining
existing items (item pools).
Constructing an instrument
When there isn’t a suitable instrument for the construct. The following needs to be
considered:
Purpose: priority for info (research) or decision-making (assessment).
Focus: knowledge-based or person-based
Content: test specification/blueprint (content and manifestation); approach to item
generation (rational or empirical)
,Format (basic): administration, response format.
Item formulation: Language, quality, applicability, possible response biases. Example:
Each question must address one construct.
Format (advanced): instructions, time limits, layout.
Evaluation and Refinement: extended administration, extended psychoanalytic analysis.
Problems with Quantification
Representation: what attribute/construct is actually represented by the numbers?
Objectivity: what rules are used to assign the numbers to the measured attribute, and
how reliable are these?
Correspondence: to what extent do the numbers accurately represent real differences in
the level of the attribute being measured?
Substance: what level of info about the phenomenon of interest is lost/captured during
the process of converting the data to numbers?
Authenticity: what level of info about the context of interest is lost/captured during the
process of converting the data to numbers? What aspects of the context are accounted
for?
1.2 BASIC CONSIDERATIONS
Interpreting how a test works
• Quantitative instrument descriptions (test descriptions) = provide useful details
about how a given quantitative instrument (test) works. Information about the
following aspects of the instrument/test is generally required in order to make an
informed decision about how useful and appropriate the instrument/test is likely
to be in a given study:
o The number of items.
o How the construct/variable is represented (subscales and item content).
o The response format: self-report and/or other report.
o The response format: closed-ended and/or open-ended items.
o The response format: response format type.
o Scoring
, • Together, these aspects provide the basis for a detailed and critical evaluation of
how a quantitative instrument (test/ measure/ scale) works.
• For convenience, the term ‘test’ is used in the following sections, but the
information given refers to all forms of quantitative instruments.
Number of Items
• Implications for the length of time to complete (amount of time and effort
required by respondent).
• No definite rules for defining ‘short’ or ‘long’ tests (depends on response format as
well).
• As a ‘rule of thumb’, less than 20 items tend to be considered short and over 100
items tend to be considered quite long.
• If a test has too few items and/or is too short, there might not be enough
questions asked to establish a consistent pattern of answers (i.e., the test might
not be sufficiently reliable) and/or there might be too few questions to adequately
represent/ capture the construct/ variable of interest (i.e. there might be too few
items to generalise/ infer effectively which can create problems with validity).
• If a test is too long, respondents might become tired or bored and start to leave
out items or answer carelessly – this is a form of response bias.
• As a general principle, longer tests tend to be more reliable and valid, but are
more likely to lead to problems with response bias such as random responding,
leaving out items, etc…. The opposite applies to shorter tests.
Representation (subscales and content)
• A subscale is a smaller set of items within the test that together represent a
specific aspect or element of the larger construct being measured by the test (e.g.
16PF).
• In order for a test to be valid (i.e., to measure what it is supposed to measure
effectively), the structure of the test (i.e., the number of subscales and what these
represent), must align with the structure of the large construct (i.e. the way in
which the larger construct is conceptualised and the number of aspects or
elements it is thought to consist of).
• Unidimensional constructs are conceptualised as consisting of a single aspect or
element (idea) and thus do not have any associated subscales E.g., measuring
self-esteem.
The benefits of buying summaries with Stuvia:
Guaranteed quality through customer reviews
Stuvia customers have reviewed more than 700,000 summaries. This how you know that you are buying the best documents.
Quick and easy check-out
You can quickly pay through credit card or Stuvia-credit for the summaries. There is no membership needed.
Focus on what matters
Your fellow students write the study notes themselves, which is why the documents are always reliable and up-to-date. This ensures you quickly get to the core!
Frequently asked questions
What do I get when I buy this document?
You get a PDF, available immediately after your purchase. The purchased document is accessible anytime, anywhere and indefinitely through your profile.
Satisfaction guarantee: how does it work?
Our satisfaction guarantee ensures that you always find a study document that suits you well. You fill out a form, and our customer service team takes care of the rest.
Who am I buying these notes from?
Stuvia is a marketplace, so you are not buying this document from us, but from seller studywithneo. Stuvia facilitates payment to the seller.
Will I be stuck with a subscription?
No, you only buy these notes for $4.57. You're not tied to anything after your purchase.
