Meaning of "population" always needs to be established.
Surveys started out as simple counting, are now exercises in measurement. Substantial technology around validity, i.e. does it measure what it intends to measure?
5 Technologies of research:
7-8 sampling technology - constructing a sample that will represent the whole.
8-9 technology of comparison - for every question we have to ask is a comparison involved.
9 validity:
10 is a survey appropriate? Five questions:
This book looks primarily at 1, 2 and 3.
13 the first aim of research is to find out true things about the world
Validating evidence very important - given an adequate definition of the problem, the rest of the planning stage involves obtaining valid evidence and demonstrating its validity.
14 most important part of project is prior analysis; the most important problem is preconception.
18 case study on travelling for surgery - political questions, prospective, choice of groups, choice of control groups, testing and refining of questionnaire.
33 4 elements in initial planning and definition of survey:
38 concern for respondents: informed consent - but note that explaining purpose might change answers. Arguably, as non-professionals, respondents might not be capable of being fully informed.
An ethical question: what harm can come from our results? Either to respondents directly, or as a result of our findings being used to make policy or practice decisions. Knowledge is power and will be used as evidence to make decisions.
39 variables:
43 importance of library work in early stages:
Other people's results - comparison base, validity check.
Theoretical, methodological, practical and experiential insights.
45 Don't start with full literature review. Start with a few core texts to get general shape, then scan backwards and forwards.
46 research proposal:
In essence the definition and proposal contain the whole of the research project.
49 Sampling as it should be so that the sample is representative of population, groups can be validly compared, size of difference or correlations between them in actual population can be assessed.
53 levels of measurement:
Simple, systematic, and stratified (including matched) random samples.
Sampling frame - list of population from which sample is to be drawn.
59 how to pick samples - random sampling gives rise to bell curve of normal distribution in which 68% lie within one standard deviation, plus or minus, of the mean, and 95% within 1.96 standard deviations. Likelihood of random sample following the population is the same.
64 standard error measure of distribution of sample means about a population mean.
68 systematic sampling may misrepresent population if population has patterns that happen to mirror the system of the sample.
69 Stratified sampling mirrors major groupings of the population.
73 when comparing means we test the null hypothesis that there is no difference between the two groups in the population; and then set a probability for how unlikely this assumption hsa to be. We begin to feel confidence at a level of about one chance in twenty, i.e. a probability of error of 0.05.
73 matched samples - e.g. in prison choose long term, then match them with medium term and short term prisoners with similar characteristics e.g. type of crime, reported mental stability, level of intelligence etc. (Problems - not representative of parent populations, difficult to match, there may be other variables for which we have not controlled.)
81 Sampling over time - e.g. hospital patients - sample frame not available but still applies principles of random sampling. Problems - sample all times of day and all days of week. Is a week long enough? At the time in question was something unusual happening?
Time frame:
83 cluster sampling In absence of sampling frame, but if location and distribution known (usually geographical), random sample can still be drawn. E.g. for schoolchildren - pick educational authorities at random; then pick schools from these authorities at random; then classes in these schools; then pupils in these classes. At each stage selection must be weighted. This may result in highly distributed samples. Solution to this is cluster sampling.
Problem is clusters will not be representative, but has substantial benefits in cost and convenience.
86 non probability sampling - two major forms quota sampling and haphazard sampling. Quota sampling defines certain characteristics then leaves interviewers to find fixed number with those characteristics. Matches population well on chosen characteristics, but perhaps not on others. Also doesn't control for interviewer bias, e.g. may choose from among their friends, who will have similar characteristics.
88 haphazard samples, samples of opportunity worst of all - poor likelihood of them being representative.
92 sampling error decreases as square root of sample size.
Note statistical significance and substantive significance not the same thing. But you might start with a small sample to test the water and then move on to a larger one if the results were satisfactory.
94 non-sampling error error built into the design or mode of collection of data.
95 sample attrition some not traceable, refusals, some may move, etc. Can omit cases, put in reserves, etc.
102 process of translation of a concept, such as "child violence" to instructions for what to count in observed behaviour is called operationalisation.
Facts and attitudes - fundamentally very difficult. Even "who was watching TV?" - who was in the room, were they actually watching, were they there all the time.
106 research into attitudes and beliefs is fundamentally based on deception.
107 projective work may uncover attitudes e.g. finishing a story.
validation of measures - i.e. do they measure what they say they do. Standardisation is first requirement. Reliability - consistent measurement of the same thing. (Note particularly inter-rater reliability.
Varieties of data collection:
Three things necessary for successful questionnaire:
Direct measures - factual. Indirect and attitude/personality scales.
123 observation schedule - largely the appliance of common sense. NB for direct observation you need to identify and categorise behaviours - e.g. interactional categories.
Interviewers need to be trained and briefed.
127 conduct of interviews. principle not stimulus equality but stimulus equivalence.
Impression management very important - try to match interviewers to interviewees e.g. for race.
Coding - as much precoding as possible is good, but some categories need expert analysis e.g. social class, nature of highest academic qualification (where examination of "other" might be helpful).
132 three ways to approach coding
134 Key characteristic is how unscientific the whole process is.
Examples - attainment and ability.
Attainment should be easy to measure - measures, e.g. tests, have high face validity. Can be further validated by comparison with other known tests - concurrent validity, or by correctly predicting scores in subsequent tests - predictive validity. Need also to test reliability.
Looking at ability - more difficult. Has to be operationalised, to show capability independently of culture, background, level of education etc.
139 Validity - not a yes/no property of evidence but a process of testing which evidence undergoes:
140 Attitudes - big problem that attitudes and observed behaviour correlate very poorly. A lot of variance in behaviour seems attributable to circumstance and history; a lot of variance in attitude seems attributable to reactivity, i.e circumstances of research and way questions are asked. hypothesise theory of planned (or rational) behaviour which suggests intention intervening between attitude and behaviour.
142 direct observation is one way of dealing with this, but is time consuming and expensive, and records may be deficient. Another option is use of vignettes - artificially constructed case descriptions which practitioners discuss or respond to.
143 Another complex construct - class. Social Class Scale; - most useful for voting; - lifestyle etc. All underrate difference in women's labour occupation. Surrey Occupational Class Scale designed to overcome this.
147 complexity and reality. Fundamental debate between instrumentalism and realism. Realism as true description of the real world; instrumentalism - science as set of techniques for prediction and manipulation judged by the extent to which they work in practice.
150 random sampling must produce errors, with a calculable probability. Distribution of possible means has a standard deviation - the standard error. Differences between subgroups will have standard deviation - the standard error of the difference. Test for statistical significance, i.e. likelihood of result obtained arising by chance - calculate Z statistic - difference divided by its standard error. For small samples t statistic - more trustworthy for small samples but more difficult to calculate. Use t for samples smaller than 100 or where characteristic you want is <20% or >80% of sample.
152 on normal curve 5% of values lie outside +/- 1.96Z, and 1% outside +/- 2.58Z on two tailed distribution.
155 non-sampling error: doesn't necessarily invalidate survey - validation is a process admitting of degrees
159 use of concepts. One can create a concept and reify it, e.g. IQ. Personality inventories used by dating agencies and psychiatrists - once a measure exists it will be taken as measuring something.
Description - publishing descriptions can change results e.g. opinion polls.
Use of statistics e.g. on gender or race - can be descriptive but can also be ascribed i.e. use for purpose of social classification and control - possesses political importance.
Politics of research - research involves the exercise of power - researchers extracting the data *they* want, by means *they* choose, from respondents *they* select. And researchers are those who categorise the world - judges of normality helping to create universal reign of normality.
163 simplest form of complex analysis - complex analysis means being able to reason from apparent association in samples to actual associations in the populations, taking account of more than two variables and allowing for sampling error.
Model fitting - what would table look like if no association between variables; then assessing observable difference. If difference is sufficiently large, null hypothesis of no difference can be rejected.
To determine significance of chi squared, we need degrees of freedom. (From another source - The maximum number of quantities whose values are free to vary before the remainder of the quantities are determined.)
173 statistical control - effects to look for:
174 tabular anlysis:
Correlation and regression used for linear modelling - seeing how much of variation in dependent variable can be explained by positing a straight line relationship with an independent variable.
Intercept coefficient measures effect of intercept value.
176 Regression coefficient measures strength of association in original units, so not comparable. Correlation coefficients provide standardisation. Most common is Pearson's product-moment - standardises both measurement scales and expresses outcome in standard deviation units.
Correlation coefficients only work if variables are integral or ratio; or for dichotomous variables.
Sampling error affects correlation coefficients - squared deviation measures this. Correlation coefficient squared tells us how much each variable explains; what is left (1-r2)is result of error or other variables. We calculate rates of variances - F.
184 Multiple correlation coefficient - determines influences of variables on each other. Partial correlation coefficient determines influences of some variables while others are controlled for.
188 this is usable on up to four variables - beyond that unwieldy - for this regression is best.
189 Multiple regression - computer gives:
190 can do multiple regression one step at a time, adding variables in order. But note this does not explain independent variance, for which beta coefficients are needed. Or can put all variables in and remove one at a time. Or can choose order on theoretical grounds - hierarchical regression
192 Causal modelling, or structural equation modelling - if one variable is temporally or logically prior, it may affect the dependent variable through its affect on other independent variables.
198 issues about causal modelling and correlational analysis in general:
Discriminant function analysis takes dependent variable on a nominal scale and attempts prediction of category it will belong to on basis of available independent variables - the null hypothesis being that the variables will be no help.
Usual statistic is Wilks' lambda (smaller value = higher correlation). F statistic - ratio of explained to residual variable - can be used to assess statistical significance of each variable's inclusion.
Factor analysis - does not use dependent and independent variables. Looks at clusters of interdependent variables to construct systematic indices.
First stage principal components analysis - first component is set of interrelated variables that explain most variance; second component is cluster that explains most of rest; etc.
Measured by Eigenvalue - where total of eigenvalues is n for n factors. Method of choosing number of factors to accept - choose all with eigenvalue over one, or scree method, looking at graph and choosing all above point where factors appear to look like rubble.
203 rotation - of orthogonal factor lines on scattergram - can make factors more apparent.
Analysis of variance
215 restrictions on use of ANOVA - data need to be (more or less) evenly distributed on all independent variables; variances may be grossly affected by outliers - need to judge whether to exclude them.
Loglinear analysis - compute expected values for every cell, then all possible effects - allows building models which specify which effects ought to be significant. Logistic regression hybrid of loglinear and regression analysis, used to explain variance in a dependent variable expressed as a dichotomy.
Great strengths of all loglinear analysis - works on nominal data, and allows testing of fit of different hypotheses.
222 principle - look for parsimonious explanation - one which provides good fit to data using minimum of information. A slightly poorer fit that is readily interpretable is preferable to a slightly better fit that is unduly complicated.
224 Saturated model - one which has variables which account for total variation in whole data set. (Example 11.2 p225) Set up this model, then look at table of partial associations to eliminate variables that do not make significant contribution. Then also trim effects that do not involve your dependent variable. Remaining then described as unsaturated model. Can then experiment with building different models to see what happens. Home in on model that explains parsimoniously.
229 important when have fitted a model to look at pattern of residuals.
230 Trend analysis collecting figers over time on same variables from different people - e.g. British Crime Survey. Panel or cohort analysis measuring same people over time.
231 Loglinear analysis also helps to measure latent variables - underlying hypothetical variables indicated by a range of observed characteristics e.g. social class from occupation, housing, status, etc.
existing sources - much can be found in libraries - your research may already have been done, or it can be supported by other work. Things to look for:
240 essential first step is to state what the problem was.
Theoretical background or analysis - depth depends closely on length of report.
241 cover nethods - what is source of data, what information was collected, what procedure was followed. Particular attention to issues such as non-participation, possible skews in the sample, etc.
244 discuss results: preliminary descriptive issues, each step in the argument from your research to your conclusions, anything else which is suggestive or starts an important line of thought.
Every factual statement wants some justification from the data - often an illustrative table or graph. Statistical significance is very important in the results section.
Should mention ethical and political framework.