Due diligence for research papers

Derek Lowe writes here [link] about “the sudden demise of Beall’s List of Predatory Publishers“.  He explains that these are publishers whose journals “publish verbatim whatever gets sent in, despite promises of peer review and editing, as soon as the funds hit their bank account“.  The list was controversial and the reasons for it’s demise are not public. However copies have been retrieved from the internet archives, for example here [link]. The problem with a blacklist such as this is that publications deserving to be on it keep mutating and growing in number, so it is difficult to keep track of them. An arguably better approach is to focus on publishers and publications that can be trusted, a list that will change more controllably. Helpful sources for locating these include: Continue reading “Due diligence for research papers”

Literature review dissertation

Chris Hart (2005) identifies three types of dissertation for a Masters degree. The ‘traditional dissertation’ is usually based on primary data collection to test a hypothesis or proposition or to  fulfill a research aim. The ‘work-based dissertation’ is focused on an issue that is relevant to an organisation with the aim of making recommendations for change. The ‘literature review dissertation’ uses the literature to explore an issue or argument or the origins of an idea or to provide evidence for decision-making.

He suggests that a literature review dissertation has four, generic stages

Hart p.140


Reference: Hart, Chris. Doing your masters dissertation. Sage, 2005.

Quality of case study design

Yin (2009, p.40) identifies four, widely-used tests for judging the quality of research design:
  • Construct validity: identifying correct operational measures for the concepts being studied;
  • Internal validity (for explanatory or causal studies only and not for descriptive or exploratory studies): seeking to establish a causal relationship, whereby certain conditions are believed to lead to other conditions, as distinguished from spurious relationships;
  • External validity: defining the domain to which a study’s findings can be generalized;
  • Reliability: demonstrating that the operations of a study – such as the data collection procedures – can be repeated, with the same results.

In the figure below he summarises, for a case study, what tactics are appropriate for each test and the phase of research in which the tactic applies.

YIN, R. K. 2009. Case study research: Design and methods, Sage publications, INC.

Content analysis

Krippendorff (2012, p.24) describes it thus: “Content analysis is a research technique for making replicable and valid inferences from texts (or other meaningful matter) to the contexts of their use“. The nature of a research design using this technique is shown in the figure below (source: Krippendorff, 2012, p.83). The framework is simple and general, employing only a few conceptual components: content-analysis_krippendorf-fig-4-1

1. A body of text, the data that a content analyst has available to begin an analytical effort .
2. A research question that the analyst seeks to answer by examining the body of text.
3. A context of the analyst’s choice within which to make sense of the body of text.
4. An analytical construct that operationalizes what the analyst knows about the context of the body of text.
5. Inferences that are intended to answer the research question, which constitute the basic accomplishment of the content analysis.
6. Validating evidence, which is the ultimate justification of the content analysis.

He goes on to explain what this means.

 As a technique, content analysis involves specialized procedures. It is learnable and divorceable from the personal authority of the researcher. As a research technique, content analysis provides new insights, increases a researcher’s understanding of particular phenomena, or informs practical actions. Content analysis is a scientific tool.

Techniques are expected to be reliable. More specifically, research techniques should result in findings that are replicable. That is, researchers working at different points in time and perhaps under different circumstances should get the same results when applying the same technique to the same phenomena. Replicability is the most important form of reliability.

Scientific research must also yield valid results, in the sense that the research effort is open to careful scrutiny and the resulting claims can be upheld in the face of independently available evidence. The methodological requirements of reliability and validity are not unique to but make particular demands on content analysis.

The reference to text in the above definition is not intended to restrict content analysis to written material. The phrase “or other meaningful matter” is included in parentheses to indicate that in content analysis works of art, images, maps, sounds, signs, symbols, and even numerical records may be included as data – that is, they may be considered as texts-provided they speak to someone about phenomena outside of what can be sensed or observed. The crucial distinction between text and what other research methods take as their starting point is that a text means something to someone, it is produced by someone to have meanings for someone else, and these meanings therefore must not be ignored and must not violate why the text exists in the first place. Text – the reading of text, the use of text within a social context, and the analysis of text – serves as a convenient metaphor in content analysis.

Krippendorff, Klaus. Content analysis: An introduction to its methodology. Sage, 2012.

Types of research

Kumar (2010, p9) suggests a good way to begin describing your type of research. He says that research can be looked at from three perspectives: (1) application of the findings of the research study; (2) objectives of the study; and (3) mode of enquiry used in conducting the enquiry.



KUMAR, R. 2010. Research methodology: A step-by-step guide for beginners, Sage Publications Ltd.

Primary and secondary data

What is the difference between primary data and secondary data? Suppose that an author has obtained primary data from ‘the field’ and published findings from an analysis of that data. If the author’s primary data is accessible you can use it yourself to carry out your own analysis and publish your own findings, which might differ from those of the original author. For you, the data is secondary data. Wikipedia describes this well. Suppose, however, that you take the author’s document and, instead of using the underlying raw data from which the document was created, you carry out a content analysis of the document itself. For example you might analyse a project feasibility study to evaluate its compliance with standard project management methodology. Your content analysis will produce primary data. There are, of course, other ways of obtaining primary data. For example you might interview people who worked in the organisations associated with the feasibility study. But the point is that you don’t have to carry out surveys, interviews and the like in order to obtain primary data.

Literature review

A literature review enables you to identify the various concepts (ideas) that are relevant to your topic and to describe those which are important. Depth and subtlety are often a result of the relationships amongst concepts. Your description should therefore explain those relationships where they are relevant to your line of reasoning. It should also compare and contrast the differing ideas and meanings that are uncovered in this way. The literature review therefore enables you to define the boundary of your topic and to find the puzzles (or gaps) that lie within it. It is these puzzles which are your potential research questions.
Note: See my postings on ‘theory’ and ‘variable concepts’ for a reminder of what we mean by a concept and how we use concepts to explain the meaning of things.

Variable concepts

A literature review should clarify the boundary of your topic and enable you to find the puzzles (or gaps) that lie within it. The review enables you to identify the various concepts that are relevant to the topic and describe those which are important. Depth and subtlety are normally a result of the relationships amongst concepts. The description must therefore explain those relationships where they are needed for your line of reasoning. Neuman (2011, p.65) explains it well.

Single versus Concept Clusters. We rarely use concepts in isolation from one another. Concepts form interconnected groups, or concept clusters. This is true for concepts in daily life as well as for those in social theory. Theories have collections of associated concepts that are consistent and mutually reinforcing. Together, the collections can form a broader web of meaning. For example, in a discussion of the urban decay, we may read about associated concepts such as urban expansion, economic growth, urbanization, suburbs, center city, revitalization, ghetto, mass transit, crime rate, unemployment, White flight, and racial minorities. Used together, these concepts form a mutually reinforcing collection of ideas that we use in theorizing and research studies.
We can simplify the concepts in daily life and social theory into two types. One type has a range of values, quantities, or amounts. Examples include amount of income, temperature, density of population, years of schooling, and degree of violence. These are variables, or variable concepts, that you will read about in Chapter 6. The other type expresses categories or non-variable phenomena (e.g., bureaucracy, family, college degree, homelessness, and cold).

Simple versus Complex Concepts. In addition to ranging from concrete to abstract and being a variable or non-variable type, concepts can be categorized as simple or complex. Simple concepts have only one dimension and vary along a single continuum. Complex concepts have multiple dimensions or many sub-parts. We can break: complex concepts down into several simple, or single-dimension, concepts. In general, the more complex concepts tend to be more abstract and simple ones more concrete, although this is not always true.

Neuman (p.218) also has helpful things to say about how to go about measuring a variable.

“The level of measurement is determined by how refined, exact, and precise a construct is in our assumptions about it. This means that how we conceptualize a construct carries serious implications.  It influences how we can measure the construct and restricts the range of statistical procedures that we can use after we have gathered data. Often we see a trade-off between the level of measurement and the ease of measuring. Measuring at a low level is simpler and easier than it is at a high level; however, a low level of measurement offers us the least refined information and allows the fewest statistical  procedures during data analysis. We can look at the issue in two ways: (1) continuous versus discrete variable, and (2) the four levels of measurement.

Continuous and Discrete Variables. Variables can be continuous or discrete. Continuous variables contain a large number of values or attributes that flow along a continuum. We can divide a continuous variable into many smaller increments; in mathematical theory, the number of increments  is infinite. Examples of continuous variables include temperature, age, income, crime rate, and amount of schooling. For example, we can measure the amount of your schooling as the years of schooling you completed. We can subdivide this into the total number of hours you have spent in classroom instruction and out-of-class assignments or preparation. We could further refine this into the number of minutes you devoted to acquiring and processing information and knowledge in school or due to school assignments. We could further refine this into all of the seconds that your brain was engaged in specific cognitive activities as you were acquiring and processing information.

Discrete variables have a relatively fixed set of separate values or variable attributes. Instead of a smooth continuum of numerous values, discrete variables contain a limited number of distinct categories. Examples of discrete variables include  gender (male or female), religion (Protestant,  Catholic, Jew, Muslim, atheist), marital status  (never married single, married, divorced or separated, widowed), or academic degrees (high school  diploma, or community college associate, four-year  college, master’s or doctoral degrees). Whether a variable is continuous or discrete affects its level of measurement.

Four Levels of Measurement. Levels of measurement build on the difference between continuous and discrete variables. Higher level measures are continuous and lower level ones are discrete.  The four levels of measurement categorize its precision.

Deciding on the appropriate level of measurement for a construct is not always easy. It depends on two things: how we understand a construct (its definition and assumptions), and the type of indicator or measurement procedure.

The way we conceptualize a construct can limit how precisely we can measure it. For example, we might reconceptualize some of the variables listed earlier as continuous to be discrete. We can think of temperature as a continuous variable with thousands of refined distinctions (e.g., degrees and fractions of degrees). Alternatively, we can think of it more crudely as five discrete categories (e.g., very hot, hot,  cool, cold, very cold). We can think of age as continuous (in years, months, days, hours, minutes, or seconds) or discrete categories (infancy, childhood, adolescence, young adulthood, middle age, old age).

While we can convert continuous variables into discrete ones, we cannot go the other way around, that is, convert discrete variables into continuous ones. For example, we cannot turn sex, religion, and marital status into continuous variables. We can, however, treat related constructs with slightly different definitions and assumptions as being continuous (e.g., amount of masculinity or femininity, degree of religiousness, commitment to a marital relationship). There is a practical reason to conceptualize and measure at higher levels of measurement: We can collapse higher levels of measurement to lower levels, but the reverse is not true.

Distinguishing among the Four Levels. The four levels from lowest to highest precision are nominal, ordinal, interval, and ratio. Each level provides a different type of information.

Nominal-level measurement indicates that a difference exists among categories (e.g., religion:
Protestant, Catholic, Jew, Muslim; racial heritage: African, Asian, Caucasian, Hispanic, other).

Ordinal-level measurement indicates a difference and allows us to rank order the categories (e.g., letter grades: A, B, C, D, F; opinion measures:  strongly agree, agree, disagree, strongly disagree).

Interval-level measurement does everything the first two do and allows us to specify the amount of distance between categories (e.g., Fahrenheit or celsius temperature: 5°, 45°, 90°; IQ scores:  95, 110, 125).

Ratio-level measurement does everything the other levels do, and it has a true zero.  This feature makes it possible to state relationships in terms of proportion or ratios (e.g., money in-  come: $10, $100, $500; years of formal schooling:  1, 10, 13).

In most practical situations, the distinction between interval and ratio levels makes little difference.”

NEUMAN, W. L. 2006. Social research methods : qualitative and quantitative approaches, Boston, Mass. ; London, Pearson, Allyn and Bacon.

Ethical approval

The requirements for ethical approval of research by my MSc students have been tightened up this year. The note here draws together the various strands of guidance.