Evaluating the Qualitative Rigour of the Gabbay and le May (2004) Paper

This report serves the purpose of evaluating the quality of Gabbay and le May’s (2004) qualitative study: Evidence based guidelines or collectively constructed “mindlines?” Ethnographic study of knowledge management in primary care.

With the rise of qualitative research comes of the question of ‘what makes a good qualitative study?’ (Yardley, 2000). The same principles used to assess the quality of quantitative research cannot be applied, therefore it is necessary to employ a framework that considers the ambiguities and complexities that qualitative research itself tries to answer (Yardley, 2000). 

For this reason, this report will use the Critical Appraisal Skills Programme (CASP) (CASP, 2018) tool to evaluate the quality of Gabbay and le May’s (2004) paper. The CASP tool is one of the most common tools for quality appraisal of health-related qualitative research papers (Long, French, and Brooks, 2020), and uses 10 questions that focus on different methodological aspects of qualitative studies to assist in the appraisal of the study’s strengths and limitations.

For the remainder of this report, Gabbay and le May will be referred to as ‘the authors’ and their article that this report is evaluating will be referred to as ‘the paper’.

CASP Assessment 

Statement of research aims 

Answering a clearly stated and important research question is the foundation of a good qualitative research paper (Frankel and Devers, 2000). The paper has a clear mechanical research statement outlining the aims of the research, which was to conduct an in-depth analysis of how primary care physicians developed healthcare decisions for patients, both individually and collectively as a practice. The paper explicitly states that the study has important implications for how research findings are disseminated and used by physicians. Whilst the importance of the study has not been explicitly elaborated upon by the authors, the relevance and implications of the study help elucidate this meaning. The paper added to existing research by observing the ways in which physicians derive healthcare decisions. This has important future relevance in the healthcare sector as it can influence the consumption of evidence-based healthcare by targeting opinion leaders and promote networking as a form of professional development. However, the papers importance and relevance were not elaborated upon until the end of the article. 

Methodology appropriateness 

A qualitative methodology is very much appropriate for addressing the research aims of this paper, as the authors sought to illuminate how physicians derive healthcare decisions, considering the influences of organisation, culture, social interactions, and time. Furthermore, the authors inquired into why the physicians had acted in a particular way when delivering healthcare. These sorts of insights could not be generated by quantitative research, which likely would have only generated an answer to ‘how many physicians use evidence-based healthcare’ (Carter, Ritchie, and Sainsbury, 2009). Using qualitative research, the authors were able to discover why physicians did not use the traditional linear-rational model associated with evidence-based healthcare and discovered what they did instead, and the value physicians placed on mindlines. The authors were also able to observe several physician-physician and physician-patient interactions and illuminate the factors that influenced whether physicians referred to evidence-based healthcare or sought the opinion of their peers. The paper placed a high value on physician’s individual and collective thought processes that contributed to the reinforcement of mindlines, further demonstrating the appropriateness of using qualitative methods. 

Research design appropriateness

The paper’s title outlines that the authors have used ethnography as a research design. Ethnography as a research design aims to study people and their behaviour in their cultural context, both individually and collectively (Draper, 2015). As the aims of the paper’s research were to study how physicians derive their healthcare decisions, both individually and collectively, in their day-to-day practice, ethnography is an appropriate research design. However, the authors also noted that their analysis was informed by a few theoretical approaches, including grounded theory. This was beneficial because it allowed the authors to take both a focussed and broad observation lens to the interactions and decision-making processes of the physicians (Najafi, Roudsari, Ebrahimipour, and Bahri, 2016). 

The researchers did not explain their justification behind using ethnography, although we can assume this is due to the apparels between the research methodology and research aims. However, they did clearly state the reason behind using a number of theoretical approaches in their analysis, including so that they could be conscious of collective sense making and the role of social and organisational structures.

Recruitment strategy appropriateness 

According to Cleary, Horsfall, and Hayter, (2014), participant selection should serve a specific purpose and provide a definitive rationale. The paper’s authors explained that they purposively chose the two practices involved in the study. Purposive selection is a key selection principle for qualitative research (Cleary, Horsfall, and Hayter, 2014), and because recruitment was driven by the concepts of ethnography, the recruitment strategy is appropriate to address the aims of the paper’s research. 

The authors did not elaborate on why the chosen practices contained participants that were most appropriate to assist them in gaining knowledge that supported their research aims, however, we can assume that because each of the practices chosen was the best in their area, they would attract the best physicians. We can also assume that the best physicians would be likely to follow the best medical practices. In line with Carter, Ritchie, and Sainsbury’s (2009) recommendations, the authors have chosen practices that were uniquely positioned to assist the authors in understanding how the best physicians derive their healthcare decisions. 

The authors also did not elaborate on the consent process for recruitment or whether some physicians or patients were not interested in taking part in the study. With all this in mind, this criterion can be considered as partially met. 

Data Collection

The setting for data collection was appropriate, taking place mainly in the medical practices and during practice meetings where observation of physicians use of knowledge and information could be analysed in their natural setting. The authors primary methods of data collection were through non-participant observations and semi-structured interviews, formal and informal. According to Clarke and Braun (2013), researchers prepare an interview guide before conducting semi-structured interviews. The paper did not detail whether an interview guide had been developed or used in the study. The authors note that observations and informal interviews were documented in field notes, and all the formal interviews were tape recorded and transcribed. Box 1 in the paper is a thorough depiction of the data collected during the study, and it does not appear that the data collection methods were modified during the study. 

Whilst the authors did not explicitly justify the data collection methods, they are appropriate for an ethnographic study (Draper, 2015) and addressed the aims of this research. The authors could have improved the data collection methods for this research by considering if they had reached saturation.   

Relationship between researcher and participants 

There was no apparent consideration of the impact of the relationship between the researchers and participants during any point of the study in this paper. The author’s note that the amount of trust or respect that physicians have for other’s who are sharing medical knowledge dictates how much they value or take note of that knowledge. The authors have not stated whether they were conscious of the fact that the physicians trust for them may have changed the way the physicians behaved or talked to patients, thereby influencing the data. However, the data was collected over a period of two years, so it is unlikely that the data would be inaccurate after this period. The authors also failed to note how their relationship with the physicians grew over the two-year period and whether this had any effect on the richness of outputs.

Ethical issues

The paper has provided no detail on the ethical considerations of this study, nor is it possible to ascertain whether ethical standards were maintained throughout the study. As noted by Guillemin and Gillam (2004), procedural ethics and ethics in practice are both major ethical dimensions that need consideration in qualitative research. However, the authors did not clarify if they received approval from an ethics committee to undertake this research, how they explained their research to the participants, or how they handled medical confidentiality agreements. The quality of the paper would have benefited if these details were provided.

Rigorous data analysis 

There is an in-depth description of the data analysis process, whereby coding and thematic analysis have both been used. Thematic analysis and coding are both appropriate methods for data analysis in this study as they are major parts of both ethnographic and grounded theory research (Liamputtong and Ezzy, 2005). The process of thematic analysis and coding was adequately explained and box 2 provides a more in-depth insight into how the original data was coded/themed and transformed into an insight. The data analysis process could have been improved if the study had ensured that each researchers analysis process was corroborated more thoroughly by the other researcher to ensure consistency. Moreover, the rigour of the data analysis would also have been improved if the researchers had considered their own role in potentially biassing or influencing the data analysis or conclusion of results; this aspect was not commented on by the authors.

Whilst the authors did not present sufficient data to support their findings, they did provide a thorough explanation of how the different aspects of the data lead to specific conclusions. Furthermore, the authors identified and used atypical incidents in their research to elevate and expand their analysis. 

This criterion can be considered partially met.

Statement of findings

The authors are explicit in their explanation of findings, detailing each key factor that contributed to how physicians derive healthcare decisions and what the implications of the research findings are for the future use and dissemination of clinical research. The statement of findings is a direct answer to the original research question. The identification and subsequent investigation of atypical incidents provides robust evidence that the researchers adequately analysed evidence for and against their argument. However, this could have been further elaborated upon to explain precisely how the atypical incidents affected the data and results of the study. 

By gathering data from a range of sources (physicians and nurses) and through multiple methods (observation, formal and informal interviews), the authors have employed triangulation to achieve a multilayered, robust understanding of how physicians derive their healthcare decisions (Yardley, 2000). Furthermore, the authors have increased the credibility and validity of the study by using the second practice to validate their findings. 

Value of the research

The authors explicitly state what was already known on this topic and what further research was needed for a more complete understanding. This was explicitly stated in its own section and was not something that needed to be explicated from the detail of the article.

The authors designed their study to address the known gap in the literature, therefore they were able to contribute new and valuable insights to the research community. The authors also explicitly state what this study adds to the existing research base and the implications this has on the way clinical research is used and disseminated. The limitations on the transferability of this study have been clearly detailed, likewise has the necessity for further research to test the model in different organisational structures been explicitly noted. The authors also explain how the findings can be used to improve the use and dissemination of clinical research and propose innovative suggestions that overcome the implications of this research.

Enhancing the quality of the paper with an additional paragraph

A patient discussing health concerns with their doctor is an exercise of trust and relies on patient confidentiality agreements.¹ The nature of our study and research aims required us to be present during doctor-patient interactions, naturally threatening existing confidentiality agreements and patient trust, creating an ethical challenge. 

To overcome this, before observing a doctor-patient interaction, we explained who we were, the aims of our research, and what we would be observing as part of our study to the patient before health discussions begun. This was supported with a short, formal information sheet that could be shared with patients and contained information about our study and the confidentiality exercises we maintain when handling patient information. 

Once explained, our researchers sought verbal consent to continue by the patient, which was documented in our field notes. If the patient was uncomfortable being observed, our researchers would remove themselves from the situation and allow the health consult to continue as normal. After the consult, our researchers would then conduct a brief discussion with the doctor to analyse the encounter. Additionally, if verbal consent was maintained but the patient’s body language seemed to indicate they were uncomfortable, our researchers were encouraged to excuse themselves from the consult and make note of this in their field notes.

1 Dickerson JE. Privacy, confidentiality, and security of healthcare information. Anaesthesia & Intensive Care Medicine. 2022 Oct 5.

References

Carter, S.M., Ritchie, J.E. and Sainsbury, P., 2009. Doing good qualitative research in public health: not as easy as it looks. New South Wales public health bulletin, 20(8), pp.105-111.

Clarke, V. and Braun, V., 2013. Successful qualitative research: A practical guide for beginners. Successful Qualitative Research, pp.77-106.

Cleary, M., Horsfall, J. and Hayter, M., 2014. Data collection and sampling in qualitative research: does size matter?. Journal of advanced nursing, pp.473-475.

Critical Appraisal Skills Programme (2018). CASP Checklists. [online] CASP – Critical Appraisal Skills Programme. Available at: https://casp-uk.net/casp-tools-checklists/.

Draper, J., 2015. Ethnography: Principles, practice and potential. Nursing Standard, 29(36), pp.36-41.

Frankel, R.M. and Devers, K.J., 2000. Study design in qualitative research–1: Developing questions and assessing resource needs. Education for health, 13(2), p.251.

Gabbay, J. and le May, A., 2004. Evidence based guidelines or collectively constructed “mindlines?” Ethnographic study of knowledge management in primary care. Bmj, 329(7473), p.1013.

Guillemin, M. and Gillam, L., 2004. Ethics, reflexivity, and “ethically important moments” in research. Qualitative inquiry, 10(2), pp.261-280.

Liamputtong, P. and Ezzy, D., 2005. Making sense of qualitative data: analysis process. Qualitative research methods, 2, pp.257-285.

Long, H.A., French, D.P. and Brooks, J.M., 2020. Optimising the value of the critical appraisal skills programme (CASP) tool for quality appraisal in qualitative evidence synthesis. Research Methods in Medicine & Health Sciences, 1(1), pp.31-42.

Najafi, T.F., Roudsari, R.L., Ebrahimipour, H. and Bahri, N., 2016. Observation in Grounded Theory and Ethnography: What are the Differences?. Iranian Red Crescent Medical Journal, 18(11).

Yardley, L., 2000. Dilemmas in qualitative health research. Psychology and health, 15(2), pp.215-228.