Editor’s note: See an update at the bottom about GM Watch’s response to this post.
In a discussion about the scientific literature on genetically engineered crops, Claire Robinson of GM Watch has previously said: “I am, as you say, unaware of your GENERA project. A comprehensive list of studies on all aspects of GMOs is badly needed but beyond our means to gather together.” Biology Fortified, Inc., with our limited resources and volunteer staff, have come to the rescue and created this “badly needed” resource. Happily, GM Watch is now aware that GENERA is in beta testing, with more to come.
In response to our recent press release about the beta test of GENERA, Claire Robinson tried the resource out, and found it useful. She was able to use GENERA to search for literature, find the information she wanted, and she agreed with our evaluation of the scientific literature in at least two cases, as discussed in a post published on GM Watch* in response to our press release about GENERA. Although she thought it both needed and useful, the GM Watch post took an odd tone in the form of a criticism – claiming that we were misrepresenting the research contained in GENERA and leaving out important information. There were so many misunderstandings in Ms. Robinson’s post that I wanted to help her organization understand what it all means and clear up their confusion.
GM Watch agrees with assessment in GENERA of at least 2 studies
While GM Watch is criticizing our conclusions, they have indicated that they found GENERA to be both useful and accurate. They mention two studies and show that they agree with how we have classified the results.
These include Pusztai’s study on GM potatoes, which the GENERA authors correctly note had a “negative” conclusion for food safety. To be specific, the GM-fed rats showed gut cell proliferation that was similar to a pre-cancerous condition.
The GENERA authors also include the multi-generational study by Kilic and colleagues, which they note had “mixed” results. These consisted of damage to liver and kidneys and alterations in blood biochemistry in rats fed GM Bt maize over three generations, though other measured parameters showed no effect.
GM Watch picks out these two studies, but they seem to misunderstand science. Science is done not by cherry-picking single studies or data points within a single study to make conclusions that are at odds with the rest of the literature. Biology research is messy and data is often hard to interpret.
Table 4 from Kilic et al.
The Kiliç et al study is a perfect example of this. They fed rats three different diets (standard diet, control maize, and transgenic maize) for three generations, as part of a masters thesis project. The data was highly variable and confusing, thus it was classified as mixed results, but there were some strange issues with the health status of these rats. One of the control groups fed non-GMO corn had an average liver size that was 1/5 the normal relative size! What is going on with these rats? This shows the perils of cherry-picking data points from individual studies to claim that GMOs are “toxic,” when if you took their approach, this table on the right would be interpreted to mean that non-GMO corn is “toxic” to female rats. The Pusztai study on GMO potatoes suffers from many other similar problems that have been discussed elsewhere.
Scientists have to take all of the evidence into account, and now GENERA gives everyone the ability to step back and see the bigger picture. While it may seem ironic to be using GENERA to dig up individual studies to argue that GENERA is “misleading” while also confirming its accuracy, GM Watch is showing us that they are finding GENERA to be a useful resource for them. In addition, based on previous comments, Claire Robinson of GM Watch has said that it is badly needed. Hence the title of this post.
Conflicts and interests
GM watch mentioned a 2011 paper by Diels et al. That paper found that studies conducted by authors classified as “industry-affiliated” were more likely to show a favorable result for genetic engineering (being safer than or as safe as conventional, or effective at achieving the desired outcome). However, the authors only considered 94 studies which is a small subset of the total research available, and they had some strange definitions for how they classified the studies. For instance, some government agencies were classified as industry-affiliated while others were not, and without seeing the actual data, there’s no way to tell if this classification system makes sense. In addition, there was no consideration in the paper of funding or affiliation to competing industries or NGOs (or agencies for that matter) that are affiliated with these industries. There was also a “one drop rule” for industry affiliation – if only one author or funding source was considered “affiliated” with industry, it was classified with studies done entirely by the industry. These issues may introduce a bias into the data analysis, which is amplified when it is done on such a small subset of the research. Even so, they found no association between funding sources and outcomes, only between outcomes and their novel “one drop rule” classification system. For another view about industry funding, including analysis of the Diels et al paper, see this post by Marc Brazeau.
When we planned GENERA we wanted to avoid these kinds of pitfalls. We included an array of classifications for funding sources that includes governments, individuals, and different kinds of industries and NGOs. Most importantly, everyone can see how we have classified each study in the Atlas – the data is entirely public and transparent. GM Watch’s comments about taking our conclusions “on faith” just don’t make any sense considering that our data is public and the data for the study they refer to is not. In fact, we encourage people to do searches, make charts, and check individual studies and tell us if they find any problems! (That’s how a beta test works!) Finally, our tentative conclusion that the clear majority of studies in the literature indicate safety is based off of 197 studies, which is more than twice the number of studies that was included in Diels, et al. And when GENERA reaches its full release, it will have many, many more.
(click for a larger image)
Additionally, GM Watch claims that the Diels review study already found that half of the research was independent, however, that is not the case. They found that 39% of the 94 studies they examined were independent by their classification system, which is not half. In fact, 39% is closer to the 1/3 estimate that we made early on before we began work on the Atlas itself. We highlighted this result because it disconfirms our previous rough estimate as well as many of the claims that are made on the internet, including in articles republished by GM Watch and the Diels et al. study. There is a perception that nearly all of the research is conducted by the biotech industry, and we wanted people to know that the GENERA beta version shows that approximately half of the research on the safety of GMOs for consumption is funded by government agencies and independent nonprofit organizations.
So how do the results of studies funded by these different sources compare? As an infographic produced by the Genetic Literacy Project shows, we have found that industry funded studies have more favorable results for genetic engineering (4 safer than, 35 as safe as, and 1 less safe) (at least with our randomly selected 400 papers from the literature). We also found that independent (non-industry) government funded studies also overwhelmingly show a favorable result for genetic engineering (7 safer than, 62 as safe as, 7 inconclusive, and 6 less safe). We hypothesize that this trend will hold as we add more studies to GENERA. Either the industry results confirm the government results because they are based on the same science and genetic engineering is indeed a generally safe technology, or there is a massive global conspiracy among scientists working for governments around the world. In my years as a scientist, I have not yet found evidence of such a conspiracy, so if GM Watch has such evidence I hope they share with the rest of us.
(click for a larger image)
Just a misunderstanding
In our press release, I was quoted as saying “Systematic reviews have concluded that genetically engineered crops are safe to eat, and when you look at the results collected in GENERA, it agrees with that conclusion.” but the press release didn’t include specific citations to reviews. GM Watch took issue with that, claiming that I had an “avoidance of specific citations”. Perhaps they think that the press release was a scientific journal article. I’m not sure why they think that, but that’s okay, it’s obviously just a misunderstanding. Press releases, as a form of prose, generally don’t have citations. Blog posts usually don’t have citations either, however at the Biofortified Blog, our blog posts often have scientific citations, so if GM Watch thought that this was a blog post then maybe that would explain their confusion?
Claire Robinson has also said that her organization purposefully didn’t provide citations in a report they published, so you’d think GM Watch would understand not including citations in a press release. Ms. Robinson said: “The reason is that such a list would be boring in a report for the public.” Of course, the citations simply weren’t included in our press release because it’s a press release. Biology Fortified, Inc. doesn’t believe the citations are boring for the public, in fact the entire goal of GENERA is to make the citations more accessible to the public.
So where are the systematic reviews?
In the case of this press release, I was thinking more of the statements produced by various organizations around the world in favor of genetic engineering as a safe technology. For example: the American Association for the Advancement of Science (AAAS) and the World Health Organization (WHO). See statements from well-respected scientific organizations by clicking on the image to the right (image by Axis Mundi).
There have, of course, been systematic reviews in the literature as well as these consensus statements from well-respected scientific organizations. One of the largest is Nicolia et al in 2014 (PDF). This review included 1784 studies about various aspects of safety of genetically engineered crops for the environment and human/animal health. They conclude:
“We have reviewed the scientific literature on GE crop safety for the last 10 years that catches the scientific consensus matured since GE plants became widely cultivated worldwide, and we can conclude that the scientific research conducted so far has not detected any significant hazard directly connected with the use of GM crops.”
Each consensus statement and review has its own focus. Once GENERA is complete, people will be able to do their own reviews of the literature and report their findings with our helpful graphs!
Perhaps GM Watch wants to help
As we say in the press release, there are only 400 randomly selected studies in GENERA right now. Surely there are other studies that GM Watch and similar organizations would like to see represented in GENERA. And we say that’s just fine! We fully acknowledge that there are many more studies than just these 400 randomly selected ones in our beta test, and we fully acknowledge that the format and function of the database still need some tweaking. Check out GENERA and take our survey to let us know if you spot any errors, and let us know if we are missing any studies (but first check our page about what is included in GENERA).
* An earlier version of this post indicated that the GM Watch post was not attributed to any author, as is usually the case for GM Watch. The author of the post was Claire Robinson, as identified in italicized text in the post.
Editor’s Note: Karl Haro von Mogel contributed to this post.
Update 9-30-2014: GM Watch now asks Questions
We were delighted to see that Claire Robinson of GM Watch responded to our post with questions about GENERA, and confirmed several of our statements while disputing others. Unfortunately, she chose to spend a great deal of the post engaging in personal attacks and name-calling, however, we do believe that there are some important and worthwhile points that she brought up in her post which we will address.
Ms. Robinson correctly noted our error that the post was indeed signed to her name, in italicized text. This is a welcome change from the historical policies of GM Watch, which usually posts unsigned criticisms. We encourage GM Watch to take it to the next level and allow comments on their site so that a true discussion can take place.
Ms. Robinson also addresses the issue of how many studies are independent, and appears to be making some creative assumptions about studies that have no funding data. If you recall, we announced that just over 50% of the studies in our beta release for GENERA are independently-funded. Ms Robinson, however said:
We didn’t need BFI or GENERA to tell us that half of GMO research is independent of the industry. That is old news…
As we point out, Diels et al only stated that 39% of the 94 studies they surveyed were known to be independent, and our result dis-confirms this number and elevates it to over 50%. But in her response, Ms. Robinson appears to be counting studies with no funding information as independent:
In other words, somewhere between 39% and 53% (just over half) of GMO studies were independent at the time of Diels’ investigation.
While it may be possible that as many as 53% of the 94 studies examined by Diels were independently funded, we don’t actually know that. It would be wrong to assume what the funding sources are, and we find it curious that GM Watch is assuming that these unknown funding sources are independent in order to create a criticism out of confirming the findings of other researchers. This is a very odd argument to make. But if you were to add the 21% of studies in GENERA that do not have funding information to our total, then that means that the independent studies are between 50% and almost 75%. This is very different from the results of Diels et al.
Biofortified accuses me of “cherry-picking” data points from the Kilic et al study, which found mixed results.
From the KQED discusssion:
Our report, GMO Myths and Truths, nowhere claims to present the entirety of the literature on GMOs (as I’ve said, that’s way beyond our means), but to present some studies that show risks and hazards of GMOs that have not been properly addressed – and which are often claimed not to exist!
Our goal is to be comprehensive, while Ms. Robinson has stated that her goal has been to present only a select subset which meet a particular conclusion. That is what it means to “cherry-pick.” This is a correct description of choosing only results you agree with in one study or from a group of studies and ignoring data that undermines it, or that calls into question the validity of the study in question. Ms. Robinson does not provide a counter-argument that negates this description so it still stands as uncontested and it seems that she is justifying it rather than negating the description.
With these errors and misunderstandings out of the way, we can now help Ms. Robinson understand what is contained in GENERA. In fact, we are glad that she is asking questions!
We’re told the chart is based on 197 studies “out of 400 randomly selected”, but no further information is given about the questions these studies addressed or which methodologies they used.
These 197 studies are the studies that made conclusions specifically about the relative risks of genetic engineering on the safety of foods for consumption. In fact, we gave quite extensive information about the kinds of studies that have been included in GENERA, and what has been excluded. This information was posted on the GENERA site since before the announcement of the beta release. This can only mean that we either 1. did not post the menu link to the “What is in GENERA” page clearly enough under the “About” tab, in which case it would be our error, or 2. Ms. Robinson instead did not find out what was actually in the Atlas before claiming that we had misrepresented its contents. We will leave it up to the reader to decide which is more likely.
Ms. Robinson also asks about how many studies are reviews, original research articles, and perspectives/opinions. There are zero perspectives/opinions included in the graphics being discussed, and there are some reviews that have been included. Again, it is easy to find out how many reviews there are, as GENERA has a full range of search and exclusion features, so anyone could repeat the same search and include only reviews, exclude them, etc. Since Ms. Robinson is asking this question after-the-fact, we can only assume that she has not taken the time to understand what is contained in GENERA before making broad-sweeping claims about it.
Ms. Robinson has similar questions about what the different outcomes mean, which we explain on the Glossary of Terms page, and what species these studies have been done on. This information is contained within the Atlas and she has had the ability to search and chart these details herself. Since she is asking this question after having declared that we are misrepresenting our Atlas, we can only assume that she has not taken advantage of this public information and was again making broad sweeping claims without first learning the facts.
Biofortified should also identify each GMO for which they are claiming safety, since GMOs are evaluated on a case-by-case basis and a finding that one GMO is toxic or safe cannot be assumed to apply to any other GMO.
Until now, no resource has existed which could allow an analysis of what they are requesting here. Indeed, on a future version of GENERA we intend to have “event-specific” features that would allow users to search for studies only conducted on specific genetically engineered crops. However, we would like to note a contradiction evident in the list of questions posted on GM Watch. Ms. Robinson both states that the control animals should not have been “fed non-GM feed,” but that no two GMOs are the same. This is a point of confusion which we note that GM Watch tends to make. If the GMOs are all different, then it doesn’t matter if the control and experimental diets both contain a GMO-derived ingredient from a crop that is not being tested in the experiment. As long as the variable that is different is the trait being tested, that is fine. If all GMOs were the same, then it would be correct to require excluding them. You can’t have it both ways.
This leads us to the final answer to her questions. We did not make judgements about the completeness or incompleteness of the individual studies. Our goal is to include every relevant study – warts and all – so that people can have greater access to them. A case in point is the inclusion of the retracted Seralini 2012 feeding study, which was criticized around the world for inadequate sample sizes for the duration of the experiment, and did not ensure that non-GM feed was fed to the control rats prior to the beginning of the experiment – both of which are criteria that Ms. Robinson identifies as study flaws. If we were to exclude studies that contain these flaws then we would be forced to exclude this study as well. We may in the future incorporate a feeding study completeness rating based on published critical reviews – which might help people gauge the overall study quality – but that is beyond the scope of our Atlas’s capabilities at this time.
It is a pity that Claire Robinson freely and readily casts aspersions when engaging in discussion and debate about genetically engineered crops, and attacks individual people with ad-hominem attacks rather than spending the time it takes to understand the science and the resource she is criticizing in the first place. We hope that these answers will help clarify her confusion and that she will be able to engage in meaningful and civil dialog and help us to improve this public resource for everyone. We are happy to clarify these and other details so that people can better understand what is in GENERA, how to use it, and what conclusions may be drawn from it. We look forward to constructive criticisms, and are thankful that Ms. Robinson has agreed with the results for what individual studies she has indicated that she has examined in the Atlas thus far.