Scale and Sample Size

I made myself quite unpopular at conferences by asking questions as to statistical significance of findings. All scientists want to prove some sweeping new concept or cure a disease, but depending on the scale and sample size, their efforts may not be relevant or useful to the world at large. The vast array in possibilities of SNPs accounts by and large for the frequency with which pharmaceuticals seem prone to causing severe complications including death because they are designed for the many and do not often take into account minute aberrations from "normal".

Many researchers came equipped with graphs and charts in a vast array of data, meant presumably to awe us with the enormity of their conclusions. However, I noticed with alarming frequency an absence of statistics validating the fit of their conclusions (not that that's always a guarantee depending on the frequency and severity of outliers for which we cannot account. More on that later.) I used my time to ask them questions on statistical relevance in order to determine how useful their science might be to me. After all, if I intend to springboard from their conclusions, I want to make sure their claims that appeal to me stand on solid ground.

For my own research, we considered both biological and technical replicates. I learned that lesson in industry at ARUP Laboratories in Salt Lake City. I would sample at least three different biological samples three different times for a total of 9 samples before plotting the data. This tripartate replication in biological and technical capacity helped me determine a better normality of data and isolate aberrations, which were usually due to operator error (me).

After that, I performed ANOVA, X2, and other tests. Please note that the R2 value in the graph from last entry is 99%. You need not do that much. I was willing to accept a simple standard deviation bar set and an n number representing sample size.

If you test one plant and then tell me you were able to raise its resveratrol levels under enhanced CO2 concentrations to 50x the normal level and then ask me to believe that will be true for every individual grape of every cultivar of every species in the genus, I won't buy it. Congress might, or maybe ASEV, and they may give you money, but it won't be useful to anyone if it was a fluke. Utility is after all what we seek.

Anything worth doing at all is worth doing well. Plus, it would prevent FDA warnings, Pfizer settlements, GSK recalls, ad infinitum, if a few scientists took the time to test a few more samples, especially if their sample size was one. Come on people.

Projections and Prognostication

While teaching undergraduate labs in graduate school, we reinforced the principle of being able to make a relevant comparison. The students were asked to analyze the protein content of tissues using various biochemical measurements in this particular example using an external standard.

After diluting a standard of 1mg/ml protein, they performed the analysis and calculations in order to produce a standard curve. The following represents a typical standard curve:



The students would then test their unknowns and compare them to this standard curve.


As luck would have it, the samples to be tested all had protein contents WAY outside the range of the curve they created. Some students foolishly extrapolated the data assuming it to be linear out to whatever value they obtained, ignoring the possibility that outside this range the curvealinear relationship observed might not hold true. We forced them to dilute their samples until the readings fell inside those covered by the standard curve and they could give an accurate number and then scale it back up using their dilution ratio.

When you project behavior beyond the measured realm, you lose all scientific credability. In economics, they continually remind you that "past performance does not guarantee future results", and in science we can only say what we have observed, not what we expect. That is a hypothesis, not data for a conclusion, so any projection represents what we THINK will be rather than what we know. This serves relevant point in politics I may address later on my other blog. It is not actually something we have measured, and represents not fact but conjecture and assumption.

Global warming advocates do this all the time. They project outside the range and extrapolate over a wide range of time and scale, the particulars of their experiment notwithstanding. Other scientists also like to apply measurements to things outside the scale and scope and make sweeping gestures which are not necessarily true.

When working on volatile compounds in grapes, the first thing I did was establish the linear detection limits for the GC/MS protocol I used. I was able for most compounds to detect them linearly to 4ppb, which is very important since the amount of volatile is not necessarily proportional to its importance or potency. Sometimes it is RELATIVE amount that makes all the difference. Even in a linear scale then, it might take 4000ppb to register a difference from 4ppb.

The proof is in the data. When scientists make broad sweeping claims across a vast array of possibilities in clime, scale, age, time, etc., I raise an eyebrow and my hand to inquire. More often than not, this error is also accompanied by another error and my next subject- economy of scale.

Why Start This Blog?

Many years ago in graduate school, I wasted quite a few months pursuing a project that would never ever work. What was even more frustrating is that other labs knew it wouldn't work, but they didn't bother to share their findings with us because science journals don't publish things that don't work.

In his book Climate Confusion, climatologist Roy Spencer makes the following observations. In speaking about why global warming alarmists and their complicit media counterparts sensationalize the armageddon scenario of world destruction, he points out that:

In science, if you want to keep getting funded, you should find something earth-shaking.

This phenomenon provides most of the impetus for hastily and poorly-drawn conclusions in science. To get published in a prestigious journal many scientists will project their findings to astronomically irrational levels and claim that "Our research on abiotic stress in creosote will one day provide all the rubber the world needs without any cost because these bushes grow wild throughout Nevada, so everyone who owns any of these shrubs on barren lots will one day be multi-millionaires" as a crude example. The truth is that, much as I like the guy, Dr. David Shintani's lab isn't remotely close to bringing any kind of alternative rubber source to market, nor will his lab by itself in our lifetime without some kind of corporate sponsorship and investiture.

In their haste to publish, graduate, and tack on a series of unintelligible vowels and consenants to the end of their names (mine are incidentally MSBMB, SSRAII, APB- whatever the heck that means) colleagues of mine have falsified data, omitted or deleted information, thrown out abnormal results without good reason, and made inaccurate claims based on a statistically insignificant number of biological and technical replicates. If you then try to piggyback on their research, by and large you may find their data and their conclusions faulty, meaning that you waste a lot of time and resources duplicating their efforts. What consequences do they face? None. I don't personally know of anyone stripped of a MS or PhD for having had their thesis/dissertation disproven.

The true tragedy is in cost to you the taxpayer and consumer. How much duplicit effort in time and money exists because people are only able to/interested in publishing breakthroughs that will exalt their own personal self-interest? Scientific journals as presently constituted concern themselves only with publishing what did work, to the exclusion of everything else we tried that didn't work with its accompanying data and explanations.

Enter the Journal of Negative Results. Would you like to know if someone already thought about trying to solve a particular scientific question with a particular technique? How did they fare? Why did they fail? Why can't they get credit for all that hard work with a publication? I think such a Journal adds value to the system of science and may save people a lot of time.

Now I lack the fudiciary means to fund such an endeavor, but I do have this- a searchable blog dedicated to any and all who would like to let the rest of us know what they have been able to disprove by their work. I may not offer a prestigious journal in which to publish, but I offer you yet another chance to get yourself on the Google or Yahoo web results for work you did and give credit where credit is due. I ask no compensation for this, and I will publish any and all information on techniques, organisms, variegations, equipment, and personnel who netted you abnormal results and didn't get you what you were aiming at, because maybe someone can serendipitously segue from your efforts and get an idea they didn't think about before all while saving everyone else time and money.

Now accepting manuscripts.