0
CarlJoos

Big Science is Broken

Recommended Posts

Nothing is real any more. Everything is a lie...
    Science is broken.


That's the thesis of a must-read article in First Things magazine, in which
William A. Wilson accumulates evidence that a lot of published research
is false. But that's not even the worst part.

Advocates of the existing scientific research paradigm usually smugly
declare that while some published conclusions are surely false, the
scientific method has "self-correcting mechanisms" that ensure that,
eventually, the truth will prevail. Unfortunately for all of us, Wilson
makes a convincing argument that those self-correcting mechanisms
are broken..."

http://theweek.com/articles/618141/big-science-broken

How the heck is one to know what to believe is true?

Share this post


Link to post
Share on other sites
Belief can be altered by scientific revision (The world is flat) and scientific revision can be altered by belief (Science proving the existence of God).
Nobody has time to listen; because they're desperately chasing the need of being heard.

Share this post


Link to post
Share on other sites
The111

***How the heck is one to know what to believe is true?



If only there was some kind of process that people could use to reach conclusions logically without needing to involve beliefs. :S

But that assumes those implementing a process allow the data to logically confirm/deny hypotheses and reach conclusions (or allow things to be inconclusive) and not fudge results to "prove" a prior assumptions. ;)

Yes, of course...one would assume or hope that all these scientists are honest with themselves...but I guess failed expirements, inconclusive evidence, etc doesn't earn grant$, is newsworthy, gets published, etc.

Share this post


Link to post
Share on other sites
BartsDaddy

And this suprises you? After all the researchers are human. And they want to keep thier houses and cars and lifestyles. To say thier findings goes against thier beliefes would doom thier research.



And btw, I think that in most situations when "findings go against their beliefs", "beliefs" in this case is not "religious beliefs" but a priori hypotheses that researcher wants to prove true.

Share this post


Link to post
Share on other sites
Well, here's your first problem, instead of reading opinion pieces about what other people write, trace down the source material.

Your quoting and linking of the opinion piece is a HUGE part of the issue in your final question of, "How the heck is one to know what to believe is true?"

Once you get to the source material on this, you should quickly realize it's also an opinion piece with a religious agenda.
quade -
The World's Most Boring Skydiver

Share this post


Link to post
Share on other sites
quade

Your quoting and linking of the opinion piece is a HUGE part of the issue in your final question of, "How the heck is one to know what to believe is true?"



The piece posted is so bad in so many ways.

It claims that science is "broken" because there has been dishonest research done somewhere. Then it mentions studies that uncovered this dishonesty, yet fails to attribute these studies (or to attribute anything period, for that matter, which is the essence of the article's horribleness). And even if it had attributed them, they are by nature scientific studies, and science is broken, so we can't trust them. So after attacking the validity of research as a general strategy to determine truth, it uses "research" to "prove" this very point. Better article title: Circular Logic is Broken.

Is every scientific researcher in existence 100% honest? Of course not. But claiming that science is "broken" because of this is a very poorly veiled attempt at justifying the idea that whatever you want to believe is equal to the truth, because there is no truth, because truth can't be known.
www.WingsuitPhotos.com

Share this post


Link to post
Share on other sites
The111

***If only there was some kind of process that people could use to reach conclusions logically without needing to involve beliefs. :S



But that assumes those implementing a process allow the data to logically confirm/deny hypotheses and reach conclusions (or allow things to be inconclusive) and not fudge results to "prove" a prior assumptions. ;)

My point was that you don't have to rely on "those implementing the process." The process is for you. Sure, Alice can lie to Bob about what is inside the box she's holding. But Bob can also just take the box and look inside of it. The entire premise behind the OP relies on mental laziness and feigns reliance on others. Science is an interactive thing. Get involved if you think scientists are lying to you. Prove them wrong. Or just bitch about it and satisfy your angst by reading shitty opinion pieces. (Note, the "you" used in this post is the generic you, not necessarily directed at the post I'm responding to)
www.WingsuitPhotos.com

Share this post


Link to post
Share on other sites
The111

But claiming that science is "broken" because of this is a very poorly veiled attempt at justifying the idea that whatever you want to believe is equal to the truth, because there is no truth, because truth can't be known.



Hell, there are people who even debate the meaning of the word "truth."

https://en.wikipedia.org/wiki/Truth

I'll stick with logic and mathematics for that.

A^2 + B^2 = C^2
quade -
The World's Most Boring Skydiver

Share this post


Link to post
Share on other sites
The111

******If only there was some kind of process that people could use to reach conclusions logically without needing to involve beliefs. :S



But that assumes those implementing a process allow the data to logically confirm/deny hypotheses and reach conclusions (or allow things to be inconclusive) and not fudge results to "prove" a prior assumptions. ;)

My point was that you don't have to rely on "those implementing the process." The process is for you. Sure, Alice can lie to Bob about what is inside the box she's holding. But Bob can also just take the box and look inside of it. The entire premise behind the OP relies on mental laziness and feigns reliance on others. Science is an interactive thing. Get involved if you think scientists are lying to you. Prove them wrong. Or just bitch about it and satisfy your angst by reading shitty opinion pieces. (Note, the "you" used in this post is the generic you, not necessarily directed at the post I'm responding to)

I first read the First Things article about a week ago, as a 'IRL' friend of mine posted it on FB and personally knows the author, who studied Mathematics at Yale. He seems to raise a bunch of valid issues. Besides dishonesty, mistakes, prejudice, etc, just the simple failure to retract or publicize mistakes (e.g. when a Parachutist article has a false statement or attributes a photo, the "correction" is hardly noticed in a subsequent issue since it is tucked into the corner of a page). You (actually both you personally and the royal "you") can't honestly expect the average person really get involved and prove scientists wrong. One needs become qualified and work in a particular field to even really question many of these studies with any sort of credibility.

The article in the OP now is just some journalist who writes a bunch of articles on various things and likes to bloviate. Yawn. I don't disagree that it's a shitty opinion piece.

Has anyone here who disagrees actually read the linked to First Things article? I know this is SC, but I thought many of us prefer to actually criticize the content and not the person (but maybe I expect too much).

Share this post


Link to post
Share on other sites
Elisha

You (actually both you personally and the royal "you") can't honestly expect the average person really get involved and prove scientists wrong.



And yet the average person still claims on a regular basis that scientists are wrong. Therein lies the heart of the issue.

There is some weight to the idea that nobody can be an expert in all disciplines. Science is a heavily connected thing, where all disciplines rely on each other. And if there truly is some massive conspiracy and they are all liars, then we truly are fucked. But if that is the case, again we hit the circular dilemma of how do you really know that? Because some other scientist told you? But you don't trust scientists! If those who we rely on for the truth (outside our areas of expertise) are all lying, to us and to each other, then we can never known the truth, and calling them liars is a waste of breath. If you call somebody a liar, presumably you know the truth, so presumably you ARE a bigger expert than them.

While not everybody can be an expert, everybody can get enough basic education to recognize bullshit from a mile away. And it's painfully evident that many people today do not have that basic education, and yet still love to call bullshit on meritorious topics, and likewise share conspiracy theories as gospel.
www.WingsuitPhotos.com

Share this post


Link to post
Share on other sites
Elisha



I first read the First Things article about a week ago, as a 'IRL' friend of mine posted it on FB and personally knows the author, who studied Mathematics at Yale.



1. Appeal to authority fallacy.

2. This about science, not mathematics. He might as well have studied art history.
...

The only sure way to survive a canopy collision is not to have one.

Share this post


Link to post
Share on other sites
Quote

Has anyone here who disagrees actually read the linked to First Things article?



I read/scanned it.

The piece had an agenda. Exactly the problem with scientific articles that the author was complaining about.

- Dan G

Share this post


Link to post
Share on other sites
metalslug

Quote

A^2 + B^2 = C^2



Hm.. is that true for all values of 2 ?

:D


I have a proof but it's slightly too large to fit in the margin of my copy of Arithmetica.
...

The only sure way to survive a canopy collision is not to have one.

Share this post


Link to post
Share on other sites
DanG

Quote

Has anyone here who disagrees actually read the linked to First Things article?



I read/scanned it.

The piece had an agenda. Exactly the problem with scientific articles that the author was complaining about.



So what if (YOU think) it had an agenda? Was the author wrong about many of his statements? Sounds more like an "appeal to motive" type of Red Herring fallacy because you just don't feel like debating.

The 111 mentioned intellectual laziness - true about the article the OP posted. But hard to say about the First Things article.

Share this post


Link to post
Share on other sites
It clearly had an agenda. I thought you said you read the article.

He points out a bunch of studies that purport to show bad science. He apparently didn't do any research into good science, or if he did he decided to keep it to himself. His premise that all science these days is suspect, but his paper itself is an excellent example of the problem he is writing about. He starts with the premise that science is junk, and then looks for confirmation. He doesn't present any evidence that doesn't comport with his beliefs.

If you read to the end you'd see that he's trying to show that science is just as flawed as religion because it is in the hands of flawed humans. It is a ridiculous premise. It would have been a more interesting article if it didn't have a pro-religion/anti-science undercurrent.

- Dan G

Share this post


Link to post
Share on other sites
Quote

Was the author wrong about many of his statements?

I think he was. Of course, how you look at things depends on how you define "broken". If you expect that every published paper will be perfect, with the conclusions impervious to reinterpretation based on new information gained from subsequent research, then I guess you could say science is "broken". In that case, though, no-one could ever publish anything until they were certain beyond all doubt that there was nothing more to be learned about a subject.

The study on reproduciblity of "social science" papers is instructive. As we all know from personal experience, human behavior is strongly influenced by environment and by prior experience (learning). All experiments in that field are highly subject to uncontrollable variables such as the personal life history of the people used in the experiment, and the results can be altered by seemingly trivial elements of the way the study is conducted, such as the precise wording of surveys or even the time of year (people behave differently [more helpful and generous] around Christmas for example, but only in societies that make a big deal about Christmas). People certainly are not chemicals, where the reaction products are highly predictable based on starting conditions. Many scientists who work on the physical sciences and biology don't really accept "social science" as a science, because everybody knows the methodological problems prevent repeatability.

Many studies that find a lack of reproducibility are really concerned with data management. For example, they will contact the authors of 100 published papers, many going back years or even decades, to obtain their raw experimental data, or get minor details of methods that were not included in the methods section of the paper. In theory, "reproducibility" includes that ability to reanalyze the original data and replicate the conclusions. If the authors did not respond within a certain period of time, the paper was scored as "unreproducible". However, no allowance was made for researchers who might have moved, retired, or died, and so never got the request. Perhaps this is an issue, but I would have to say that it would take me months to comb through all my student's lab notebooks to pull out the raw data that pertains to every paper ever published from my lab. If someone was to contact me and want all the raw data for a 10 year old study, and only give me a couple of weeks, it is not going to happen. In a pinch I might have the whole notebook copied and send it and tell them to figure it out. I cannot stop all my other responsibilities, tell the students in the courses I teach to take two weeks off, cancel every meeting, ignore every deadline, just to provide someone with a data point.

The BICEP2 study on plight polarization by gravitational waves associated with inflation in the first milliseconds after the Big Bang was instructive too. The study, which involved very precise and difficult measurements of incredibly small amounts of polarization, was rigorously done, measurements were repeated several times, and the data was analyzed according to state of the art theory. After it was published a different group of researchers (the Planc group), who were using satellite based observation to quantify dust in interstellar space, realized that the amount of dust they were finding (which was more than had previously been measured using less precise ground-based approaches) could explain most or possibly all of the observed light polarization. This group published their dust measurements six months after the BICEP2 study as published. They then contacted the first group who reanalyzed their data in light of the new observations and agreed that the higher amount of dust could account for at least 40%, and possibly all, of the light polarization, leaving it uncertain of there was any signal resulting from cosmic inflation. This new analysis was published jointly by the two groups. In what way does this reflect science being "broken"? A study was published, subsequently new data came to light, the data from the first study was promptly reanalyzed including the new data, the new analysis disagreed with the first, and the results were promptly submitted for publication. No-body hid the new data, nobody denied that the initial conclusions were undermined by the new data, instead everything was made public.

Perhaps we should consider Newton to have been a fraud, because his work was supplanted by Einstein's work on relativity? We still use Newtonian mechanics to calculate lots of things because it works very well, as long as you're not talking about close to light speed. Not many would consider Newton to be "broken". Not many would say that Newton should not have published the "Principia" because of the possibility that there might be something more to learn about physics.

As far as the crap in the opinion piece article about researchers having to conform to the theories of their elders, I have never experienced that or seen it happen to anyone.

Science is not perfect, after all it is an enterprise carried out by humans and humans are subject to a number of foibles, including becoming overly enamored of their own ideas and unwilling to tolerate criticism. Such people are rare in my experience, though, and quickly become marginalize and bypassed by the rest of the field. Everyone I know is much more motivated by curiosity and a desire to really figure out what is happening. Everyone can tell lots of stories about their great ideas that turned out to be compelling but wrong. There is no shame in that.

Only theologians and politicians are convinced of the unassailable correctness of their ideas, and neither group ever allow their ideas to be subject to critical examination.

Don
_____________________________________
Tolerance is the cost we must pay for our adventure in liberty. (Dworkin, 1996)
“Education is not filling a bucket, but lighting a fire.” (Yeats)

Share this post


Link to post
Share on other sites
GeorgiaDon



Only theologians and politicians are convinced of the unassailable correctness of their ideas, and neither group ever allow their ideas to be subject to critical examination.

Don




+1
"Here's a good specimen of my own wisdom. Something is so, except when it isn't so."

Charles Fort, commenting on the many contradictions of astronomy

Share this post


Link to post
Share on other sites
GeorgiaDon

If someone was to contact me and want all the raw data for a 10 year old study, and only give me a couple of weeks, it is not going to happen. In a pinch I might have the whole notebook copied and send it and tell them to figure it out. I cannot stop all my other responsibilities, tell the students in the courses I teach to take two weeks off, cancel every meeting, ignore every deadline, just to provide someone with a data point.



Is the research not done with reproducibility in mind from the very beginning?

I don't know how common it is, but I make an effort to provide sufficiently detailed documentation such that any analysis I perform can be exactly reproduced, given the raw data (which I also make available as appropriate) and similar hardware & software. (I avoid using GUI tools wherever possible, since programming and scripting languages allow easier documentation. It's a PITA to document scrolls and mouse clicks.)

I'm not criticizing or implying anything; I'm genuinely curious.
Math tutoring available. Only $6! per hour! First lesson: Factorials!

Share this post


Link to post
Share on other sites
Quote

I don't know how common istalloraphy persont is, but I make an effort to provide sufficiently detailed documentation such that any analysis I perform can be exactly reproduced,y cry given the raw data...

Fair enough question. I don't know what your field of research is, so it's hard to make comparisons. In my case, I may have one person who does flow cytometry, another who handles multiplex assays for cytokines and chemokines, someone else who handles mass spec, an X-crystallography person, a bioinformatics person, and so on; I usually handle cloning and expression and HPLC, and some bioinformatics. Everyone works on multiple projects simultaneously. So, the raw data is distributed across several people's lab books (generally multiple books/person), and each lab book records experiments for multiple projects. When we meet to work on a specific paper, people bring the experimental results that are relevant to that paper to the meeting. Bear in mind that not all these people are in my lab, some may be collaborators at other institutions, and the meetings are often done online. I have some collaborators I have never met in person, so I have no access to their notebooks. In the case of my students and technicians, I require that they keep a "table of contents" for each lab book so I can figure out where to look for (as an example) some specific gel or Western blot. So, it is possible for me to provide the raw data for any figure or table that was generated from my lab, but I would have to go through many lab notebooks copying a page here, three pages there, and so on. In the case of collaborators, people would have to contact them directly to get their raw data. I have a few collaborators who have retired or left science (for example, to go into medical practice), and I have no idea where to go to get their lab notebooks. I am not aware of any University policy to ensure lab notebooks are archived in perpetuity; as far as I know they are discarded when people leave (if they don't take them when they go) or die.

In many ways, it is easier to archive mass sequencing data such as RNAseq, as that data is already digitized so you just have to keep the raw sequence files and an accurate record of anything you did to analyze it. There is no physical "stuff" to keep track of, such as dried gels or Westerns. Any script that needs to be written can be reported as supplementary information. I am not a computer geek, so I do use some GUI-driven analysis, but current versions of some analysis packages such as Geneious do a good job of recording every step of the analysis so it is always possible to retrace your steps precisely.

Personally, I consider it most important to report the methods in enough detail that anyone (with the appropriate equipment) can repeat exactly what I did. Also I keep glycerol stocks of anything I clone/express, and provide those to anyone who asks, so they can re-sequence them or use them to express and bioassay recombinant protein and so verify my results.

Don
_____________________________________
Tolerance is the cost we must pay for our adventure in liberty. (Dworkin, 1996)
“Education is not filling a bucket, but lighting a fire.” (Yeats)

Share this post


Link to post
Share on other sites
GeorgiaDon

In many ways, it is easier to archive mass sequencing data such as RNAseq, as that data is already digitized so you just have to keep the raw sequence files and an accurate record of anything you did to analyze it. There is no physical "stuff" to keep track of, such as dried gels or Westerns. Any script that needs to be written can be reported as supplementary information. I am not a computer geek, so I do use some GUI-driven analysis, but current versions of some analysis packages such as Geneious do a good job of recording every step of the analysis so it is always possible to retrace your steps precisely.

Personally, I consider it most important to report the methods in enough detail that anyone (with the appropriate equipment) can repeat exactly what I did. Also I keep glycerol stocks of anything I clone/express, and provide those to anyone who asks, so they can re-sequence them or use them to express and bioassay recombinant protein and so verify my results.

Don



Thank you for the detailed explanation.

I'm used to working with data that are already in digital form. I forget that for some research, the raw data aren't digital. I've never had to develop a workflow for non-digital data. Clearly it would be more difficult than what I'm used to.

My tool of choice for data analysis is R, so reproducibility is very easy, assuming I have digital raw data. Since I strive to write neat, well commented code, it's usually just a matter of setting seeds (for random processes), saving the code (and data), perhaps integrating it into a Markdown document as appropriate.

A GUI-driven analysis tool that records steps sounds very useful. What is quite nightmarish is when someone cleans their data in Excel (or any other typical spreadsheet app) and takes no steps to document the process!
Math tutoring available. Only $6! per hour! First lesson: Factorials!

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

0