Experimental vs Nonexperimental

Today we will look at the differences between experimental and nonexperimental methods and find out why scientists prefer to avoid nonexperimental methods where possible.

REMEMBER, ANOTHER WORD FOR NONEXPERIMENTAL IS NON-SCIENTIFIC

Some non-experimental approaches to psychology have led to errors and abuse

Insulin Shock Therapy


It was discovered accidentally, that by causing convulsions with an overdose of insulin, patients afflicted with psychosis, particularly schizophrenia, appeared to improve.

Scientific Reasoning for insulin shock

"My supposition was that some noxious agent weakened the resilience and the metabolism of the nerve cells ... a reduction in the energy spending of the cell, that is in invoking a minor or greater hibernation in it, by blocking the cell off with insulin will force it to conserve functional energy and store it to be available for the reinforcement of the cell."

Scientific Claims for insulin shock

1). This approach is a practical and workable physiological method to attack the debilitating effects of schizophrenia.
2). More than 70 % of patients improve after insulin shock therapy.
3). This is one the most important contributions ever to be made to psychiatry.

Scientific Research on insulin shock

The 1939 study, published in the American Psychiatric Association by John R. Ross and Benjamin Malzberg, among 1757 cases of schizophrenia (isn't that a lot for one hospital?) treated by insulin shock therapy, 11% had prompt and total recovery, 26.5% were greatly improved and 26% had some improvement.
Notice that this is not 70% improvement it is 63.5%

The 1942 study, carried out at the SAME Hospital, had an improvement rate of 63%, with 42% of the patients still well after two years of follow-up.
The rate is still not "more than 70%" and is surprisingly close to the 63.5% found in the first study.
They used a rating scale for "not improved, slightly improved, etc." Does this give you any insight into the "scientific" value of rating scales?


However, look what happened when the true scientific research occurred

Initial enthusiasm was followed by a decrease in the use of insulin coma therapy, after further controlled studies showed that real cure was not achieved and that improvements were many times temporary.
Since this method was still in use in many countries until recently.

Famous People who have undergone ECT


Ernest Hemingway (who shot himself after undergoing ECT treatment at the Mayo Clinic), poets Silvia Plath (who also committed suicide), Paul Robeson, rock star Lou Reed, film actresses Frances Farmer and Gene Tierney, pianists Vladimir Horowitz and Oscar Levant, and talk show host Dick Cavett.

The History of ECT Use

ECT stands practically alone among the medical/surgical interventions in that curing was not the goal, but instead the goal was controlling the patients for the benefits of the hospital staff

ECT TODAY

It is used for a variety of conditions.
Adults can have their treatments as in-patients, or they can come in for a monthly treatment as an out-patient and then go to work later in the day.

Children

Children still account for a small percentage of shock patients, and no national estimates exist. But at a seminar for shock therapy doctors in May, one-third of psychiatrists raised their hands when asked if they did shock on young people.
California and Texas ban shock therapy on kids under 12. Most states permit it with approval of two psychiatrists and a parent or guardian.

Non-scientific
CLAIM

"There's no evidence that electroconvulsive therapy affects brain development of children in any permanent way," says researcher Kathleen Logan, a Mayo Clinic psychiatrist.

Research

A 1947 study by psychiatrist Lauretta Bender reported on 98 children (ages 3-11) shocked at Bellevue Hospital in New York.
Bender claimed a 97% success rate: "They were better controlled, seemed better integrated and more mature."
In 1950, Bender shocked a 2-year-old who had "a distressing anxiety that frequently reached a state of panic." After 20 shocks, the boy had "moderate improvement."

In a 1954 follow-up, other researchers could not find improvement in Bender's children: "In a number of cases, parents have told the writers that the children were definitely worse."
In spite of this, today's researchers interpret Bender's study as evidence that shock works, at least temporarily.

New studies are again reporting great success. A UCLA study had 100% success in nine adolescents.
The Mayo Clinic found 65% were better.
At Sunnybrook Hospital in Toronto, 14 who received shock spent 56% less time in the hospital than six who refused the treatment.

In addition to the history of abuse(and continued abuse)


Non-experimental approaches often use techniques that are seem scientific but aren't

LOGIC

Non-experimental approaches often rely on "logic" and logic alone does not necessarily lead to truth.

Look at the following example.

Published in InfoWorld August 6, 2001
There is an opinion that the right brain is something of a parallel processor, integrating information far faster but just as reliably as the left brain's linear approach to thinking.
Tain't so. Want proof (or at least strong evidence)? Here's a quick experiment: Consider a chess game between equally matched opponents. Give one player five seconds to make each move -- plenty of time to use his or her judgment but not much time for logical analysis. Give the other player five minutes per move, so that he or she has time to think. Assuming the players are evenly matched, is there any doubt who will win?
This is strong evidence or proof that the right brain is the sole operator during the 5 seconds per move and the left brain is incapacitated and contributes nothing.

EXPERTISE or AUTHORITY


Non-experimental approaches often rely on experts or authorities.

Are experts or authorities objective presenters of scientific truth or do they perhaps have idiosyncrasies or biases in their observations?

Look at the following example.

Correlation Between Bipolar Disorder and Reactive Attachment Disorder by John F. Alston, M.D.
Historically, mental health professionals have long associated Attention Deficit Disorder with Reactive Attachment Disorder. Experts have put this correlation of ADD and RAD at between 40% and 70%.

In my experience as psychiatric consultant to the Attachment Center since 1977, as well as within my own private practice and consultations with other attachment programs and adoption agencies in which I supervise psychotherapists who work with attachment disorders, I have come to realize that correlations between Bipolar Disorder and Reactive Attachment Disorder are indeed much more common than believed and the correlation between ADD and RAD is much less common than believed.
Additionally, In my experience, this miscorrelation between ADD and RAD is international. This conclusion has led to different, and in my experience, much more effective medical treatment plans for these children.


Popularity

Non-experimental approaches often rely on what people accept as truth (popular truisms such as "absence makes the heart grow fonder" -- what about "out of sight out of mind?)
In recent years, a popular topic in psychology has been about "educating the right hemisphere".
Look at the following example.

Brain hemisphericity and academic majors: a correlation study by Amany Saleh

METHOD


The Hemispheric Mode Indicator has 32 items; each item consists of a continuum between two adjectives, such as "neat" and "sloppy." The subject chooses an adjective and rates it as "a lot" or "somewhat".

RESULTS


1. There was a significant difference between arts and literature majors and business majors.
2. There was a significant difference between education majors and business/commerce students.
3. There was a significant difference between nursing, communication, and law students, and engineering/science, business and commerce students.

DISCUSSION

The results appear to confirm that students choose to study subjects that accommodate their cognitive/learning styles.

The knowledge from this study is vital to educators since the literature on learning styles in the last three decades shows that teachers cater to only one learning style that correlates only with left hemispheric skills.

Personal Experience & Testimonials


Non-experimental approaches often rely on Personal Experience and testimonials.

BUT do we always interpret experiences correctly?

Look at the following example.

This is the story of a man who had cancer and was told by his doctor that the two months of radiation therapy he had been getting had done no good and that he had two months to live.
Nine months later, the man is "alive and well" because a spiritual healer treated his cancer with Therapeutic Touch (TT).
Here is how TT works.

The healer directs his energy to the patient's body without touching it. In this case, the healer would place his hands over the man's chest and move them in a slow motion around his lungs for about 30 minutes.

What are the alternatives to the TT explanation?

1) The doctor was wrong in his prediction of how long his patient had to live.
2) Maybe the man misunderstood the doctor, or the doctor may have qualified his prediction, such as, "my best guess is..." or "based on similar cases, I would estimate...."
3) It is possible that the doctor was wrong about the effectiveness of the radiation therapy and that it had worked better than he or she thought.
4) It is possible the cancer went into spontaneous remission.
5) It is possible the man was misdiagnosed and mistreated and he's alive only because his doctors gave him up for dead.

All we know for sure is that the man lived longer than a doctor predicted and that both radiation therapy and TT were administered.

This is where nonexperimental and experimental methods clash.


Misusing data

Non-experimental approaches often present numbers and give only one point of view as to how the data can be interpreted.

Look at the following example of interpreting data.

A careful analysis of traffic accidents showed that about 70% of car accidents happen when the driver is within 10 miles of home. This was interpreted to mean that people are careless when they are close to home (familiarity breeds complacency).

While the statistic is true, the interpretation is not. The reason why most people have accidents close to home, is that most of the time, they are close to home.


Personal Investment in the issue affects interpretation of the data

Non-experimental approaches often are biased because people expect their point of view to be right.

Look at the following example of two different points of view seeing the same data differently.

At the University of California, women sued the university claiming they were being discriminated against by the graduate school.
In countering this suit, administrators showed that there was actually a positive bias for women.

It turns out that BOTH the women and the administrators were right (depending on how you look at the data).

Suppose there were only two departments in the graduate school, economics and psychology.
70 of 100 men (70 percent) who applied to the economics department were admitted
15 of 20 women (75 percent) who applied to the economics department were admitted
5 out of 20 men (25 percent) who applied to the psychology department were admitted
35 of 100 women (35 percent) who applied to the psychology department were admitted
(a)The administration noted that a higher percentage of women were admitted (75 to 70 and 35 to 25)
(a)However, overall, 75 of the 120 male applicants (62.5 percent) were admitted whereas 50 of the 120 female applicants (41.7 percent) were admitted.


For nonexperimentalists, data can be close enough

In nonexperimental psychology, it tends not to matter what the standards are for saying that some outcome is meaningful.
If you are "close enough" or "in the ballpark" in nonexperimental psychology, then your outcome is considered significant.

For example, one researcher hypothesized that men's financial stability would be preferred more by women than women's financial stability would be preferred by men. This was believed to be so because previous research had shown this relationship.
However, when the data didn't come out supporting the hypothesis, the researcher decided that it actually did and reported ñ "though my hypothesis was not fully supported it was marginally significant."

"Marginal" or "nearly significant" statistical outcomes (which up to now have been called "non-significant") are a big controversy in psychology among the experimentalists versus the nonexperimentalists.
One faculty member at RIT has taught the idea "near significance", in classes, suggesting that we should be using the term "marginally significant" because it advances science.
The American Psychological Association (APA) disagrees, but what do they know?


Nonexperimental methods often say that one good result is enough

Non-experimental methods often say that a single study is evidence of truth.
Experimental methods expect replication.

Look at this example.

A computer generated a randomized 100-item list of the digits 1, 2, 3, 4, and 5. 179 student participated. Each student received a list in a sealed envelope and had to guess which number was in each of the positions.
By chance, 20 guesses out of the 100, should be correct.
When told beforehand that ESP was beneficial, Ss averaged 20.66 correct out of 100. When told that ESP was harmful, they averaged only 19.49 correct.
This was an experimental method, but no journal would publish this study without replication.
Similar nonexperimental studies HAVE been published.

Therefore the researchers repeated their experiment.
The result of the replication showed NO statistically significant effects were present.
Therefore, the misinformation was not published as indicating a truth about ESP.
This is the value of using experimental vs nonexperimental methods.


Much nonexperimental research is dependent on surveys

Surveys are notoriously non-scientific but are publicized all the time as if they are scientific.
Sometimes they are even called "Scientific Surveys".
Let's look at a couple problems with surveys.

Surveys are among the most inaccurate and poorly constructed research tools in use and are unlikely to get any worthwhile data just because of the way they are constructed.

Even given a perfectly constructed survey, there is no reason to believe any data obtained from surveys because people lie on them.
One reason they lie on surveys is because they are encouraged to lie on surveys.
Wired Magazine, DC Newspress, and other sources are pointing out that surveys are:
getting not only demographic data from you but also information on personal habits, tastes and opinions. This information is stored, processed, analyzed and traded.
Their advice is, if you are not already lying on surveys, please start.

Even without being encouraged to intentionally lie, many people are liars to begin with. A Christian survey found the following results.
How honest would you say you are:
1% lie constantly
4% mostly lie
27% lie a lot
57% lie sometimes
11% never lie
By these numbers shouldn't many of them have been lying on this Christian lie survey?

Another survey found the following:
If you could live in a world where everyone told the truth at all time, would you want to
65% said no.
These results probably aren't accurate either, so do we really know anything based on this result (other than we can't trust survey results).

Then there are lots of topics people just don't seem to be honest about.
Sex is one of those issues.
The evidence is that people just don't tell the truth about their sexual practices on surveys.

For example, in most surveys the average number of sex partners reported by heterosexual males is significantly greater than the average number reported by heterosexual females.

The number of men reporting impotence on surveys is typically less than 10%, while the number of men surveyed who say they want Viagra to treat impotence is much higher.

Additionally, there is evidence that people don't tell the truth even on surveys as innocuous as ones on midlife satisfaction.
This is what was found:
The responses to the MacArthur Foundation study on midlife satisfaction didn't compare with other response measures. For example, self-protective and jocular responses like "Oh, we're happy and everything's fine" were frequent, while more serious responses like -- "My job is often boring" were infrequent.

The evidence is that colleges don't tell the truth on surveys
In Money magazine's 1994 college guide, New College of the University of South Florida was ranked No. 1 overall. Listed among the school's strengths was the freshman class's average SAT score -- an impressive 1296. This placed New College among the most selective schools in the nation.

The score - as well as the pretense of exclusivity - was false.
For years, the Sarasota-based school concedes, it deliberately inflated its SAT scores by lopping off the bottom-scoring 6% of students, thereby lifting the average about 40 points. Admission Director David Anderson describes the practice, which he says he recently discontinued, as part of the college's "marketing strategy."

Even when people are trying to be honest on surveys, there is the problem that people don't really know themselves and tend to overestimate their worth and abilities.

Take a look at a couple of examples

Ninety-four percent of university professors think they are better at their jobs than their colleagues.
Twenty-five percent of college students believe they are in the top 1% in terms of their ability to get along with others.
Seventy percent of college students think they are above average in leadership ability. Only two percent think they are below average.

Additionally, surveys are often reported in ways that make them seem better than experimental data.

Consider this psychotherapy survey

A Consumer Reports nonexperimental survey (Consumer Reports. Mental health: Does therapy help? 1995, November, pp. 734-739) of 2900 readers who had received psychotherapy concluded that patients report substantial benefit from psychotherapy.

This is in spite of the fact that over 500 scientific studies show psychotherapists' credentials and experience are not related to patient improvement from counseling.

Experimental evidence tells us that psychologists and psychiatrists are no better than intelligent, minimally trained people at diagnosing mental problems or predicting which people will have problems
Psychologists and psychiatrists with 30 or 40 years of experience are no better at helping people than are intelligent, minimally trained lay people.
Self-help groups often do as well as professional psychologists, and many members of the clergy do good counseling with little or no training.
Friends do just as well as professional therapists in both the long and short run.


Generalizing from other research

Nonexperimental methods are notorious for taking evidence from other sources and building a case for their argument (whether or not that evidence legitimately fits their argument).
Take a look at the following study.
The Correlation Between Animal Abuse and Human Abuse by Katie Linnes. Mankato State University

Katie argues that -- We all know that serial killers don't wake up one morning and say, "Today I think I will kill someone." Something has to lead to their fascination of suffering and death.
She claims -- Studies show that there is a definite cycle of abuse that starts with animals and leads to humans. Here is her evidence:
The FBI has found that a history of cruelty to animals is one of the traits that regularly appears in its computer records of serial rapists and murderers (Goleman, 1991).
The future killer's childhood centration on violence will lead to an adulthood violence-focus" (Anderson, 1994). (correlation/causation)
One confessed he had killed so many cats he'd lost count (PETA, 1998).
"If an animal is being neglected in a house, there's real good proof that a child is probably being neglected too" (Fisher, 1997).
One in four women admit to staying in the abusive home because of their pets and 57 percent of the total women admitted that their partner had harmed or killed their pet (Rizk, 1997)

Katie Linnes CLAIMS that -- These gruesome examples obviously show a direct link between animal abuse and human abuse.
The cycle of abuse can potentially be prevented during childhood, as long as the signs are caught early. Schools, parents, communities, and courts who shrug off animal abuse as a "minor" crime are ignoring a time bomb (PETA, 1998). Instead, they should be aggressively punished and counseled.


Shark Cartilage Claims

Powdered shark cartilage has been touted as a cancer cure, especially by William Lane, Ph.D. Lane got his inspiration from the work of real scientists who injected bovine and shark cartilage into the bloodstreams of rabbits and mice with cancer. The stuff greatly inhibited angiogenesis, the growth of blood vessels which supply nutrients to the cancerous cells.

EXPERIMENTAL EVIDENCE SHOWS
However, not all cancers rely on angiogenesis.
Sharks do get cancer, even cancer of their cartilage.
Experimental evidence does not support the efficacy of shark cartilage in treating cancer.

Do you believe the nonexperimental evidence, the experimental evidence or both?

As you can hopefully see, there is a lot of garbage out there masquerading as scientific research.
You can get anything published if you just work hard enough at it.
Look at the following example.


Correlations
between soup preferences and personalityby Brian Wansink

RESULTS: someone who orders chicken-noodle soup is high on the church-going scale, fond of pets and more likely to be stubborn and less outdoorsy.
Minestrone's fans were more likely to be physically fit, nutritionally conscious, family-spirited, unlikely to own a pet and on a restricted diet.
Vegetable-soup eaters are a homebody at heart, less likely to be a world traveler or spontaneous and more likely to read family and home magazines.
Tomato-soup lovers are more along the lines of adventure lovers, are more social and tend to enjoy books and their pets.

How the study was done

In this study, Wansink correlated the soups people prefer with their personality traits.
In order to do this, Wansink went to the experts: experienced waitresses. "We needed someone familiar with people and soup, so we went to 32 different waitresses in the Midwest who had around an average of eight years experience."

The study used 27 waitresses from the Elite Diner and Merry Ann's Diner. The waitresses had strong opinions about what different types of soup-eaters were like.
Wansink then conducted a random telephone survey across the 50 states, assessing people's opinion of 12 common soup products. The survey included adults over 18 with 602 women and 401 men.
Wansink took the people with strong preferences and then went back to the waitresses, asking which profile went with which soup. He said the waitresses found the task easy.

He gives an example of Lisa Moreno, a waitress at the Elite Diner in Urbana.
She has eight years of waitressing experience and said she often has an idea of what people want when they come in.
When asked about a typical tomato-soup eater, Moreno said it will probably be a woman who is middle-aged or older


This study was published nationallyWhat do you think?

Your answer should be that just because something is published does not mean that it is accurate or meaningful.

WATCH OUT

As you have seen, many nonexperimentalists try to make their studies look and sound like experiments.
As you have heard, even some of the psychologists in our own department do this.
You have to learn not to be fooled.

COMPARING EXPERIMENTAL METHODS WITH NONEXPERIMENTAL METHODS

In the experimental method, psychologists manipulate variables and observe how subjects respond to the manipulation.
This manipulation of variables allows the establishment of cause and effect.

Nonexperimental methods cannot establish cause and effect.

In the experimental method, psychologists apply the manipulation to one group (the experimental group) while another group does not receive the manipulation (the control group). This allows the psychologist to know if the manipulation actually causes some effect.

The nonexperimental method typically examines relations between variables. Thus, groups are often left as they are and are merely observed.

In the experimental method, participants are randomly assigned to all groups. Thus all groups are considered equivalent.

The nonexperimental method uses already self-assigned groups, thus the groups cannot be considered to be equivalent. The outcome is that biases can make the results meaningless.

In the experimental method, all variables are controlled. This means that opportunity for biases and error are basically eliminated.

In the nonexperimental method, the relations between variables usually already exist, there is no manipulation or control of variables, thus biases and errors are not eliminated.


What all of this means

The experimental method is considered to be "scientific".

The nonexperimental method is considered to be "non-scientific".


Drawbacks of the Experimental Method

Although experimental methods are the most "rigorous" of all research designs, the "gold standard" against which all other designs are judged, there are few major limitations to be considered.

1) Experimental methods can only be used when it is practical and ethical for the researcher to manipulate the conditions.
For example, a psychologist might want to know if parent's method of disciplining their children has an effect on how their children behave.
It would be neither practical nor ethical to make parent's discipline their children in a certain way just to see if that effects their child's behavior.

2) Experimental studies are usually done in the highly controlled setting of the laboratory.
These conditions are artificial and may not reflect what really happens in the less controlled and infinitely more complex real world

3) Experimental methods can only be used when it is possible to manipulate variables.
For example, if you want to study deaf/hearing, male/female, psychologist/engineer, etc. differences, then you couldn't use the experimental method.
You can't randomly assign someone to be deaf or hearing or male or female or a psychologist or an engineer.

FINALLY: No research (including Experimental Research) proves anything.

A scientific proof is not known with absolute logical certainty.
A controlled study can never be completely controlled-there are just too many possible variables. Findings cannot be known in the sense of known beyond any logical or conceivable doubt.
The best that science can offer us is that we can distinguish between what is "conceivably" true and what is "reasonably" true.

DON'T USE THE WORD "PROVE" IN ANY OF YOUR PSYCHOLOGY WRITE-UPS.

Go over assignment


THE END