Briefly, anthropology is... oh, heck Webster's 1913 wins:
"The science of man, including the study of the ditribution
of physical and cultural attributes in relation to man's
origin, location, history, and environment;..."
That should include "culture". In other words, _why_ we are the way we are, and why we do the things we do, from an evolutionary standpoint. The things we do, and the things we tend to think and the way we tend to think about were selected for through bleah bleah years of evolution. They're not always spot on, but there is almost always a reason for them. Understanding why we do things helps us evaluate why it might or might not be appropriate.
Okay, back on topic. Why so much project failure? Because shooting too high and coming up short is usually a good strategy. Trying to hunt too large of game and failing (non-spectacularly) is a win over hunting small game. You're better off bringing back a bore once a week and failing 6 out of 7 times than bringing back a rabbit every night. If we're not slightly out of our comfort zone, we're not comfortable -- we feel like we should be trying harder. Also, we normally exist in a skill and strength realm, where we get stronger, and we develop techniques for accomplishing again what we accomplished once. In short, optimism is good, just as in sports.
That doesn't map well to something with hard deadlines, composed of thousands or millions of subtasks which have cascading failures, where skills are very nearly already maxed out and improvement at the high level programmers are hired at is very gradual.
Methodologies check rampant optimism and force thinking of projects in terms of the thousands of subtasks. Some do that better than others, but beyond that, I don't think any are silver bullets.
Just as it's generally advantageous to be optimistic (unless we have specific knowledge forbidding optimism), it's generally advantageous to be _un_-happy.
Times when it's advantageous to be happy: when you're trying to attract a mate, so he or she doesn't know how so very much you are the center of your universe and are displeased when all do not bow before you. When you're on the bottom of a social structure and your best hope is to be able to do your job without being given any additional misery. Baristas in chic coffee shops and ski instructors teaching ski bunnies to ski benefit from being happy.
Most of the time, being unhappy sends the message of "I want more". Like a baby bird screeching for a worm, that's our social cue that we want more. You don't succeed in bushing being a pushover. Most of these people aren't happy. The music industry and anything near CA is famous for this -- you get people who pretend to be spectacularly happy and chummy but are actually miserable, sad humans. That's just turning the "I like you, and I want more" dial up... er, all the way. The most successful humans are the ones who most brazenly and creatively redefine "fair" in their favor.
Going way off topic here... in most cultures, obesity is a stigma. I wonder if we haven't adapted to hate the obese as eating more than their fair share, and shun them as such. But in some cultures, obesity is considered extremely attractive -- tribal leaders, who are generally very generous as the small time dictators they are. Similarly, intelligence is often shunned, except by the intelligent. Intelligent humans are sometimes benevolent but most often are just willey, better at weaseling out more than their fair share. And for most of human evolution, any thing one person figured out, every else could easily be taught -- making arrow heads, building pits and cages for boars, etc. Just as being fat, being intelligent means that you're probably taking more than your fair share.
Benevolence is always loved except by creatures too cynical, cheesed off by false benevolence, to accept it. Abuse victims fit that category...
Wow, I go way off topic when I go off topic...
Okay, tying that all up...: Projects fail because humans are over ambitious, never happy, and hate intelligent people (who stereotypically work on the projects) anyway.
-scott
