In a wonderful book “Stalking the Black Swan”, the author, Kenneth A. Posner, writes on the practice and difficulties of exercising judgment.

“Under the textbook approach, judgment is considered a process of weighing the evidence before coming to a decision. Experts often cite the advice of Benjamin Franklin, who came up with a simple method for weighing the pros and cons over 200 years ago:

My way is to divide half a sheet of paper by a line into two columns; writing over the one Pro, and the other Con. Then, during three or four days of consideration, I put down under the different heads short hints of the different motives, that at different times occur to me, for or against the measure.

When I have thus got them all together in one view, I endeavor to estimate their respective weights; and where I find two, one on each side, that seem equal, I strike them both out. If I find a reason pro equal to some two reasons con, I strike out the three . . . and thus proceeding I find at length where the balance lies. . . .

And, though the weight of reasons cannot be taken with the precision of algebraic quantities, yet when each is thus considered, separately and comparatively, and the whole lies before me, I think I can judge better, and am less liable to make a rash step, and in fact I have found great advantage from this kind of equation. . . .

Franklin's approach has many merits-for example, reducing the load on working memory, as most people have trouble focusing on more than a handful of variables at once. Some experts would replace the pros and cons with numerical quantities and use a linear model (like a spreadsheet) to add them up. They argue that using linear models leads to a significant improvement in accuracy over intuitive judgments. Models are not only more precise in their calculations, but they eliminate emotion, bias, and other forms of human error.

But there is a major problem with any process of weighing or adding up multiple factors, regardless of whether the process is conducted intuitively, more deliberately (with pen and paper, as Franklin suggested), or using high-powered computer models. I have pointed out different aspects of this problem throughout this book. To summarize:

Ø       The analyst must zero in on critical issues, or be forced to weigh an infinite number of factors.

Ø       In practice, we may assign weights to variables on the basis of confidence as an emotional signal, which does not always reflect the accuracy of the analysis.

Ø       We need to be careful not to overweight the critical issues that we study most carefully, ignoring or underweighting other variables in the environment, a sign that we may have fallen into the representativeness trap.

Ø       In an uncertain world, we may need to consider multiple outcomes for critical issues so that we recognize the full range of the risks. We cannot add up the variables properly if we miss asymmetries (recall Jensen's inequality). That is why the weighing-up process must often be conducted using probability trees.

Ø       The process of adding up can become intractably complex because of the combinatorial explosion of interactions between variables. For this reason, analysts must map out the problem, seeking to balance gains in accuracy against the costs of complexity.

Ø       Analysts cannot add up variables properly without grasping the correlations among them, which requires a sound understanding of the causative drivers.

Ø       Finally, all these weighing processes are essentially algorithmic in nature. Algorithms cannot solve all problems, because for some, calculations will never stop-this is the “halting problem” identified by Alan Turing in the 1930s

For this reason, some kind of oversight, which we are calling judgment, is necessary to assess whether an analytic process is producing useful results, or whether some other process should be used instead.

Sometimes judgment is seen in the imposition of a seemingly arbitrary constraint on the output of a sophisticated computer model. For example, portfolio managers often use statistical models to search for the "efficient frontier," where they hope to maximize risk-adjusted returns by optimally allocating capital across a portfolio of investments. These kinds of optimization models have a tendency (no doubt mathematically correct) to concentrate portfolios in a small number of assets with high expected returns. But portfolios managers are rarely comfortable with such concentrated allocations. More often they spread their bets, even at the cost of earning suboptimal returns. In doing so, they are acknowledging the presence of what Martin Leibowitz calls "dragon risk," a reference to the practice of medieval mapmakers, who labeled uncharted territories as places where dragons might dwell. Leibowitz uses the term dragon risk to refer to the possibility that the statistical model might be invalid, or that if many managers allocate assets similarly, the returns might become correlated with the broader market and collapse in a downturn.

Judgment may also be recognized in a give-and-take debate, where decision makers weigh computer models against other factors”.