Background:  This article is licensed under the GNU Free Documentation License and Creative Commons Attributions-Share-Alike Unported. It was posted to Wikipedia by an author who wished to remain anonymous, known variously as "24" and "142".  It was subsequently removed from view on Wikipedia, but its text has been preserved by a number of mirrors.  While it could be seen as no more than a basic primer in moral philosophy, it is arguably required reading to anyone unfamiliar with the philosophical background of such concepts as Friendly AI and Coherent Extrapolated Volition.

The search for a formal method for evaluating and quantifying ethicality and morality of human actions stretches back to ancient times. While any simple view of right, wrong and dispute resolution relies on some linguistic and cultural norms, a 'formal' method presumably cannot, and must rely instead on knowledge of more basic human nature, and symbolic methods that allow for only very simple evidence.
By contrast, modern systems of criminal justice and civil law evaluate and quantify social and moral norms (usually as a fine or sentence or ruling on damages) rely usually on adversarial process and forensic method, combined using some quasi-empirical methods and many outright appeal to authority and ad hominem arguments. These would all be unacceptable in a formal method based on something more resembling axiomatic proof, which by definion relies on some axioms of morality.

Religious moral codes provide such axioms in most societies, and to some degree, following those strictly could be considered formal in that no more trusted or respected method existed. But our modern concept of what is formal and thus universally trustworthy and transparent is derived from that of the ancient Greeks:

Pythagoras and Plato sought to combine moral and mathematical elements of reality in their work on ontology. This was very influential and the work of both is still consulted to this day, although, the social and political implications of their methods are often rejected by more modern philosophers.

Thomas Aquinas, Francis Bacon and some of the Asharite philosophers shared a belief in some kind of over-arching ethical reality provided by a deity. But while Aquinas and Bacon integrated this with methods of Aristotle and ultimately inspired Jesuit and other Catholic methods of assessing and dispensing justice, resulting in Catholic canon law and other forms of Christian church law, the Asharite influence on Islam rejected parallel Mutazilite work on Aristotle, and eventually resulted in the "classical fiqh" and the shariah now being revived in some parts of the Islamic World. Thus it could reasonably be said that Catholic and Islamic thought diverged on Aristotle's ideas in the middle ages.

Some consider the debate to continue to this day in economics, with the neoclassical economics based firmly on Aristotle's methods via Friedrich Hayek and Karl Popper, against Islamic economics and feminist economics which reject some aspects of Aristotle's logic, e.g. law of excluded middle, and seek to build on some intuitive and morally defensible ontology, as Plato did. This is probably no less of a controversy today than it was in Plato's time, or among the Asharites:

Today, few accept that economics is a means to any ethical or moral end, but more of a technology that serves the ends of those who control and refine it. It remains however that economics does "evaluate and quantify" relationships of such importance, e.g. food, labour, that most humans literally cannot live without an economy around them. Thus an economy embodies assumptions about ethics and morality, and Karl Marx thought that this was itself proof that capitalist economics had subsumed the role of the old feudal methods. This view is current to this day in Marxist economics.

However, the longest-lived view of formal methods as applied to morality comes not from Western but Eastern traditions. Confucianism with its stress on honesty and transparency and etiquette, and moral example of rulers and elders, has at times been seen as a formal method among the Chinese, its "axioms" often respected as much or more than any from science.

Buddhism also stresses notions of right livelihood which seem to be possible to measure and compare in a quasi-formal manner. The Noble Eightfold Path is a set of priorities, ordinal not cardinal, not strictly quantities, but still, a useful framework for any more formal or weighted value theory.

During The Enlightenment the various traditions became more unified:

Immanuel Kant, in his "categorical imperative", sought to define moral duty reflectively, in that everyone was obligated to anticipate and limit the impacts of one's own actions, and "not act as one would not have everyone act.". This can be seen as a restated Golden Rule. In the 20th century it was restated as the ecological footprint, a measure of one's use of the Earth's natural capital, which later became a keystone of green economics.

Other related practices are means of measuring well-being and assessing the implied value of life of various professional ethical codes and infrastructure decisions. While these systems rely on empirical methods for gathering data, and are more interested in "is" than "should", they are at least "transparent" and "repeatable" in a sense that could be called "pre-formal" or "pre-requisite to formal". Some think that they verge at times on the reliability of the quasi-empirical methods in mathematics, in that no conceivable disproof seems possible, but evidence "for" is not disputed - an example being the observation of Marilyn Waring that actions which prepare for war have measurably higher economic values than those within family.

A formal method could reconcile many points of view by excluding forensic or audit methods which passed morally-undesirable outcomes, e.g. war or genocide, or worse which valued them highly. It could not validate any one view a "true" but it could find a "best" or "best next step" for some given time horizon or limited list of models or choices to evaluate. Most proposals for moral purchasing employ some such process. Given a very large number of socially-shared semi-formal economically-committed methods, one might take a mean or other stochastic measure of ethical and moral acceptability to those participating, and thus produce very nearly a species-wide informal method that would have as much reliability as one could expect from any "formal" method. Such are the goals of some NGOs in civil society and peace movement and labour movement and anti-globalization movement circles.

An alternative but less popular view is that "human nature" can be so well understood and modelled mathematically that it becomes possible to assess with formal and mathematical methods, the cognitive bias or moral instinct, e.g. altruism of humans in general, perhaps with measurable variations due to genetics. This view has been popular since the emergence of the theory of evolution, and E. O. Wilson and George Lakoff are among those who have asserted a strong "biological basis" for "morality" and "cognitive science of mathematics" respectively.

Some combination of these views may effective at posing a starting point for models of moral cores and instincts and aesthetics in human beings. However few see them as routes to new moral codes that would be more reliable than the traditional religious ones. A notable exception is B. F. Skinner who proposed exactly such replacement in his "Walden Two", a sort of behaviorist utopia which had many characteristics in common with modern eco-anarchism and eco-villages. Most advocates of such co-housing and extended family living situations, e.g. Daniel Quinn or William Thomas, consider informal, political, "tribal" methods sufficient or more desirable than those involving any kind of "proof".

If so, the long search for a formal method to evaluate and quantify ethical outcomes, even in economics, may come to be seen as a sort of mathematical fetishism, or scientism, or even commodity fetishism to the degree it requires the reduction of quality of life to a series of simple quantities.