The social sciences, like much of biology but unlike most fields of the physical sciences, have to deal with structures of essential complexity, i.e. with structures whose characteristic properties can be exhibited only by models made up of relatively large numbers of variables.
…This corresponds to what I have called earlier the mere pattern predictions to which we are increasingly confined as we penetrate from the realm in which relatively simple laws prevail into the range of phenomena where organized complexity rules. As we advance we find more and more frequently that we can in fact ascertain only some but not all the particular circumstances which determine the outcome of a given process; and in consequence we are able to predict only some but not all the properties of the result we have to expect. Often all that we shall be able to predict will be some abstract characteristic of the pattern that will appear – relations between kinds of elements about which individually we know very little. Yet, as I am anxious to repeat, we will still achieve predictions which can be falsified and which therefore are of empirical significance.
Of course, compared with the precise predictions we have learnt to expect in the physical sciences, this sort of mere pattern predictions is a second best with which one does not like to have to be content. Yet the danger of which I want to warn is precisely the belief that in order to have a claim to be accepted as scientific it is necessary to achieve more. This way lies charlatanism and worse. To act on the belief that we possess the knowledge and the power which enable us to shape the processes of society entirely to our liking, knowledge which in fact we do not possess, is likely to make us do much harm. In the physical sciences there may be little objection to trying to do the impossible; one might even feel that one ought not to discourage the over-confident because their experiments may after all produce some new insights. But in the social field the erroneous belief that the exercise of some power would have beneficial consequences is likely to lead to a new power to coerce other men being conferred on some authority. Even if such power is not in itself bad, its exercise is likely to impede the functioning of those spontaneous ordering forces by which, without understanding them, man is in fact so largely assisted in the pursuit of his aims. We are only beginning to understand on how subtle a communication system the functioning of an advanced industrial society is based – a communications system which we call the market and which turns out to be a more efficient mechanism for digesting dispersed information than any that man has deliberately designed.
If man is not to do more harm than good in his efforts to improve the social order, he will have to learn that in this, as in all other fields where essential complexity of an organized kind prevails, he cannot acquire the full knowledge which would make mastery of the events possible. He will therefore have to use what knowledge he can achieve, not to shape the results as the craftsman shapes his handiwork, but rather to cultivate a growth by providing the appropriate environment, in the manner in which the gardener does this for his plants. There is danger in the exuberant feeling of ever growing power which the advance of the physical sciences has engendered and which tempts man to try, “dizzy with success”, to use a characteristic phrase of early communism, to subject not only our natural but also our human environment to the control of a human will. The recognition of the insuperable limits to his knowledge ought indeed to teach the student of society a lesson of humility which should guard him against becoming an accomplice in men’s fatal striving to control society – a striving which makes him not only a tyrant over his fellows, but which may well make him the destroyer of a civilization which no brain has designed but which has grown from the free efforts of millions of individuals.
— Friedrich August von Hayek in his 1974 Nobel Prize Lecture: The Pretence of Knowledge