If you are lucky enough to find an unabridged version of
Swift's classic book, you will be able to read (among descriptions of
the people of many other unbelievable countries) about the people of
Laputa. In Laputa, people are interested only in two things:
mathematics and music. Their love of these things goes so far that,
for instance, all food is served either in the shape of a mathematical
figure or the shape of a musical instrument. I'm afraid I can't say I
appreciate some of Swift's stereotypes of mathematicians! I do not
mind so much that mathematicians are said to be interested in
politics:
(quoted from Gulliver's Travels)
But what I chiefly admired, and thought altogether unaccountable, was the strong disposition I observed in them towards news and politics,
perpetually enquiring into public affairs, giving their judgments in matters of state, and passionately disputing every inch of a party opinion. I have
indeed observed the same disposition among most of the mathematicians I have known in Europe, although I could never discover the least
analogy between the two sciences; unless those people suppose, that because the smallest circle hath as many degrees as the largest, therefore the
regulation and management of the world require no more abilities than the handling and turning of a globe.

but I would hate to think that his description of their clumsiness is
true of mathematicians today:
(quoted from Gulliver's Travels)
Their houses are very ill built, the walls bevel without one right angle in any apartment, and this defect arises from the contempt they bear to
practical geometry, which they despise as vulgar and mechanic, those instructions they give being too refined for the intellectuals of their workmen,
which occasions perpetual mistakes. And although they are dexterous enough upon a piece of paper in the management of the rule, the pencil, and
the divider, yet in the common actions and behavior of life, I have not seen a more clumsy, awkward, and unhandy people, nor so slow and
perplexed in their conceptions upon all other subjects, except those of mathematics and music. They are very bad reasoners, and vehemently given
to opposition, unless when they happen to be of the right opinion, which is seldom their case. Imagination, fancy, and invention, they are wholly
strangers to, nor have any words in their language by which those ideas can be expressed; the whole compass of their thoughts and mind being shut
up within the two forementioned sciences.

Presumably, the portrayal of mathematicians as being politically powerful was influenced by Isaac Newton's political connections, as David Fowler mentions (or quote Whitehead mentioning):
Contributed by
David Fowler in Mathematics as Science Fiction
Rarely are mathematicians described as foolish, although Swift’s Island of Laputa persists as an image of mathematicians as fools. As Alfred Whitehead pointed out, “Swift describes the mathematicians of that country as silly and useless dreamers. . . . On the other hand, the mathematicians of Laputa . . . ruled the country and maintained their ascendancy over their subjects.” Whitehead also remarked that Newton had just published his Principia and suggested that “Swift might just as well have laughed at an earthquake.”

Note: A very mathematical "sequel" was published in 1947 called Gulliver's Posthumous Travels to Riemann's Land and Lobachevskia.
