This is certainly a well-written and engaging thriller. My wife and I both read it and had trouble putting it down. But, I also had trouble deciding whether to consider it "mathematical fiction". To some extent, whether it is depends on how you answer the question "What is a mathematician?"

In several places, the book refers to Hoffmann as a physicist...and prior to founding his investment company he worked at CERN. But, his work at CERN was not really focused on elementary particle physics but rather on creating software to analyze the data generated by the physics experiments. So, arguably, he could be considered either a computer scientist or a data scientist. And, since he ends up being successful in finance, one could also call him a "quant" or even a "businessman". Any of those terms are valid descriptions of this character, and none would necessarily justify classifying him as a fictional mathematician.

However, I am going to consider him a mathematician with the following justifications:

• The book itself indicates that Hoffmann is a mathematician:

  (quoted from The Fear Index) [H]is head of section had considered him one of the most brilliant mathematicians on site.

  (quoted from The Fear Index) [Hugo] saw from the start that Hoffmann's mystique as a mathematics genius, like that of Jim Simons, would be an important part of selling the product.

  (quoted from The Fear Index) It was the self-absorbed mathematician -- his social artlessness, the strange innocence of him -- that she had fallen in love with; it was the new Alex, the billionaire hedge-fund president, she found difficult to take.

• When physicists stop answering questions about the fundamental laws which govern the dynamics of matter and energy and instead switch to another topic like finance, what they are really doing is using the math that they learned in their physics training and applying it to a different area of application. In that sense, they have become mathematicians rather than physicists.
• One of the first things we learn about Hoffmann is the four digit code he selected for the burglar alarm at his home: 1729, the smallest number expressible as a sum of two cubes in two different ways. This is a matter of pure math, not physics or finance, and it seems to be of great importance to Hoffmann.

• As you might guess, the narration repeats the oft-told tale about 1729 being the number of the cab which Hardy took to the hospital when he visited Ramanujan.
• It is clearly implied that mathematics is useful for making money on investments. For example, at one point it refers to "immensely intricate mathematical formulae" that are used to ensure that all of their risky bets are appropriately hedged. Furthermore, it says that Hoffmann only hires those with PhDs in either math or physics to work at his firm.
• More specifically, a formula for computing the VIX is described in words:

  (quoted from The Fear Index) If you want the math, it's calculated as the square root of the par variance swap rate for a thirty-day term, quoted as an annualized variance. If you don't wan the math, let's just say that what it does is show the implied volatility off the market for the coming month.

• Hoffmann is not particularly interested in either physics or finance. It is made clear that both of those things are merely sources of data that he can use to train his "AMR".
• Alex Hoffmann is a stereotypical mathematician character, displaying many of the traits of the anti-social mathematical genius as often portrayed in fiction. For example:

  (quoted from The Fear Index) It was ... an ordeal for him to attend his wife's first exhibition. From the moment he stepped out of the car and crossed the crowded pavement and entered the noisy gallery, he wished he could turn around and leave. People he suspected he had met before, friends of Gabrielle's, loomed up and spoke to him, but although he had a mind that could perform mental arithmetic to five decimal places, ha had no memory for faces. It was as if his personality had grown lopsided to compensate for his gifts. He heard what others were saying, the usual trite and pointless remarks, but somehow he didn't take them in. He was conscious of mumbling things in reply that were inappropriate or even downright odd.

  In addition, to being anti-social, Hoffmann has diagnosed mental health issues and was prescribed a variety of psychoactive pharmaceuticals at about the time he lost his job at CERN, thus justifying my tagging of this post with the Insanity motif.

• At one point when Hoffmann is late to a meeting with investors, his partner Hugo lies and says "He sends his apologies, he's working on a very complex theorem."
• The scene in which Hugo explains to Hoffmann what a hedge fund is was reminiscent of the bar scene in A Beautiful Mind. Here, Hugo points out a woman eyeing Hoffmann at a party (a woman we know will become his wife) and uses a bet about the color of her underwear along with elementary probability to illustrate his point.
• As already indicated above, Jim Simons (a former Stony Brook math professor who became quite rich on Wall Street and now funds a lot of research in math and physics) is mentioned, but it says a bit more including describing his company Renaissance Technologies as "the daddy of all algorithmic hedge funds".

Perhaps in 2011 the big plot twist of the book would come as a surprise, but I can say that in 2021 when I read the book I saw it coming from a mile away and so I don't feel that it is much of a spoiler. Still, just to be safe:

It turns out that the strange things going on in the book are not caused either by Hoffmann's own insanity or a person out to ruin him as we are led to suspect. Instead, the culprit is none other than the artificial intelligence "VIXAL-4" that Hoffmann created. This may be a bit of a cliche now, but it is still well-done here. My favorite thing about it is that unlike other thrillers in this same genre, this rogue AI does not simply pursue its own selfish desires but actually sticks to what Hoffmann had programmed it to do: capitalize on human fear to make huge profits in the stock market. That's certainly what VIXAL-4 does...it just uses methods that its creators did not imagine or condone, such as killing an employee in an elevator shaft.