The main math element in Infinite Jest is the Eschaton chapter/endnote. In particular, these contain a statement of the Mean Value Theorem with a few accompanying basic-calculus type diagrams, and then some gibberish about computing integrals from only the extrema of the function in question. There is some question as to whether the latter mistake is intentional. I tend to think that it is, for three reasons: the author's knowledge of mathematics and perfectionist style, the unreliable-narrator aspect of the passage in question, and the consistency of such a mistake with the themes of the work.
To the first reason, the author has written other books with much stronger, more advanced mathematical content, to apparently positive reception. Attention to detail - to the point of pedantry - is both a feature and explicit theme of Infinite Jest in general (the Militant Grammarians of Massachussetts, for example) and in this particular passage (which is littered with a comical overuse of [sic]'s). Given all of that, such an elementary error in a central point of the endnote seems incongruous.
To the second reason, the endnote in question is presented as a transcript produced by one character (Hal), of an explanatory dictation given by another (Pemulis) for inclusion in a reference text for younger tennis students who play a nuclear geostrategy game called Eschaton (a mix of Risk and tennis lobs, with a lot of gratuitous computer statistics). So, this is not the omniscient narrator voice presenting the mistake. Moreover, if one reads carefully, it is pretty clearly implied that the characters are smoking marijuana during the transcribed (from Hal's memory) exchange. Note that this is far from the only instance of an unreliable narrator in Infinite Jest; see also the conflicting accounts given on the question of Joelle Van Dyne's disfigurement (or not).
Lastly, the reading that Pemulis and/or Hal are too stoned to notice that they're botching the math seems to fit with many of the book's and chapter's themes. For example: that both characters are suffering more psychological fall-out from their recreational drug use than they are willing to recognize, or that the tennis academy's academic pretensions are perhaps just that. Note that the Eschaton game the younger students play later in the chapter devolves into a brawl which culminates with a student's head being smashed through the statistics computer in question, all while Hal and friends watch ineffectually (stoned again, of course). To take the contrary reading, that the author didn't realize his mistake and that the math presented is intended to be "correct," seems to make the passage in question a waste of time.