The "notoriously powerful" Callistus was a freedman who rose to a position of great influence under Caligula and Claudius, Pliny records having seen for himself thirty large onyx columns in his dining room (Natural History, XXXVI.xii.60). It was indeed a change of fortune for a slave once sold by his master as being of no use. Seneca relates the story.
"I have seen standing in the line, before the door of Callistus, the former master of Callistus; I have seen the master himself shut out while others were welcomed,—the master who once fastened the 'For Sale' ticket on Callistus and put him in the market along with the good-for-nothing slaves. But he has been paid off by that slave who was shuffled into the first lot of those on whom the crier practises his lungs; the slave, too, in his turn has cut his name from the list and in his turn has adjudged him unfit to enter his house. The master sold Callistus, but how much has Callistus made his master pay for!" (Epistles, XLVII.9-10).
Josephus speaks of Callistus, as well, in the Antiquities of the Jews as among those who conspired to assassinate Caligula. "And besides these, Callistus also, who was a freed-man of Caius, and was the only man that had arrived at the greatest degree of power under him,—such a power, indeed, as was in a manner equal to the power of the tyrant himself, by the dread that all men had of him, and by the great riches he had acquired; for he took bribes most plenteously, and committed injuries without bounds, and was more extravagant in the use of his power in unjust proceedings than any other" (XIX.1.10ff). He goes on to relate how, fearful of losing his wealth, Callistus ingratiated himself with Claudius, although not so much that he felt obliged to inform the emperor of Messalina's infidelities, "remembering that prudent rather than vigorous counsels insure the maintenance of power" (Tacitus, Annals, XI.29.1ff; also Dio, Roman History, LIX.29.1).
In the Life of Galba, Plutarch indicates that Caligula "while still a young man, had been intimate with the mother of Nymphidius, a woman of comely appearance and a daughter of Callistus, Caesar's freedman, by a hired sempstress. But this intimacy, as it would seem, was later than the birth of Nymphidius, and it was believed that he was a son of Martianus, the gladiator (with whom Nymphidia fell in love on account of his fame), and his resemblance to Martianus was thought to favour this connection" (IX.1-2).