This phrase has been a mainstay of popular American culture for well over 100 years. Exactly where it came from nobody knows, however it’s been traced as far back as this satyrical ad published in The Knickerbocker in 1849:
For sale, a Poll Parrot, cheap. He says a remarkable variety of words and phrases, cries ‘Fire! fire!’ and ‘You rascal!’ and ‘Polly want a cracker,’ and would not be parted with, but having been brought up with a sea-captain he is profane and swears too much for the subscriber, being a pious man and having children in the family, to whom his example is bad. On this account he will be sold at a bargain, but hoped that the purchaser will use his endeavors to break him of the habit, which may be done by depriving him of his coffee and not scratching his hed for him when he puts it through the bars.
Notable examples of the use of “Polly want a cracker?” include this 1937 Warner Brothers cartoon and the lyrics of the song Polly on Nirvana’s Nevermind. And while the exact origin is a mystery, knowing what you now know about crackers, it’s probably easy to imagine how the stereotype of a cracker-munching sea captain with a parrot on his shoulder got started. “Polly”, it seems, was as popular a name for a parrot in the 19th Century as “Spot” was for a dog in the 20th.