The pair are up for four awards and go up against each other in the favourite female pop/rock artist category.
Justin Bieber and One Direction are in the running for three awards each, while Kelly Clarkson and Katy Perry are among those up for two.
The AMAs take place on 18 November in Los Angeles.
Rihanna is also up for artist of the year, as well as favourite female soul/R&B and favourite soul/R&B album for Talk That Talk.
The star will be hoping to do better than at the recent US MTV Video Music Awards, where she only won in one out of five categories.
Nicki Minaj’s other nods at the AMAs are for favourite rap/hip hop artist and for favourite pop/rock album and favourite rap/hip-hop album, both for Pink Friday: Roman Reloaded.
Two British bands are nominated in the favourite rock/pop band, duo or group category, with One Direction going up against The Wanted.
At the MTV VMAs One Direction came out on top for the new artist award, with the video for What Makes You Beautiful beating Glad You Came.
The full list of nominations is:
Artist of the year: Justin Bieber, Drake, Maroon 5, Katy Perry and Rihanna
New artist of the year: J. Cole, fun., Gotye, Carly Rae Jepsen and One Direction
Favourite male artist – pop/rock: Justin Bieber, Flo Rida, Pitbull and Usher
Favourite female artist – pop/rock: Kelly Clarkson, Nicki Minaj, Katy Perry and Rihanna
Favourite band, duo or group – pop/rock: fun., Maroon 5, One Direction and The Wanted
Favourite album – pop/rock: Justin Bieber – Believe, Maroon 5 – Overexposed, Nicki Minaj – Pink Friday: Roman Reloaded and One Direction – Up All Night
Favourite artist – rap/hip-hop: Drake, Nicki Minaj and Tyga
Favourite album – rap/hip-hop: J. Cole – Cole World: The Sideline Story, Drake – Take Care and Nicki Minaj – Pink Friday: Roman Reloaded
Favourite male artist – soul/R&B: Chris Brown, Trey Songz and Usher
Favourite female artist – soul/R&B: Beyonce, Mary J. Blige and Rihanna
Favourite album – soul/R&B: Chris Brown – Fortune, Rihanna – Talk That Talk and Usher – Looking 4 Myself
Favourite artist – alternative rock: The Black Keys, Gotye and Linkin Park
Favourite artist – adult contemporary: Adele, Kelly Clarkson and Train
Favourite artist – electronic dance music (EDM): David Guetta, Calvin Harris and Skrillex