I don't know how they do it in the film industry, but in the music industry there's a couple of different types of royalty checks.
First, there's the mechanical royalties to the artist. Say nortabihpmA is signed to a label and releases his first CD. His manager/attorney negotiated a royalty rate for him, and once he pays up on his recoupable expenses (the money fronted by the label), he'll get that royalty rate for each copy of the cd. Supposedly. That's the way it's supposed to work, anyway.
Second, there's performance rights royalties. When nortabihpmA's song is on the radio he's supposed to get paid for that. ASCAP and BMI go around to the radio stations in the country and have them write down the songs they play, and then use funky algorithms to determine how many times the song's actually been played and how many people have heard it (in reality, they audit a station for a week and extrapolate how many stations of that size are like it, and how many listeners they serve, and a whole lot of other crap). Then nortabihpmA gets a check for the supposed number of radio plays for that quarter.
My guess is that the headliners of the movie industry accept a certain upfront salary, plus a percentage on the back end....a percentage of the "profits" of a flick...plus a percentage of additional ancillary income from tv broadcasts and whatnot. One would assume they'd negotiate a percentage of dvd sales, too.
I can't imagine anyone other than the main players getting offered anything beyond an upfront salary, unless there's a formula mandated by their union.