I kinda get the feeling that McKellen won't be returning, but it's time for a new villain anyway. Patrick Stewart knows all about sticking with a sci-fi franchise until there's nothing but dust left in the gas tank, so I imagine he'd be on board for future films.
Berry, I think, would be a problem. She needs the franchise more than it needs her, but I reckon she'd be asking for a huge payday to keep coming back. Grammer is possible - depends how he liked the make-up process, I guess. As for Jackman, maybe for a goodwill cameo, but not as a lead. This is why it sucks that they dumped Cyclops and lost Nightcrawler. Maybe with Singer not involved, Cumming might return (though I understand he hated the make-up as well) but you can always recast.
As for the kids, I think an X-Men Academy approach is the most likely way forward, but none of the characters left have the presence or backstory to support an entire film. Even in an ensemble, you need a lead - and the closest they've got is Iceman. Fun though his arc has been, I don't see it being enough to promote him to Jackman's position in the franchise.
The big question is, why would anyone think that a fourth movie would be an improvement over X3 - a movie that scrapes by on whizz-bang action and momentum from the previous films.
I can't think of a single franchise that has gone from an average third movie to a strong fourth. Superman, Lethal Weapon, Alien - they all just about scraped through Part 3 and then came apart utterly when they tried for a fourth.
Much as I'd like to see it continue, common sense says they should quit while they're marginally ahead.