One of the forgotten pleasures of Dungeons and Dragons is reaching name level and being permitted to establish your own stronghold. This early feature of DnD is obviously inspired by the wargaming traditions of the first roleplayers.
The problem is, there are woefully inadequate rules in Original DnD to govern stronghold and empire building, and stonghold-building precipitates player-versus-player conflict, particularly when the name-level players want to expand into the same territories. Not to mention that the level of abstraction necessary to handle a game of empire building is different from the level of abstraction necessary for a game of dungeon-delving resource and hit point management.
Though not perfect by any means, I get my fantasy empire building and PvP conflict fix from our occasional games of Settlers of Catan. Settlers is at just about the right level of abstraction to avoid getting bogged down in the minutae, and allows for a certain amount of PvP conflict without resulting in the total elimination of the other players.
Settlers would be a suitable base upon which to build a really interesting high-level fantasy empire building game. I've heard good things about Small World as well, but have yet to give it a try.