And once you have decided on some arbitrary height as "sea level," you then have to project this imaginary line across the land. Older map-makers did this by assuming that the earth was a perfect sphere, but this turned out to be so far off that even nineteenth-century geographers began making corrections for the flattening of the sphere at the poles. But even assuming the earth is a smooth ellipsoid is wrong, because the earth is not a perfectly even anything. Therefore, all our maps are wrong, and they are more wrong the farther you get from our old reference, the north Atlantic Ocean.
The US is currently creating a new model of height from which everything will be measured, based on data from satellites, that promises to be the most accurate ever. Floating in space, satellites are just a lot more stable than the heaving ocean, and thus a lot easier to measure from.
Part of a Cut and Fill Map of the Anacostia River in Washington, DC; red means cut down since 1894, green means built up
I know something about this because archaeologists have long tried to understand what has happened in our cities and other much disturbed locations using something called "cut and fill analysis." Let's say you have a reasonably accurate contour map of your city from 1890. You might think that by comparing the elevations of that map with those on a modern map, you could figure out which locations are now much taller, because they have been built up by fill (which is more than 50 feet thick in some parts of DC and New York), and which are now shorter, which means that the old ground (along with the archaeological sites) has been cut down. This is extremely useful for planning your archaeological dig, and some locales, like DC, have started requiring it as part of preliminary archaeological studies.
1892 US Coast & Geodetic Survey Map of the district, so accurate that a small cellar hole I once found in Rock Creek Park, about 2 feet deep and 12 feet across, shows up on the map as a wiggle in a contour line. Yet comparing this with modern maps does not give you the correct depth of fill – in my experience it has in fact never given the depth of fill to within a foot. This is because the standards of elevation, and the methods used to project them across the landscape, have changed so much that it requires detailed local knowledge to convert between the old and the modern systems.
One day maybe some mathematician will work out a formula for comparing maps of this period to modern ones. For now, though, the state of the art is that you make your cut and fill analysis, then go out and do a bunch of mechanical soil cores, and use the cores to correct your map. Even this is difficult because it often happens that Core A will show your model is off by 4 feet, while Core B 200 feet away will be off by 2 feet or 10 feet. As I said, these maps are in some ways wonderful, so I don't think these are just errors; they show that we don't understand how they thought about the globe well enough to compare their maps to ours.