The stadium was built on a "nice deserted moor," near several fields used as campgrounds (run by Muggles named Mr. Payne and Mr. Roberts.)
Diagram of the stadium