I live really close to the coast, and my gf’s family is from a coastal town, and when we stop seeing tourists there and stop getting in the ocean ourselves, that’s the end of summer. In my mind summer is the season for swimming outdoors, and when the ocean gets cold that’s the end of that, because pools and lakes get cold even quicker than the ocean. But yeah I guess if you don’t swim or live by water, you’d have a different metric for it