Curious about why we wanted to make a neighborhood-based map to explore NYC’s data? Curious about how we made the map you’ve been clicking around? Read on to find out!
We wanted to present our robust data in a neighborhood by neighborhood fashion because we know how meaningful neighborhoods are for many people. Unlike cities, or election districts, or census blocks, however, neighborhoods do not have officially assigned edges. While their general location is often widely agreed upon, their actual boundaries are notoriously fuzzy. They might overlap or not meet at all, leaving no-man’s-lands between known neighborhoods. Smaller sub-neighborhoods may nestle within larger neighborhoods. Or, they simply might change: neighborhoods are where we say they are, so their boundaries are constantly re-defined by the people who proudly claim identity with, blithely pass through, or eagerly visit them.
Crafting a map that accurately represents these fuzzy, flexible boundaries presented us with a delectable challenge — one that we thought was best solved by crowdsourcing. After all, neighborhoods are a user-defined geography, so why not let the user define them cartographically, too? Flickr is a great source of this sort of user-defined geography, as it provides its huge and growing trove of geotagged data to anyone interested (check out their API here). Since the Flickr data includes the lat-long coordinates of the posted photo as well as the neighborhood tag chosen by the photographer, this data makes it possible to make a map of neighborhoods as a group of people (in this case, Flickr users) believe them to be — which is, as we’ve pointed out, the only way they really exist in the first place. Luckily for us, David Blackman, the lead geographic infrastructure engineer at Foursquare, had already created a set of open-source, completely editable NYC neighborhood polygons from this Flickr data. We highly encourage you to check out David’s polygon project at zetashapes, because it’s super cool (and we found it a very useful starting place for our own neighborhoods project).
We want to emphasize that the neighborhood polygons we created are only one version of the manifold ways to represent New York’s neighborhoods. Other versions are available on our on our Data Wrangler, and we hope to add more variations in the future. If you have comments for how we can improve our neighborhood boundary file, drop us a line. And if you decide to build your own neighborhood boundaries by downloading the geoJSON from us or the Flickr-based zetashapes, we’d love to see what you come up with!
If you’ve been waiting to read about exactly how we arrived at the neighborhoods map you’ve seen on our site, the fun starts here. Nathan kicked off the project by downloading the zetashapes and spending some in time in QGIS, performing a rough normalization of each of the neighborhood shapes so that they would correspond more closely to the contours of NYC’s streets and coastline. After joining the 256 separate neighborhood polygons into a single file, Nathan exported them as a geoJSON and passed them off to Zack for further refining.
Zack then went through the painstaking process of identifying and removing the gaps and between neighborhoods. After converting the geoJSON to an ESRI shapefile, he ran the union function in ArcMap without letting the program allow gaps. This process identified 213 blank polygons — that is, gaps between neighborhoods. You can see the gaps highlighted in blue below.
Zack combed through these 213 gaps, deciding which neighborhood they should belong to and manually assigning them the corresponding name. Finally he used the dissolve function to bring each former gap safely into the neighborhood it belonged to, creating a gap-free neighborhood file.
Zack’s next task was to make sure that our neighborhoods file would correspond accurately to NYC’s complicated shoreline. He accomplished this by buffering the neighborhoods file to a distance of 50 meters (ensuring that the neighborhoods extended well beyond the shoreline) and then running ArcMap’s symmetrical difference function between the neighborhoods buffer file and borough boundaries file to identify areas that would need to be removed from the buffer.
Zack then used the merge and dissolve function to arrived back at a file with only 256 unique neighborhoods. Finally, used a borough boundaries shapefile to clip the neighborhoods file, ensuring the removal of any neighborhood area that extended beyond the city’s actual shoreline.
The remaining GIS task was to remove overlaps from the neighborhoods shapefile. After manually editing the entire stretch of the shoreline, deleting and adjusting errant polygon points as he went, Zack used ArcMap’s intersect function to identify overlaps between interior polygons. Once the conflicting polygons had been deleted, he again dissolved the file to arrive at a clean, 256 neighborhood file. The file was exported as a geoJSON and uploaded into our data wrangler, where you can download it for your own use. As you might imagine, Zack’s main challenge was to design a “repeatable method” that accurately synthesizes this multi-step process and facilitates the maintenance of our neighborhoods file.
Our next task was to design a functional, attractive neighborhood map to embed in our site. Aykut, our resident expert in GIS-related database and programming challenges, took command of this step of the process. Because it was our first map using CartoDB, Aykut took a moment to familiarize himself with the platform. He then added all the technical functionality that makes using this map an intuitive experience: hovering labels and polygons that highlight as you move your mouse across the map, one-click navigation to any specific neighborhood sub-page, a dropdown neighborhoods menu, an address search function, and a “find me” function to locate users’ position on the map.
We hope you click around the map and tell us about anything you find, from gems to glitches. You can also grab the neighborhoods geoJSON from our data wrangler and explore it. As always, we really appreciate your contributions to making NYC’s open data even better!