Connecticut/CT Building and Address Import

From OpenStreetMap Wiki
Jump to navigation Jump to search
STATUS: Completed

Duration: 28 Feb 2021 - 9 Apr 2022

Current progress: 100%
Note: Additional gaps filling in address points coverage using NAD is still in progress.

Participating users: Mashin, jnighan, roptat

Import account: CT-import

Changeset comment: #CT-building-address-import (searchable on osmcha)

For comments send PM to OSM account Mashin or leave a message at OSMUS Slack channel #local-connecticut.

Connecticut has still most of its buildings missing in OSM. Building in south-west CT were recently imported and the largest cities (New Haven, Hartford, Bridgeport, Middletown, Norwich,...) have various level of building coverage. Completing the building coverage would allow community to dedicate their limited time to improvements of businesses locations or road network. Local CT community was contacted and all responses were positive.


>>> Link to progress timeline animation
Building density CT.png

Data

Three datasets from two sources will be use for the import:
Sample of merged and processed data can be found here: Sample 1, Sample 2
Code: All code used for processing is available on GitHub - Mashin6/CT_building_address_import

1. Building footprint data for CT @ data.ct.gov

Building footprints from data.ct.gov

This data contains 1,490,668 buildings and was most probably created by extracting features from 2012 impervious landcover images. Building walls are therefore sometimes jagged due to extra nodes or improper angles and require deeper processing. Comparison to Esri World Imagery (clarity) and Maxar Premium Imagery showed that absolute vast majority of buildings are still up to date and occasional discrepancies will be solved during import.

Processing

  • All building polygons were simplified to remove extra nodes
  • Buildings were then squared using custom Orthogonalize python script that forces all angles to 180˚ or 90˚, but still allows for walls (e.g. bay windows) close to 45˚ to remain untouched. Distortion of buildings with complex non-rectangular shape was prevented by allowing squaring only for buildings for which the overlap between old and new building shape is more than 95%.
  • Additional round of simplification to remove nodes with 180˚
  • All buildings are tagged as building=yes


2. Building footprints for City of Hartford @ Hartford Open GIS

Building footprints from Hartford data

This data was created from aerial flight imagery from April 2015 and was yearly updated. Overall data is very good quality and requires minimum processing and contains 40380 polygon objects.

Processing

  • Every polygon was tagged as building=yes.
  • For buildings with a deck attached, the deck part was tagged as building:part=deck and the adjacent structure as building:part=yes. Outline for connected building:parts was tagged as building=yes.
  • Greenhouses were tagged as building=greenhouse
  • Damaged buildings/ruins were kept and tagged as building=ruins
  • Objects in close to completion marked as 'Foundation' were tagged as building=yes.
  • Objects labelled as "Cement Pad" or "Patio" were removed.


3. Address points information @ data.ct.gov

Address points for CT (without WestCOG area)

Dataset contains 741,050 address points from 2013 Address data collected by the Connecticut state for Towns where it was available and overlaid with 2012 Parcel data and buildings form source 2.

Processing
Because all structures on one parcel shared the addresses, they were de-duplicated by assigning address point to the largest structure on the parcel (in vast majority of cases this is the main structure). Following tags were used if available:


4. Building footprints for Stonington @ Town of Stonington GIS

[6 Apr 2021] This data was included after public announcing and review of this proposal. Contains 19,905 polygons of very good quality data.

Processing
Objects tagged as

  • BLDG or BLDG_UC were kept and tagged as building=yes
  • MOBILE were kept and tagged as building=yes
  • FOUNDATION, DECK were removed

5. Building footprints for Glastonbury @ Town of Glastonbury REST API

[25 May 2021] This data was included after public announcing and review of this proposal. Contains 17,759 polygons of very good quality data.

Processing
Objects tagged as

  • BLDG, OUT-BLDG, FOUNDATION, CONSTRUCT and MOBILE were kept and tagged as building=yes
  • RUIN were kept and tagged as abandoned:building=yes
  • BASE_ELEV converted to ele=* in meters
  • ELEV_SL - BASE_ELEV used to calculate height=* in meters

6. Address points for CT @ National Address Database release 6

Processing
Objects tagged as

Import Strategy

Harford area: Data from 2. will be used
Outside Hardford and outside WestCOG: Data from 1. will be used
Address points (3.) will be used for the whole CT except WestCOG area.

  • Import will split into small parts and progress will be tracked using task manager.
  • Every part will be loaded into JOSM and checked for errors with validator and visual comparison with Maxar/ESRI imagery.
  • If necessary building shape and/or position will be adjusted.
  • In areas with existing building data, conflation will be done on case to case basis only if the import version is better than the one present in OSM.
  • Conflation will be done with JOSM function "Replace geometry (Shift+cmd+G)", which preserves history of the object.

Separate account will be used for data upload CT-import

Attribution will be noted in import catalogue and changesets will be marked with source=State of Connecticut or source=State of Connecticut; City of Hartford.

License

2.:
OSM:
Dear Brett,

On the Hartford web page, I found a building dataset that was created by Sanborn company [1]. The page also mentions that the associated licence is CC BY 4.0 and therefore wanted to ask if it would be possible to release the data under different license i.e. CC0 1.0 (https://creativecommons.org/publicdomain/zero/1.0/).

The reason why I am asking is because I would like to import these data into OpenStreetMap database and make them easily accessible for the general public. CC BY 4.0 is unfortunately not exactly compatible with the ODbL license that OSM is using and would require a signed waiver about certain aspects of attribution. This makes it more complicated and if you would release the data for use in OSM under CC0 1.0, this would make it much more easier.

Brett Flodine:
I apologize the dataset should be licensed under the 1.0 version Im not sure why it is set to 4.0, I will have it fixed today. Brett


1. and 3.:
From conversation with Yury Yatsynovich: https://lists.openstreetmap.org/pipermail/imports/2020-August/006343.html
OSM:
Hi Scott!

Are there any constraints on who and how is allowed to use the data posted on http://geodata-ctmaps.opendata.arcgis.com <https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgeodata-ctmaps.opendata.arcgis.com%2F&data=01%7C01%7CScott.Gaul%40ct.gov%7C7d7679549bd943e31e6f08d837ac29e1%7C118b7cfaa3dd48b9b02631ff69bb738b%7C0&sdata=PYF2JL9cVk0JZ50ndaEao7mH9iaA%2BGGlsKjsmTkO8is%3D&reserved=0> ? I'm asking this because I'm interested in importing the data on CT buildings with addresses into OpenStreetMap (OSM) -- to do so I need to make sure that the imported data's license is compatible with OSM. With kind regards, OSM contributor, -- Yury Yatsynovich

OSM:
Thank you, Scott!

The above mentioned terms describe mostly (lack of) liabilities of the data provider, but don't mention any constraints on who and how can use the data. Does it mean that there are no such constraints? There is also no attribution requirement -- does it mean that I'm not required to cite the source of the data? With best regards,

CT:
Hi Yury, At this point, yes, I think that’s correct. Obviously, we’d like if you cite the state, but we don’t have set guidelines for that. ... Thanks, Scott

4.:
OSM:
Apr 2, 2021
Hello!
I am writing to ask permission/constraints for using Stonington GIS Data. I'm interested in importing building data into the OpenStreetMap database as part of a larger project (more info: https://wiki.openstreetmap.org/wiki/Connecticut/CT _ Building_and_Address_lmport). We are currently using statewide data, but yours is of superior quality. The town website says "All information provided on this website is public infmation"-is this data available for use under CC0 1.0?


Roger Kizer (Information Technology Manager, Town of Stonington):
Apr 6, 2021
The files are for public use and available for such projects. We don't have a creative commons restriction, but would remind you of the usage disclaimer listed below.
THE DATA AND ANY RELATED MATERIALS CONTAINED THEREIN ARE PROVIDED "AS IS," WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NONINTERFERENCE, SYSTEM INTEGRATION, OR NONINFRINGEMENT. THE ENTIRE RISK OF USE OF THE DATA SHALL BE WITH THE USER.

5.:
OSM:
May 25, 2021
Hello!
Is Glastonbury GIS DAta considered public domain under CC0 1.0? I want to use it for an OpenStreetMap project.


Jena Gosselin (Information Technology Manager, Town of Glastonbury):
May 25, 2021
Yes, you can use building footprint layer. Just note it's from our 2014 aerial flight.

Note: This data was originally extracted from the 2012 Land Classification by Esri (Arthur Crawford) and provided at no cost and no restrictions on use to Connecticut. 

Guideline

JOSM setup

  • Enable Expert mode (Preferences > tick in lower left corner)
  • Install plugins: utilsplugin2, opendata
  • Set account to CT-import: (Preferences > OSM server > Use Basic Authentication > fill in credentials)
  • Enable Show user name in title (Preferences > Look and Feel)

Data handling

  • Login to tasks.openstreetmap.us under your user account.
    • Set default editor to JOSM: Settings > Default editor > JOSM
  • Select square and click on Map selected task
  • Download import data
  • On every square perform following:
    • Run JOSM validator (Shift + V) and fix any issues
    • Adjust buildings position according to ESRI clarity imagery. You can move multiple buildings by selecting area (S) and subsetting for ways (Shift + E).
    • Visually scan and correct distorted buildings. (Optionally: improve shape by adding details from imagery. Helpful is extrusion tool (X))
    • Merge import layer to OSM Data layer: select both layers, right click > Merge > OSM Data as target layer > Merge layer
    • Run validator again (Shift + V) and fix any errors: overlap with road, river,...
    • If there is a building overlap, because building exists in OSM:
      • Visually check which shape is better.
      • Compare with Maxar premium layer to get a sense of the more current state of the building.
      • If import shape is better, select both buildings and Replace geometry (Cmd + Shift + G) or (More tools > Replace Geometry)
      • If OSM shape is better, delete import shape.
  • Upload:
    • Fill in Commit message: #CT-building-address-import
    • Fill source: State of Connecticut (or State of Connecticut; City of Hartford in Hartford)
    • Add Changeset tag: documentation = wiki.openstreetmap.org/wiki/Connecticut/CT_Building_and_Address_Import (Tags of new changeset > fill value: documentation | wiki.openstreetmap.org/wiki/Connecticut/CT_Building_and_Address_Import)
    • Hit upload
  • In tasking manager set Task status: yes and Submit task
Retrieved from "https://wiki.openstreetmap.org/w/index.php?title=Connecticut/CT_Building_and_Address_Import&oldid=2707090"