Madrid taxi Import

From OpenStreetMap Wiki
Jump to navigation Jump to search
Madrid taxi Import - Other languages

Goals

The goal is to manually merge and import the taxi ranks and taxi ranks information from the Madrid city council database, see: Import_Ayuntamiento_Madrid (in spanish)

In Madrid there are 106 taxi ranks already in OSM. The dataset from the city council contains 424. It contains useful information like address and capacity.

Existing hand mapped information will not be replaced by the import without on-the-ground verification.

Schedule

  1. October 16 2016: We started the preparation and the discussion. See the discussion in the talk-es list (in spanish)
  2. October 17 2016: Requested information about the taxi ranks with time restrictions to the City Council. We'll wait some time before starting the import.
  3. October 28 2016: Got the response with the time information and more.
  4. November 1: We started the import
  5. November ?: We finish doing the QA after the import
  6. November ?: Send a report to talk-es and the city council

Import Data

Data description

The original dataset uses the xls format. It includes a total of 424 spots. 3 mappers checked some known ranks and the locations are correct.

We'll use the xls file for processing. You can download the data here.

Background

ODbL Compliance verified: YES

The license can be consulted here (in Spanish). It requires attribution of the source, so all nodes and the changeset will include source=Ayuntamiento de Madrid. The Madrid City Council (Ayuntamiento de Madrid) is already on the contributors list.

After sending the report with the problems and errors found during the drinking water import, including a section about the license, got a response from the "Subdirección General de Transparencia del Ayuntamiento de Madrid" (the data owner), and they confirmed me that we comply with their attribution clause using their data in OSM if we clearly specify the source and the source:date, the way it was done on the Madrid_Drinking_Water_Import. And they do like the work we are doing :) Kresp0 (talk) 13:30, 16 October 2016 (UTC)

Compatibility with the ODbL was already discussed in the imports and talk-es mailing lists.

Cumplimiento de la cláusula de atribución de los datos abiertos del Ayto. de Madrid en OSM.pdf

Import Type

The import will be done manually, using the openstreetmap.es Tasks Manager, so it can be assigned to different volunteers. JOSM will be used to edit the data.

Data Preparation

Data Reduction & Simplification

Some of the fields aren't relevant and will be ignored.

Tagging Plans

Here are the original fields, their meaning and how they will be converted to the resulting OSM file:

Tagging Conversion
Original Fields Meaning OSM tag Comments
Número Internal reference number Not used
Gis_X EPSG:23030 X X coordinate Reprojection to EPSG:4326 needed
Gis_Y EPSG:23030 Y Y coordinate Reprojection to EPSG:4326 needed
Fecha de Alta Not used
Distrito District Not used
Barrio Suburb Not used
Calle Street addr:street=*
Nº Finca housenumber Not used
Tipo de Reserva kind of parking reservation Not used all "taxi"
Línea / Batería line parking or not Not used
Número de Plazas Capacity capacity=*
Texto Cajetines Text with times of operation opening_hours=* Translation from spanish to opening_hours format needed

We added the following tags:

Translation to opening_hours format

Original text opening_hours translation
DE 10:00 A 20:00 H Mo-Su 10:00-20:00; PH 10:00-20:00
De 17 a 3 h. Mo-Su 17:00-03-00; PH 17:00-03:00
De 6 a 24 h. Mo-Su 06:00-00:00; PH 06:00-00:00
De Lunes a Viernes Mo-Fr 00:00-24:00; PH 00:00-24:00
De Lunes a Viernes de 8 a 15 h. Mo-Fr 08:00-15:00; PH 08:00-15:00
De Lunes a Viernes de 8 a 16 h. Mo-Fr 08:00-16:00; PH 08:00-16:00
De Lunes a Viernes de 8 a 19 h. Mo-Fr 08:00-19:00; PH 08:00-19:00
De lunes a viernes de 8 a 20 h. Mo-Fr 08:00-20:00; PH 08:00-20:00
De Lunes a Viernes de 8 a 20 h. Mo-Fr 08:00-20:00; PH 08:00-20:00
De lunes a viernes de 9 a 14 h Mo-Fr 09:00-14:00; PH 09:00-14:00
DE LUNES A VIERNES LABORABLES DE 9 A 15H. Mo-Fr 09:00-15:00; PH off
DÍAS LABORABLES DE LUNES A VIERNES DE 8 A 19 H Mo-Fr 08:00-19:00; PH off
EN DIAS DEESPECTACULO Mo-Su,PH "En días de espectáculo"
En linea NOT USED
Excepto de 20 a 8 h. Mo-Su 08:00-20:00; PH 08:00-20:00
Excepto Sabados y Festivos Mo-Fr 00:00-24:00; PH off
EXCEPTO SABADOS Y FESTIVOS Mo-Fr 00:00-24:00; PH off
LAB. DE 8 A 20 H. EXCEPTO TRANSPORTE URBANO DE MERCANCIASPERMITIDO TAXIS DE MIERCOLES A DOMINGO DE 20 A 8 H.. We-Su 20:00-08:00
Laborables de 8 a 14 h. Mo-Fr 08:00-14:00; PH off
Laborables de 8 a 20 h. Mo-Fr 08:00-20:00; PH off
Laborables de 9 a 14 h. Mo-Fr 09:00-14:00; PH off
Laborables de Lunes a Jueves de 8.30 a 17.30 h. Mo-Th 08:30-17:30; PH off
Laborables de Lunes a Viernes de 6 a 16 h. Mo-Fr 06:00-16:00; PH off
Laborables de Lunes a Viernes de 6 a 19 h. Mo-Fr 06:00-19:00; PH off
Laborables de Lunes a Viernes de 6 a 19 h. Mo-Fr 06:00-19:00; PH off
Laborables de Lunes a Viernes de 6 a19 h. Mo-Fr 06:00-19:00; PH off
Laborables de Lunes a Viernes de 6 a 21 h. Mo-Fr 06:00-21:00; PH off
Laborables de Lunes a Viernes de 8 a 14 h. Mo-Fr 08:00-14:00; PH off
Laborables de Lunes a Viernes de 8 a 19 h. Mo-Fr 08:00-19:00; PH off
Laborables de Lunes a Viernes de 8 a 20 h. Mo-Fr 08:00-20:00; PH off
Laborables de Lunes a Viernes de 9 a 19 h. Mo-Fr 09:00-19:00; PH off
Laborables de Lunes a Viernes de 9 a 21 h. Mo-Fr 09:00-21:00; PH off
Laborables de Lunes a Viernes de a 6 a 19 h. Mo-Fr 06:00-19:00; PH off
Lunes a viernes Mo-Fr, PH 00:00-24:00
Lunes a Viernes Mo-Fr, PH 00:00-24:00
Lunes a Viernes de 6 a 21 h. Mo-Fr 06:00-21:00; PH 06:00-21:00
Lunes a viernes de 8 a 21 h. Mo-Fr 08:00-21:00; PH 08:00-21:00
RESERVADO TAXIS DE 20 A 22 H. EN DIAS DE EXPECTACULO Mo-Su 20:00-22:00 "En días de espectáculo"
SUPLEMENTO DE TARIFA AUTORIZADO SALIDA DE ESTACION NOT USED
SUPLEMENTO DE TARIFA AUTORIZADO SALIDA DE ESTACIÓN NOT USED
SUPLEMENTO DE TARIFA AUTORIZADO SALIDA ESTACION NOT USED

Changeset Tags

We will use the following changeset tags:

Data Transformation

Data is in xls format. We already:

  • Converted to the OSM format following this steps:
  • 1.- Delete the first row
  • 2.- Save as reserva_paradas_taxis.csv using semicolon (;) as field separator and quotes (") as text separator.
# 3.- Extrae los campos a usar y añade FIXME a la dirección:
  awk -F ';' '{print $2";"$3";FIXME "$7";"$11";"$12}' reserva_paradas_taxis.csv | perl -pe 's/;FIXME Calle;/;Calle;/g' > r ; mv r reserva_paradas_taxis.csv

# 4.- Prepara la reproyección de EPSG:25830 a EPSG:4326 con la rejilla:
  echo '<OGRVRTDataSource>
  <OGRVRTLayer name="reserva_paradas_taxis">
  <SrcDataSource>reserva_paradas_taxis.csv</SrcDataSource>
  <GeometryType>wkbPoint</GeometryType>
  <LayerSRS>+init=epsg:25830 +wktext</LayerSRS>             
  <GeometryField encoding="PointFromColumns" x="Gis_X" y="Gis_Y"/>
  <Field name="addr:street" src="Calle" />
  <Field name="capacity" src="Número de Plazas" />
  <Field name="opening_hours" src="Texto Cajetines" />
  </OGRVRTLayer>
  </OGRVRTDataSource>' > paradas-taxi-25830.vrt

# 5.- Reproyecta usando ogr2ogr
  ogr2ogr -lco GEOMETRY=AS_XY -overwrite -f CSV -t_srs EPSG:4326 paradas-taxi.csv paradas-taxi-25830.vrt

# 6.- Prepara la conversión a shapefile
  echo '<OGRVRTDataSource>
  <OGRVRTLayer name="Paradas_taxi_Madrid">
  <SrcDataSource relativeToVRT="1">.</SrcDataSource>
  <SrcLayer>paradas-taxi</SrcLayer>
  <GeometryType>wkbPoint</GeometryType>
  <LayerSRS>WGS84</LayerSRS>
  <GeometryField encoding="PointFromColumns" x="x" y="y"/>
  </OGRVRTLayer>
  </OGRVRTDataSource>' > paradas-taxi.vrt

# 7.- Convierte a shapefile
  ogr2ogr -f "ESRI Shapefile" . paradas-taxi.csv && ogr2ogr -f "ESRI Shapefile" . paradas-taxi.vrt
  • 8.- Open the shp with JOSM (opendata plugin needed)
  • 9.- Save as "Paradas_taxi_Madrid.osm" and close JOSM
# 10.- Transforma, añade y quita etiquetas:
grep -v "'Y'" Paradas_taxi_Madrid.osm | grep -v "'X'" | perl -pe 's/addr_stree/addr:street/g' | perl -pe 's/opening_ho/opening_hours/g' | grep -v 'S/N' | perl -pe "s/  <\/node>/    <tag k=\'amenity\' v=\'taxi\'\/>\n  <\/node>/g" | perl -pe "s/  <\/node>/    <tag k=\'operator\' v=\'Ayuntamiento de Madrid\'\/>\n  <\/node>/g" | perl -pe "s/  <\/node>/    <tag k=\'source\' v=\'Ayuntamiento de Madrid\'\/>\n  <\/node>/g" | perl -pe "s/  <\/node>/    <tag k=\'source:date\' v=\'2016-10-27\'\/>\n  <\/node>/g" > r ; mv r Paradas_taxi_Madrid.osm


# 11.- Traduce al formato de opening_hours:
perl -pe 's/'\''/xx/g' Paradas_taxi_Madrid.osm | perl -pe 's/xxDE 10:00 A 20:00 Hxx/xxMo-Su 10:00-20:00; PH 10:00-20:00xx/g' | perl -pe 's/xxDe 17 a 3 h.xx/xxMo-Su 17:00-03:00; PH 17:00-03:00xx/g' | perl -pe 's/xxDe 6 a 24 h.xx/xxMo-Su 06:00-00:00; PH 06:00-00:00xx/g' | perl -pe 's/xxDe Lunes a Viernesxx/xxMo-Fr 00:00-24:00; PH 00:00-24:00xx/g' | perl -pe 's/xxDe Lunes a Viernes de 8 a 15 h.xx/xxMo-Fr 08:00-15:00; PH 08:00-15:00xx/g' | perl -pe 's/xxDe Lunes a Viernes de 8 a 16 h.xx/xxMo-Fr 08:00-16:00; PH 08:00-16:00xx/g' | perl -pe 's/xxDe Lunes a Viernes de 8 a 19 h.xx/xxMo-Fr 08:00-19:00; PH 08:00-19:00xx/g' | perl -pe 's/xxDe lunes a viernes de 8 a 20 h.xx/xxMo-Fr 08:00-20:00; PH 08:00-20:00xx/g' | perl -pe 's/xxDe Lunes a Viernes de 8 a 20 h.xx/xxMo-Fr 08:00-20:00; PH 08:00-20:00xx/g' | perl -pe 's/xxDe lunes a viernes de 9 a 14 hxx/xxMo-Fr 09:00-14:00; PH 09:00-14:00xx/g' | perl -pe 's/xxDE LUNES A VIERNES LABORABLES DE 9 A 15H.xx/xxMo-Fr 09:00-15:00; PH offxx/g' | perl -pe 's/xxDÍAS LABORABLES DE LUNES A VIERNES DE 8 A 19 Hxx/xxMo-Fr 08:00-19:00; PH offxx/g' | perl -pe 's/xxEN DIAS DEESPECTACULOxx/xxMo-Su,PH xxEn días de espectáculoxxxx/g' | perl -pe 's/xxEn lineaxx/xxBORRAMExx/g' | perl -pe 's/xxExcepto de 20 a 8 h.xx/xxMo-Su 08:00-20:00; PH 08:00-20:00xx/g' | perl -pe 's/xxExcepto Sabados y Festivosxx/xxMo-Fr 00:00-24:00; PH offxx/g' | perl -pe 's/xxEXCEPTO SABADOS Y FESTIVOSxx/xxMo-Fr 00:00-24:00; PH offxx/g' | perl -pe 's/xxLAB. DE 8 A 20 H. EXCEPTO TRANSPORTE URBANO DE MERCANCIASPERMITIDO TAXIS DE MIERCOLES A DOMINGO DE 20 A 8 H..xx/xxWe-Su 20:00-08:00xx/g' | perl -pe 's/xxLaborables de 8 a 14 h.xx/xxMo-Fr 08:00-14:00; PH offxx/g' | perl -pe 's/xxLaborables de 8 a 20 h.xx/xxMo-Fr 08:00-20:00; PH offxx/g' | perl -pe 's/xxLaborables de 9 a 14 h.xx/xxMo-Fr 09:00-14:00; PH offxx/g' | perl -pe 's/xxLaborables de Lunes a Jueves de 8.30 a 17.30 h.xx/xxMo-Th 08:30-17:30; PH offxx/g' | perl -pe 's/xxLaborables de Lunes a Viernes de 6 a 16 h.xx/xxMo-Fr 06:00-16:00; PH offxx/g' | perl -pe 's/xxLaborables de Lunes a Viernes de 6  a 19 h.xx/xxMo-Fr 06:00-19:00; PH offxx/g' | perl -pe 's/xxLaborables de Lunes a Viernes de 6 a 19 h.xx/xxMo-Fr 06:00-19:00; PH offxx/g' | perl -pe 's/xxLaborables de Lunes a Viernes de 6 a19 h.xx/xxMo-Fr 06:00-19:00; PH offxx/g' | perl -pe 's/xxLaborables de Lunes a Viernes de 6 a 21 h.xx/xxMo-Fr 06:00-21:00; PH offxx/g' | perl -pe 's/xxLaborables de Lunes a Viernes de 8 a 14 h.xx/xxMo-Fr 08:00-14:00; PH offxx/g' | perl -pe 's/xxLaborables de Lunes a Viernes de 8 a 19 h.xx/xxMo-Fr 08:00-19:00; PH offxx/g' | perl -pe 's/xxLaborables de Lunes a Viernes de 8 a 20 h.xx/xxMo-Fr 08:00-20:00; PH offxx/g' | perl -pe 's/xxLaborables de Lunes a Viernes de 9 a 19 h.xx/xxMo-Fr 09:00-19:00; PH offxx/g' | perl -pe 's/xxLaborables de Lunes a Viernes de 9 a 21 h.xx/xxMo-Fr 09:00-21:00; PH offxx/g' | perl -pe 's/xxLaborables de Lunes a Viernes de a 6 a 19 h.xx/xxMo-Fr 06:00-19:00; PH offxx/g' | perl -pe 's/xxLunes a viernesxx/xxMo-Fr, PH 00:00-24:00xx/g' | perl -pe 's/xxLunes a Viernesxx/xxMo-Fr, PH 00:00-24:00xx/g' | perl -pe 's/xxLunes a Viernes de 6 a 21 h.xx/xxMo-Fr 06:00-21:00; PH 06:00-21:00xx/g' | perl -pe 's/xxLunes a viernes de 8 a 21 h.xx/xxMo-Fr 08:00-21:00; PH 08:00-21:00xx/g' | perl -pe 's/xxRESERVADO TAXIS DE 20 A 22 H. EN DIAS DE EXPECTACULOxx/xxMo-Su 20:00-22:00 xxEn días de espectáculoxxxx/g' | perl -pe 's/xxSUPLEMENTO DE TARIFA AUTORIZADO SALIDA DE ESTACIONxx/xxBORRAMExx/g' | perl -pe 's/xxSUPLEMENTO DE TARIFA AUTORIZADO SALIDA DE ESTACIÓNxx/xxBORRAMExx/g' | perl -pe 's/xxSUPLEMENTO DE TARIFA AUTORIZADO SALIDA ESTACIONxx/xxBORRAMExx/g' | grep -v BORRAME | perl -pe 's/xx/'\''/g' | perl -pe 's/'\''En días de espectáculo'\''/"En días de espectáculo"/g' > p ;  mv p Paradas_taxi_Madrid.osm
  • 12.- Manually drawn the working area using JOSM: create new layer, draw the area and save as geojson
  • 13.- Upload the working area to the TM
  • 14.- Split the osm file to have one file for each task on the TM using the osmboxes script
  • 15.- Upload the splitted files to some webserver
  • 16.- Add the special link (see the osmboxes README) to the per task instructions on the TM

Data Import Workflow

Team Approach

Import will be undertaken by experienced OSM volunteers, following a strict workflow.

References

This manual import will be discussed in the Talk-Es list

Workflow

As most (if not all) of the volunteers will be Spanish speaking, they can follow the detailed workflow in Spanish language on the TM.

The workflow will be as follows:

  • 1. Install the TODO list JOSM plugin if the volunteer don't have it.
  • 2. Create, if we don't have one yet, an import specific user account, like username_madrid_taxi_import (you will need a different email address from the account you used for your regular OSM account). Change the OSM username to that specific account in the JOSM preferences.
  • 3. Configure JOSM to allow remote access.
  • 4. Login into the Tasks Manager.
  • 5 Select one square to work on.
  • 6 Open the OSM data and the taxi ranks data by clicking on the JOSM button and the link on the extra instructions.
  • 7 Select all nodes on the taxi ranks layer and add them to the ToDo list. For each node we'll:
    • Edit the addr:street=*, fix the name to match the current OSM street name
    • Conflate the node data with the taxi rank area/node if there is already one or create a new one if not. Don't replace existing hand mapped information without on-the-ground verification.
  • 8. After the import is done, there will be a review process using the TM.

Updates

Each year I'll download the dataset, do a diff and update the data to OSM manually. Until I get bored or automate it. Kresp0 (talk) 20:44, 16 October 2016 (UTC)

Reverse plan

In case of any trouble, JOSM reverter will be used.

Retrieved from "https://wiki.openstreetmap.org/w/index.php?title=Madrid_taxi_Import&oldid=1551897"