Travel Mapping Manual: Deal with data errors

The developer tools execute different kind of data checks to indicate potential errors. Highway data managers need to check for these errors to ensure consistent and correct highway data. Some of the data checks are available in the waypoint editor before submitting changes, but many need attention after the next site update.

Contents

Data errors
Highway data check list
Marking errors false positive (FP)
Concurrency check

Load graph with mapview
Load graph with Highway Data Examiner (HDX)
Check for broken concurrencies

Near-miss points
Marking NMPs false positive FP

🔗 Data errors

When working with the WPT file editor, a data check for the loaded route is executed with every modification. The code of a data error is indicated in the last column of the waypoint table. Use the tool tip for additional info. After manual changes within the editor field, the data check is done on the next editor action, e.g. when pressing the Load button.

Check the table for errors before saving the .wpt file. Fix unintended errors or mark intended errors false positive.

Note: Not all data errors are detected by the WPT file editor, see below: supported, proposed but not yet supported, not possible to be detected in WPT file editor.

Link	Code	Description	WPT File Editor	FP possible
🔗	ABBREV_AS_CHOP_BANNER	The route's chopped routes .csv file entry has the Abbrev erroneously entered into the Banner field.	NO	NO
🔗	ABBREV_AS_CON_BANNER	The 2 potential underlying causes are: The route's chopped routes .csv file entry has the Banner erroneously entered into the Abbrev field. If the same route is also flagged as ABBREV_NO_CITY, this is probably the case. The route's connected routes .csv file entry has the chopped route's Abbrev entered as its Banner.	NO	NO
🔗	ABBREV_NO_CITY	The route's chopped routes .csv file entry has an Abbrev, but no City. If the same route is also flagged as ABBREV_AS_CON_BANNER, the `Abbrev` data should probably be moved to the `Banner`.	NO	NO
🔗	BAD_ANGLE	Angles cannot be computed for two adjacent points at the same coordinates. Instead, use `AltLabels` or fix coordinates of one point or both.	YES	NO
🔗	CON_BANNER_MISMATCH	The route's Banner field in its chopped routes .csv file mismatches the Banner field in its connected routes .csv file.	NO	NO
🔗	CON_ROUTE_MISMATCH	The route's Route field in its chopped routes .csv file mismatches the Route field in its connected routes .csv file.	NO	NO
🔗	BUS_WITH_I	Label looks like an Interstate with `Bus` banner instead of `BL` or `BS`.	YES	YES
🔗	DISCONNECTED_ROUTE	The 3 potential underlying causes are: Routes that are not connected in the first place. Roots are out of order within the `_con.csv` line. Just plain mismatched coordinates at a border, whether NMP or too far apart to be flagged. How to fix: Make sure all roots in the `_con.csv` line are in fact connected. If not, split into multiple `_con.csv` lines. Make sure the roots are in sequential order within the `_con.csv` line. Make sure coordinates match at regional boundaries. If this means changing a `.wpt` file in a region maintained by somebody else, coordinate as needed with that person on making the changes.	NO	NO
🔗	DUPLICATE_COORDS	Duplicated coordinates for two or more waypoints	YES	YES
🔗	DUPLICATE_LABEL	Duplicated labels for more than one waypoint. Labels must be unique for each route.	YES	NO
🔗	HIDDEN_JUNCTION	Concurrent route splits off at a hidden waypoint. The concurrency is most likely broken by accident, or the waypoint needs to be visible.	NO	YES
🔗	HIDDEN_TERMINUS	Hidden point at the beginning or end of a route. Routes must begin and end with visible waypoints so that users can mark all of them as traveled.	YES	NO
🔗	INTERSTATE_NO_HYPHEN	Label looks like an Interstate without a hyphen between the `I` and numerals.	NO	NO
🔗	INVALID_FINAL_CHAR	Disallowed character at end of label	YES	NO
🔗	INVALID_FIRST_CHAR	Disallowed character at beginning (after any leading `*` characters) of label	YES	NO
🔗	LABEL_INVALID_CHAR	Label contains at least 1 invalid character. Labels may only contain letters, numerals, parentheses, slashes, underscores, hyphens, or periods, with an optional leading plus sign or asterisk. An `Info` value of `UTF-8 BOM` indicates a UTF-8 byte order mark before the first waypoint, and the file should be saved as plain ASCII text.	YES	NO
🔗	LABEL_LOOKS_HIDDEN	`X123456` style label without a leading `+`.	YES	YES
🔗	LABEL_LOWERCASE	Label starts with a lowercase letter.	YES	NO
🔗	LABEL_PARENS	Number of parentheses do not match. Opened `(` must be closed with `)`.	YES	NO
🔗	LABEL_SELFREF	Label appears to reference own route. This datacheck is designed to flag cases where a line was directly copied from another `.wpt` file without the label being changed. The site update program looks at intersecting routes in order to eliminate false positives. Several error subtypes are listed in the `Info` column, with different ways they can be fixed before marking false positive: 🔗 `NO_COLOC`: An intersecting route with matching Route, Banner, and/or Abbrev wasn't found. Ensure this point has the same coordinates as intersecting/concurrent routes if appropriate. 🔗 `NO_SUFFIX`: The route forms a loop, and has two intersections with itself at these coordinates. You'll need underscored suffixes to distinguish the two waypoints, unless one or both of the labels mentions two routes. 🔗 `NO_ABBREV`: The route intersects another with the same name/number and same banner, without a city abbreviation in the label. You probably want to add the city abbreviation (without an underscore, in order to match the intersecting route's .list name) for disambiguation purposes. 🔗 `TRUE_ERROR`: The label is an exact match of the route's full .list name, including Banner and Abbrev. 🔗 `FULL_MATCH`: The label is a full match of the .list name followed by extra characters other than a slash or underscore.	NO	YES
🔗	LABEL_SLASHES	Too many slashes in label (`> 1`).	YES	NO
🔗	LABEL_TOO_LONG	Label is too long to fit in the space allocated for the DB field.	YES	NO
🔗	LACKS_GENERIC	Label begins with `Old` followed immediately by a number without a preceding generic designation type such as `US`, `E`, `A`, `Hwy`, `Rte`, etc.	YES	YES
🔗	LABEL_UNDERSCORES	Too many underscored suffixes (`> 1`)	YES	NO
🔗	LONG_SEGMENT	Long segment (distance `> 20 mi`, `32 km`) between this and the previous hidden point.	YES	YES
🔗	LONG_UNDERSCORE	Label has long underscore suffix: `> 4` characters after underscore, or `4` characters not ending in a capital letter.	YES	NO
🔗	LOWERCASE_SUFFIX	Underscored suffix starts with a lowercase letter.	YES	NO
🔗	MALFORMED_LAT	Invalid argument after `lat=` in URL which cannot be converted to a numeric value	NO	NO
🔗	MALFORMED_LON	Invalid argument after `lon=` in URL which cannot be converted to a numeric value	NO	NO
🔗	MALFORMED_URL	URL is missing `lat=` and/or `lon=` argument(s)	NO	NO
🔗	NONTERMINAL_UNDERSCORE	Label has underscore suffix before slash	YES	NO
🔗	OUT_OF_BOUNDS	Latitude `> 90°` or `< -90°`, or longitude `> 180°` or `< -180°`. False positives are allowed for the rare potential case of routes spanning the international date line.	YES	YES
🔗	SHARP_ANGLE	Sharp angle (`> 135°`) with previous and next waypoint.	YES	YES
🔗	SINGLE_FIELD_LINE	WPT file line only contains one field. Each line must have a minimum of 2 space-separated fields: A waypoint label followed by an OpenStreetMap URL. Deprecated alternate labels may optionally be included in between.	NO	NO
🔗	US_LETTER	Label uses `USxxxA` or `USxxxB` rather than `USxxxAlt`, `USxxxBus`, `USxxxByp`, etc.	YES	NO
🔗	VISIBLE_DISTANCE	Long distance (Visible distance `> 10 mi`, `16 km`) between this and the previous visible point. Not reported for active routes!	YES	YES
🔗	VISIBLE_HIDDEN_COLOC	Visible waypoint is hidden on intersecting/concurrent route(s).	NO	YES

🔗 Highway data check list

When changes to highway data have been submitted and are live on the site, all routes of the project have been checked during the site update process. They are reported on the data check list. The list is sorted by the system categories active, preview and in-development.

Check the table for errors. It is possible to filter the list by system or by region.
Click on the link in the Route column to load the route into the HB.

Fix unintended errors by using the WPT file editor.
Mark intended errors false positive.
Some of the errors are likely fixable from the information in the Log of Near-Match FPs from datacheckfps.csv. These are FP entries which have previously been added but due to minor changes, e.g. repositioning a waypoint, they no longer match the actual FP entry.

Note: It is possible to edit the url in the address bar of the browser to filter for more than one region or system. For instance, it is possible to create a link to all regions a highway data manager maintains. Save it to your browser bookmark and load it when needed for the check.

🔗 Marking errors false positive (FP)

The last column of the data check list contains the FP entry to Submit.

If no FP Entry is specified, it is always a true error and cannot be marked false positive.

If a FP entry is specified, select and copy it.

To quickly mark the whole FP entry, some web browsers allow to triple-click the entry.

Paste it into your copy of the datacheck FP list.

The list is sorted by region codes. Insert your entry at the right position.

Submit the change. The data error will disappear from the data check list with the next site update.

Do not forget checking for unmatched FPs. Remove them from the datacheck FP list.

🔗 Concurrency check

Concurrent segments of different routes are detected and considered for calculating user stats. If a concurrency is broken by accident, stats are affected. Highway data managers need to pay attention to check broken concurrencies in their regions. The check can not be done before submitting changes but is currently only possible with the data of the last site update.

The best practise to do this, is as follows: Load the region graph either with mapview or with HDX and make a manual check.

🔗 Load graph with mapview

Open mapview.
Select the region(s) you want to check.
Press OK.
Select Color by Concurrencies under Visible Routes.
Select Highlight None.
Zoom-in where you want to check concurrencies.

🔗 Load graph with Highway Data Examiner (HDX)

Open the Highway Data Examiner (HDX).
Go to Option 1 on the left and enter the name of the region you want to check.
Press enter.
Wait. The graph is automatically loaded after a few seconds but it might take longer with large graphs.

You might open up a small graph first, e.g. the District of Columbia graph, then deselect Show Markers and select whatever region graph you are actually interested in to speed up loading.

Deselect Show Data Tables to close the table to the right.
You see a map of the selected region with yellow dots.
If you hardly see anything but only yellow dots, deselect Show Markers.
You should see a map of the selected region with funny colors now.
Select Show Markers again, it was just to show you what happened.
Zoom-in where you want to check concurrencies.

🔗 Check for broken concurrencies

The colors are created by lines connecting the waypoints.

Blue line = one route only
Green line = two concurrent routes
Magenta line = three concurrent routes
Yellow line = four concurrent routes
Red line = five concurrent routes
Aqua line = six concurrent routes
Purple line = seven concurrent routes
Olive line = eight concurrent routes
Orange line = nine concurrent routes
Grey line = ten or more concurrent routes

When there is a green line - 2 routes concurrent - but interrupted by a dark blue line at visible or hidden waypoints, it's a sign that concurrencies are broken

The dark blue is created when two normal blue lines are overlapping.
The funny colors are automatically created when different lines are overlapping. Zoom-in to distinguish them.

Non-standard colors can potentially indicate broken concurrencies.
- When there is a green line (2 concurrent routes) interrupted by a dark blue line at visible or hidden waypoints, it is a sign of potentially broken concurrencies. The dark blue is created when two normal blue lines are overlapping.
- Other non-standard colors are caused by multiple lines overlapping in the same place.
When you spot errors, click on the lines or markers to get info about the route and waypoint labels. It is sometimes difficult to click all routes when they are mostly overlapped.
Fix the coordinates in the corresponding .wpt files.
Load the changed .wpt files into the WPT file editor to avoid causing unintended data errors.
Broken concurrencies of short segments can hardly be found this way. It is recommended to check NMPs to find these errors.

🔗 Near-miss points

Where two or more routes intersect, the routes must have a waypoint. If the coordinates of the waypoints are identical, the graph is connected and showroute can indicate intersecting routes to ease navigation through the routes when mapping travels. Near-miss points (NMPs) are waypoints very close together. They should be checked whether they are candidates to merge to fix broken intersecting links, and broken concurrencies.
NMP files can also be loaded into HDX to visualize their positions on a map. It is easiest to view NMPs by region as described below.

Open the NMP files filtered by region index page.
See if the region you are checking has any unmarked pairs (second column of the table).
If so, select the "HDX" link for the region you want to check.

You can see your region with some colored dots on the map now. These are NMP hotspots.

Zoom-in to investigate the points. Use the table on the left to go through the NMP hotspots.
Since all pairs of all involved routes are reported, very often more than just one line of the table on the right corresponds to a NMP hotspot.

Green dots are those which are already marked FP.
Yellow dots are off by exactly 0.000001°. This was likely done intentionally to break concurrencies and the waypoints are candidates to be marked FP.
Red dots must be checked more carefully. These could be legitmate nearby points that need to be separate (very close intersections). Or these could be broken concurrencies or intersecting routes where the waypoints do not match precisely, but should.

Click on the NMP hotspot lines or their endpoints to get info about the involved routes and waypoint labels.
Since you only see the points but not the whole network graph, you might need to open another HDX instance on load the region graph by choosing "Basic Search" and typing the region name to find the correct graph. To figure out which routes should intersect, what's going on there etc. For instance, it's possible that concurrent routes are only broken on a very short segment you don't see (or missed) when viewing individual routes.
Fix the coordinates in the corresponding .wpt files.
Load the changed .wpt files into the WPT file editor to avoid causing unintended data errors.

🔗 Marking NMPs false positive FP

Marking NMPs false positive is trickier than for simple data errors but works the same way.

The FP Entry to Submit can only be found in nearmisspoints.log which contains all NMP entries including those which are already marked FP. The list is sorted by region code.
- Best practice is to use the browser search function to find lat or lon coordinates (or route code) which one can copy from the HDX table.
- There are minimum two FP entries per hotspot but very often even more, please refer to the issue with multiple lines on the HDX table mentioned above.
The entry must be added to nmpfps.log.

The list is sorted by region codes. Insert your entry at the right position.
If the entry ends with [LOOKS INTENTIONAL], this part of the entry must be removed.
If not all entries for a NMP hotspot have previously been marked FP, only the missing entries need to be added. Entries ending with [MARKED FP] are already entered as FP.

Do not forget to check for unmatched NMP FPs. Remove them from nmpfps.log.

Credits and Sources Manual/Developer Tools/Links Raw Stats Stats/Logs Archive TM on GitHub