Mathematically Verifying South African ID Numbers with Survey123

This blog post describes how South African ID numbers can be verified mathematically in Survey123. South African ID numbers have the following format:

{YYMMDD}{G}{SSS}{C}{A}{Z}

YYMMDD : Date of birth.
G  : Gender. 0-4 Female; 5-9 Male.
SSS  : Sequence No. for DOB/G combination.
C  : Citizenship. 0 SA; 1 Other.
A  : Usually 8, or 9
Z  : Control digit

The most challenging part of verifying the ID number is the control digit which is calculated by using the Luhn algorithm – this will be the focus of this blog post.

The best way to tackle complex mathematical functions in Survey123 is to break it up into separate mathematical calculations and using calculated fields:

The check digit is the last digit of the SA ID number so it can be retrieved with the following function: substr(${idnr}, string-length(${idnr}) – 1, string-length(${idnr})) where ${idnr} refers to the captured ID number.

Once you have an understanding of the substr() function the rest of the calculations used to verify the ID number is pretty much straight forward.

The survey’s XLSForm file can be found here (and can be freely used): XLSForm for SA ID Number verification

  1. Copy the file to your downloads folder
  2. Open Survey123 Connect
  3. Create a New Survey and base it on an existing file
  4. Choose the Excel file that you have downloaded
  5. Your survey will be generated

Notes:

  1. The SA ID Number does not indicate if a user was born in 19yy or 20yy so both options are catered for – with a logic test to see if the birth date is in the future (age not greater than zero)
  2. Race is no longer indicated in the SA ID Number

How to save over 70GB of hard drive space in one click!

Drives

Recently I found myself wondering where exactly all the space on my hard drive was going. One day it was there, and the next it was gone.

I did my usual Windows clean-up but still wasn’t happy with the outcome so I did a bit more exploring into the Esri side of things to see what could be done. And the answer, quite simply is, A LOT, with absolute minimal effort!

Today I am going to introduce you to a lesser known tool from the Data Management Toolbox (and definitely finding its way into my Top 10) called Compact.

The tool does what the name implies, specifically for file (and personal) geodatabases which we all characteristically have scattered across our hard drives.

The underlying architecture of these types of geodatabases relies on binary files – as you add, remove and edit data within the geodatabase these files become fragmented which ultimately decreases the performance of your database and takes up wasted space.

What compact does is rearrange how these files are stored on your disk, reducing the overall size and improving overall performance. WIN-WIN!

To explore just how much a difference this could possibly make, I wrote a script that could iterate through all of the directories on my computer, searching for these geodatabases to perform a compact operation on them. If you’re working with a specific feature class or a database is locked for whatever reason, the script will gracefully skip over it and continue on its hunt for free space in your directories. Your overall savings may vary based on the type of work you’re doing with your databases on a day-to-day basis, I personally saw a total of 70 GIGABYTES of data released back into the system. That’s a lot of 0s and 1s.

Geodatabase Compactor

I’ve made the script into a geoprocessing tool which you can download here. If you’re the more inquisitive type, you can right click on the tool in a Catalog window and click Edit to see the nuts and bolts – it’s a very good example of Python’s os.walk function to step through files and directories.

You can choose the nuclear option like I did, and scan an entire drive, or choose a specific directory for it to iterate through.

If you have background geoprocessing enabled, progress messages will be logged to the Results Window.

Depending on the number of geodatabases you have on your PC, the first run of the tool may take some time. Subsequent runs will be faster as your databases will already be optimised.

Happy space saving!

Check to see if a field exists using Python

Ever wanted to know if a certain field exists in a feature class or attribute table? This could be to either populate it with something if it does exist or create it first if it does not exist, then populate it. The easy steps below will show you how to check if a field exists. If it does not exist, it will be created then perform a field calculation on it.

First is the code (function) to check if a field exists (Note that the green text is purely some metadata about this function):

def fieldExists(dataset, field_name):
    """fieldExists(dataset, field_name)

       Determines the existence ofa field in the specified data object. Tests
       for the existence of a field in feature classes, tables, datasets,
       shapefiles, workspaces, layers, and files in the current workspace. The
       function returns a Boolean indicating if the element exists.

         dataset(String):
       The name, path, or both of a feature class, table, dataset, layer,
       shapefile, workspace, or file to be checked for existence of the
       specified field.

         field_name(String):
       The name of the field to be checked for existence"""

    if field_name in [field.name for field in arcpy.ListFields(dataset)]:
        return True

Next we will work with this code (known as calling this function) to check if a specific field name exists in our feature class. The path to our feature class is C:\data\MyData.gdb\TestFeatureClass and the field name we are going to check for is CATEGORY. These we will set in a variable as such:

featureclass = r"C:\data\MyData.gdb\TestFeatureClass"
fieldName = "CATEGORY"

Because the fieldExists function with return a Boolean of True, we can use an if statement to do something if it does
exist. We do that by using the following:

if fieldExists(featureclass, fieldName):

Now we need to do something if it does exist. For now we will just return a message to say that it does exit (if it actually does exist in the feature class):

    print ("Yes, {0} field exists in {1}".format(fieldName, featureclass))

If this field does exist in the feature class, the message returned will look like this:

image005
At the moment, if the field does not exist, no message will be shown. This also means that if it does not exist, you cannot do anything else. What we now need to do is write something to say that if the field does not exist in my feature class, I must do something else. This is done by using the else statement under the if statement (that’s logical, don’t you think). This is done like so:

else:

Pretty simple so far? Great!

Now we need to add a message to say that the field does not exist in the feature class:

   print ("No, {0} field does not exist in {1}".format(fieldName, featureclass))

If this field does not exist in the feature class, the message returned will look like this:
image008
After that and using the same indentation as the print statement you can now use something like arcpy.AddField_management() to add the missing field which needs to be populated.

The completed script looks like this (You can copy and paste the code below and re-use it in your own script):

import arcpy # don't forget to import arcpy

def fieldExists(dataset, field_name):
    if field_name in [field.name for field in arcpy.ListFields(dataset)]:
        return True

featureclass = r"C:\data\MyData.gdb\TestFeatureClass"
fieldName = "CATEGORY"

if fieldExists(featureclass, fieldName):
    print ("Yes, {0} field exists in {1}".format(fieldName, featureclass))
else:
    print("No, {0} field does not exist in {1}".format(fieldName, featureclass))
	# arcpy.AddField_management()
  

6 things you can do with ArcGIS in 60 minutes or less

Stopwatch

The ArcGIS product stack can sometimes be overwhelming. Quite often I get asked if we “can do something” – before I even hear the “thing” my answer is yes! The challenge is always how to do it and with which set of tools. So, with all this amazing technology, we sometimes forget how easy it is to do the basic things – getting data into the system and sharing it in powerful and meaningful ways. So here is my list of 6 things you can do in a very short amount of time with ArcGIS, do you have any other ideas?

1

Make your own field data app

Want to capture pictures and the location of graffiti in your neighbourhood? The location and a photo of birds on a walk in the bush? Just open an editable web map on Collector for ArcGIS (iOS and Android) and start capturing! More info here.

2Create a website that tells a story with maps

Want to show your friends all the places you have visited this year? Or maybe want to show off sights of your neighbourhood in your local community meetings? Use a web map and create a story map using a template and your data and share! More info here.

3

Configure a mobile-ready web app

You want to be able to create a native mobile app for your children’s school to use on their outings? Or to brand an information app for your cousin’s tour company? Use a web map and configure the app with the AppStudio for ArcGIS! More info here.

4Deploy an app to any mobile platform

Want to create a basic mapping app that allows people to click on a feature and get a popup? Or change the basemap to imagery and view their own house using their mobile phone? Use a web map and configure a web app using the Web AppBuilder for ArcGIS! More info here.

5Add your Excel data to a map

Have some Excel data about schools in your province and want to add them to a map? Or maybe you have some Excel information about households in your local club and want to add them to a map? Use Excel with Esri Maps for Office and you can quickly add that data to the map using a wizard! More info here.

6Embed a map into your existing website

Have an existing website and want to include a map with directions to your office or house? Use a web map and embed it using code provided for you! More info here.

Getting domains into the Geodatabase the easy way

verosha1We are often faced with the challenge of having to create (or update) a vast number of domains within a Geodatabase. The common way to do this is through the Properties dialog in Catalog – but this is not helpful if you have tens or hundreds of domain values to type in.

Table to Domain to the rescue! This is a tool in the Data Management toolbox that you can use to directly translate a table (Excel, csv, database table, etc.) into a Geodatabase domain. Since it is a GP tool, you could add it to a model and schedule it to run periodically too – this is handy if you ever need to keep your domains up-to-date from another system.

The tool aims to save the user time, specifically if the original state of data is in an excel format. With this tool users are able to create or update a coded value domain with values from the table. One of the main benefits of domains is to reduce errors in data entry by eliminating invalid data entries. It also reduces data entry time by creating a series of automated drop-down menus.

Here is an easy 3-step guide:

Step 1: Add the table to ArcMap

Input data in Excel

Step 2: Search for the geoprocessing tool, Table to Domain. Fill out the parameters as follows:

Geoprocessing Tool

Step 3: Click OK, the tool will take a moment to complete

You can now see the domains in the Geodatabase properties window as follows:

Domains in the Geodatabase

For more information, see the online help at http://resources.arcgis.com/en/help/main/10.1/index.html#//001700000025000000