Broadband in South Africa

Broadband in South Africa

Whether it is mobile or fixed broadband services, consumers are demanding cheaper and faster speeds. Consumers have become brand agnostic and have become data hungry.

How do you differentiate yourself from your customer? Better services, faster speeds and rapid deployment of broadband services. This requires huge capital expenditure and exploring unknown markets.

Esri South Africa have helped our customers in the rapid build and design of their networks using the ArcGIS Platform. Using mobile applications our customers can update, capture and report on their network. This has changed the way in which our clients operate; instead of using paper based maps, our clients are enabled to manage their network from the field. With the introduction of BYOD, it helps organizations reduce hardware expenses.

Our mobile applications have enabled our clients to engage directly with the client by means of integrating CRM’s and allows the rapid deployment of their sales force into the field.

Esri helps you answer the question of “where” and solve problems. You transform your telecommunication company provide better service via your network assurance team, plan and build team and sales team. Esri provides a complete system that allows you to integrate disparate data, access and update information from the office or the field, and maintain a real-time view of all operations. More than maps and applications, Esri gives you the location analytics you need to save time, lower costs, and satisfy customers.

fibre_editing

Modelling the holiday-based redistribution of South Africans in December

MAP SERIES

Over the coming months, I’m planning on developing a map series to showcase often overlooked aspects of Cartography and GIS. The idea is to explore topical subject matter to create insightful and (hopefully) beautiful maps.

This is the first in the series.

OLYMPUS DIGITAL CAMERA

Every December hundreds of thousands of South African holiday-makers push pause on their lives and scatter across the country; making time to explore, relax and unwind.

I got to wondering if there would be a simple way of modelling this behaviour. Surely there must be some universal underlying factors that could be used to help explain where people go in December? I also knew I wanted to represent my data in a non-traditional way.

For the sake of simplicity, I limited my sights on South Africans moving within South Africa for the holiday season and eventually settled on four broad factors to consider:

  • F1 [-] Distribution of population during the rest of the year
  • F2 [+] Accessibility (using major roads as a proxy)
  • F3 [+] Distribution of holiday accommodation
  • F4 [+] Distribution of National Parks

There are obviously many more factors at play however these four seemed to interact spatially in a dynamic enough way across the country that I was happy to move forward with my investigation.

The density per factor was calculated per municipality, normalised across the country and combined into an equation that attempts to model the interaction between these factors as a linear function.

equation

In the formula, population density acts as a push factor – people will be moving away from areas of high population density towards areas with low population density. The availability of accommodation, how accessible the area is and the distribution of national parks all act as pull factors.

The amount that each factor contributes towards the final index is controlled with weights and the global difference within each variable is exaggerated by squaring it’s normalised value to highlight the most favourable areas more clearly.

The final index can be used to rank order each municipality based on the likelihood that it will be visited in December by people who do not live in that region.

These values were then used to generate the following cartogram:

dec_mapseries_cartogram_screenshot

  • You can explore the map right down to the municipal level
  • The shades of blue represent the percentage change in surface area relative to the region’s usual size. This is affected by the rank as well as the relative difference in the ranks surrounding the area.
  • National parks are included as well as major cities as you zoom in for context
  • The top 20 sites are highlighted with the concentrically banded points
  • Clicking anywhere on the map will return the overall rank for that region

Cartograms have been around since the 1800s. They provide us with a new perspective to our world by taking a thematic variable and typically substituting it for the area of the land that it represents.

The creation of cartograms comes with several challenges as regions must be scaled and still fit together. A recent (2004) and popular method of generating contiguous cartograms is the Gastner-Newman Method. This method is faster, conceptually simpler to understand and produces easily readable cartograms. The algorithm guarantees topology and general shape preservation (albeit with some distortion). This method allows its users to choose their own balance between good density equalization and low distortion of map regions, making it flexible for a wide variety of applications.

Now I need YOUR help.

Taking this one step further, I’ve configured a crowd sourcing web application which will allow users to post about their holiday destinations in a collaborative manner.

You will be able to access this from anywhere on any device and see information contributed by all users of the application. My hope with this is that this information will further support the outcome of the formula and cartogram produced in this exercise.

destinationwhere

Please share far and wide and happy holidays!

Mathematically Verifying South African ID Numbers with Survey123

This blog post describes how South African ID numbers can be verified mathematically in Survey123. South African ID numbers have the following format:

{YYMMDD}{G}{SSS}{C}{A}{Z}

YYMMDD : Date of birth.
G  : Gender. 0-4 Female; 5-9 Male.
SSS  : Sequence No. for DOB/G combination.
C  : Citizenship. 0 SA; 1 Other.
A  : Usually 8, or 9
Z  : Control digit

The most challenging part of verifying the ID number is the control digit which is calculated by using the Luhn algorithm – this will be the focus of this blog post.

The best way to tackle complex mathematical functions in Survey123 is to break it up into separate mathematical calculations and using calculated fields:

The check digit is the last digit of the SA ID number so it can be retrieved with the following function: substr(${idnr}, string-length(${idnr}) – 1, string-length(${idnr})) where ${idnr} refers to the captured ID number.

Once you have an understanding of the substr() function the rest of the calculations used to verify the ID number is pretty much straight forward.

The survey’s XLSForm file can be found here (and can be freely used): XLSForm for SA ID Number verification

  1. Copy the file to your downloads folder
  2. Open Survey123 Connect
  3. Create a New Survey and base it on an existing file
  4. Choose the Excel file that you have downloaded
  5. Your survey will be generated

Notes:

  1. The SA ID Number does not indicate if a user was born in 19yy or 20yy so both options are catered for – with a logic test to see if the birth date is in the future (age not greater than zero)
  2. Race is no longer indicated in the SA ID Number

Make your own 3D web app

jsapiWe all know that ArcGIS comes with a very large number of out of the box apps that seemingly do just about everything! Surprisingly though, we often come across the problem of finding an app that “fits just right” with what you need to achieve. The solution to this is to build your own app using an API or Runtime. The beauty of the ArcGIS APIs and Runtimes is that they extend the platform – meaning you can leverage all the power of ArcGIS such as Server, map services, web maps, popups, geoprocessing, etc all in a front-end that suits your workflow, user needs and styling choices. This may seem daunting, but don’t be fooled, anyone can do it!

This post will take you through some basic steps to create your own custom web application using a 3D scene. We will use the ArcGIS JavaScript API 4.0 to make the app.

1. Getting started – setting up your environment

Like I said, this is really simple. There are only two things you need. Firstly, you will need an ArcGIS Online subscription to make a web scene that we will load into our custom app

Secondly, you will need is a JavaScript IDE (editor) such as Notepad++ or WebStorm (link).

And that’s it, we are ready to go!

2. Creating the web scene

Log in to your ArcGIS Online account and click the menu option entitled “Scene”. This is a familiar interface very similar to the “Map” tool that lets you other webmaps. Use this tool to create a web scene with your own data layers, popups and symbology. In my example, I have added the provinces of South Africa as a layer and extruded them to show some 3D.

blog1
Creating a web scene in ArcGIS Online

Make sure to save the scene when you are done and take note (copy into notepad) of the item id for the web scene. You can grab this from the address bar in your browser. My id is “0390e2ec01fa488a847d4e413f015cd0”.

blog2
Getting the web scene’s ID

Note that to make things easier, you can share your web scene publically. This will avoid your app needing to authenticate you when opening it (i.e. logging in). In future, you can add security to your app as required.

More info on how to create a web scene, go to this link.

3. Creating your web page structure

Open up a new, empty html file in your IDE of choice. We will first put in the basic structure of the HTML page to get us going (TIP: I have created myself a template that I can re-use as a starting point each time I want to create a new app – saving me the time of creating the structure). An HTML page consists of some mandatory sections:

  • HTML doctype declaration – so that a web browser knows that the page is an HTML file (and what version – we are using 5)
  • Head – this includes all necessary ‘meatadata’ for the web page including the styling and logic (javascript)
    • Title – be sure to set a readbale title that you will see on your browser tab
    • CSS – this is the styling file and its easiest to pull in the CDN hosted Esri stylesheet
    • Javascript libraries – this is the ‘core’ JS library that will enable all kinds of web mapping goodness, we will pull in the CDN hosted JSAPI from Esri (link)
    • Javascript – this is the logic of your won app, here will enter our code to load a scene view and any other logic we want from the page
  • Body – this is the HTML which provides the structure of the page using HTML tags (link)

Here is my basic skeleton template including the links to the CDN CSS and JavaScript libraries that make up the ArcGIS JavaScript API (v4.0):

<!DOCTYPE html>
<html>
<head>

<title>My First 3D Web App</title>

<link rel="stylesheet" href="https://js.arcgis.com/4.0/esri/css/main.css">
<script src="https://js.arcgis.com/4.0/"></script>

<script>
/* section for my custom code */

</script>

<body>
<!-- page layout will go here --> 

</body>
</html>

More info on creating HTML pages, go to this link.

4. Building the HTML page structure

Before we can add a map (or other elements) to the UI, we need to create the structure of the page itself. Now, there are many ways to skin this cat which I wont go into in this post, but I am going to use a couple of layers (“DIV”s in HTML parlance) to layout my page. This is where you can really start to have fun – since you are in full control of everything, you can choose layout, colours, fonts, graphics, etc to make you page look great!

Here is the HTML layout code that is inserted into the “body” section:

<div id="wrapper" style="position: absolute; top: 0; bottom: 0; left: 0; right: 0; background-color: rgb(200,200,200)">

  <div id="mapPanel" style="position: relative; float: left; width: 60%; height: 100%">
  </div>

  <div id="mapDescription" style="position: relative; float: left; width: 40%; height: 100%; background-color: rgb(220,220,220)">
    <p style="margin: 10px; font: normal 10pt verdana;">Welcome to my very first 3D web app! You can click around and navigate the scene.</p>
  </div>

</div>

5. Adding the JavaScript “Scene View”

Since v4.0 of the ArcGIS JavaScript API, Esri now uses a new pattern for adding maps to the view. This is done by first creating the map (which is a container) and then adding a view to that map.

Here is the code for loading my map which is inserted directly into the <script> section:

require([
  "esri/views/SceneView",
  "esri/WebScene",
  "dojo/domReady!"
], function(SceneView, WebScene) {

  var scene = new WebScene({
    portalItem: {
      id: "0390e2ec01fa488a847d4e413f015cd0" // replace with your web scene's ID
    }
  });

  var view = new SceneView({
    container: "mapPanel",
    map: scene
  });

});

Note that I have entered the Item ID from the web scene I created earlier as the reference. This means that the logic will now create a scene container, find the scene item by its id and load that into the scene as a view. Too easy!

More info on the ArcGIS JavaScript API, go to this link.

6. Revel in your mastery

And that’s it! You have created your first 3D web app. To view it, simply open the HTML page in your favourite browser (JSAPI v4.0 is supported in Chrome, Firefox, and IE 11).

Here is a screenshot of my first app:

blog3
My First 3D Web App!

Happy coding!

– Richard

ArcGIS Earth is here!

ArcGIS Earth Logo

Version 1 of ArcGIS Earth was officially released mid-January 2016. The application offers functionality to share data in a similar way that Google Earth does.

For Esri customers ArcGIS Earth offers additional value as it makes data viewing in realistic 3D and data sharing possible across the platform – from the desktop, mobile, server or custom developments, the same authoritative data can now be viewed in ArcGIS Earth.

Some advantages of ArcGIS Earth are:

1. Basemaps

ArcGIS Earth offers a choice of 10 global Basemaps ranging from street maps, to terrain and imagery at the click of a button. Simply set the Basemaps to suit the data that is displayed. This means you always have access to high quality, global data that is being constantly updated for you.

ArcGIS Earth screen shot

Multiple datasets from various online or offline sources can be viewed in context of a Basemaps of your choice.

 

2. Collaboration & Content

If you are an existing Esri client with a Portal (or ArcGIS Online) identity you have full access to your organization’s authoritative content in the form of map and feature services, which means you can do your work quicker and easier than before. Sharing data requires no conversion, saving you time and money.

3. Ownership & Security

With ArcGIS Earth you have the ability to share GIS content that is 100% secure in an existing ArcGIS Online or Portal environment.

  • The level of data access is controlled by your Portal identity. Users can only access data they have been granted access to.
  • When information is added to ArcGIS Earth it remains the property of organisation/person who published it. This is different from other software providers may keep data even after you have removed it.
  • It is not possible to extract or download data from ArcGIS Earth. You can share a view of your data without giving it away. This is great because you can rest assured that your company’s data is safe and secure while using the latest technology to do your work.

4. Save your last session

There are several setting that can be customized. Among them is the possibility to have the Start-up view to continue where you left off. This setting remembers your location, Basemaps and all the other layers that were added during your last session. This can save you time when starting to work each day!

5. Limitations

ArcGIS Earth is great as a free tool for realistic 3D data visualization and sharing data in collaboration with your colleagues and customers. As with all software, there are currently a few limitations:

  • The file based data formats are limited to shp and kmz/kml. If you wish to use data from other Esri sources it must be published to a map or feature service first.
  • Where the symbology of file based features can be changed and the popups are visible, this is not the case for feature and map services. It is not possible to change the symbology or transparency of service layers to view data in context of layers below. It is also not possible to label or set popups for a service layer.

In conclusion

ArcGIS Earth version 1 has many useful features and boasts unrivaled global Basemap content. It is a great way to share your data securely with anyone, any place, anytime. So, have a go! You can download it for free.

 

Adding a Custom Widget to the Web App Builder

Note that the functionality to deploy custom widgets in the Portal-hosted Web AppBuilder is now supported at 10.5.1. See this Esri blog for more details: https://blogs.esri.com/esri/arcgis/2017/06/30/web-appbuilder-for-arcgis-now-supports-custom-widgets-in-arcgis-enterprise-10-5-1/

The Web AppBuilder from Esri has already proven to be a powerful tool for creating new web mapping apps incredibly quickly – for us and many of our users. One challenge we keep coming across is that ‘it provides us with 80% of what the users actually need’. So, how do we go about giving them the last 20%, without building a new app from scratch with the SDKs? Well, the answer is to create your own custom widgets – this is fairly well documented by Esri (see link). But what if you have Portal for ArcGIS in your organisation and want to embed the widget in there? Well, let’s walk through the steps on how to go about doing this.

For detailed instructions and screenshots of every step, please click on the link below to view the pdf documentation:
Portal Widget Install Guide rev02

P.S. Please remember to make a backup of all the files we will be editing in these steps.

Step 1:

First copy your custom widget with all its files into the following widgets directories:

“<install_location>\ArcGIS\Portal\apps\webappbuilder\stemapp\widgets\”

“<install_location>\ArcGIS\Portal\apps\webappviewer\widgets\”

 Widget_Picture01

If you are uncertain about the structure in which your widget should be, have a look at some of the other widgets in these directories.

Step 2:

Next you will want to add your widget to the config.json files:

“<install_location>\ArcGIS\Portal\apps\webappbuilder\stemapp\”

“<install_location>\ArcGIS\Portal\apps\webappviewer\”

 Widget_Picture02

  • Set “widgetManifestsMerged”: true

  • Add custom widget to the “WidgetPool”\”widgets” section

Step 3:

Add your widget’s manifest to the widget manifest file “widgets-manifest-builder.json” to the following locations:

“<install_location>\ArcGIS\Portal\apps\webappbuilder\stemapp\widgets\”

“<install_location>\ArcGIS\Portal\apps\webappviewer\widgets\”

To save time you can copy one of the existing widget’s manifests and change the details accordingly.

Step 4:

Congratulation, you are all done!

Now all that is left is to restart Portal for ArcGIS or JavaScript Application Builder and you will be able to use your custom widget in your app builder.

Widget_Picture03