How to add new indicators to existing core layers of a data catalog

Related posts:

This article is particularly aimed at clients that use the InstantAtlas National Data Service and wish to add their own new indicators to their existing Data Catalog. It is assumed that you have already followed the instructions on the Data Catalog | National Data Service Integration page.

Before carrying out any of the steps described below, we recommend that you read the full article to make sure that you have a clear understanding of the entire process. You should make a plan of what you need to do and note down the names that you want your geodatabases, hosted layers and fields to have. Think carefully about these and be consistent, as it may be difficult to change them later once you have created outputs (reports or dashboards) based on the new core layers and data. Contact support@instantatlas.com if you are unsure about anything before you start.

For this example, let’s assume that you would like to load some fuel poverty indicators for four of the existing core layers (LSOA, UTLA, Region and Country) containing four years’ worth of data (2012 – 2015). These are being treated as completely new indicators for the data catalog.

To load new indicators into your data catalog, the data needs to be saved in your ArcGIS Online account as one or more feature layers . If you want to use the data in either the Report Builder or Dashboard Builder apps, and you wish to see comparison values in the outputs, it is essential that the feature layers contain relationship classes between the core layers and also between the data layers. The Comparison Areas and Relationships page provides further information on this topic. At time of writing it is not possible to create relationship classes within ArcGIS Online so you will need to use either ArcGIS Pro or ArcMap to create the feature layer(s).

When you received your master table and metadata table from the Geowise Support Team you would have also been sent a zip archive containing a file geodatabase with the core layers of your data catalog, likely called <MyOrganisation>_CoreLayers.zip. If you have not received this, please contact support@instantatlas.com. This file geodatabase is a good starting point as it already contains the relationship classes between the core layers. You should extract the zip archive and take a copy of the file geodatabase. Keep the original one as a reference and rename the copy so that the name describes the data you wish to load; for this example, this could be FuelPoverty.gdb. Now connect this file geodatabase to your ArcGIS Pro project. To do this, right-click on Databases in the Catalog Project pane and select Add Database. Then browse to the file geodatabase.

Expand the contents of the database and you will see the core layers of your data catalog and the relationship classes between them.

As the new indicators do not have data for wards, you can delete the Ward layer. This will automatically delete all related relationship classes as well. The database now only contains the four layers you would like to load data for, as well as their relationship classes. As you want to add data to these layers, you no longer call them core layers but data layers.

Now let’s have a look at the required format of the data you wish to load. The data needs to be in CSV format, one file per data layer. The first row needs to contain the column headers and each file needs to contain a column with the feature codes. These codes need to match the code column in the respective data layer in the file geodatabase so that you can connect the data to the data layers. The column headers of the indicator columns should be specific enough to be unique and identifiable. To save having to rename the indicators later, it is advisable to enter the full indicator names into the column headers. You should also include the dates in the column headers, separated from the indicator name by a pipe character ‘|’. The dates should be sorted in ascending order. The image below shows the CSV file for the UTLA data layer.

Now you can load the CSV files into your file geodatabase as tables. To do this, right-click on the database and select Import – Table.

The Geoprocessing Tool Table To Table opens. Select one of the CSV files as Input Rows and give it a suitable Output Name (this table is just a temporary item in the database so the name does not matter).

It is now important to check that all of the output fields are loaded in the Field Map with their correct data type. Click on the first field and select Properties. The Type property is derived from the first value of the field so you should check that it is correct for each field and adjust if necessary. For example, if you add a rate indicator and the first value of the column happens to be an integer value, the whole field will be imported as Long Integer, stripping the decimal places from all other values of the column.

Click Run to import the data. When complete, the table will appear within your database in the Catalog Project pane. To check whether the import was successful, you can add the table to a map. You can open the table and see the values if you right-click on the entry in the Content pane and select Open.

Repeat the data loading steps for each of the CSV files.

The next step is to join the data from the imported tables to the data layers. In the Geoprocessing pane, click the Back icon and find the tool called Join Field. Choose your data layer as the Input Table (you can drag and drop it from the Catalog Project pane) and the imported table as the Join Table. Select the matching code fields as Input and Output Join Field.

Open the Join Fields drop down and toggle all checkboxes to select all fields. There is a button at the bottom of the drop-down list that does this for you. You may wish to uncheck the columns containing the feature code and names as they will already exist in the data layer.

Then click Add and Run to join the selected fields to the data layer. The layer will automatically be added to your map if you have one open. You should check that the join was successful by opening the attribute table of the layer (right-click on the layer in the Content pane and select Attribute Table).

Once the joins are complete, you can delete the temporary tables you imported from the CSV files from the database (right-click on the item in the Catalog Project pane and select Delete).

Now close ArcGIS Pro and browse to your file geodatabase in Windows explorer. Zip it and delete the ‘.gdb’ from the zip file name.

Using your Internet browser login to your ArcGIS Online account. Click on the Content menu item at the top and then select the folder you wish to upload the data to. Alternatively create a new folder and select it. Now click on Add Item – From my computer and browse to the zipped file geodatabase.

In the Contents drop down select File Geodatabase and keep the box for Publish this file as a hosted layer checked. You will have to add at least one tag (tags can help you find the item later through searches), then click Add Item.

Your file geodatabase is converted to a feature layer and you will be redirected to the item details page. Share the item with the Everyone (public) or with a specific group if you don’t want the data to be publicly visible. Please be aware that if you do not set it to be public, users will be asked to login to ArcGIS Online to see the data or any outputs you will create using it.

Now you should sign in to https://hub.instantatlas.com/ and click on the Manage Catalog button. You should see your National Data Service Data Catalog with Core Layers and Data Model for each layer.

Select one of the core layers that you have data for in your new feature layer. You may wish to add a new theme and/or subtheme using the icon on the right of the data model. The theme hierarchy is the same for each core layer so you only need create a new theme once. Select the theme you wish to load the indicators into and click on the Add Indicator Connection button.

In the dialog that opens, select the new feature layer you added to ArcGIS Online. If there are many feature layers in your organisation’s account you can use the search field or sort options at the bottom of the dialog to help you find the right one. Click Choose. Now select the correct data layer that corresponds to the active core layer in your data catalog and click OK. You will see a dialog with the fields you can add as indicators. Fields that have been formatted with a pipe character followed by a date are automatically recognised as different dates for the same indicator and will be pre-selected.

Select the indicator(s) you wish to add and click Save. The new indicators should then appear in your data model.

To check that the import was successful you can click on the icon to see the instances (dates) belonging to the indicator.

Now import the indicators for the other core layers in the same way.

You can add metadata for the new indicators by clicking the icon and then clicking on the Edit button.

If you have a large number of indicators, adding metadata through the Data Catalog Manager interface may not be the most efficient method. As an alternative, you can create a CSV file containing the metadata for your new indicators and append it to your metadata table. Please contact support@instantatlas.com for information on how to do this.