Onboarding an analytic to Lumen

In this section we detail out the process for onboarding your analytic to Lumen. This assumes you have already completed the preparing analytic stage.

Onboarding an analytic to Lumen

Start by logging in to the Lumen platform.

The first page is the Lumen landing page. This displays any apps that are available to a user. To onboard an analytic, the user must have permission to the Lightkeeper App. If this is available, click on it to proceed.

This opens up the Agent view. This displays all the components currently running in the system, and their interactions with one another. The pink circles represent data sources, pushing data into the system. The blue circles represent analytics, which consume data either from a data source or another analytic and produce results.

Starting the onboarding process

To start the onboarding process, select “Onboard New Agent” from the top navigation menu.

This opens a page providing a choice of onboarding a new data source, or an analytic. Select the analytic icon.

Uploading the Analytic file

Lumen onboarding

The first stage is to upload the analytic file on to Lumen. From the first box, select the text button labelled “Select File”. This will open a pop-up file browser. Navigate to the folder where the analytic is stored and select the appropriate file. This should be a zip file containing all the required analytic files.

Select the name field and provide a name for the analytic. This is what other users in the system will see if they have the correct permissions.

If the zip file contains multiple programming files, the user may be required to select the correct one which will execute the analytic. Selecting the Executable field will provide a drop-down menu with all the available files for selection.

Select the description field to add some additional details of what the analytic does. This allows other users in the system to understand the analytic, and if they can use it in their own workflows.

When complete, select next from the bottom navigation bar. In the background Lumen will create a new environment for the analytic within its own container, and customise this environment with the correct libraries and dependencies required to run.

Analytic inputs

The next stage defines the analytics inputs. Selecting the “+ Add Input” button creates an new input field on the form. The number of inputs created should match the number of inputs the uploaded analytic requires, NOT including the UUID.

Lumen inputs

When creating inputs, the ordering of them is important. The order of the inputs created in Lumen must match the order the parameters would get passed into the analytic. The first input created will pass in its data to the first parameter of the analytic. If the inputs are arranged in the wrong order, it can lead to incorrect executions or errors.

On the input form select “Name” to label the input field. This name is used for display purposes only.

Click on “Type” to open a drop-down menu of available data types. Select the one appropriate for the current input field.

Tags can then be applied to each input field to describe the data that is required to run the analytic. This is how Lumen integrates analytics with available data. The tags form a common language across the platform, and by using them to describe the data each input requires, the components within the system can communicate and connect together.

The more tags used to describe the data, the more selective the system will be in what it can connect to. For example, using a single tag Temperature for an input will bring back every piece of data labelled temperature available in the system. Adding an additional tag such as Celsius will now refine the data retrieved to temperatures labelled only as Celsius. Adding a tag representing a location, such as Glasgow will reduce the connections further to only temperature data, measured in Celsius, for Glasgow.

Multiple inputs cannot share an identical set of tags. If two inputs share the exact same tags it would lead to duplicate values being passed in for every execution.

There are multiple ways to add tags to a field. Clicking on the Tag textbox under the field name will open a drop-down menu of all tags. As the user types this will automatically refine this list based on what is being entered. Clicking on the item or pressing enter will add the tag to the field.

On the right-hand side of the page is also a Tag search box. By entering terms here, a list of relevant tags will be returned. Once the correct tag has been identified, it can be selected and dragged across to the desired input. Using the search box also allows tags to be applied to multiple fields at the same time. When dragging the tag, an “Apply All” box will appear on the screen. Drop the tag into this box to apply this tag to all available inputs.

Lumen tagging

Once all the inputs have tags applied, select next from the bottom navigation menu.

This moves to the stage where the analytic outputs can be defined. This allows Lumen to understand the data the analytic is producing and provides an opportunity for the user to describe this data so that other components in the system can use it.

Results and outputs

Lumen defining outputs

The process of defining outputs is very similar to creating inputs. Select the “+ Add Output” button from the top right-hand corner to add the number of outputs required.  Each output can then have a field name and type selected.

One important distinction between defining inputs and outputs is that for outputs the field name is important and must match the name of the results from the analytic. When creating the analytic all outputs must be captured in a dictionary, and the keys used in this dictionary will become the field names in Lumen.

Tags the can then be applied to the output fields. These tags are what allows other components in the system to discover the results being produced and use it for their own analysis. When applying tags to the output of an analytic, they should be as descriptive as possible.

Once tagging has been completed, select Next from the bottom right-hand navigation menu.

The next stage configures how the analytic should be executed.

Execution condition

Selecting execution conditions provides a drop-down menu with a couple of options for how to run the analytic. If the user selects “Any Input Received” this sets the analytic to run as soon as any input receives a new piece of data. If the user selects “All Input Received” this sets the analytic to run only once all the inputs have received new data.

The user can also choose to ignore identical data or not. If yes is selected, sequences of identical data will only produce one execution result. This is useful for APIs or systems that broadcast data on repeat until new measurements appear. If no is selected results will be produced for every input regardless if it is identical or not.

Lumen run conditions

The final option is to set if the results from the analytic are to be stored or not. If “Save Outputs” is set to “No”, then the analytics’ results will not be stored in Lumen. These results can still be used in the system by other analytics, and form parts of pipelines, but the results will not be available for viewing or graphing within the platform. If “Yes” is selected the results will be stored and can used for visualisations and graphing.

Click Next on the bottom right-hand side once all options have been set.

Preview agent

Lumen preview

The final screen provides a preview of the analytic before it is fully deployed. This allows the user to see the interactions their analytic will have with the system and evaluate if it has made the correct connections to both upstream input sources, and any downstream analytics.

As on the main page, pink circles represent data managers, sources of data being brought into the system. Blue circles represent analytics that process the data. There are now also purple circles, which represent the input and output parameters from these sources.

The user should confirm all the input parameters to the analytic are connected to a relevant output. If any input does not have a connection the analytic will not run, and the user will need to revisit the Inputs section to update the tags. Input parameters may be supplied by many sources.

If all the inputs are fulfilled, the user can select finish from the bottom right hand navigation menu. This will deploy the analytic to the cloud, integrate it with all the identified data sources, and begin to execute it as configured.

If the analytic has been deployed, the user should see the Success screen. Here they can then go back to the main page of Lumen or begin to onboard another analytic.