Chapter 4 Remote Kinetic EFM Dataflows

Prerequisites

Ensure you have completed all the steps in the following documents before continuing:


You should still be in the Dataflow Editor interface from the previous chapter.


Create Demo Dataflows on the Remote Broker

Connect to the ParStream Server

  1. Navigate to Data > downstream > ParStream-Import
  2. Right-click on ParStream-Import and select Add Database
  3. Use the following options:


Save All Control Data to ParStream

  1. Select Data > downstream > dataflow

We are creating these next dataflows on the remote broker, which is where the ParStream database resides and where the ParStream connection is configured above. This makes sense given the main purpose of the remote broker is to aggregate the data from local brokers.

  1. Right-click on dataflow and select Create Dataflow
  2. Name the dataflow save-all-control-data and click Invoke
  3. Expand ... > dataflow > save-all-control-data
  4. Open the save-all-control-data dataflow in the right-hand editor workspace by clicking on its dataflow icon
  5. Under the Blocks section, drag Table Operations > JSON Parser to the editor workspace Dataflow section
  6. Double-click on the new block and rename jsonParser to parsed-control-data
  7. Navigate to ... > downstream > DQL
  8. Right-click on DQL and drag Query to the editor workspace Dataflow section

  1. Double-click on the new block and rename query to get-all-control-data
  2. Select get-all-control-data and edit the following values in the right-hand Properties section:

This is Distributed Query Language (DQL), which provides a powerful, flexible way to extract published data throughout all the brokers within an EFM deployment. This particular DQL query gets any control data published by any broker the remote broker knows about. In this lab environment, the only other broker is the local broker you configured previously.

  1. Click the Invoke button associated with the invoke field
  2. Click the Table button associated with the output field
  3. Drag the value in row 0 for the value column to the parsed-control-data input field

If you don’t see any data in the table, actuate the Sense HAT joystick to have the publish-local-control dataflow on the local broker send data.

  1. Close the table
  2. Navigate to ... > downstream > ParStream-Import > historian > Tables > control_data

If you don’t see the control_data table listed, right-click on historian and select Introspect and then check again.

  1. Right-click on control_data and drag Insert Row to the editor workspace Dataflow section

  1. Double-click on the new block and rename insertRow to save-control-data
  2. Select save-control-data and edit the following values in the right-hand Properties section:
  1. Select get-all-control-data and click the Table button associated with the output field in the Properties section
  2. Drag the value in row 0 for the timestamp column to the save-crash-data ts field

  1. Close the table
  2. Select parsed-control-data and click the Table button associated with the output field in the Properties section
  3. Link the table values in row 0 to the associated column field inputs for save-control-data

Since we are filtering on our local broker to only publish released (i.e. completely pressed) joystick control actions we don’t need to store this column in the control_data ParStream table.

  1. Close the table

You will fully confirm whether this dataflow is working correctly when you configure a dashboard in the next chapter. For now you should see the save-control-data fields update when you actuate the Sense HAT and see a value of INSERTED for the Status field in its Properties section.


This dataflow saves all control data into the ParStream database to enable a use case where all data is considered operationally important. The next dataflow will show how to perform a moving average for cases where data trends are more important than the individual data points.

  1. Close the save-all-control-data dataflow by clicking the x icon next to its name above the editor workspace

Save Sampled Sensor Data to ParStream

  1. Select Data > downstream > dataflow
  2. Right-click on dataflow and select Create Dataflow
  3. Name the dataflow save-sampled-sensor-data and click Invoke
  4. Expand ... > dataflow > save-sampled-sensor-data
  5. Open the save-sampled-sensor-data dataflow in the right-hand editor workspace by clicking on its dataflow icon
  6. Under the Blocks section, drag Table Operations > JSON Parser to the editor workspace Dataflow section
  7. Double-click on the new block and rename jsonParser to parsed-sensor-data
  8. Navigate to ... > downstream > DQL
  9. Right-click on DQL and drag Query to the editor workspace Dataflow section
  10. Double-click on the new block and rename query to get-all-sensor-data
  11. Select get-all-sensor-data and edit the following values in the right-hand Properties section:

This DQL query gets any sensor data published by any broker the remote broker knows about.

  1. Click the Invoke button associated with the invoke field
  2. Click the Table button associated with the output field
  3. Drag the value in row 0 for the value column to the parsed-sensor-data input field

You should see this table data update every 3 seconds.

  1. Close the table
  2. Under the Blocks section, drag Table Operations > Realtime Recorder to the editor workspace Dataflow section

A Realtime Recorder block builds a table of data from individual data points over time.

  1. Double-click on the new block and rename realtimeRecorder to buffered-sensor-data
  2. Click on the small plus icon in the upper-right of the buffered-sensor-data block (3) times to add 3 more name and value fields
  3. Mouse over the blue dot associated with the name 0 field in the Properties section
  4. Click it and check the Pinned box
  5. Mouse over the blue dot associated with the value 0 field in the Properties section
  6. Click it and check the Pinned box
  7. Edit the following values in the right-hand Properties section:
  1. Select parsed-sensor-data and click the Table button associated with the output field in the Properties section
  2. Link the table values in row 0 to the associated column field inputs for buffered-sensor-data

  1. Close the table

If you select the buffered-sensor-data block and click the Table button associated with the output field in its Properties section you will see that it is generating a table with 60 seconds of rolling sensor data. This table is the data source for the 60-second moving average that will ultimately end up saved into ParStream. This is an example of how EFM can intelligently sample data to drive insight and efficiency or otherwise enforce a customer’s desired operational policies.

  1. Under the Blocks section, drag Tables Operations > Aggregation to the editor workspace Dataflow section

An Aggregation block aggregates columns of data into individual data points based on the specified operation (e.g. average, median, maximum value, last value, etc.)

  1. Double-click the new block and rename tableAggregation to last-ts
  2. Select last-ts and edit the following values in the right-hand Properties section:
  1. Link the buffered-sensor-data output field to the last-ts input field
  2. Under the Blocks, drag Date Time Operations > Date Math to the editor workspace Dataflow section

A Date Math block allows for adding and subtracting time periods from dates, however in this case it is used to normalize the timestamp into UTC, which automatically happens just by passing the timestamp “through” the block.

  1. Double-click the new block and rename dateMath to ts-to-utc
  2. Select ts-to-utc and unpin the following fields by right-clicking on the blue dot and unchecking the Pinned box:

This will make the block much smaller and easier to position in the editor workspace by only exposing the needed input and output fields.

  1. Link the last-ts output field to the ts-to-utc input field

  1. Under the Blocks section, drag Table Operations > Aggregation to the editor workspace Dataflow section
  2. Double-click the new block and rename tableAggregation1 to avg-temp_c
  3. Select avg-temp_c and edit the following values in the right-hand Properties section:
  1. Link the buffered-sensor-data output field to the avg-temp_c input field
  2. Under the Blocks section, drag Table Operations > Aggregation to the editor workspace Dataflow section
  3. Double-click the new block and rename tableAggregation2 to avg-press_hpa
  4. Select avg-press_hpa and edit the following values in the right-hand Properties section:
  1. Link the buffered-sensor-data output field to the avg-press_hpa input field
  2. Under the Blocks section, drag Table Operations > Aggregation to the editor workspace Dataflow section
  3. Double-click the new block and rename tableAggregation3 to avg-humidity
  4. Select avg-humidity and edit the following values in the right-hand Properties section:
  1. Link the buffered-sensor-data output field to the avg-humidity input field

  1. Navigate to ... > downstream > ParStream-Import > historian > Tables > sensor_data
  2. Right-click on sensor_data and drag Insert Row to the editor workspace Dataflow section
  3. Double-click on the new block and rename insertRow to save-sensor-data
  4. Select save-sensor-data and edit the following values in the right-hand Properties section:
  1. Link the ts-to-utc output field to the save-sensor-data ts field
  2. Link the buffered-sensor-data value 0 field to the save-sensor-data station field
  3. Link the avg-temp_c output field to the save-sensor-data temp_c field
  4. Link the avg-press_hpa output field the save-sensor-data press_hpa field
  5. Link the avg-humidity output field to the save-sensor-data humidity field

As before, you will fully confirm whether this dataflow is working correctly when you configure the dashboard in the next chapter. For now you should see the averaged save-sensor-data fields update every 3 seconds (the ParStream INSERT itself only happens every 60 seconds per the interval field) and see a value of INSERTED for the Status field in its Properties section.

  1. Close the save-sampled-sensor-data dataflow by clicking the x icon next to its name above the editor workspace
  2. Close the Dataflow Editor

On to Chapter 5 Data Visualization

left-chevronright-chevron