Chapter 3 Local Kinetic EFM Dataflows

Prerequisites

Ensure you have completed all the steps in the following documents before continuing:

• Chapter 2 Kinetic EFM Configuration

You should still be in the Dataflow Editor interface from the previous chapter.

Create Demo Dataflows on the Local Broker

Make sure your Raspberry Pi is connected and running (with green status LEDs) before starting this section. If not, ask the instructor for assistance.

Publish Unfiltered Local Sensor Data

1. Select ... > l-broker > downstream > dataflow
2. Right-click on dataflow and select Create Dataflow
3. Name the dataflow publish-local-sensor and click Invoke
4. Expand ... > dataflow > publish-local-sensor
5. Open the publish-local-sensor dataflow in the right-hand editor workspace by clicking on its dataflow icon
6. Expand ... > l-broker > downstream > mqtt > local-broker > Data > sensor

7. Under the Metrics section, drag pi-sense to the editor workspace Dataflow section

If you don’t see any data listed under Metrics from pi-sense check that the Raspberry Pi is powered on and the status LEDs are green. Make sure you have correctly set up the MQTT DSLink and MQTT configuration as detailed above as well.

If you do see pi-sense data listed, you probably noticed that it is changing every 3 seconds. The EFM Dataflow Editor shows data updates in realtime.

8. Double-click on the new block and rename pi_sense to pi-sense-sensor-mqtt
9. In the right-hand Properties section, mouse over the lastUpdate field until a blue dot appears

10. Click on the blue dot and check the Pinned box

Pinning a field adds it to that block’s listed fields in the Dataflow section so it is easy to link it to the input field of other blocks.

11. Navigate to ... > l-broker > data
12. Right-click on data and drag publish to the editor workspace Dataflow section

13. Select the new block and edit the following values in the right-hand Properties section:

• autoRun: true
• Path: /data/sensor/pi-sense

Make sure you are updating the Path property toward the bottom and not the path property near the middle of the Properties section. The Path is how other EFM brokers are able to subscribe to the local broker’s published data.

14. Link the pi-sense-sensor-mqtt value field to the publish Value field by dragging the red arrowhead from one block to the open spot on the other

If you ever accidentally link the wrong fields, you can simply double-click the arrowhead and the link will be removed. If you ever drag the wrong block to the editor workspace Dataflow section, just select the block and hit the delete key on your keyboard.

15. Link the pi—sense-sensor-mqtt lastUpdate field to the publish Timestamp field

You can confirm that the data is successfully published by navigating to ... > l-broker > data and expanding the sensor > pi-sense entry under the Metrics section. The data values should change every 3 seconds, just like the source data. This is an example of how EFM can publish data upstream as it is received locally without modification. The next dataflow will add some additional filtering logic to its publish activity.

16. Close the publish-local-sensor dataflow by clicking the x icon next to its name above the editor workspace

Publish Filtered Local Control Data

1. Select ... > l-broker > downstream > dataflow
2. Right-click on dataflow and select Create Dataflow
3. Name the dataflow publish-local-control and click Invoke
4. Expand ... > dataflow > publish-local-control
5. Open the publish-local-control dataflow in the right-hand editor workspace by clicking on its dataflow icon
6. Expand ... > l-broker > downstream > mqtt > local-broker > Data > control
7. Actuate (move around) the Raspberry Pi Sense HAT joystick to make some control data appear

8. Under the Metrics section, drag pi-sense to the editor workspace Dataflow section

If you don’t see any data listed under Metrics from pi-sense when actuating the joystick check if the sensor MQTT data is still coming in every 3 seconds. If not, double-check the status LEDs on the Raspberry Pi and make sure it is still connected to the MQTT server. Ask the instructor for assistance if needed.

9. Double-click on the new block and rename pi_sense to pi-sense-control-mqtt
10. In the right-hand Properties section, mouse over the lastUpdate field until a blue dot appears
11. Click on the blue dot and check the Pinned box
12. Under the Blocks section, expand Table Operations
13. Drag JSON Parser to the editor workspace Dataflow section

This block will parse the JSON data that is received over MQTT and convert it into table format. Take a minute to look at all the different blocks that are available for dataflow logic and processing. As you create progressively more complex dataflows below you will become familiar with many of these blocks.

14. Double-click on the new block and rename jsonParser to control-table
15. Link the pi-sense-control-mqtt value field to the control-table input field
16. Navigate to ... > l-broker > data
17. Right-click on data and drag publish to the editor workspace Dataflow section
18. Select the new block and edit the following values in the right-hand Properties section:

• autoRun: true
• Path: /data/control/pi-sense

19. Mouse over the enabled field in the Properties section until a blue dot appears
20. Click on the blue dot and check the Pinned box
21. Under the Blocks section, drag Logic > If to the editor workspace Dataflow section
22. Double-click on the new block and rename if to released-filter
23. Select control-table and click on the Table button for the output field in the Properties section to see the underlying table data

24. Drag the value in row 0 for the action column to the released-filter input 0 field

The resulting field input 0 indicator for the released-filter block will be gray with a circle indicating it’s from an external resource that can’t be easily represented graphically within the dataflow editor.

25. Close the table dialog
26. Select released-filter and edit the following values in the right-hand Properties section:

• input 1: released
• then: true
• else: false

This one if block is providing the main logic for this dataflow. It is evaluating the joystick action and filtering out only released events. This will prevent publishing unnecessary pressed and held joystick events when we link this logic to the publish block.

Most dataflows can be broken down logically to an “input-logic-output” cycle. It can be helpful to keep this in mind when trying to understand more complex dataflows. You may have also noticed that data-oriented dataflow blocks (usually used for input and output) are red while logic and operations blocks are orange.

27. Link the released-filter output field to the publish enabled field
28. Link the pi-sense-control-mqtt value field to the publish Value field
29. Link the pi-sense—control-mqtt lastUpdate field to the publish Timestamp field

30. Actuate the Sense HAT joystick

You can confirm the data is successfully published by navigating to ... > l-broker > data and expanding the control > pi-sense entry under the Metrics section. The data values should change with each completed joystick move, but only show released events. You can confirm this within the publish-local-control dataflow as well, watching the enabled field in the publish block, which should change to true during released events and stay false otherwise.

This is an example of how EFM can intelligently filter input data streams locally before they are published upstream, which is especially critical for input data that comes at an extremely high rate or is likely to contain invalid or operationally uninteresting data. The next dataflow will use even more logic to tie filtered local control actions to local display output at the edge.

31. Close the publish-local-control dataflow by clicking the x icon next to its name above the editor workspace

Local Control-Display Integration

1. Select ... > l-broker > downstream > dataflow
2. Right-click on dataflow and select Create Dataflow
3. Name the dataflow local-control-display and click Invoke
4. Expand ... > dataflow > local-control-display
5. Open the local-control-display dataflow in the right-hand editor workspace by clicking on its dataflow icon
6. Expand ... > l-broker > downstream > mqtt > local-broker > Data > control
7. Under the Metrics section, drag pi-sense to the editor workspace Dataflow section
8. Double-click on the new block and rename pi_sense to pi-sense-control-mqtt
9. In the right-hand Properties section, mouse over the lastUpdate field until a blue dot appears
10. Click on the blue dot and check the Pinned box
11. Under the Blocks section, expand Table Operations
12. Drag JSON Parser to the editor workspace Dataflow section
13. Double-click on the new block and rename jsonParser to control-table
14. Link the pi-sense-control-mqtt value field to the control-table input field
15. Under the Blocks section, drag Logic > If to the editor workspace Dataflow section
16. Double-click on the new block and rename if to held-filter
17. Select held-filter and edit the following values in the right-hand Properties section:

• input 1: held
• then: true
• else: false

Does this process seem similar to the last dataflow? It is almost identical except the if logic will filter for held events instead of released and there is no publish block (yet). The rest of the dataflow will be new.

18. Under the Blocks section, drag Logic > Case to the editor workspace Dataflow section
19. Double-click on the new block and rename case to translate-direction
20. Click the small plus icon in upper-right of the translate-direction block (3) times to make three more cases appear

The Case block allows for evaluating a single input value against an arbitrary number of different known cases and provides distinct output values for each one. Here you will be using the Case block to translate the published direction into a prettier value that will be displayed on the Sense HAT LED matrix.

21. Select translate-direction and edit the following values in the right-hand Properties section:

• case 0: up
• then 0: N
• case 1: down
• then 1: S
• case 2: right
• then 2: E
• case 3: left
• then 3: W
• case 4: middle
• then 4: *

22. Select control-table and click on the Table button for the output field in the Properties section to see the underlying table data
23. Drag the value in row 0 for the action column to the held-filter input 0 field
24. Drag the value in row 0 for the direction column to the translate-direction input field
25. Close the table dialog
26. Under the Blocks section, drag String Operations > Concatenate to the editor workspace Dataflow section
27. Double-click on the new block and rename concat to display-json
28. Click the small plus icon in the upper-right of the display-json block (1) time to make another input field appear

The Concatentate block joins text strings together in the input order listed. You will use it to form the JSON message that will be published to the display topic via MQTT back to the Raspberry Pi so it will display the specified text message.

29. Select display-json and edit the following values in the right-hand Properties section:

• input 0: { "action": "text", "text": "
• input 2: ", "color": [0, 0, 255] }

30. Link the translate-direction output field to the display-json input 1 field
31. Expand ... > l-broker > downstream > mqtt > local-broker
32. Right-click on local-broker and drag the Publish action to the editor workspace Dataflow section

33. Double-click the new block and rename publish to set-display
34. Select set-display and edit the following values in the right-hand Properties section:

• topic: display/pi-sense

35. Mouse over the enabled field in the Properties section until a blue dot appears
36. Click on the blue dot and check the Pinned box
37. Link the held-filter output field to the set-display enabled field
38. Link the held-filter output field to the set-display invoke field

Why use both the enabled and invoke fields for this block? This approach will “debounce” the joystick output so that only direction changes are pushed to the display rather than an ongoing set of joystick held events, which would overwhelm the display with continuous messages.

39. Link the display-json output field to the set-display value field

40. Hold the joystick toward a certain direction and confirm that the Sense HAT displays the appropriate message

If the Sense HAT is blank, double-check your display-json block and make sure there are no typos in the input 0 and input 2 fields. Ask the instructor for help if you get stuck.

This dataflow showcases a major feature of Kinetic EFM and fog computing: the ability to resiliently perform a multi-part input-logic-output cycle with both filtering and translation entirely at the edge in the local broker. If this were a roadway, factory, utility substation, or offshore drilling platform, this entire dataflow would continue to run locally even if the upstream connection were disrupted.

41. Close the local-control-display dataflow by clicking the x icon next to its name above the editor workspace

On to Chapter 4 Remote Kinetic EFM Dataflows