Home Security/Automation System - Sockets

Design Update  (April 25, 2014)
Previous     Next     Table of Contents

The design currently consists of a set of three (3) distributed processes, each running on a different node.  Messages between the processes are via Sockets, with the messages being JSON messages.

• The Sensor: running on a BeagleBone Black, written in Python.  This reads the various sensors and reports any state changes (e.g., from open contact to closed contact) to the Controller.  It uses Sockets to communicate with the Controller, the only process it directly communicates with.  It is dedicated to this work.  Testing is performed via py.test.  In a Model View Controller (MVC) paradigm, this is the Model.
• The Controller written in Kivy for Python: running on a desktop node, currently an Ubuntu 13.04 system.  This system stays up all of the time.  All logging will be located here.  It communicates with the Sensor, the GUI, and the outside world, all of which are via Sockets.  This node is not dedicated to this activity; it is one of many activities taking place on this node.  In a MVC paradigm, this is the Controller.
• The GUI: running on multiple nodes on the local network, possibly running on non-local networked devices, written in Python.  Uses Kivy for the GUI.  Some nodes may be dedicated, while other nodes may have other activities as well.  It communicates with the Controller via Sockets.

An issue with Sockets.

Interestingly, as I have been programming for decades, I was unaware that SOCK_STREAM type Sockets are not message oriented!  While UDP deals with complete messages, TCP deals with byte-streams.  So, the data between the processes aren't complete messages, but instead, byte-streams.  So, while you may think you are sending a whole message you've constructed, in fact, you are sending a series of byte-parts of the message.  Your code will have to reconstruct the byte-parts into a single message.

In running tests in py.test, I became aware that sometimes the messages between the Sensor module and the Controller module would fail from one test run to another, even with no code changes.  

Looking into this, I found that data placed onto the network via the sock.send, sock.recv are not guaranteed to be delivered in whole, as a complete message.  Instead, the message may be chopped up in as many small messages as the underlying network components implementing Sockets desire.  A message that Sensor is sending, that in it's entirety might be 64 bytes long, could be delivered as 64 separate 1-byte messages, or two separate 32-byte messages, or one single 64-byte messages, or any other combination.  The only guarantee is the order they will be received; they will be received in the correct order.

This means the sock.recv call in the Controller code (or anywhere a socket sock.recv is being used) must be capable of building up the complete message from multiple partial-messages.  This is usually implemented via a loop in the code.

But the problem now becomes: how do the receiver know it's received all of the partial messages for the complete message, and now can send this message off for processing while it starts receiving the next message?

In researching this, I have found the following recommended approaches:

• Always use the same fixed-length messages, then loop until this length of message has been obtained.  This means you have to either have a message construct that content never varies, or you need to pad the message to reach some agreed-upon maximum length.
• Include the length of the message as part of the message, preferably at the message head.  (BTW - in my testing, I found that Python dictionary contents are not in any specific order, so placing a key of 'MSG_LEN' with the length value may be the last dictionary element in the stream!).  The message length does not include the length-part, just the message part.  I plan on using this method, but an issue cropped up because I'm using JSON for the messages.  This involves two messages - the first message of a known type and length (struct 'I' 4-byte is frequently recommended) that contains the length value of the following message, and the second message being the actual data.
• Delimiter notifying the end of the message.  This entails receiving byte-streams, scanning for the end delimiter, then when found, packaging the result into a final message.  This means setting aside some special byte combination that won't be replicated as part of a normal message.  This may work, as I'm in control of the messages that will be sent/received, and can dictate the message contents.  Currently, I'm not using this approach.
• Using UDP instead of TCP SOCK_STREAM.  This is a true message construct, and either the whole message is delivered, or the message isn't delivered; partial messages aren't delivered.  Since this is a system whereby a missed message may be an alert that the back door was just opened, this isn't an option for me, so I'll stay with TCP SOCK_STREAM.
• Using some sort of framework that performs the underlying message construction for me, such as ZeroMQ.  I'm hesitant to use this approach, as I'm not certain how much overhead this approach will introduce, particularly on the BeagleBone Black.  More research will need to be spent here for analysis, so for right now I'm not taking this approach, but may come back to it if the message-length approach isn't satisfactory.

JSON Message Length Issue

I thought I could just create a new message that consisted of the message length (in a string format) concatenated with the JSON message already developed.  However, the receiving end, where it rebuilds the JSON message to the original construct, objected to this non-JSON length portion.  This means the length has got to be part of the JSON message.

Decision: Using embedded message length

For right now, I'm going to use the embedded message length approach, which entails a two messages: the first message being the length of the following data content message, and the actual data content message.  If that doesn't pan out, I'll try the delimiter end notifier, then finally try the ZeroMQ.

A Home Security/Automation System - Monitor changes

Monitor Update  (April 9, 2014)
Previous     Next     Table of Contents

Was finally able to get a little bit of insight into Kivy (GUI for Python), and I've updated the Monitor Screen.

Here's a picture.

I'm not going to post the code right now, as I'm debating on whether the project has reached the point where I need to move it to github.

Anyway, here's several pictures.

The first shows the yellow separator line between Zones, all Zones NORMAL, all radio buttons ACTIVE:

The next shows Zone 1 in an alarm state (it blinks first, then I acknowledge which seals it in).  Notice on the right, I've selected various radio buttons to change the state.  Zone 1 is ACTIVE, and in ALARM.  Zone 2 is INACTIVE, Zone 3 is BYPASS, Zone 4 is MONITOR, Zone 5 is TEST.  The various colors are associated with that Zone's radio button - e.g., any Zone in BYPASS will have a BLUE Zone Button.

Kivy - Using ObjectProperty

I'm learning Kivy to use as the GUI interface to my Home Secutiry/Automation system.

This is a fairly new language, but looks to me like its going to be a good fit for my project, as well as growing in popularity.

Because it's new, there's not a lot of examples that I've been able to find, and the two books I've looked at jump a little too fast for me.

I solve this by writing a lot of small example programs to teach myself the underlying structure - this post is one of those examples.

In a MVC (Model, View, Controller), Python is performing the Control function, and Kivy is performing the View function (presentation); I don't really have a Model function in this example.

I've been having a lot of difficulty understanding the usage Kivy's ObjectProperty() call in Python.

Here's what I understand, and the examples (Python code and Kivy code) follow.

--UPDATE 4/5/14: I think it would be easier to use dynamic classes in .kv instead of the empty Python classes with 'pass'.  For example, below for Python, remove 'class Zone1' and in .kv change : to I haven't changed the code below to reflect this.
--

In Python, the following entries are made in a class:

class Zones(BoxLayout):
    labelText = ObjectProperty(None)
    basement   = ObjectProperty(None)

The two objects ('thispage' and 'basement') are references to the classes defined in the associated .ky file.

Here's the .ky file entry for those two class definitions:

:

    some_text: "This is Label text"

:

    zoneName: "Basement - Door and Windows"

Those two class definitions in the .ky file, must also have an entry in the in the .py file.  So back to the .py file to add these two classes (they aren't going to perform any logic, so I'm just setting their work to be performed to 'pass'):

class LabelText(Widget):

    pass

class Zone1(Widget):

    pass

Now back to the .ky file.  Within this file, so far, are the class definitions for Kivy - the LabelText class and the Zone1 class.  This represents the layout work that's going to be done by Kivy - the presentation.

Next I'm going to associate the Python objects 'labelText' and 'basement' with the two Python/Kivy classes 'LabelText' and 'Zone1'.

Adding a new class : to the Kivy .kv file, will give part of the association to the labelText and basement references, but it's not the final reference; there will be two more yet, where the Zone class is using the two widgets (classes) LabelText and Zone1.  

:

    labelText:   label_text

    basement:   zone1

    LabelText:

        id: label_text

    Zone1:

        id: zone1

So following Python code 'basement' entry:

    basement   = ObjectProperty(None)

Refers to the reference name assigned in Kivy .ky file to the widget id (where 'basement' is the tie to Python, and 'zone1' is the widget's ID in .ky):

    basement:   zone1

Then within Kivy, the zone1 is finally defined as the widget ID for the class instance :

    Zone1:

        id: zone1

So, bare bones, for basement/Zone1 we've got:

==== Python .py file ====

class Zone1(Widget):

    pass

class Zones(BoxLayout):

    labelText = ObjectProperty(None)

    basement   = ObjectProperty(None)

==== Kivy .ky file ====

:

    zoneName: "Basement - Door and Windows"

:

    basement:   zone1

    Zone1:

        id: zone1

Here's a PNG of the output, followed by the Python and Kivy code, with some additional features added (couple of more buttons), and a print utility (to your terminal window):

==== Python Code ====

#!/usr/bin/env python

# -*- coding: utf-8 -*-

#

#  TestProperties1.py

#  

from kivy.app import App

from kivy.uix.widget import Widget

from kivy.properties import ObjectProperty

from kivy.uix.boxlayout import BoxLayout

class LabelText(Widget):

    pass

class Zone1(Widget):

    pass

class Zone2(Widget):

    pass

class Zone3(Widget):

    pass

    

class Zones(BoxLayout):

    labelText = ObjectProperty(None)

    basement   = ObjectProperty(None)

    sunroom    = ObjectProperty(None)

    livingroom = ObjectProperty(None)

    

    def printzonenames(self):

        print("labelText: ", self.labelText.some_text)

        print("basement: ", self.basement.zoneName)

        print("sunroom: ", self.sunroom.zoneName)

        print("livingroom: ", self.livingroom.zoneName)

class TestProperties1(App):

    def build(self):

        theZones = Zones()

        theZones.printzonenames()

    

def main():

    TestProperties1().run()

    

    return 0

if __name__ == '__main__':

    main()

=======Python Code End=========

=======Kivy Code ============

#testproperties1.kv

Zones:

:

    some_text: "This is Label text"

:

    zoneName: "Basement - Door and Windows"

:

    zoneName: "Sunroom - Door and Windows"

    

:

    zoneName: "Living Room - Area"

:

    orientation: 'vertical'

    

    labelText:   label_text

    basement:   zone1

    sunroom:    zone2

    livingroom: zone3

    

    LabelText:

        id: label_text

    Zone1:

        id: zone1

    Zone2:

        id: zone2

    Zone3:

        id: zone3

    Label:

        text: label_text.some_text

    Button:

        text: zone1.zoneName

    Button:

        text: zone2.zoneName

    Button:

        text: zone3.zoneName

    

=======Kivy Code End =====    

Home Security/Automation System - Monitor Screen

Monitor Screen - under development

Previous     Next     Table of Contents

As mentioned on the previous post, I'm performing the GUI portion of the Home Security/Automation System using Kivy.

I've had quite a learning curve to get the Monitor Screen (below) to the point where it is now.  While I like Kivy, the documentation is still a struggle for me, so I have to build a lot of simple GUIs and work with them, then gradually adding complexity.

The screen below is a mockup of what I'm envisioning.  It doesn't have any logic behind it, just ability to show the zone buttons and checkboxes.

I don't like the way the checkboxes are being displayed.  With the buttons, it's easy to see the button boundary, but the checkboxes just all run together to me; can't really tell where one zone's checkboxes ends and another's begins.

So I'm researching, trying to find a way to add some sort of visual delimiter that separates the zone's checkboxes from the next zone's checkboxes.

Here's what I've got so far (first the picture, then the code - note: Kivy code is two part; a Python portion (*.py) and a Kivy language portion (*.kv).  I've posted both below.

First - the picture of the Monitor Screen (currently titled as 'Dynamic Button Text')

Here's the Python code:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
#
#  DynamicButtonText.py
#
#  http://BBQandBanjos.blogspot.com
#  

# Given a Dictionary of dynamic names,
# pick each up and assign it to the Button text.
# 

from kivy.app import App
from kivy.uix.boxlayout import BoxLayout
from kivy.uix.gridlayout import GridLayout
from kivy.uix.button import Button
from kivy.uix.label import Label
from kivy.uix.checkbox import CheckBox

class ZoneList():
    _zoneL = ["Basement","Sun Room","Den","Living Room","Front Door"]

class ZoneElements(GridLayout):
    pass

class ZoneCheckBoxes(GridLayout):
    _instance_count = -1
    _zoneNames = ZoneList._zoneL

    def __init__(self, **kwargs):
        super(ZoneCheckBoxes, self).__init__(**kwargs)
        ZoneCheckBoxes._instance_count += 1
    

class ZoneButton(Button):
    _instance_count = -1
    _zoneNames = ZoneList._zoneL
    
    def __init__(self, **kwargs):
        super(ZoneButton, self).__init__(**kwargs)
        ZoneButton._instance_count += 1

class ZoneLayout(BoxLayout):
    def __init__(self, **kwargs):
        super(ZoneLayout, self).__init__(**kwargs)
        for i in range(len(ZoneList._zoneL)):
            self.add_widget(ZoneElements())

class DynamicButtonText(App):
    pass

def main():
    DynamicButtonText().run()   #run kivy app
    
    return 0

if __name__ == '__main__':
    main()

    

========

And here's the Kivy code:


#dynamicbuttontext.kv

ZoneLayout:

:
    orientation: 'vertical'
            
:
    text: "Zone " + str(root._instance_count + 1) + ": " + root._zoneNames[root._instance_count]
#    text: "Button: "
        
:
    cols:2
    CheckBox:
        active: True
        group: "Zone " + root._zoneNames[root._instance_count]
    Label:
        text: "Active"
        
    CheckBox:
        group: "Zone " + root._zoneNames[root._instance_count]
    Label:
        text: "Inactive"
    
    CheckBox:
        group: "Zone " + root._zoneNames[root._instance_count]
    Label:
        text: "Bypass"
    
    CheckBox:
        group: "Zone " + root._zoneNames[root._instance_count]
    Label:
        text: "Monitor"
    
    CheckBox:
        group: "Zone " + root._zoneNames[root._instance_count]
    Label:
        text: "Test"


:
    cols: 2
    ZoneButton:

    ZoneCheckBoxes: