My Attempt to Protect from Ransomware

Background

About a year ago, a local newspaper published an article regarding Ransomware.  It was the first time I had heard of this scam and intrusion.  It alarmed me!  The business was able to restore their data because they had daily off-site backups; most victims weren't as fortunate.

With that in mind, I decided it was high-time to get my backups in order and methodical.

This will be a series of postings detailing what I've done, and what I'm going to do regarding this.

First Steps

Backups

Except for a single laptop, all of my various systems are Ubuntu or Raspbian; all a derivitive of linux. So, naturally, my backup plan was going to involve 'tar', 'gzip'.  The backups were going to be on USB Hard Drive devices.

I have a lot of data, as well as a lot of programs that I want to retain permanent access to, and not lose to Ransomware.  

After a lot of research, I decided I was going to implement:

• Daily Backups.  These would occur on Sunday Morning 00:30, through Friday morning 00:30.  These would be differential in nature, meaning the backup would be for any file that had changed since the last Weekly Backup on Saturday. 
• Weekly Backups.  These would be a full backup of the system.
• Monthly Backups.  Same criteria as Weekly Backups - full backup of the system.
• Image Backups: Entails bringing the system down and rebooting into FileZilla, then producing two (2) images: one of the root file system '/', and the other of home '/home'.  Performed monthly, this is an important step, as it lets me also have the ability to restore my 'virtualbox' system, which has a Windows instance on it.  Using images, I'm able to restore to a new hard drive or partition and Windows doesn't complain - it doesn't see the new hard drive or restore as being different, so it runs correctly.  This is the only way I've found to do this without Windows refusing to run.
• Backups of Backups
• Since I still have vulnerabilities associated with times the filesystems are either mounted or can be mounted if root password is obtained, the above backups can still be destroyed by someone malicious.  In order to prevent this, I have a separate BackupsOfBackups Raspberry Pi that is not connected to any network, that has it's own 5TB USB hard drive attached to it.  Once a week, I manually disconnect the USB connection from my Weekly and Monthly backups, and manually connect  them to my BackupsOfBackups Raspberry Pi, where I do a copy to the backup filesystem on that BackupsOfBackups Raspberry Pi.  I do not run any software that is located on any of the primary backup filesystems; only software that I installed on the Raspberry Pi.  I do not connect this BackupsOfBackups Raspberry Pi to the internet, even to get updates - it stays as it was when I created it.  I only use Raspbian OS as distributed from the Raspberry organization.  By doing this in this way, I see no way for a malicious person to destroy the filesystem located on the BackupsOfBackups Raspberry Pi.
• Criteria:
•  Backups would utilize 'tar' and 'gzip', as well as 'exclusion files' for tar.  For instance, I excluded the '/tmp' directory.
• Each backup type (e.g., Daily) were to be kept on their own USB hard drives.  This means at least four (4) separate USB hard drives are dedicated to this approach.  This is a personal choice; your choice could be to go to a single USB hard drive instead.
• USB hard drives would be unmounted and mains unplugged in the morning and reattached in the evening in order to shorten the window for direct attack; this however would not alleviate an attack that managed to place executables onto my system that was stored to run any time a partition was mounted.
• An automated backup script would be created to run each morning at 00:30.  This would mount the necessary hard drive partition, then perform the correct backup based on the day and date (end of month) for the system.
• Notes
• All filesystems are owned by root, and permissions set to 400, with directories set to 500.  This doesn't protect if someone has root password!
• The hard drives would be unmounted except when needed for the backup, and also disconnected from mains electrical, in order to preclude someone malicious that had somehow gained root access to mount those filesystems.  I have a multi-strip that has individual switches on it for each outlet that is very useful for this mains isolation.
• The backups are automated in a 'bash' shell script that detects the day (when to do Daily vs Weekly) and the date (month end).  It also mounts the appropriate partition at the start, and unmounts it at the end, and makes status log entries into /var/log/Backup log-files.
• If the mount fails, then the backup is aborted and an error created, as my '/' filesystem isn't large enough to store the backup.
• The 'bash' shell script is run via root's cron, every day at 00:30.
• An email is sent to me showing pass/fail.

Future

I sometimes have forgotten to manually turn on the mains switch to the USB hard drive associated with that morning's backup, resulting in the backup aborting.  I also have a larger than desired window of opportunity for someone to get into my system, as I typically turn on the mains switch for the drive around 9:00 PM, then back off around 6:00 AM (unless the backup is still running!).  I am in the process of installing a relay board (for mains isolation) that will be controlled by a Raspberry Pi 3.  This will utilize node.js to receive API commands from the main system backup 'bash' shell script, instructing the Raspberry Pi to turn on the appropriate mains switch relay at the start, then back off at the completion/abor and unmount completing.

I have purchased the relay board, and the Raspberry Pi, so am now in the process of creating a driver board in order to limit the current the Raspberry Pi was going to have to sink.  Looking at software CAD programs that will produce a perfboard (or similar) drawing output.

I'll create a new blog entry on the development of the driver board and the node.js software running on the Rapberry Pi, once I get this worked out.

Security System - Local Alarm From Kit

As part of my Security System, I wanted to place an audible alarm near whatever sensor I'm using.

My sensors are going to be networked, feeding information to the main security control.  Currently I'm using a NodeMCU which makes use of a ESP8266's WiFi network capabilities.  This also means I can send a request, from the main security control, to turn on the alarm remotely, should I desire to do so, as its under the control of the ESP8266 device.

In this post, I'm not showing any type of sensor.  However, in most cases, I'll be using a PIR or a ultrasonic sensor to set the ESP8266 in alarm output for pin 5, and then enabling the local sensor alarm sound.  So, I'm just showing the alarm device and ESP8266 here.

Because the ESP8266 is a 3.3 vdc device, it can only supply 3.3 vdc and a limited amount of current; this wasn't enough to directly drive the audible alarm, so I've added a switching transistor to supply the switching current.

Scroll down for circuit description, construction, and construction photos.

In this post, I'm showing the use of a cheap commercial Window and Door Open/Closed Sensor Alarm, that I picked up for $1.00 at a local Dollar Store.  It contains a circuit board with:

• square wave generator
• battery holder
• transformer
• on/off switch
• piezo electric speaker
• 3 1.5 vdc batteries

I'm not making use of the magnetic switch in this post.  Instead, I'm making use of the audible piezo alarm (hereafter referred to as ALARM), when enabled by the NodeMCU device.

Circuit Description
In this design (software in a later post), the NodeMCU is driving pin 5 high to 3.3 VDC.  The output of this pin is connected to a 6.8K resistor, then to the base of a 2N3904 NPN transistor, which is biased as a switch.

The negative battery post of the ALARM has been separated from the ALARM circuit board, with the transistor inserted between the negative battery post and the circuit board, with the ALARM circuit board connected to the transistor's collector, and the negative terminal connected to the circuit common ground.

When a high value (3.3vdc) is applied to the base of the transistor, it turns on, basically connecting the circuit board through the transistor to ground - the ALARM sounds.

When a negative value (0vdc) is applied to the base of the transistor, it turns off, stopping current flowing from the collector to the emitter, so that the circuit is open, and no ALARM sounds.

Any 3.3vdc supply can be applied to the base to turn the transistor on - you don't have to use a NodeMCU.

If other voltages are used, such as a 5vdc from an Arduino, then you should resize the base resistor.

The 2N3904 has a beta of around 200 (varies somewhat), and is limited to 200ma and 625mwatts.

I wanted to restrict the current through the transistor to 100ma.

For 100ma at 3.3vdc and beta of 200, this equates to:
CollectorCurrent (Ic) = BaseCurrent(Ib) x Beta(200)
Ic = Ib x Beta
Ib = Ic / Beta
Ib = 100ma / 200
Ib = .5ma

Base Resistor (Br) = Vdc/ma
Br = 3.3/.5ma
Br = 6.6Kohm

While I'm showing 3.3 vdc above, a closer approximation would be 3.3 - .7 = 2.6 vdc, which includes the voltage drop across the base-emitter, which would yield about 76 ma instead of the 100 ma.  However, that's good enough for this circuit, and the 6.8K is close enough, and it sound loud enough for a local alert for me; the main security controller will driving the loud stuff.

The battery voltage of 4.5 (3x1.5)vdc is flowing from the ALARM's batteries, and is being switched via the transistor's base of 3.3vdc.

This ALARM can be used in any device with a small voltage to switch the transistor on and off, by making adjustments to the base resistor.  I've put one, with a much larger value resistor into a box, so I can use it as a beeper for simple circuit tests and continuity tests; the higher value resistor limits the sound!

BBQ

Construction Pictures:


The device I bought for $1.00 at the local Dollar Store


The contents of the device: 3 1.5vdc batteries, a magnet, and the sensor/alarm/battery case.


Remove the battery cover, then remove the screw I'm pointing at in order to get into the back of the case.


After you've opened the case, this what it will look like.

Two board types in stock!  Both work the same for our purposes!  We are only using the red wire where it connects to the negative battery terminal post.


Newer Device Circuit Board - pointing at black blob.
While building the circuit, I realized there were two (2) versions of the device, both bought at the same time from the same Dollar Store.  This is the newer device, showing a 'black blob' where the square wave generator is located.


Older Device Circuit Board - pointing at discrete components.
This works the same as the newer board.


Unsolder the red wire from this junction - it is the negative battery post.  Leave the other end of the red wire still connected to the circuit board.  We will be inserting the transistor between this negative battery post and the red wire, so the transistor, in acting like a switch; allowing current to flow back to the negative battery post when transistor enabled, and no current when transistor disabled.


Connect a green wire to the negative batter post you just removed the red wire from above.  This will be connected to our circuit common.


Next we are going to connect a yellow wire to the red wire that was removed from the negative battery post previously.  So, solder it in already!  Our transistor will be going between this yellow wire and green wire.


This is how the circuit will function.  The green wire is connected to ground, the yellow wire is connected to the collector of the 2N3904 NPN transistor.  The emitter is connected to ground.  The base is connected to a 6.8K resistor, and then to the output pin 5 of the NodeMCU device, which will switch 0 - 3.3vdc.


There isn't anywhere in the ALARM case for the two new wires to pass, so I used my soldering iron to melt a small hole in the case, then put the wires in, and closed up the case.


Testing!  A simple test of just the wiring changes!
Before transistor and software are added to the circuit!

I've inserted the batteries (polarity as indicated in the battery compartment bottom), and made sure to slide the switch to 'On' position - it won't sound if you don't turn it on!



With everything on, and the two wires not connected, the ALARM should not be making an alarm sound.


With everything on, and shorting the yellow and green wires together, the ALARM should be making an alarm sound!


Circuit schematic.


Using the NodeMCU with software I created, testing.  High output from NodeMCU pin 5 (also showing on blue LED on NodeMCU) causes ALARM to sound alarm.


Using the NodeMCU with software I created, testing.  Low output from NodeMCU pin 5 (also showing no LED on NodeMCU) causes ALARM to not sound alarm.

I'll upload the software in a later, separate post.

BBQ

Home Security/Automation System - New Design!

Design Update  (February 17, 2016)
Previous     Next     Table of Contents

Well, if you've been following this blog from the start, then you know that I've had a lot of hurdles to overcome!

My background in programming, while extensive (beginning in 1977) and including many programming languages, did not have any real front-end presentation layer activities.  So, with no front end stuff, then no HTML, no Javascript, no Nodejs, no Express, no Jade.....just a lot of "no's!".

Since I had a lot of Java background, that was my first approach.  But it was just too heavy for my target hardware - a Beaglebone Black.  Also, I was tired of programming in it!  Since I was now retired, I decided I wanted something more efficient than Java, both from a programming standpoint, as well as a system load standpoint (Beaglebone remember!).

My search led me to Python.  I considered Scalia and Ruby, but Python looked good to me, and was something I could easily use on future code for around the house.  So, I started planning on Python.  But I ran into a problem on the front end presentation layer - not much better than what I was used to in Java!   But I kept going...see my posts on Kivy on this blog.

Life intervened, and I had to stop the project for a while.  But I kept occasionally researching, and somewhere came across nodejs.  This really peaked my interest.

I dug into nodejs (note: this is the new, correct spelling for this product; after their recent merger, then changed the name from node to nodejs).  I really liked what I saw.  That, along with the ability to push data to a HTML5 browser (I refused to do polling) meant I could move off of Kivy etc. bandwagon, and move to a browser, which would simplify a lot of things, particularly with respect to system monitoring while away from home via the internet.

So, I started leaning Nodejs, which begat getting a firmer knowledge in Javascript.  Then backtracking to HTML, then CSS - none of which I needed in my professional life before; well, OK, I did some stuff, but barely anything.  Then came Jade, and document databases, and Nedb.  Then somewhere Express.  I stayed away from CSS, Express, Jade for a while, because I was concerned about the Beaglebone being able to host all of this, but it turned out not to be a problem.

So, if you were like me, experienced programmer, but needing web based knowledge, then I was suggest the following learning path, in this order:

1. HTML
2. CSS
3. Javascript
4. Nodejs
5. Express
6. Jade

So, my redesign!

• Uses the Beaglebone Black.  I'm going to also develop on the Raspberry PI 2; will update with that info.
• Hosts a web server via Nodejs
• Uses the Google Chrome browser.  Others may work, but I'm not testing or programming to those other platforms.
• Provides dynamic web page security updates.
• Displays current weather information while no security work or events taking place.

Since the base display is weather displays, and I needed somewhere to start, I first programmed the weather display system.  The design for this is:

• Runs on Beaglebone
• Hosts a web server via Nodejs
• Uses the Google Chrome browser.
• Provides dynamic web page weather updates.
• Obtains live data from wunderground.com for my local, every 5 minutes.
• Obtains static radar jpg images from weather.gov and wunderground.
• Displays a digital clock, data from the Beaglebone pushed to web page.  If the network drops, the Beaglebone freezes, the server freezes, the browser freezes - then the seconds no longer update.  In the future, the time will also change color, and maybe an alarm is generated.
• Will switch to security mode on work or event.

The weather system is functional.  I'm currently refactoring the code and adding a few new features.

When those updates are complete, I will publish.

Banjo

Nodejs: Passing values from Express to jade to javascript!

Passing values from Express to Jade to Javascript

Note: jade version 1.11.0
Today's date: 2/17/16
Note: assumes you are somewhat familiar with the technologies mentioned (nodejs, jade, express, javascript, html)

I've been hard at work on my security system, but came across an issue that took several hours to resolve.  I'm a very experienced programmer (C, C##, Java, Python, assembly, Fortran, Basic, etc), but almost all of that development has been done on back-end stuff; not much GUI front end stuff, no HTML stuff.  So, I'm somewhat weak on the front-end stuff; just now leaning nodejs, jade, express, javascript, html!

The security system I'm designing and building requires a lot of front-end stuff; it's showing the status via sms text messages and the browser.  This means the architecture is going to make use of:

• Nodejs (note this is now the correct spelling for this product)
• Jade (used with Nodejs)
• Express (used with Nodejs)
• Javascript
• HTML

Instead of having an image of the current security system on display in the browser all of the time, I decided to have the current weather shown, along with the current time.  Weather in the Southeastern US can be quite volatile, so I have kept a radar image up on an old laptop in our living room for years, so weather is important to me to keep aware of.

The idea is the weather will be shown until either I select to view/work on the security system, or a security event occurs, in which cases the browser will display the appropriate security screen.  As part of the weather, I'm getting current weather information from the Wunderground.com API.  For items such as radar images, I'm looping through some current jpg files that are offered from weather.gov and wunderground.com

Now - the problem I was encountering.

In looping through the various web pages, I have to have a delay between pages in seconds.  I want this value to be passed in at the time the application is started, for the default value.  Later, I intend to add a change that will let me interactively change the delay time.

The current architecture, using Nodejs and Express, renders a loopingPages.jade file.  I'm passing the the DELAY value obtained at startup via process.env.DELAY in app.js, where it stores the value, before rendering the looping pages.  So, the DELAY value is passed from the startup command line to the app.js (the server), then to the jade rendered page via:

     app.get('/loopingPages', function(req, res) {
          res.render('loopingPages', {delay:DELAY}
     });

The jade file loopingPages.jade includes a script file.  So, to be clear, the value that is originating from app.js server (using Express), needs to be passed to the rendering jade file loopingPages.jade, and this file includes a javascript file loopingPages.js, where the DELAY is actually used.

Well, I couldn't get it to work.  I could get the DELAY value from the environment variable passed at startup using the 'process.env.DELAY' term, but I could not get it to propagate properly.

After a lot of searching on the internet, I was unable to find an example that matched the one I needed, and most were old.  Seems this was an exercise in futility and esoteric programming!

But, I finally got it to work!

Below is a simplified version, which I was able to get to work.  It assumes you have created a fresh nodejs Express architecture for using this, which creates the proper directory structure and contents that I'll add to.  All directory locations are relative to your base directory for the application where app.js is located.

1. Modify routes/index.js to add name:'Banjo'
        res.render('index', {title: 'Express', name: 'Banjo' });

2. Modify views/index.jade to add p My name is index.jade #{name}
It should look like this:
     h1= title
     p Welcome to index.jade #{title}
     p My name is index.jade #{name}

The #{name} will show the name value passed from the app.js on the web page.  The #{title} was already available from the initial build.

3. At the bottom of index.jade (above), add the following inline script and script src:
     script.
          var theName = "#{name}"
          var theTitle = "#{title}"
          console.log("name passed in to index.jade is " + theName)
          console.log("title passed in to index.jade is " + theTitle)
     script(src="/javascripts/indexInclude.js")

Note the new file indexInclude.js which you will create next.

4.  Using your programming editor, create the file
           public/javascripts/indexInclude.js
It should have the following content:
     console.log("This is from indexInclude.js");
     console.log("+++++++++++FINAL: IF Banjo SHOWS THEN SUCCESS!+++++++");
     console.log("indexInclude.js theName is " + theName);
     console.log("+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++");

5. Run the app.js server
6. Open your browser to your localhost:port needed for nodejs
7. Confirm you see 'Banjo' printed on the page
8. Right-click on the page, then do 'inspect element' or similar for your browser.
9. Select 'console' tab
10. The console.log item should show text that includes the above (step 4) Banjo.

I hope this has helped you!

Banjo

Python and Kivy: out of here!

Home Security System
Previous     Next     Table of Contents
Date: 9/19/15

A while back I started a Home Security project.  After examining several different approaches, I settled on Beaglebone Black for the hardware, programming with Python and presentation using Kivy.

After much work on Kivy, and continuing issues with the Linux Angstrom that was on the Beaglebone, I came to the conclusion that it just wasn't ready for prime time, and I suspended my project waiting for the technology to catch up.  This was easy to do, as I always have more to do on my plate than time to do it!

Recently, I decided it was time to take another look at this project.  I purchased a new Rev C Beaglebone Black, then started working with it.  This was a major improvement, as this version also moved from Linux Angstrom to Debian.  I was unwilling to try and work with Angstrom again, so the move to Debain was great news.

I became convinced that the pyvenv development environment would work best, but that entailed moving to Debian release 8 or 8.1 instead of the current 7x version, and Python 3.4.  So I quickly got well acquainted downloading and installing various test releases of Debian 8.1 for the Beagle.

Once the OS and Python versions issue was resolved, I set up the project on GitHub, and started moving forward, with the first task to set up do to builds on the Beaglebone while using pycharm as the IDE on my host Ubuntu system.  I mounted a host directory onto the Beagle using sshfs, so that code changes I made within pycharm would be directly available for code running on the Beagle via the shared directory.  I would edit in the IDE on my host, then use a terminal session that was logged into the Beagle to run and test the code.   This worked really well, and all my source code was maintained on my host and pushed to github as well.

With those issues resolved, I began to look into the presentation layer.  I really was concerned about trying to use Kivy after all the disappointment from a while back.  Looking at the emails from the Kivy group - there were still a lot of issues.

All of my previous work in IT had primarily been devoted to development on the back end, with very little work on front end presentations.  However, I had come across some articles regarding html5 and it's ability to work with websockets, so I wanted to take this opportunity to do a little deep dive into html5 to see how that might better fit my presentation layer problem.

The first thing was to take a look at the Beagle's onboard webserver.  I had not really taken a look at that in the past, but now with all presentation options on the table, I did take a look.

Wow!  What an insight that was!  A little bit of nodejs code, and batta-bing, I was serving up simple web pages!  And there was a direct ability to push data from the Beagle to the presentation via a web page.  This really looked like the approach I had been seeking: a user has a web page up (a security monitor), the Beagle first supplies the current state of all sensor elements, then pushes any updates (sensor state change) to be displayed on the web page without the user having to refresh the web page.

This was it!  While I was happy to drop Kivy, it also looked like the easiest approach would be to move totally away from Python and over to JavaScript as well!  That was a shock!  This also meant I no longer needed pycharm; I no longer needed the sshfs (well, it may be a little early yet to decide that), I no longer needed pyvenv, so I no longer needed Python 3.4, and that meant I no longer needed Debian 8.1 either.......holly cow!  The last weeks worth of changes to the OS, pyvenv, pycharm, Python - all now relegated to the trash heap!

I'm going to take a look at Debian, nodejs, and JavaScript to see if there's any benefits to staying with Debian 8.1 (which at this time is in 'test' mode; prelease), but I'm hoping it's not needed, so I can stay with the standard deployment.

So: this project has abandoned Python and Kivy!!!

Moving to JavaScript and Nodejs!!!

I will be documenting the development of the Beagle project, utilizing JavaScript, nodejs and bonescript here.  Stay tuned for updates!

I will be placing my detailed notes on setting up a development environment for the Beagle for Python here on this blog at a later date, as they may be of use to someone doing Python development for use on a Beagle for their project.

February 2016 update:  Recently, TOS pushed an update out that moved from Java 1.7 to Java 1.8.  This has corrected most, if not all, problems outlined below.  I do not think the information below is still needed, but will leave it here in case it helps someone now or in the future.
Thanks,
Banjo


ThinkOrSwim (TOS) on Linux!

Please check the date you are reading this - because TOS makes frequent changes, the information I'm providing here may not work for you.

I'm currently running TOS on three (3) different platforms: Ubuntu 14.04, Windows 7 (on VirtualBox in Ubuntu 14.04), and on LinuxMint 17.02.

All are working, although the Windows version is, by far, working the best.  There just doesn't appear to be much interest from TOS in making this a quality experience on a Linux based machine!

I'm not going to detail any installation on Windows - the TOS installation is fine.

Instead, these are are my 'notes' from installing on LinuxMint - it was the same as on Ubuntu.

- from a terminal, verify the version of Java you are running, using the command
        java -version

The response to this on my machine is: "build 1.7.0-88-b15".

The main thing here is that it must be version 1.7.......!  TOS will not work with 1.8..... or newer!   If you aren't running version 1.7, then you will need to resolve this before you go any further.  In my case, it took me a while to resolve, but ultimately I ended up (first) verifying I didn't have any applications using Java 1.8, (next) I uninstalled Java 1.8, then I obtained Java 1.7 from Oracle and installed it.

The following assumes you running 1.7...!


- pull the Linux distribution down from TOS website.  Specify the Ubuntu/Debian version.

- save it to your download folder.

- run the TOS install, being sure to install for just yourself, not for all users (this is very critical) at the popup window.  We won't be launching via the desktop icon, so there's no need to let it install this.  Instead, we will be launching from via a terminal shell command.

- once installed, you'll have to make some mandatory edits to some of your TOS files.

- the first edit change is associated with the amount of memory that can be used by the Java Virtual Machine.  It comes preconfigured with a really low value.  So, we're going to change this.  I'm using a starting value of 1024 MBytes, and a max value of 2048 MBytes.  That means TOS will start with a minimum of 1.024 GBytes of memory allocated, and can consume up to 2.048 GBytes.  Make sure you have enough memory to support those values, or set to your own system capabilities!

- in terminal, enter: cd  #to make sure we are starting at your home directory
- in terminal, enter: cd thinkorswim
- in terminal, enter: cp thinkorswim.vmoptions thinkorswim.vmoptions.bak #to make a backup copy you can revert to if needed.
- in terminal, enter: gedit thinkorswim.vmptions
- in gedit, find the line: -Xmx<>m  , and change the value to -Xmx2048m # I no longer know what my original values were, but seems like it was 512m
- in gedit, find the line: -Xms<>m  , and change the value to -Xms1024m #remember - adjust these values so as not to exceed the amount of free memory you have available, or else you will start swapping, and the system will be too unresponsive for you to use at that point.
- exit and save from gedit.



- the next edit change is associated with changing the TOS shell script that launches TOS.  It's located in the same directory you were in above (~/thinkorswim).  So, you are going to gedit the directory/file ~/thinkorswim/thinkorswim

- in terminal, find out where the Java executable is located, by issuing the following command in the terminal: whereis java
- my response to this contains multiple fields, separated by 'space'.  The installation on my machine makes use of file links, so I'm going to show you how mine resolved.  The first field response to 'whereis java' is the important field.  On my machine, it begins with /usr/bin/java

-- Steps to see linkage:
-- in terminal, enter: ls -al /usr/bin/java   #responds /usr/bin/java -> /etc/alternatives/java

-- in terminal, enter: ls -al /etc/alternatives/java #responds /etc/alternatives/java -> /usr/lib/jvm/java-7-oracle/jre/bin/java

--this is the actual location of the file.  You should be able to use any of these values in your thinkorswim shell script, but if you have problems, use the last one - the actual file and location.

- Verify this on your system!  If it doesn't resolve to Java version 7, then you can't use it.  I haven't tested it, but even if you have a Java version 8 installed, if you point to this final location of Java 7, then it probably should work.

- in terminal, enter: cp thinkorswim thinkorswim.bak #making a backup copy you can revert to if needed.
- in terminal, enter: gedit thinkorswim
- in gedit, find the line:  #INSTALL4J_JAVA_HOME_OVERRIDE
- you are going to remove the comment (#) character, so do that!
- next, you will need to enter (assuming you are going to use the final location from above), or the links (if they are all set up properly): /usr/lib/jvm/java-7-oracle/jre/bin/java   
- Your line should look like this:
 
INSTALL4J_JAVA_HOME_OVERRIDE=/usr/lib/jvm/java-7-oracle/jre/bin/java

(unless you used your verified link /usr/bin/java instead.  Note that these links can change if Java updates to 1.8 or newer on you at some point, so a good reason to use the hard coded version until TOS ever gets around to upgrading their JVM version reliance.)

- At this point, the changes required to get going should be complete.

- However, this would not work correctly on either my Ubuntu system, nor my LinuxMint system, if I used the desktop launcher for TOS.  However, if I opened a terminal, and from the ~/thinkorswim directory entered:
   sh thinkorswim
then my system would work correctly.  If I did it from the launcher, it may work for a while, then crash, or not work at all......no idea what the issue was.

The TOS system on my Linux based machines are not as robust nor as reliable as running on Windows, even though it is running inside of a VirtualBox instance on the Ubuntu machine!  However, it does work.  I use it on the LinuxMint node to display some graphs, while on the Ubuntu/VirtualBox/Windows node I do my trading, with a total of 4 monitors - 2 on Ubuntu/VirtualBox/Windows, and 2 on LinuxMint.

Hope this helps!!!

BBQ

Learning Kivy - Part 6

Prev

Table of Contents

Learning Kivy - Part 1

Learning Kivy - Part 2

Learning Kivy - Part 3

Learning Kivy - Part 4

Learning Kivy - Part 5

Learning Kivy - Part 6

After a LOT OF WORK, I was finally able to get some code working that incorporates the following:

• Thread, with a simulation of a Socket.  This writes to a Queue.
• Main thread - Kivy GUI thread, that reads from the Queue
• Queues

Where I've been having issues is:

• Getting both threads up and running (Thread for Socket simulation, thread for main Kivy GUI), while not causing either thread to block.  Over the years, I've written a lot of Thread software, yet there still issues.  I still don't know what the problem was.
• Getting both threads to use the same Queue.  For some brain-fart reason, I was creating two (2) Queues, one in each thread, then was surprised that the data I was pushing into the Queue wasn't available to be read on the other Queue!  Clearly a misunderstanding on my part.  I've used Queues before, and knew how they worked, so I don't even have the excuse of ignorance to justify this - it was just a plain brain-fart.  In the code, I create a global variable that has the Queue, which makes it available for each thread.  I'm not saying that's the best approach, but I wasn't trying to research the best approach for Queues, I was trying to get Kivy going.
• These were the two (2) main problems.

Here's the code.  It does the following:

1. Builds a simple Kivy Language BoxLayout (via my ShowGUI class) via the 'Bulder.load_string' command.
2. Creates a global variable q and assigns Queue instance to it.
3. Creates an instance of my SimSocket class.  The class has methods that will create some data, then push it onto the Queue.
4. Creates a Thread named 'simSocket' that links to the callback SimSocket method "put_on_queue".
5. Starts the 'simSocket' thread.
6. Runs Kivy.

Once running, the following takes place:

1. On the Terminal, initialization messages, followed by comments that say data is being written to the Queue, and shows the data.
2. A pop-up GUI appears.
3. Data is read from the queue.
4. The GUI updates with the  the data that is being pulled from the queue, at a rate of 1/60 of a second.
5. It continues to run until you stop it, or it reaches 1 million messages.

======Python Code Follows=========

#!/usr/bin/env python

# -*- coding: utf-8 -*-

#

#  threads_and_kivy.py

#  

'''threads_and_kivy.py

Trying to build up a foundation that satisfies the following:

    - has a thread that will implement code that:

        - simulates reads data from a Python socket

        - works on the data

        - puts the data onto a Python Queue

    - has a Kivy mainthread that:

        - via class ShowGUI

            - reads data from the Queue

            - updates a class variable of type StringProperty so it will

                update the label_text property.

          

'''

from threading import Thread

from Queue import Queue, Empty

import time

from kivy.app import App

from kivy.uix.boxlayout import BoxLayout

from kivy.properties import StringProperty

from kivy.lang import Builder

from kivy.clock import Clock

kv='''

:

    Label:

        text: str(root.label_text)

'''

Builder.load_string(kv)

q = Queue()    

        

class SimSocket():

    global q

    def __init__(self, queue):

        self.q = queue

    def put_on_queue(self):

        print("<----- ..threaded..simsocket.put_on_queue="" entry="" font="">

        for cntr in range(1000000):

            print(".....threaded.....SimSocket.put_on_queue(): Loop " + str(cntr))

            self.some_data = ["SimSocket.put_on_queue(): Data Loop: cntr: " + str(cntr)]

            self.q.put(self.some_data)

        print("..threaded..SimSocket.put_on_queue(): thread ends")

class ShowGUI(BoxLayout):

    label_text = StringProperty("Initial - not data")

    global q

    

    def __init__(self):

        super(ShowGUI, self).__init__()

        print("ShowGUI.__init__() entry")

        Clock.schedule_interval(self.get_from_queue, 1.0 / 60.0)

    def get_from_queue(self, dt):

        print("---------> ShowGUI.get_from_queue() entry")

        try:

            queue_data = q.get(timeout = 5)

            self.label_text = queue_data[0]

            for qd in queue_data:

                print("SimKivy.get_from_queue(): got data from queue: " + qd)

        except Empty:

            print("Error - no data received on queue.")

            print("Unschedule Clock's schedule")

            Clock.unschedule(self.get_from_queue)

    

    

class KivyGui(App):

    def build(self):

        return ShowGUI()

    

def main():

    global q

    ss = SimSocket(q)

    

    simSocket_thread = Thread(name="simSocket",target=ss.put_on_queue)

    simSocket_thread.start()

    

    

    print("Starting KivyGui().run()")

    KivyGui().run()

    

    

    return 0

if __name__ == '__main__':

    main()

BBQ Review - Hickory Pit Bar-B-Queue, Chattanooga, Tennessee

BBQ Review - Hickory Pit Bar-B-Queue, Chattanooga, Tennessee

Note: I don't publish any negative reviews at all, so you only see reviews for places I stop at and enjoy.  I don't think it's fair to write a negative review for an establishment that might have had an off day.

These guys (Mr. and Mrs. Ford) just won Fifth in a Nation-wide award from TripAdvisor!  That's no small feat; per TripAdvisor, their "community of millions of travelers" had decided that this BBQ restaurant is worthy of 5th place in the USA!

The award was posted Thursday, May 8, 2014.  I learned of it on Friday, May 9, 2014.  I was there eating a big-ol' BBQ Pork sandwich on Saturday, May 10 - the next day!

My wife and I planned the trip for Saturday, May 10th, 2014, to see relatives in Chattanooga TN that would require us to go right by this restaurant on our way to visit!

Coming from Atlanta on I-75 N., we took Exit 1B for Ringgold Ave, then headed West for one (1) mile, and located the restaurant on the right side of the highway.

It's small!  Like 2 tables outside, 4 or 5 tables inside!

We tried to time our arrival at 11:30, so that hopefully we might avoid the major crush I was expecting.  Well, I was wrong - there was already a major crush, and it just got bigger and longer as we waited....45 minutes to be served, after waiting about 15 minutes to order.

The staff was doing all they could, but they were overwhelmed.  Everyone was pleasant though, and we didn't hear anyone complain about the wait.

They were, however, in addition to taking orders from the people that stood in line to place their order at the counter, also taking orders at the drive-up window, as well as orders over the phone.  That meant anyone placing an order via their car or over the phone was jumping to the head of the line and this meant the line was at least 3 times as long as what we saw standing in line. 

At some point Mrs. Ford realized this wasn't fair to the people that had traveled to eat with them, and stopped taking any further orders over the phone, which continued to ring non-stop!

There was no where to sit for us, so we took 'take-out', and ate it in our vehicle.  That was a little difficult, because this was a huge BBQ Pork sandwich, heavily laden with sauce!  Careful attention was required, after multiple wraps with napkins, to try and keep it off of our cloths - we both succeeded, which I would have thought would be impossible!

So - what did we think?

WE LIKED IT A LOT!

Worth the wait?  No, but then nothing would have been worth the wait!  The wait will change as the excitement wears thin, or they adjust their staff to the new reality.

WOULD WE GO BACK!  YES!  After the excitement has died down somewhat.

This is ONE HUGE BBQ PORK SANDWICH.  Next time, I'll ask for it to be NAKED, so I can better judge the meat, but it's the real-deal!

BBQ in the News - Smokin' Ribs

This is from an article on smoking ribs on a bullet-style smoker that caught my attention, and wanted to share with you.

Link to complete article at HoddyToddy.com:


Excerpts from the article:

Pork Belly Slim’s Method for Mouth-Watering Barbecued Ribs

POSTED ON MAY 1, 2014

This photo, from D’Artagnan (dartagnan.com) shows ribs finished with coffee barbecue sauce (not the method used by Pork Belly Slim, but similar).

It’s that time of year. The flowers are in bloom, the weather is warm and we’re all excited to get outside after a colder winter than we’ve had in a few years.  For me, and a lot of others, spring means BBQ season is open, and it’s time to dust off the smoker, grab some charcoal and wood for smoke, and start the fire.

More in complete article at HoddyToddy.com here:

BBQ in the News - Best BBQ Restaurants

BBQ joints travelers say are best in ratings for TripAdvisor.

There are more restaurants in the articles, but the following are close to home


Article From FoxNews reporting on an article from TripAdvisor:

Wiley’s Championship BBQ, Savannah, Georgia

TripAdvisor

With more than 30 years of experience winning awards on the national barbecue competition scene, husband and wife team Wiley and Janet McCrary opened their first brick and mortar restaurant in 2008. The joint pleases palates with low-and-slow cooked meats including savory St. Louis cut ribs.

Hickory Pit Bar-B-Que, Chattanooga, Tennessee

TripAdvisor

Amid the Ridge-and-Valley Appalachian Mountains in southeastern Tennessee, this log cabin style restaurant feeds famished barbecue fans with an emphasis on Southern hospitality. Dry-rubbed meats are smoked over hickory wood, and can be served on a plate, as a sandwich, or even stuffed inside a “killer” baked potato.

Jim’s Smokin’ Que, Blairsville, Georgia

TripAdvisor

Open Thursday through Saturday, this northern Georgia smokehouse advises guest to arrive early, as their succulent meats – including ribs, brisket, pulled pork, chicken and turkey – often sell out. Those lucky enough to score a meal can also enjoy their choice of satisfying sides, from Brunswick stew to banana pudding.

Joe’s BBQ, Blue Ridge, Georgia

TripAdvisor

In the midst of the Blue Ridge Mountains, this joint satisfies hungry guests with tender and moist meats complemented by a selection of sauces including “Alabama white,” a mayonnaise-based mixture that has found fame in several Southeastern states.

Learning Kivy - Part 5

Prev

Table of Contents

Learning Kivy - Part 1

Learning Kivy - Part 2

Learning Kivy - Part 3

Learning Kivy - Part 4

Learning Kivy - Part 5

Learning Kivy - Part 6

This is a continuation of Learning Kivy - Part 4. 

As noted in Part 4 for Pong.py and pong.kv, we've come across an important type in the Player class, of NumericProperty, the instance of which is assigned to player1 class variable.

With respect to this program, it is an important part of being able to dynamically update the Label's text with the changing score.

Looking at kivy.org site to get some insight into the NumericProperty, I noticed the following:

Theses properties implement the Observer pattern. They help you to:

• Easily manipulate widgets defined in the Kv language
• Automatically observe any changes and dispatch functions/code accordingly
• Check and validate values
• Optimize memory management

I know from previous experience what an Observer pattern is, but in essence, it allows one part of code to inform other interested parts of code that something has changed - the change is an event, and those parties that have registered their interest through a callback function, are informed of this change.  Here's a more detailed explanation from Wikipedia: Observer Pattern

It appears that, by setting the class PongGame's attribute player1 to an instance Player, and Player is of type Widget, and Player has an attribute of score which is of type NumericProperty, we are making use of the Observer Pattern via the NumericProperty, which will(from kivy.org) "...automatically observe any changes and dispatch functions/code accordingly".  So that appears to be the magic that let's us change.

In other words, I'm thinking it means: when we update the 'score' attribute, since it's of type NumericProperty, then it gets updated in the pong.kv class Label text, because Label text is set to "str(root.player1.score)".  Thus the new value is displayed.

Recommended additional info
Here's a good question/answer thread on Kivy NumericProperty.  Here's a clip from it.

[Alexandar Taylor]: Properties hook into kivy's event system to reference each other in automatically updated ways, dispatch events when changed etc. In comparison, a normal python class level property...doesn't. Kivy properties also take care of class uniqueness, whereas class attributes are shared by all class instances which can cause you unexpected problems if they're mutable.

For the ReferenceListProperty, this creates a property that looks like a list, but whose elements when accessed or changed automatically reference or change the component velocity_x or velocity_y. In comparison, your example of just using a tuple does none of that - if you access velocity[0] you'll probably not even get a number but instead the NumericProperty object you defined it to point at. In contrast, the ReferenceListProperty acesses and returns the velocity_x property.

Same goes for the NumericProperty. You could create class level python attributes if you wanted (or, more usefullly, standard instance attributes created in __init__), but then you can't bind to their changes etc. with kivy's event system. Learning more about kivy will make it clear how it ties together, you generally always will be thinking in terms of properties affecting each other via the bindings that kivy properties allow.

The guy that wrote the above, Alexander Taylor, also has a series of youtube casts.  Here's one dealing with this issue.

Learning Kivy - Part 4

Prev

Table of Contents

Learning Kivy - Part 1

Learning Kivy - Part 2

Learning Kivy - Part 3

Learning Kivy - Part 4

Learning Kivy - Part 5

Learning Kivy - Part 6

So, what did I learn from Part 3?  I learned that I can instantiate a thread, and have that thread block via a sleep, and not impact the Kivy run method.  

That's good to know.

And, that's all I need to know, at this point, about creating a thread for a socket outside of the main thread.

However, that begs the question: how to get the data from the socket thread to the Kivy thread?

Well, one way is through Python's global values.  I don't like this approach myself, but let's take a look at it anyway.

For this example, I'm going to want Kivy to change the displayed GUI text, from
text: "Hello World from inside simple.kv, via Box_a"
to something more dynamic, meaning something not static - I want to be able to have the text changed that is being displayed to something originating from the thread.

Kivy has an ObjectProperty class that allows class attributes to be exchanged between Python code and Kivy Language attributes.  Some additional information on Kivy's Properties.

This requires a new import statement:
from kivy.properties import ObjectProperty 

To recap where we are, and where we're needing to go:

• Have gotten Threads to work
• Need to find a way to update the Kivy display

The kivy.org site has a tutorial called Pong.  I had run this, and played around a little bit a while back.  Today it came to mind when I was trying to find a way to update displayed information.  It occurred to me that Pong has the following elements that have to be updated:

• The paddles.  The displayed position of these has to be updated as you play the game.
• The ball.  The displayed position of the ball is constantly changing, and as such, it has to be updated.
• The score.  This was my main focus: it's a text item.  Checking the code, sure enough, it's a Label.  So it's worth digging deeper into this!

After studying it, I wanted to reduce it to the bare bones, and then have it display the 'score'.  I didn't want to work through all of the code that is playing a game, no ball nor paddles, just the 'score'.  So I edited the .py code and the .kv code to a minimal point, invoked it, and sure enough - it was updating the score!

Here's the reduced .py file:

=====Pong.py====

from kivy.app import App

from kivy.uix.widget import Widget

from kivy.properties import NumericProperty

from kivy.clock import Clock

class Player(Widget):

    score = NumericProperty(0)

class PongGame(Widget):

    player1 = Player()

    

    def update(self, dt):

        self.player1.score += 1

class PongApp(App):

    def build(self):

        game = PongGame()

        Clock.schedule_interval(game.update, 1.0/2.0)

        return PongGame()

def main():

    PongApp().run()

if __name__ == '__main__':

    main()

===============pong.kv===========
#I've modified this for a test I'm doing on how to dynamically update
#Label text
:    
    
    
    Label:
        font_size: 70  
        text: str(root.player1.score)
        

Saving these to two files in the same directory (that's very important for Kivy), running it, and I've got a numeric display that's counting up at a rate of twice per second.

That's a demonstration of what I needed (to be able to dynamically update the text of a Label), so now how does it work?

Here's a breakout of the code elements that are important:

• class PongApp(App):
• This extends the App class.  it's the class called within the main() element that starts the program: PongApp().run()
• def build(self):
• This is the method that is called by Kivy itself.  You don't see this call, but ultimately it's invoked by your invocation of 'run()'
• game = PongGame()
• This creates an instance of PongGame and assigns it to the build method variable game.
• class PongGame(Widget):
• This extends the Widget class, and this is important to Kivy, since everything that's displayed is a Widget.
• It has a class variable: player1, which is of type Player.
• Note that there is also a <PongGame>: statement in the pong.kv file.  There is a direct association between this class definition of PongGame in the Pong.py file, and the class definition in the pong.kv file.
• player1 is instantiated as a type Player instance.  In creating this Player instance, note that Player extends Widget, and that it has a class variable of score, which is of type NumericProperty.  This is an important notation, as it will directly affect our ability to dynamically update our Label's text (Label is in the pong.kv file).

At this point, I'm going to move to Part 5, and go into a little more detail of the NumericProperty object.