So it wasn’t such a smooth experience as expected…. So this post is merely a reminder to myself on how to get this all set up and working properly.

There are 2 Lua package managers for Windows;

• LuaDist
• LuaRocks

LuaDist seems to be the favourite as a successor for LuaforWindows, but I ran into trouble because its dependencies (CMake) interfered with my Git installation. Hence I picked LuaRocks. Despite my problems with LuaDist, LuaRocks is also the one with a larger selection of packages available. On the other hand LuaDist has tighter control over the packages, so is less likely to run into trouble.

Setting up LuaRocks

There are some dependencies, for c modules, a compiler is required to build them. And besides the packages themselves (rocks) I also need an editor, and my current favorite is ZeroBrane Studio. One big problem I ran into is that the defacto standard socket library for Lua (LuaSocket) does not compile with LuaRocks and the MinGW compiler. But a workaround is available.

The paths are assumed to be;

• “c:\MinGW” for the MinGW compiler
• “c:\users\public\Lua” for the Lua related stuff

MinGW

To compile the binary modules, LuaRocks needs a compiler. Simplest one is the MinGW, it’s good and it’s free.

• Download it as a windows installer package from its downloads page
• Start the installer and make sure to;
  • install in “c:\MinGW” (for this tutorial this is used, anyway; do not use paths with spaces!)
  • check “Download latest repository catalogs”
  • check “C compiler” and “C++ compiler”

LuaRocks

The main package required, install as follows;

• Download the windows package and pick the newest marked with “-win32.zip”
• extract it into a temporary directory eg. “c:\temp\LuaRocks”
• Open an elevated command prompt (in startmenu type “cmd” in the search box, then right-click and select “run as administrator”)
  • cd c:\temp\LuaRocks
  • install /p c:\users\public\lua\LuaRocks /MW
  • copy C:\Users\Public\Lua\LuaRocks\2.0\lua5.1.exe C:\Users\Public\Lua\LuaRocks\2.0\lua.exe

The “install” command invokes the LuaRocks installer and installs it in the directory provided. The “/MW” parameter instructs it to use the MinGW compiler. The copy command last in the list makes sure Lua can be accessed as “lua.exe” instead of the more awkward “lua5.1.exe”.

Configure paths

The paths (both system paths and Lua paths) are required so the individual elements can all find the tools and libraries they need.

• Right click on “My Computer”, “Properties”, ”Advanced system settings”, “Environment variables…”
• Lookup “PATH” in the bottom list, and append “;c:\mingw\bin;C:\Users\Public\Lua\LuaRocks\2.0;C:\Users\Public\Lua\LuaRocks\bin”
• The following two should be added as new enviornment variables;
• “LUA_PATH” with “.\?.lua;.\?\init.lua;C:\Users\Public\Lua\LuaRocks\share\lua\5.1\?.lua;C:\Users\Public\Lua\LuaRocks\share\lua\5.1\?\init.lua”
• “LUA_CPATH” with “.\?.dll;C:\Users\Public\Lua\LuaRocks\lib\lua\5.1\?.dll”

LuaSocket

This is such a standard library that many packages require it. But as mentioned MinGW cannot build it. So here is the workaround;

• Download the binary LuaSocket package into “c:\temp”
• Open a command prompt (as administrator)
  • cd c:\temp
  • luarocks install luasocket-2.0.2-3.win32-x86.rock

ZeroBrane studio

This one is simple

• Download the ZIP archive from the ZeroBrane website
• unzip in “c:\users\public\Lua\ZeroBraneStudio”
• add shortcuts to ZeroBrane to the startmenu or the quick launch bar

Finalizing the installation

• configure extension “.lua” to open with either ZeroBrane Studio (edit as default action) or to “C:\Users\Public\Lua\LuaRocks\2.0\lua.exe” (run as default action)
• configure extension “.luac” to open with “C:\Users\Public\Lua\LuaRocks\2.0\lua.exe” (compiled Lua code, so can’t edit anyway)
• configure extension “.rockspec” (which is just a Lua file) to open with ZeroBrane Studio

Some warnings…

• The LuaSocket implementation used is a binary rock, it has probably been compiled against VS2005, which may cause problems on Windows (DLL dependency hell). So far I haven’t seen any, but no guarantees…
• ZeroBrane also includes some packages, and when running from within the IDE, it might use those instead of the LuaRocks versions (type “print(package.path)” in the ZeroBrane console to see where it gets its modules)

Finally

LuaRocks is a command line tool, which remains slightly awkward on Windows. But it is easy to understand the basics. Simply type “luarocks” on the command line and it will provide some basic help.

A driver template is included so new drivers can quickly be setup. A fully functional xplrfx driver is included as well as 3 demo drivers. All c code is now compiled for distribution, no more development/debug libraries required.

Get the download and extract the zip file in a new folder. From there simply run the batch file LuaUPnP.bat to start the gateway. You’ll get instructions on command line parameters and usage.

To start the samples try; LuaUPnP –debug=debug demo1 demo2 demo3
This will start the gateway with verbose logging and the three demo drivers loaded (located in “lua/upnp/drivers“). The demodrivers are identical, but give a good idea of what you get when loading multiple drivers.

Once started checkout through devicespy the gateway device with;

1. its own service with a set of methods for controlling (restarting/stopping) the gateway
2. each driver as a separate sub device (demo 1/2/3)
3. within each driver more sub devices, being the actual devices published by that
   specific driver

Device Spy showing the gateway with demo 1/2/3 drivers

To create your own driver, check out the directory “lua/upnp/drivers“ which contains the fully functional xplrfx driver, the 3 demo drivers and a template for creating your own.

Through the xplrfx driver (and an RFXLAN device with xplrfx firmware) the gateway is now functional for the following homeautomation hardware;

• ANSLUT
• ByeByeStandBy
• Chacon
• DI.O
• Digimax
• DomiaLite
• Düwi
• ELRO AB400
• ELRO AB600
• Flamingo
• Harrison curtain
• HomeEasy
• HomeEasy (including HE105)
• HomeEasy EU
• Ikea Koppla
• Intertechno
• Intertek/Unitec
• KD101
• KlikAanKlikUit
• NEXA
• Waveman
• X10 lighting

And the fun part is that those devices that support preset-based dimming (homeeasy for example has levels 0 – 15) now also support ramping (load from 0% to 100% over 30 minutes for example) simply because that functionality has been integrated as standard feature in the gateway, indepedent of the underlying hardware.

Give it a try and let me know what you think. Source code is on github, as is the documentation.

Here’s the download (downloaded 22 times)

How does all this work?

Here’s a diagram I drew that would have helped me big time, if it would have been available when I needed it (it wasn’t so I drew it myself).

• This is the application that embeds the Lua environment. In case of the standalone Lua implementation, it’s just a wrapper around the Lua engine

• This is the code part of the Lua environment, it includes the compiler, the virtual machine, etc. It does not have any data nor any Lua code, all its data will always be stored in the Lua states. From an OS perspective this makes it easy to run multiple Lua states (on the same Lua engine) in parallel on different OS threads.

• Looking at a state from the perspective of a piece of Lua code, it is basically its universe. All Lua code runs within its own LuaState and cannot extent beyond that (unless specialized libraries like Lua Lanes, Rings or likewise, are being used)
• Multiple states can run in parallel on top of the same Lua engine
• A LuaState is not thread safe, only a single thread can safely access the LuaState at any given time without causing problems. Multiple states can run on separate OS threads on top of the same Lua engine.
• The state itself is in C represented as ‘lua_State *L‘ it is just the same as a coroutine, except that in this case (as the main thread) it has the global environment attached to it.

Where to store and access data

In general, data should be stored in the LuaState. If you store data local in the C library, then you need to take precautions so the library can be used with multiple Lua states. When used with multithreading, then this shared data also requires locks for safe access.

Here’s a table showing access to library data depending on where it is stored and from where it is being accessed;

IMPORTANT: all of the above assumes ‘end-user’ Lua. So in several occasions where it is mentioned that something is not accessible from Lua then it might very well be possible to access it through the debug library (eg. the registry through debug.getregistry()).

Once you get the way it works it isn’t all that hard. It’s just that the official docs are sometimes hard to grasp if you don’t get the definitons entirely right. And especially for newcomers that is pretty hard (at least that’s what I found).

If you happen to find an error or an omission, please drop me a note below.

I’ve uploaded the slideset, available here (downloaded 132 times). Furthermore some additional comments and links;

• Home automation site; check out bwired.nl, awesome. Nothing to add to that.
• Bwired also has a generic home automation forum, not specifically related to a server product.
• xPL and all its related info can be found on the xPL project site
• For the internet-of-things there is a great open source project started by Sierra-Wireless. It is being run under the Eclipse umbrella and already delivered some nice results. Check out Koneki (LDT and OMA), Mihini and Paho.

For a start with UPnP there is a lot of documentation on the forum pages. But there is no real shortcut to UPnP, but here’s where to start

• To get a feel of what it can do, download the UPnP Developer Tools (runs on .NET, so might also work on Mono, but never tried). Use the DeviceSpy utility to examine some of your own devices and fiddle with the actions (Tip: right-click on a service and subscribe to events). Also give the included “Network Light” sample application a try (to fiddle with from Device Spy).
• Read the UPnP Device Architecture 1.1 document (UDA), even if you don’t make it through, focus on sections;
  • 1 Discovery (introduction only)
  • 2 Description (introduction only)
  • 2.3 Device Description
  • 2.5 Service Description
  • 3 Control (introduction only)
  • 4 Eventing (introduction only)
• Browse the device and service specifications. Start with “Lighting Controls_1” which are fairly easy to read, and once you get the concept of those try something more challenging like the Media Server/Renderer stuff.

It was a fun evening, thanks for the invitation guys! If you have any questions or want some more info, just drop a message below.

The code now includes a BinaryLight and a DimmableLight (both standardized DCPs), they are based on standardized services SwitchPower and Dimming (also included). Implementing a fully functional DimmableLight device now only takes 4 lines of code;

The promise of all this is that any device type needs to be coded only once, and can then be reused over and over again. Will be nice to implement the HEMS (Home Energy Management Systems) DCPs that are currently being developed on this engine as one of the initial implementations.

Please give the code a try. It’s fully functional, including dimming, ramping and stepping. Same as before, windows only, unpack and run the batchfile, then control through the UPnP DeviceSpy. Documentation is also available now from the github pages.

Here’s the download (downloaded 55 times)

• Todo: UPnP security service has not been implemented (must have!)
• Todo: Some of the wrapper code (mutexes, waithandles and sockets) are not complete for all platforms yet, currently it only runs on Windows, but this should be limited as the wrappers themselves have all been created already
• Todo: Solid logger, as its designed to be used in the background without an interface, it must provide solid logging, even if only for troubleshooting and debugging
• Todo: Serial library; currently network connections are supported through the LuaSocket library, but serial connections must be added to (probably through ser2net, but that lacks a windows version AFAIK)
• Todo: The demo is a single Lua file (which is totally crap code), and should be replaced by a Lua side object oriented framework to access and modify UPnP devices/controlpoints
  

The framework is ready and working (probably needs to see some more testing, but its here). So check the download and compare the previous NetworkLight.lua file with the new NetworkLight2.lua file. Major cleanup. Also when you run the sample, all output is now generated by the added logger module. Allows for logging to files, console, email, tcp, etc. (thank you Kepler project!)

The demo code now only consists of the web folder (xml device/service descriptions) and the NetworkLight2.lua file (the remainder is the UPnP-Lua engine). The framework will parse the xml descriptions and generate the objects from it. All that is left is creating a list of action implementations and optionally set default values, these can be set using a ’creator’ function, and that is exactly the code in the NetworkLight2.Lua file.

Here’s the download (downloaded 105 times)

So what is it?

Its a Lua scripting engine, glued to the pupnp library. Lua is a small, fast and very portable dynamic scripting language and pupnp is a portable UPnP library. This creates an engine that will run on any platform (Windows, Mac, Linux, or even embedded on a NAS or your wifi router), while at the same time allowing users/developers to write code in Lua. So it takes away the need to do all the complex stuff in hardcore C, threading, platforms, etc. The Lua scripts will simply run on all platforms, unmodified.

Why create it?

This is a bit of a long-term vision I have on the subject; homeautomation is very fragmented, lots of opensource stuff and many commercial products, each can do a lot, but none can do all. Check out EventGhost, HomeSeer, Girder, and then some. That’s software, hardware is even worse, X10, HomeEasy, Zigbee, Zwave, EIB-KNX either unreliable or competing for market dominance and hence closed technology (especially Zwave). Now all server products try to implement as much hardware support as they can, so they write drivers for a lot, but still not all, hardware. Hardware vendors have to implement drivers for their products for as many products as possible as well, to be able to sell to customers who are well bought into existing hard and software.

A lot of energy is wasted here, with a sub-optimal result. So put the UPnP script engine in between, de-couple the homeautomation server and the hardware support layer. Develop  1 driver for each hardware type, using the UPnP engine. And have each homeautomation server implement 1 UPnP controlpoint. All servers will be able to get along with all hardware. Server developers can focus on implementing state-of-the-art GUI’s and compete on that without having to worry about hardware support. Hardware suppliers can focus on delivering top-notch hardware instead of wasting energy on supporting all kinds of different setups. Users can run it on any platform, even small 24×7 devices.

In the end; everybody wins!

How will this compete with existing hard and software?

Not. The gateway is solely intended to bridge proprietary hardware to generic and open UPnP. I have no intent to create anything else, no homeautomation server/GUI, nor a logic engine. Lets just make sure evereybody does what he/she does best.

Is it ready?

Nope, its a first demo. Lots needs to be done, in no particular order;

• UPnP security service has not been implemented (must have!)
• Some of the wrapper code (mutexes, waithandles and sockets) are not complete for all platforms yet, currently it only runs on Windows, but this should be limited as the wrappers themselves have all been created already
• Solid logger, as its designed to be used in the background without an interface, it must provide solid logging, even if only for troubleshooting and debugging
• Serial library; currently network connections are supported through the LuaSocket library, but serial connections must be added to (probably through ser2net, but that lacks a windows version AFAIK)
• The demo is a single Lua file (which is totally crap code), and should be replaced by a Lua side object oriented framework to access and modify UPnP devices/controlpoints

Show me the demo!

The demo is a Lua version (Windows only until the todo’s have been fixed) of the NetworkLight from the former Intel UPnP toolset, but lacking the fancy images. Download the zip file and extract it in a single folder, then start the batchfile (‘_start.bat’). Now you can use the DeviceSpy utility (also from the UPnP toolset) to inspect the variables and execute actions. If you’re not familiar with the tool;

• right-click a service and tick ‘subscribe’ to get updates
• double-click an action to set parameters and execute it

Here’s the download, at only 272Kb! (downloaded 66 times), and below a screenshot of a testrun.

Here’s the article

Here’s the download (downloaded 146 times)

Here’s how to set it up (on Windows that is, but with some scripting also on other platforms);

1. Download and install the standalone version (that’s what I used)
2. Start it and create a simple new project called ‘test’; menu File > New > Lua project
3. Create a new external tool config; menu Run > External Tools > External Tools Configurations…
4. On the left pane right-click ‘Program’ and select New, then configure it as follows;
5. Obviously, ‘Location’ should point to your local Lua VM
6. Now add a statement to your test project; print(“Hello world”) and click the ‘External Tools’ button and select ‘lua_run’ as configured in step 4.
   IMPORTANT: you must first select ‘main.lua’ in the script explorer or you’ll get an error;
7. You should now see “Hello world” in the console window

Now that’s for running a Lua script directly, so how about debugging? The debug features go through a network connection to a remote device (or ‘localhost’), but it will require the script to be running with the Lua debugger modules loaded. So I created a batch file to do this.

1. Download the debugger files (DBGP client) and store them in a folder ‘luadebug’ inside the directory where Eclipse is installed, for example; “C:\Program Files\EclipseLDT\debuglua”
2. Download this batch file (downloaded 320 times) and store it in the same directory as the debugger files
3. Create another ‘external tool’ config as detailed above, with the following settings;
4. The path to the Lua VM is now in the ‘Arguments’ pane, so make sure it is correct for your local Lua VM.
5. Add a debug configuration via menu; Run > Debug configurations…
6. Configure it with the following settings;
7. Start the debugger by selecting ‘lua_debugger’ from the debug button drop-down;
8.  Then start the application by selecting ‘lua_debug’ from the ‘External Tools’ button;
9. The application will start with the debugger module loaded, it will connect to the debugger GUI within Eclipse.

In the debug configuration, a batch file is used. The batch file takes two parameters; a) path to the Lua VM, b) path to the debug target. The batch file will run the following steps;

1. update the LUA_PATH environment variable such that the debugger modules will be in the search path
2. start the Lua VM and initially execute the debugger modules
3. then start the target Lua script
4. after exiting it will restore the LUA_PATH variable

This approach prevents me from having to add the debugger code to the actual code. Though it is a windows batch file, it should be easy to port it to other platforms.

What’s to be improved still;

• start debugger and debug target with a single click
• allow for adding command line parameters to the target
• having some sort of default start file, instead of manually having to select the correct file before starting it
• the debugger requires a link to a specific project in its setup, so probably a specific setup is required per project (haven’t tested that yet)
• the batch file does not support setting the remote debugger IP, port or IDE key, so it always uses the defaults (127.0.0.1, 10000, luaidekey)

I’m interested in solutions to these, so if you find one let me know…

.

Supporting for the UPnP gateway showed that several devices would simply not show up. Some fiddling with other UPnP tools and the debug facilities showed that the devices had faulty description xml’s (bad UPnP implementations by their vendors). The cases were the Grace Digital tuner and the Vera home automation box (the Grace Digital has a firmware update, beta, available that fixes the issue)

I had to take a pretty steep dive into the code to fix all the undelying elements. What happened was that the code for parsing device and service xml’s only had exception handling on the highest level. This means that you get either all or nothing.

The update now includes more error checking so that individual elements that do not adhere to the UPnP standards are now left out. So any faulty device or service will still get parsed, for all the proper elements it has. To know whether there are any issues with the description xml’s, the debug window can be used.

Showing the debug window can be done from the menu, or via the ‘/DEBUG’ commandline option.

As an example using the faulty service description of the Grace Digital tuner, that device has/had a RenderingControl1.xml service description with an action argument that referenced a non-existing statevariable. If you load this xml into the Service Author tool, it would simply display an error and do nothing;

Error message before the fix, simply wouldn't load

With the fixed version of the stack, it now logs the errors, but commences with parsing the proper parts of the device;

The debug window indicates what's wrong, but it still loads

In the screenshot you can see in the top pane of the debug window that the errors pertain to the actions GetLoudness and SetLoudness. In the bottom pane it is mentioned that there is an invalid reference to the Loudness statevariable. In the main Service Author window, all proper defined actions are now shown (note that Get/SetLoudness are absent here).

The practical use for the Service Author application is very limited, but I have no access to the device itself, so this was the only way to show the effects. Any way, the same UPnP stack can be used with the Device Spy, which basically means that from now on even the bad behaving devices will show up in the Device Spy, which is great for troubleshooting UPnP devices, and of course, the stack can be used to build controlpoints that are also capable of controlling those bad behaving devices.

I mailed the input to Ylian Saint-Hilaire (Intel), who currently maintains the Developer Tools for UPnP Technologies. So I’m expecting the update to become available soon.