It wasn’t just the coverage that expanded. While I started out focussing on Renewable Obligation certificates (ROC) and their value, I was soon being asked about other aspects of finance related to renewable energy. The Ofgem data provided a way of viewing the output for stations but my early focus on ROC was soon shown to be an error as other financial schemes such as the Feed In Tariff needed to be considered. Then there was the selling of electricity that I hadn’t thought about and the constraint payments made were another area that I had not originally anticipated tracking but something I added to the site.

As with all projects that grow in such an organic manner my initial design of data structures turned out to be totally inadequate. I added the pieces I needed but over time it grew arms and legs on it’s arms and legs. It worked, but adding things was getting harder and harder. Simple maintenance was proving a challenge. Developing the site was also getting harder.

As a result of everything above I finally decided that the time was right to bite the bullet and use the my additional knowledge to develop a better set of structures for the data. This finally allows me to add some of the extra things that I’ve been struggling with – hopefully in a manner that also allows for better maintenance and easier upgrades!

I’ve made a good start on getting things changed and have a lot of the new design work done, so it’s just a question of finding and correcting all the small (and some not so small) bugs before pushing it live. I’m hoping to get that done this week, but it may drift into next week.

“Scottish wind farms paid £1 million to shut down one day” ¹

“Record Breaking Day for Wind” ²

The above headlines have both been published recently. They both quote impressive facts and figures – or do they?

Emotions tend to run high when discussing green issues and even more so when discussing renewable energy. It’s an important debate but one that I feel is regularly trivialised by the sort of reporting evidenced by the above headlines. I have some sympathy for the journalists as due to the complexity of the energy market writing about it in an accessible manner isn’t easy, but that’s not an excuse.

In order to have the informed debate that I feel is missing it’s important to establish the facts and it’s at this first obstacle that problems start to appear. People make money from the energy market – lots of money. There are markets where trading takes place and as with stocks, shares or other commodities these markets require real time data. Whenever you mix finance with information you come across barriers, restrictions and payments to the companies that grow to control access to the data.

Without free and unencumbered access to such data it’s impossible to have an honest and open debate of the issues.

Much of the information about the UK energy market is provided by Elexon UK. They have a lot of information available, for free, but when trying to use the information things soon become less friendly. Given what the data relates to and how important it is, the fact that Elexon UK consider they “own” (their words not mine) all information they make available should be of concern.

I previously wrote about my dealings with Elexon UK in April 2013.

Disclaimer: I maintain a website that attempts to provide an unbiased view of wind energy production in the UK, http://www.variablepitch.co.uk/.

References:

1. http://www.telegraph.co.uk/news/uknews/scotland/10038598/Scottish-wind-farms-paid-1-million-to-shut-down-one-day.html
2. http://www.renewableuk.com/en/news/press-releases.cfm/2013-03-23-record-breaking-day-for-wind

When I started looking into what was required there was the usual very steep learning curve. All the new terms and concepts to grasp coupled with people keen to sell you things without telling you the details meant it has taken me a while to bite the bullet and start assembling the parts I’ll need.

I’ll try and document what I do to try and help others and remind myself of what not to do next time. It’ll probably also help when reconstructing from the inevitable crashes!

Underpinning this process is a complex set of rules and procedures, detailed on the National grid website here. Given the scale of the UK energy market and the significance of the service, the volumes (and therefore cashflow) involved are very large. An obvious question when developing Variable Pitch was how involved wind energy producers were?

The information is available via the BMRS webiste (Elexon) which allows you to get figures for all Balancing Mechanism Units for any given date or settlement period. This had been available on Variable Pitch since Nov 2012 and showed the volumes and financial amounts involved. However, as with any set of numbers, taken out of context it was difficult to fully assess them. While the amounts paid to wind generators seem large – are they really out of proportion with those being paid to the other participants? Context is important and was lacking.

Once the lack of context had been pointed out I started looking at the available data again and started collecting additional information to allow me to provide the context that was lacking. Along the way I had to delve deeper into what the data meant in order to avoid misrepresenting it, but that required me to contact Elexon UK. Initially they were happy to answer questions, but they then delivered this statement.

The information is available, but the data is owned by Elexon and permission to reproduce will need to be obtained.

Variable Pitch has always had a strict policy about the data it uses.

Only data that is freely available and can therefore be validated from the source direct should be used.

It may not have been stated on the website (but perhaps it should be?), but that’s been the underlying policy I have always applied. All sources of data are noted on the site and no attempt is made to disguise where information comes from. As far as possible the “raw data” is available on the site. This is the minimum I feel is required for an open and honest representation.

I could contact Elexon UK and request permission to use the data, but then anyone else who wished to do likewise would need to contact them, which isn’t acceptable to me. Sadly I have reached the decision that I will have to remove all data that is sourced from their website.

To say I do this with a heavy heart is an understatement and it marks a retrograde step by decreasing the transparency and honesty with which the energy market operates. Elexon UK make money by selling the data so I can fully understand their desire to control the data, but as it relates to an industry that is an essential service it really should be freely available.

Without free and open access to such data honest, open and informed debate is impossible.

As an example, look at the Glyndebourne Wind turbine REGO issuance (http://www.variablepitch.co.uk/station/392/REGO/). In January 2012 there were 2 sets of certificate issued, both for 158MWh of production and both with the same certificate numbers (click on the i to see the details). The only difference appears to be that they were issued to different holders. Hmm. Similar patterns are visible for the rest of the periods covered until July 2012 when things got more predictable.

Why does this matter? Well, by simply looking at the certificates issued in a naive manner such double issuances would lead to double counting so to use the collected data a smarter approach is needed. In fact looking at the data things sometimes get even stranger.

For instance, look at the REGO issuance for An Suidhe in August 2012 (http://www.variablepitch.co.uk/station/98/REGO/). It appears that only 334 certificates have been issued, but the process of arriving at this number is anything but straightforward.

Until I changed the method of storing and collecting the certificate data for Variable Pitch these changes weren’t as obvious or as easy to view. Now that the data is easier to work with I’m working towards new processing that should allow the site to better track the correct output levels and who has ultimately ended up with the certificates. It’s proving to be a difficult problem due the variety of possible paths that can be taken, but hopefully I’ll get it working before long.

Watch this space.

Of course, that’s only half the battle as the metadata is then used by the media server to serve the files – destroying much of my good work in the process I really wish there was a decent media server for linux.

Why do I think CouchDB is a good fit? It’s NoSQL foundations means that data can be easily arranged and stored in ways that make sense for each type. The replication features will allow the various devices to sync with each other without any extra effort. Thanks to the Android Couchbase project it’s possible to use CouchDB on Android (albeit with a larger binary than I would like) allowing for offline storage.

As I’ll be using multiple devices, I’ve decided to start by letting the server allocate it’s default random _id which should avoid duplicates and not require me to devise some more complex scheme.

With a conventional SQL database the decision of how to arrange the data would be simple – a series of tables connected with joins. However, when contemplating how to arrange the data with NoSQL there doesn’t seem to be as simple a solution. This could be my inexperience with them but having read a lot of articles I think it’s just there are so many possible ways of arranging the data that the right solution will depend on the situation.

I’m keen to keep the complexity down but at the same time I don’t want to have a simpler database at the expense of making the interface more complex to work with. My original thought was to store all the related documents in an array embedded within a single document, for example

{
"_id": "12345",
"title": "Some Subject"
"notes": [
{
"type": "text",
"content": "Simple text note"
},
{
"type": "text",
"content": "Another simple piece of text"
}
]}


This is nice and simple and allows me to use a single request to get the entire document, but it adds a lot of complexity to the interface. When editing one of the notes, I would need to

1. request the document
2. edit the document
3. update the entire document

Far simpler from an editing point of view would be to have the notes as seperate records with a link back to their parent document. For example,

{
"_id": "12345",
"title": "Some Subject"
}
{
"_id": "23456",
"parent": "12345",
"type": "text",
"content": "Simple text note"
}
{
"_id": "34567",
"parent": "12345",
"type": "text",
"content": "Another simple piece of text"
}


This allows me to edit each document separately and so avoids the need to request the entire parent document and then manipulate the returned data prior to updating, but does raise the question of how I could efficiently request the document and all it’s related child documents. With some simple changes, a complex key in a view and careful use of parameters it turns out to be very possible.

First I alter the documents slightly by adding a type field to every document.

{
"_id": "12345",
"title": "Some Subject",
"type": "subject"
}
{
"_id": "23456",
"parent": "12345",
"type": "subject:note",
"content-type": "text",
"content": "Simple text note"
}
{
"_id": "34567",
"parent": "12345",
"type": "subject:note",
"content-type": "text",
"content": "Another simple piece of text"
}


Next I write a simple view function.


function(doc) {
if (doc.type) {
if (doc.type == "subject") {
emit([doc._id, 0], doc);
} else if (doc.type == "subject:note" && doc.topic) {
emit([doc.topic, 1], doc);
}
}
}

Running this query (without any parameters) produces the following result,


{"total_rows": 3, "offset": 0, "rows": [
{
"id": "12345",
"key": ["12345", 0],
"value": {...}
}, {
"id": "23456",
"key": ["12345", 1],
"value": {...}
}, {
"id": "34567",
"key": ["12345", 1],
"value": {...}
}
]}


The ‘…’ in the value fields will have the contents of the document, so this query returns me all the information I need. Filtering is also possible thanks to CouchDB and the usual key, startkey and endkey parameters. Additionally I can add fields to the complex key without changing how it functions. Nice.

Getting a single subject with all the related notes is as simple as requesting the view with the parameters ?startkey=["12345"]&endkey;=["12345", 2].

If I add additional document types then I can simply adjust my view and return them as well, using the number to differentiate between them. Parsing the data for an interface becomes simple and each document can be edited/updated in isolation.

Having recently updated the mail server an old issue resurfaced that caused one of my users to be unable to send mail correctly. We use SMTP AUTH to authenticate users and only allow relaying for authenticated users. It’s not an unusual configuration but as postfix is installed by Ubuntu it has this line in the main.cf config file

smtpd_tls_auth_only = yes

With this line only connections that are encrypted will offer the AUTH command. While this is great for security it means that users who ignore my advice to use encrypted connections are unable to relay mails.

I’ve commented out the line in main.cf allowing the default value of ‘no’ to be used which has allowed all users to send correctly. Of course, warnings have also been given to the affected users along with a reiteration of the advice to use an encrypted connection

bash: android-sdk/platform-tools/adb: No such file or directory

The solution is simple enough, but it took me a few minutes to find, so maybe this post will save someone else that time.

The issue was that the apps are 32-bit and my install is 64-bit, so you need to install the 32-bit versions of the system libraries to make them happy.

sudo apt-get install ia32-libs

Et voila! The apps now run.

I’d suggest doing this before installing the eclipse ADT as it will check whether the apps can be run and if they can’t will produce some very strange and unhelpful errors when you try and look at an android project.

I did find a web page describing installing the apps via a ppa which claimed to avoid the need for the 32-bit libs, but when I tried it I couldn’t find any way of telling eclipse where to find the apps installed via the ppa and so was no further forward. If anyone knows the magic incantations to make this work then let me know as I’d prefer not to need the 32-bit libs.

Additionally, when trying to run the emulator I saw the following errors

[2013-01-06 17:58:48 - Emulator] Failed to load libGL.so
[2013-01-06 17:58:48 - Emulator] error libGL.so: cannot open shared object file: No such file or directory
[2013-01-06 17:58:48 - Emulator] Failed to load libGL.so
[2013-01-06 17:58:48 - Emulator] error libGL.so: cannot open shared object file: No such file or directory

The solution was provided by this blog post.