GroveStreams
lkeyes private msg quote post Address this user
Hi...I've been experimenting with sending data from my Arduino Uno + Ethernet shield with the goal of sending real-time heart-rate data for charting in GroveStream. I got the example to run fine, but now seem to be having reports back that the connection couldn't be made after four tries; this after modifying the original example code.

Didn't know if anyone else had been experimenting with Arduino and if so, I'd love hear of your experiences.
Post 1 IP   flag post
MikeMills private msg quote post Address this user
It seems like the biggest culprit for these types of errors is lack of memory on the board. That could be your issue.

The Arduino forum has some threads on how to monitor memory.

A large clue that memory is an issue that we've noticed in the past is that when we try and trace out the URL in our sketch, prior to it being used, only part of the URl is traced out. I imagine the lack of memory is causing memory to be overwritten in places.

Also, avoid doing string appending like this:
a = b + c + d;
Do this instead:
a = b;
a += c;
a += d;

This lowers required peak memory.
Post 2 IP   flag post
ctmorrison private msg quote post Address this user
I did a fair amount of development with the Arduino Uno platform and implemented a production monitoring system using a Ruggeduino which is a bit more tolerant of real world mishaps. The device was fairly reliable at reading, packing up data and sending it off to a cloud service called ThingSpeak on a per minute basis. It ran for over a year with few issues. We then moved to the GroveStreams cloud platform for many reasons, but the micro-controller end of things is fairly similar.

The next move was to the SolderCore micro-controller platform and had significantly greater success due to higher performance, more memory and a simple development environment.

Most recently, I've made the move to the Electric Imp platform and don't see going back. The Imp is a pleasure to work with and my development has progressed at a much quicker pace. We've also found the cloud connectivity to be far superior to either the Arduino or SolderCore platforms. Yes, we're still on the GroveStreams cloud service have have never regretted that move!

I'm not sure this is what you wanted to hear, but thought I'd offer the feedback.
Post 3 IP   flag post
steveb private msg quote post Address this user
I had the devil of trouble with the Arduino in the beginning - similar problems where the base example ran but when I tried to expand it ,"bad things" happened - it used to run for 30 minutes (6 posts) and then die. And as I put in more debugging (serial.prints etc), the worse it got - although sometimes it did print some corrupted data out which pointed me towards memory management.

As the sample code suggests, I think that the problem is with the Arduino memory handling - something which I had to learn about (not being a C programmer I had no background in this)

I fixed it eventually and now my much more advanced sketch, complete with calling in external libraries to decode infrared meter readings, has been rock solid for a number of weeks.

Just as soon a I tidy what I did I will post it but the code is VERY messy as I don't know how to pass pointers into functions etc so the main grovestreams HTTP call is pieced together in one place, and I dropped a few unnecessary (to me) functions - getMacReadable and getIpReadable.

So, I changed all of the "Strings" to char[] - and the ones in the header to to const char[] - the logic being that constants don't use memory, as the compiler compiles them in the code? (This may or may not be true, but makes sense in my mind)

const char gsApiKey[] = "Your API key here";

I also defined

char url[190] ; //note this needs to be long enough to hold your longest post. 190 is OK for me!
char buf [10]; //our biggest number is 9 digits long when converted to strings

then in UpdateGroveStreams I replaced the String addition with things like this. Strcopy makes the intial string (well, character array), then strcat appends the extra pieces. sprintf formats my unsigned long variables into the array "buf" which is then used in the strcat process.

strcpy(url, "PUT /api/feed?compTmplId=meter&compId=90:a2:da:0e:60:a4");
strcat(url,"&compName=");
strcat(url, gsComponentName);
strcat(url, "&api_key=");
strcat(url, gsApiKey);
strcat(url,"&");
strcat(url,gsStreamId1);
strcat(url,"=");
 
sprintf (buf, "%lu", PVReading);
 
strcat(url, buf);  
strcat(url,"&");
strcat(url,gsStreamId2);
strcat(url,"=");
 
sprintf (buf, "%u", currentWatt);
 
strcat(url, buf);  
strcat(url, " HTTP/1.1");


You may be able to back-piece this into your solution, but I think the master arduino sample sketch could perhaps do with being updated by someone more competent that me, since it runs quite "close to the wire" on memory usage.

-Steve

--edit-- can't get rid of the smileys, sorry, even inside "code" tags
Post 4 IP   flag post
MikeMills private msg quote post Address this user
Thanks Steve. I think we'll eventually add a second Arduino example without String class usage as you suggest for advanced users.

Ninja Post (our forum software) is looking into the smileys inside code tags issue. The code tag was just added by them.
Post 5 IP   flag post
mikew private msg quote post Address this user
Quote:
Originally Posted by MikeMills
Ninja Post (our forum software) is looking into the smileys inside code tags issue. The code tag was just added by them.


Sorry about that @steveb and @MikeMills -- code should render much better now!
Post 6 IP   flag post
fixingthingsguy private msg quote post Address this user
I have been struggling with buffering data using strings as in the Ard. example. I'm trying to avoid the 10 sec. rule. I wish there is a GS workaround to help, but I have read the posts.
The results using strings are not repeatable and inconsistent.
I hard coded the data and sent it up in the URL and GS responded all the time accurately.
So, I'm going to attempt Steveb's approach. I also endorse the suggestion to modify the GS/Arduino example to make things easier for new potential GS users.
Post 7 IP   flag post
ctmorrison private msg quote post Address this user
I've had the Ruggeduino (ruggedized Arduino) running for over a year, pushing data to GroveStreams every minute of the day and night. Would it help if I share that code? I'm simply pushing temperature and humidity now, but it could easily have sent several additional streams updates as well, as it did in the past at another site. I needed to do more sophisticated processing and didn't want to deal with what I perceived to be limitation of the Arduino Uno platform. That being said, it has been a rock solid platform!
Post 8 IP   flag post
JosAH private msg quote post Address this user
Greetings,

I use an Arduino Yun for this purpose; I've been testing for some time now and all the glitches came from the embedded devices; not from the internet connection nor from the Grovestreams server. I personally think that 2.5KB or 4KB or RAM is too small to use for string manipulation (without hacking); I let the Yun's Linux side handle it all an leave the Arduino (a small AtMega 32U4 processor) do for what it's good at: control small devices.

kind regards,

Jos
Post 9 IP   flag post
fixingthingsguy private msg quote post Address this user
CTM,others:
I'm using an Arduino UNO R3 and the problem I'm up against is that I am monitoring (testing now) door open/close, that can happen fairly fast. So during testing I noticed a number of events were being missed and found out(remembered,is more like it) that GS rejects anything more frequent than once in 10 secs. That's fair, thats the business/performance decision.
So I tried to buffer these events and send them along after 10 sec had passed. Therein lies the problem(not at GS). I'm using the URL as a String from the example and its using up SRAM which is limited to 2048 bytes. I traced the usage and it keeps successively eating up SRAM and then fails.

If you have a working example of not using strings to form the URL for the http message that would be useful to me and I'm sure others. I hear that working with characters is more reliable probably because you reserve that space and there is less fragmentation.That's the option I have to work with unless I chose a different processor, which I don't want. So, I'm going to take a break and dive into characters from scratch to see if this problem on the Arduino is solvable, for me, anyway.

The code I wrote essentially says, if I last updated GS within 10 s, buffer the data. And that buffering kills the SRAM.
I have another system running for months monitoring my garage door with no failures, of course, because I rarely break the 10 sec rule for an open and close. For an entry door, that's different and during testing I was being quite fast.
Maybe I'm being too rigid in my requirements.
Regards
Post 10 IP   flag post
steveb private msg quote post Address this user
this is my live system, been flawless for 6 months - all I removed was the API key

let's see if ninjapost mangles this...

first thing to do would be to remove some junk functions and comments - it REALLY hasn't been tidied, as you can see, but hopefully it explains how I handled the string to char problem. I also removed a load of "useless" functions like ip to string

This is fairly SRAM light as it uses a large external library as well (ElsterA100C) to count infrared meter reading data and translate it into a reading.

-Steve

It mangles it - try here:
the code hosted on google drive


 
 
/*
 
 
 **changes - LED pin from 12 to 9 (avoid conflict with ethernet shiedl)
 **turn off LED to avoid burnout 
 **update frequency to 60 instead of 20 and int to long as maybew overflows to a negative?
 **turn it back to int/20 - probelms!
 ** used pulldown resistor to hopfully stop internittent crazy values
 ** changes int and longs to unsignedlong
 **change 20 * 1000 to 30000 (maths causes odd results)
 
 **v3 
 ** add elster code
 ** remove delay
 **remove all strings, leave it working in a roundabout kind of way...
 
 **v4 - v3 hangs after ~30 minutes
 **move url and buf to main space 
 
 
 Arduino GroveStreams Stream Feed via Ethernet
 
 The GroveStreams client sketch is designed for the Arduino and Ethernet.
 A full "how to" guide for this sketh can be found at https://www.grovestreams.com/developers/getting_started_arduino_temp.html
 This sketch updates several stream feeds with an analog input reading,
 from a temperature probe, via the GroveStreams API: https://www.grovestreams.com/developers/apibatchfeed.html#apu1a
 The Arduino uses DHCP and DNS for a simpler network setup.
 The sketch also includes Internet Connection Reset logic to ensure the
 Arduino stays connected and can regain connectivity after a network outage.
 Use the Serial Monitor on the Arduino IDE to see verbose network feedback
 and the GroveStreams connectivity status.
 
 GroveStreams Setup:
 
 * Sign Up for Free User Account - https://www.grovestreams.com
 * Create a GroveStreams organization and select the Arduino blueprint
 * Enter a unique MAC Address for this network in this sketch under "Local Network Settings"
 *    (Newer shields have the mac address on a sticker on the shield. Use that.)
 *    (A MAC address can also be generated within a GroveStreams organization: Tools toolbar button - Generate MAC Address)
 *    (The MAC address is used to identify this device within GroveStreams. It must be unique within your GS organization)
 * Enter the GroveStreams api key under "GroveStreams Settings" in this sketch  
 *    (The api key can be retrieved from a GroveStreams organization: click the Api Keys toolbar button,
 *     select your Api Key, and click View Secret Key. Paste the Secret Key below)
 
 Arduino Requirements:
 
 * Arduino with Ethernet Shield or Arduino Ethernet
 * Arduino 1.0 IDE
 
 Network Requirements:
 
 * Ethernet port on Router    
 * DHCP enabled on Router
 * Unique MAC Address for Arduino
 
 Additional Credits:
 Example sketches from Arduino team, Ethernet by David A. Mellis
 Sketch review and advice from Arduino enthusiast and expert David Thompson, http://www.desert-home.com/
 */
 
#include <SPI.h>
#include <Ethernet.h>
#include <elster.h>

ElsterA100C meter(meter_reading);
 
// Local Network Settings
byte mac[] = {
  0x90, 0xA2, 0xDA, 0x0E, 0x60, 0xA4 };  // Change this!!! Must be unique on local network.
                                         // Look for a sticker on the back of your Ethernet shield.
  char url[190] ;
  char buf [10];
// GroveStreams Settings
 
const char gsApiKey[] = "your aip key here";   //Change This!!!
const char gsComponentName[] = "Temperature";        //Optionally change. Set this to give your component a name when it initially registers.
 
//const char gsDomain[] = "grovestreams.com";   //Don't change. The GroveStreams domain.
//const char gsComponentTemplateId[] = "meter";  //Don't change. Tells GS what template to use when the feed initially arrives and a new component needs to be created.
 
 
//GroveStreams Stream IDs. Stream IDs tell GroveStreams which component streams the values will be assigned to.
//Don't change these unless you edit your GroveStreams component definition and change the stream ID to match this.
const char gsStreamId1[] = "g1";   //Don't change. Temp C - Random Stream.
const char gsStreamId2[] = "c1";   //Don't change. Temp F - Random Stream. Don't change.
 
const unsigned long updateFrequency = 300000; // GroveStreams update frequency in milliseconds (the GS blueprint is expecting 20s)
                                       // You will need to change your GroveStreams interval stream base cycles to match the new value if you change updateFrequency.
const int temperaturePin = 0;          // You might need to change depending on the Pin you are using. The Temperature pin number.    
 
// Variable Setup
//String myIPAddress;  //Don't Change. Set below from DHCP. Needed by GroveStreams to verify that a device is not uploading more than once every 10s.
//char myMac[20];        //Don't Change. Set below from the above mac variable. The readable Mac is used by GS to determine which component the
                     // feeds are uploading into. It must match an existing GroveStreams component's ID
 
unsigned long lastSuccessfulUploadTime = 0;    //Don't change. Used to determine if samples need to be uploaded.
boolean lastConnected = false;        //Don't change. Used for Internet Connection Reset logic
int failedCounter = 0;                //Don't change. Used for Internet Connection Reset logic
 
// Initialize Arduino Ethernet Client
EthernetClient client;
 
 
 //for pulse counter
volatile unsigned long counter = 0;
volatile unsigned long lastImpulse = 0;
volatile unsigned long diffImpulse = 0;
volatile unsigned long currentWatt = 1;  //start at 1 just to check data is being recieved by open.sen.se
volatile unsigned long PVReading = 1;  //start at 1 just to check data is being recieved by open.sen.se
 
unsigned long timeRef;
 
int val = 0;
char buffer[30];
int ledPin = 9;
int ledState = LOW;             // ledState used to set the LED
 
 
void setup()
{
  // Start Serial for debugging on the Serial Monitor
  Serial.begin(9600);
   pinMode(ledPin, OUTPUT);  
  attachInterrupt(1, blink, FALLING);
  // Start Ethernet on Arduino
    meter.init(0); //interrupt 0,pin D2
  startEthernet();
}
 
void loop()
{
 
  // Print Update Response to Serial Monitor
  if (client.available())
  {
    char c = client.read();
    Serial.print(c);
  }
 
  // Disconnect from GroveStreams
  if (!client.connected() && lastConnected)
  {
    Serial.println(F("...disconnected"));
    Serial.println();
 
    client.stop();
  }
 //Serial.println((millis() - lastSuccessfulUploadTime > updateFrequency));
 
 //Serial.println(millis() );
 //Serial.println(lastSuccessfulUploadTime );
 //Serial.println( updateFrequency);
 
  // Update sensor data to GroveStreams
  if(!client.connected() && (millis() - lastSuccessfulUploadTime > updateFrequency))
  {
 
    Serial.println(F("Trying upload:"));
    //char temps[] = getTemperatures();
    //Serial.println(temps);
 
    updateGroveStreams();
  }
 
  // Check if Arduino Ethernet needs to be restarted
  if (failedCounter > 3 ) {
 
    //Too many failures. Restart Ethernet.
    startEthernet();
  }
 
  lastConnected = client.connected();
 
    // Decode the meter stream
  const int byte_data = meter.decode_bit_stream();
  if (byte_data != -1) {
    meter.on_data(byte_data);
  }
  //delay(1000);
 
}
 
void updateGroveStreams()
{
  unsigned long connectAttemptTime = millis();
 
  if (client.connect("grovestreams.com", 80))
  {        
 
    //Assemble the url used to pass the temperature readings to GroveStreams.
    // The Arduino String class can be memory intensive. char arrays should be used instead. But,
    // to make this example simple to understand we have chosen to use the String class.
    //We are passing temperature readings into two types of GroveStreams streams, Random and Interval streams.
 
   // url[] = "PUT /api/feed?compTmplId=meter&compId=90:a2:da:0e:60:a4";
 
    strcpy(url, "PUT /api/feed?compTmplId=meter&compId=90:a2:da:0e:60:a4");
  //    strcat(url, gsComponentTemplateId);
  //   Serial.println(url);
 
     // strcat(url,"&compId=";
    //  strcat(url, myMac);
strcat(url,"&compName=");
strcat(url, gsComponentName);
strcat(url, "&api_key=");
strcat(url, gsApiKey);
  strcat(url,"&");
 strcat(url,gsStreamId1);
strcat(url,"=");
//strcat(url,String(0,DEC)); 
 
 
sprintf (buf, "%lu", PVReading);
 
strcat(url, buf);  //Temp F - Random Stream
 
//strcat(url,"0"; 
strcat(url,"&");
strcat(url,gsStreamId2);
strcat(url,"=");
//strcat(url, String(currentWatt,DEC));  //Temp F - Random Stream
 
 
sprintf (buf, "%u", currentWatt);
 
strcat(url, buf);  //Temp F - Random Stream
 
 
 
strcat(url, " HTTP/1.1");
 
 Serial.println(url);
 
     currentWatt=0; //reset to zero
  digitalWrite(ledPin, LOW);   //turn off LED to avoid burnout
 
 
    client.println(url);  //Send the url with temp readings in one println(..) to decrease the chance of dropped packets
    client.println("Host: grovestreams.com" );
    client.println(F("Connection: close"));
    //client.println("X-Forwarded-For: "+ myIPAddress); //Include this line if you have more than one device uploading behind
                                                      // your outward facing router (avoids the GS 10 second upload rule)
    client.println(F("Content-Type: application/json"));
    client.println();
 
    if (client.available())
    {
      //Read the response and display in the the console
      char c = client.read();
      Serial.print(c);
    }
 
    if (client.connected())
    {
      lastSuccessfulUploadTime = connectAttemptTime;
      failedCounter = 0;
    }
    else
    {
      //Connection failed. Increase failed counter
      failedCounter++;
 
      Serial.println("Connection to GroveStreams failed ("+String(failedCounter, DEC)+"");  
      Serial.println();
    }
 
  }
  else
  {
     //Connection failed. Increase failed counter
    failedCounter++;
 
    Serial.println("Connection to GroveStreams Failed ("+String(failedCounter, DEC)+"");  
    Serial.println();
  }
}
 
void startEthernet()
{
  //Start or restart the Ethernet connection.
  client.stop();
 
  Serial.println(F("Connecting Arduino to network..."));
  Serial.println();  
 
  //Wait for the connection to finish stopping
  delay(2000);
 
  //Connect to the network and obtain an IP address using DHCP
  if (Ethernet.begin(mac) == 0)
  {
    Serial.println(F("DHCP Failed, reset Arduino to try again"));
    Serial.println();
  }
  else
  {
 
    Serial.println(F("Arduino connected to network using DHCP"));
    Serial.println();
 
    //Set the mac and ip variables so that they can be used during sensor uploads later
 
   // snprintf(myMac, 100, "%02x:%02x:%02x:%02x:%02x:%02x", mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]);
    //myMac = getMacReadable();
    //Serial.println(strcat("MAC:", myMac);
 
  //  myIPAddress = getIpReadable(Ethernet.localIP());
  //  Serial.println("IP address: " + myIPAddress);
  }
 
}
 
 
//String getIpReadable(IPAddress p)
//{
//  //Convert the ip address to a readable string
//  String ip;
//  for (int i =0; i < 3; i++)
//  {
//    ip += String(p[i], DEC);
//    ip += ".";
//  }
//  ip +=String(p[3], DEC);
//  return ip;
//}
 
char getTemperatures()
{
  //Get the temperature analog reading and convert it to a string
 /* float voltage, degreesC, degreesF;
 
  voltage = (analogRead(temperaturePin) * 0.004882814);
  degreesC = (voltage - 0.5) * 100.0;
  degreesF = degreesC * (9.0/5.0) + 32.0;
 
  char temp[15] = {0}; //Initialize buffer to nulls
  dtostrf(degreesC, 12, 3, temp); //Convert float to string
  String tempC(temp);
  tempC.trim();
 
  char temp2[15] = {0}; //Initialize buffer to nulls
  dtostrf(degreesF, 12, 3, temp2); //Convert float to string
  String tempF(temp2);
  tempF.trim();
 */
 
  //char temps[];
 // strcat(temps,"&"
 // + gsStreamId1 + "=" + String(0,DEC);   //Temp C - Random Stream
 // temps += "&" + gsStreamId2 + "=" + String(currentWatt,DEC);  //Temp F - Random Stream
 
 
 //currentWatt=0; //reset to zero
 // digitalWrite(ledPin, LOW);   //turn off LED to avoid burnout
 
 // return temps;
}
 
void blink() {
  unsigned long currentImpulse = millis();
 
//  Serial.println("Pulse";
 
  if (ledState == LOW)
      ledState = HIGH;
    else
      ledState = LOW;
 
    // set the LED with the ledState of the variable:
    digitalWrite(ledPin, ledState);
 
 
  // Increment counter
  counter++;
 
  // Calculate current Watt usage by measuring the time difference between two impulses
  if (lastImpulse != 0) { // we can now calculcate
    if (currentImpulse > lastImpulse) {
      diffImpulse = currentImpulse - lastImpulse;
      currentWatt = long((3600UL * 1000UL) / diffImpulse);
    } else if (lastImpulse > currentImpulse) { // overflow
      diffImpulse = currentImpulse + (4294967295UL - lastImpulse);
      currentWatt = long((3600UL * 1000UL) / diffImpulse);
    } else {
      // We should never get here UNLESS THEY ARE THE SAME millisecond
      diffImpulse = 0;
 
    }
  }
 
  lastImpulse = currentImpulse;
 
}
 
void meter_reading(unsigned long r)
{
  Serial.print(r);
  Serial.print("rn");
 
  PVReading = r;
 
 
 
 
}
 
 
Post 11 IP   flag post
seckford private msg quote post Address this user
I've been running a test program using Python on a laptop as a preliminary to running some ex-Cosm Arduino code, and I'm getting dropouts up to four hours long, typically in the PM (GMT). At the moment I'm trying to work out where the dropouts occur - on the LAN, on the Internet, or within GroveStreams; when I find out more I'll post again.

Will
Post 12 IP   flag post
seckford private msg quote post Address this user
Quote:
Originally Posted by seckford
I've been running a test program using Python on a laptop as a preliminary to running some ex-Cosm Arduino code, and I'm getting dropouts up to four hours long, typically in the PM (GMT). At the moment I'm trying to work out where the dropouts occur - on the LAN, on the Internet, or within GroveStreams; when I find out more I'll post again.

Will


In the end the problem was in either the local router, or the ISP connection; nothing to do with GroveStreams at all. Both the Python program and an Arduino version have now run for several days without problems.
Post 13 IP   flag post
ctmorrison private msg quote post Address this user
Fortunately, in (nearly?) every instance, the problems I've experienced had nothing to do with GroveStreams.
Post 14 IP   flag post
fixingthingsguy private msg quote post Address this user
Steve_b: Thanks for posting your code. I converted most of the code that was tracking state changes to characters, including use of PGMEM to store the large chunk of constant data in program space. Used strcat liberally.
However, for testing the concept I stripped my streams down to 1 from 3.
The results are good:
-no pac_man effect on SRAM. Stays constant throughout my testing(at 740 bytes).
-able to buffer multiple states and batch them to GS. Not hitting the 10sec limit, internally I have set to 10 sec, assuming there is some more time available in transmission,etc.
-verified GS receiving them (in the same order naturally).
-used &freq parameter to separate the batched data(don't know yet if I can use millis, I thought we had to use epoch, don't know how to do that yet). When I figure that out, the data will correspond to actual times the state change occurred.
Next step is to clean up the code so I can present it here! Won't be a lot different than your code.
Also will try to put back my additional streams but I'm not optimistic that I'll have enough SRAM.
Thanks for inspiring me on.
Post 15 IP   flag post
3033 15 15
Log in or sign up to compose a reply.