GroveStreams
bioagbob private msg quote post Address this user
Is there a way from within the GS website to download all (or date delineated) data into a file (all or some streams from all or some components)?

I know there are API calls to get stream data in large chunks, but I would like the user to be able to get access to it on the site. I also know about the "Export to spreadsheet" button when viewing a stream, but that's limited to 1000 data points.

Thanks,
Bob
Post 1 IP   flag post
MikeMills private msg quote post Address this user
The Table View dashboard widget is probably the best way to export streams today as it lets you choose the streams and the format of times and values. It allows up to 9999 rows to be viewed/exported. This restriction is mainly due to the grid widget performance.

Try it out and give us feedback where it falls short of what you need.
Post 2 IP   flag post
bioagbob private msg quote post Address this user
Thanks Mike. I didn't see that before. It works pretty well. We're working with lots of folks that would like access to all their data for offline analytics, so it would be nice to have a way to get a raw data dump between 2 dates but with unlimited data points (if wide range).

Thanks!
Post 3 IP   flag post
MikeMills private msg quote post Address this user
That makes sense. We'll add it to the features pipeline. We'll leverage much of what that widget offers with the exception of the interval limit.
Post 4 IP   flag post
MikeMills private msg quote post Address this user
Removed row limit for this widget and improved its performance. Tested with 50,000 rows (one stream) and it took about 45 seconds to load. This widget is a great way to export stream data until we implement a better export solution.
Post 5 IP   flag post
bioagbob private msg quote post Address this user
Mike - Any new methods for downloading data? The dashboard table method takes forever. We have over 20 components with several streams each. We'd like to download 2 years of data for offline analysis.
Post 6 IP   flag post
MikeMills private msg quote post Address this user
What takes forever? Adding the streams to the widget and editing the settings or the runtime after you click the export button?
Post 7 IP   flag post
bioagbob private msg quote post Address this user
It takes forever to load the table in the web browser. For example, 30,000 points for a 4 stream component took 4 minutes. The dashboard just hangs while it's waiting for the widget to return. In Google Chrome it complains with the "Page Unresponsive" popup about 5 times. After it loads the widget, the download is fast (was about 1.5MB). For 20 components, most with more than 4 streams, it could take me hours to load widgets and download. Just wish there was a direct download option to avoid the long wait from the table widget. Maybe a special download widget that calls for a background csv download.
Post 8 IP   flag post
MikeMills private msg quote post Address this user
OK. We'll see what we can do. Others have asked for this too.
Post 9 IP   flag post
MikeMills private msg quote post Address this user
The problem with the table widget is that it brings all of the data into the browser, formats it via java script, then sends it back to the server for export. Most of the time is spent with java script formatting the data and displaying it. That's why it can take a significant amount of time.

I have some questions:
Do you always export the same streams or do you manually select different ones each time?

Do you have to format the export (modify the time format, change column names, ...) or can you deal with a fixed format?
Post 10 IP   flag post
bioagbob private msg quote post Address this user
So you're saying that to export data with this method it's actually downloaded, uploaded, and then downloaded again!?

For now, the need is just a data dump. I usually select specific streams and change the cycle to a longer timeframe to slim down the data set for faster download. It would tend to be the same streams every time. All reformatting would be done offline after download for now.
Post 11 IP   flag post
MikeMills private msg quote post Address this user
Please try this and give us some feedback. Do a Browser F5 to clear your cache and pickup the changes:



Select a stream Group to export.
Update Stream Group Membership prior to the export.
All or a time range ofsamples



1st Column is the time (ISO UTC):
Each other column is a stream.
1st Row is the Component Name.
2nd Row is the Component ID.
3rd Row is the Stream Name.
4th Row is the Stream ID.



We could add a Cycle and Function picker (i.e. Day SUM) if anyone requires that.
Post 12 IP   flag post
bioagbob private msg quote post Address this user
I don't see the Export option. Should be under Admin and between Import and Tools, right? I've cleared cache, restarted browser, and tried 3 different browsers including one that's never accessed GS (fully updated Chrome, Safari, and Opera on Mac).
Post 13 IP   flag post
MikeMills private msg quote post Address this user
Ensure a User Group that you belong to has Manage Export Rights:

Post 14 IP   flag post
bioagbob private msg quote post Address this user
That works. Don't know why it wasn't enabled before. Thanks. This is great.
The last helpful piece would be for it to estimate or warn how much data will be used before downloading.
Post 15 IP   flag post
2965 15 15
Log in or sign up to compose a reply.