GroveStreams

File Stream - Download in Chunks511

MikeMills private msg quote post Address this user
Moved from another thread:

@crevicedweller:

File storage looks interesting. I tried it and it works well to download the file to a PC with essentially unlimited storage.
I would like to use it with an embedded processor so would need to read the file a line at at time (maybe 512 bytes per transaction), save the line to local non-volatile storage, probably on an SD card, and keep repeating until the end of the file is encountered. This would provide the essential elements of an over the air upgrade capability to the embedded processor product.

Is there a way to read the file as described? If not has that been considered as a future addition?
Post 1 IP   flag post
MikeMills private msg quote post Address this user
@crevicedweller Your suggestion makes sense in the device world.

Eventually we will probably add streaming capabilities for video files, but I doubt that would help you.

We could extend our API to do what you're asking.

What would work for you?

1. Requesting by line number?
2. Requesting by startByteNumber and numberOfBytes
Post 2 IP   flag post
crevicedweller private msg quote post Address this user
The second method looks good, in part since it is similar to what I am used to with opening and reading a file directly.

Pardon if this is redundant. For background, the file i posted is a binary image of the application code that runs on the embedded processor. I imagine a small protected portion of code on that processor that would read the file and program it into the normal run area on the processor then jump to that code for normal operation. The generic term for that is a "Bootloader" and is a common way to upgrade code on an embedded processor product. Placing the file on Grovestream, where we already have several streams monitoring normal operation solves several obstacles to the "over the air" upgrade process for a small company with limited ability to provide the servers etc to support access to the file. Probably what you meant by "...makes sense in the device world".
Post 3 IP   flag post
MikeMills private msg quote post Address this user
Would you be able to convert base64 encoded bytes if we return the bytes that way?

Would it be a pain if we include the base64 as part of a JSON response:

 {
    "streamId": "deviceCode",
    "data": "alskdjfal",
    "isBase64" : true,
    "time": 1380225122000
 }
 


Or is life easier if we just return the base64 string only?
alskdjfal
Post 4 IP   flag post
crevicedweller private msg quote post Address this user
Base64 is OK, I prefer the string only but could work with the JSON response also.
Post 5 IP   flag post
MikeMills private msg quote post Address this user
Implemented. See start and limit parameters in the API here: https://www.grovestreams.com/developers/api.html#7

Decided not to return JSON. Easier to handle in embed code.

Note that in the API, each chunk can contain padding as per base64 specs. So, you will want to decode each chunk as it's returned.

Here's some Java code we used to test it:

 
		String compId = "fileComp";  					//CHANGE!!!
		String streamId = "fileStream"; 				//CHANGE!!!
		String api_key = "d8a1708b-553b-399e-bc2e-f8b2036xxxxx"; //CHANGE!!!
		String fileName = "statement.pdf";      //CHANGE!!! 
		int limit = 200;
		int start = 0;
 
 
		DefaultHttpClient client = new DefaultHttpClient();
		client.getParams().setParameter(CoreConnectionPNames.SO_TIMEOUT, (int)60000);
		client.getParams().setParameter(CoreConnectionPNames.CONNECTION_TIMEOUT, (int)60000);
 
		byte[] fullFileBytes = new byte[0];
		boolean done = false;
		while (!done) {
			String url = String.format("http://grovestreams.com/api/comp/%s/stream/%s/feed/file/%s?api_key=%s&limit=%d&start=%d", compId, streamId, fileName, api_key, limit, start);
			HttpRequestBase meth = new HttpGet(url);
 
			HttpResponse response = client.execute(meth);
 
			if (response.getEntity() != null) {
				long responseSize = response.getEntity().getContentLength();
				if (responseSize > 0) {
					HttpEntity entity = response.getEntity();
 
					byte[] chunkBytes = Base64.getDecoder().decode(EntityUtils.toString(entity));
					fullFileBytes = Bytes.add(fullFileBytes, chunkBytes);
				} else {
					done = true;
				}
			} else {
				done = true;
			}
 
			int statusCode = response.getStatusLine().getStatusCode() ;
			if (statusCode != HttpStatus.SC_OK && statusCode != 204) {
 
				throw new RuntimeException("Http statusCode = " + statusCode);
			}
 
			start += 200;
		}
 
		if (fullFileBytes != null && fullFileBytes.length > 0) {
 
			FileUtils.writeByteArrayToFile(new File("/home/gs/Dev/testFile.pdf"), fullFileBytes);
		}
code
Post 6 IP   flag post
crevicedweller private msg quote post Address this user
This looks great. Thanks Mike.
I am away from the office until Wednesday but will try this out as soon as i am back.
Post 7 IP   flag post
crevicedweller private msg quote post Address this user
I now have demo version of code running on my target hardware that reads chunks from the file i posted. Looks like exactly what we need!
Post 8 IP   flag post
crevicedweller private msg quote post Address this user
cancelled post
Post 9 IP   flag post
crevicedweller private msg quote post Address this user
Cancelled post
Post 10 IP   flag post
3002 10 10
Log in or sign up to compose a reply.