how do extract and upload data recorded sma sb3600 before installation sbfspot

Topics: 1. General
Aug 4, 2015 at 7:01 PM
I've successfully installed sbfspot on a rapberry pi & uploading new gathered data to pvoutput.org works like a charm. But how can I extract and upload data from before the sbfspot installation ? Inverter is SMA SB3600, connected over speedwire.
Aug 5, 2015 at 8:55 PM
Edited Aug 20, 2015 at 5:55 PM
dear tommekes,

you can extract the historical data from your inverter by using the command line options (see help) for sbfspot
the default options for sbfspot are "to-day" - "this-month" (see below for -ad1 -am1)
an alternate system is to use SunnyExplorer (a windows based tool from SMA) to export to .csv tables

basic information: -> an inverter has the following information
-spot values
-daily file
-monthly file
-events (user / installer)

uploading -> see at the end of this reaction

kind regard wim


!!!! as long as inverter's parameters (eg date-time, . . . . ) are not correctly set
it might be that not all information is correctly stored into inverter's memory
for example - if the date-time was never set then you're inverter's clock will have started at 01jan2008 00h00

be aware of this feature:
if "date-time" of your inverter is not yet set correctly or if you want to extract only a part of history data
then you can use the "start-date" option as the starting point for the look-back (option since v308)

-a- spot values -> these values are not stored in inverter's memory (no known way to recuperate them)
"sbfspot" is the perfect tool to read these values and store them into "tables/database"
for further analysis / usage

-b- daily table -> the inverter writes each 5minute a record with "energy-produced" in a "continious" table
inverters keep ~61days in memory (some inverters keep more days) (61 corresponds to actual info)

"sbfspot" allows to extract that table from the inverter's memory
use the command line option -adnn to extract the "available" history (nn 0 <--> ) nn is number of days
sbfspot writes the history to .csv tables and/or database -> see sbfspot.cfg for options
-b.1- output into .csv tables -> for each day sbfspot will write a table
-- todays holds only the data till "now" the table for the oldest day in memory will not be a complete day !!!
-b.2- !!! sbfspot rewrites the .csv tables when output is done to day/day .csv tables
-- for the above you might loose information by requesting export for more days then your inverter has in memory
-- for the above for the oldest-day in memory -> that daily file is not complete (part of a previous .csv is lost)

-b.3- output into database -> sbfspot ADDS the records to the database

c- monthly table -> the inverter writes early mornng a record with "energy-produced" for the PREVIOUS day
inverters keep all monthly files in memory (a limit is not yet known)

"sbfspot" can be used to extract that table from the inverter's memory
use the command line option -amnn to extract the "available" history (nn 0 <--> ) nn is number of months
sbfspot writes the history to .csv tables and/or database -> see sbfspot.cfg

-c.1- output into .csv tables -> for each month sbfspot writes a table
-- this months table holds only the data till (and including) "yesterday"
-c.2- !!! sbfspot rewrites the output into month/month .csv tables
-- in general you will not loose information when requesting for a high number of months

-c.3- output into database -> sbfspot ADDS the records to the database


d- events table -> the inverter writes all events in records
inverters keep about 12months of events memory (limit is not exact known)

"sbfspot" can be used to extract the events table from the inverter's memory
("user" is default option for installer event apply the options: "installer" and "password")
use the command line option -aenn to extract the "available" history (nn 0 <--> nn is number of months
sbfspot writes the history to .csv tables and/or database -> see sbfspot.cfg

-d.1- the output by sbfspot (cfr sunexplorer) writes all events to a single file
-- the filename includes information about start-end of the requested period


-f- about uploading older data to pvoutput (after extrracting the older data)
that depends on your subscription to pvoutput's service
  • in standard mode there is a limit on the number of data
  • in donation mode you can upload more data (limited according pv-output rules)
Marked as answer by SBF on 8/12/2015 at 2:04 PM
Aug 7, 2015 at 12:30 PM
Thanks Wim, that did the trick.
Aug 14, 2015 at 9:49 AM
Edited Aug 14, 2015 at 9:49 AM
hi,

I have the same issue, but I'm not advanced as you; could you please write exactly what to do (i.e.: same sample strings to extract data in a certain period)?

thanks and ciao,
gabriele
Coordinator
Aug 15, 2015 at 6:18 AM
Edited Aug 15, 2015 at 6:20 AM
Pvoutput allows only 15 days (30 when you have donated)
Tot get the data from your inverter, enter this command:
Sbfspot -finq -v -ad15
This will fill the DB with 15 days of archive day data and the daemon will upload everything to pvo
Aug 20, 2015 at 12:11 PM
thanks.

I've donated, and seems the period is 90 days. My question is (reading also Wim, point b.2):

If I run the script
Sbfspot -finq -v -ad15 or Sbfspot -finq -v -ad90
all data already stored and updated to pvoutput will be overwritten with the data stored in the inverter?

thanks and regards,
gabriele
Coordinator
Aug 20, 2015 at 2:45 PM
course66 wrote:
If I run the script
Sbfspot -finq -v -ad15 or Sbfspot -finq -v -ad90
all data already stored and updated to pvoutput will be overwritten with the data stored in the inverter?
No, the data already uploaded to PVO will not be uploaded a second time.
The problem is your inverter holds a maximum of +/- 60 days, so if the data older than 60 days is not yet in the DB, you'll have to do this manually - CSV import is a feature request.
Aug 20, 2015 at 6:16 PM
course66 wrote:
I've donated, and seems the period is 90 days. My question is (reading also Wim, point b.2):

If I run the script
Sbfspot -finq -v -ad15 or Sbfspot -finq -v -ad90
all data already stored and updated to pvoutput will be overwritten with the data stored in the inverter?

dear gabrielle - as mentioned by sbf the information that was already send to PVoutput is not lost

my warning about "loosing data is:
-- not related with DataBase (sbfspot adds records to the database) cfr -b.3-
-- but is related to outputs in .csv tables

if you request more then 60days (in your example 90) then (depending on inverter type):
-- days 1--60 will be handled without problem (new readings replaces previous == tables)
-- days 61-63 output depends on the memory contents - read below
-- days 64-90 nothing will happen - if there is no information then SBFspot will not attempt to write a daily file

depending on ???
for day 61-63 not predictable - but assume day62 to be the day with the oldest information in memory
-> that daily file will not be complete -- missing part from 00h00 -> . . . .
-> by rewriting the table - if there was a previous .csv table then a part of the table is lost
Aug 21, 2015 at 3:19 PM
Edited Aug 21, 2015 at 3:20 PM
thanks Wim,

I've just run the script: /usr/local/bin/sbfspot.3/SBFspot -finq -v -ad90
and something it did... but seemed to me that upload to pvoutput.org was not running; so I did the following (don't tell me why :)):
  • run the script: /usr/local/bin/sbfspot.3/SBFspot -finq -v
  • run the script: /usr/local/bin/sbfspot.3/SBFspotUploadDaemon
  • made a reboot
    and seems to work live, but I don't see old dat. Maybe I should wait one day.
thanks and ciao,
gabriele
Aug 21, 2015 at 9:55 PM
Hi,

I think there may be a bit of confusion as to how things work in relationship to pvoutput.org.
1) pvoutput will allow a user to store "past" data that is up to 90 days old. But only if you are in "donation mode", meaning you donated money to pvoutput.org.
2) Data sent to pvoutput.org may be no more granular than 5 minutes. Also, you notice the "daily" files that are created by SBFspot have entries at five minute intervals. Why, because this is a granular as a SMA inverter will produce data records (at least my inverter anyway). This data is sent to pvoutput.org as live 5 minute record data and displayed on the live page. I don't think it is coincidence that the minimum granularity of pvoutput.org and SMA inverters are the same :)
3) The data retention time of SMA inverters varies with model. You may set SBFspot to obtain any number of days of previous data, but obviously you may only go as far back as the inverter retains the data.
4) OK, addressing the issue of how pvoutput.org handles storing 5 minute live data records. Data records are always stored by time stamp. If a record is sent with a time stamp of "d=20150821" -d "t=16:30" it is stored with that time stamp. If days from now, a record is sent to pvoutput.org with the same time stamp ("d=20150821" -d "t=16:30") the previous record is overwritten.
The new record is immediately visible via the live graph. Most of pvoutput.org's subsequent data processing is performed "on the fly" meaning it calculated as or shortly before it is displayed (daily,monthly,yearly).

Forgive me if I stated the incredibly obvious, but sometimes it is the obvious stuff folks over look.

73
Ron / W4MMP
On 8/21/2015 15:03, course66 wrote:

From: course66

thanks Wim,

I've just run the script: /usr/local/bin/sbfspot.3/SBFspot -finq -v -ad90
and something it did... but seemed to me that upload to pvoutput.org was not running; so I did the following (don't tell me why :)):
  • run the script: /usr/local/bin/sbfspot.3/SBFspot -finq -v
  • run the script: /usr/local/bin/sbfspot.3/SBFspotUploadDaemon
  • made a reboot
    and seems to work live, but I don't see old dat. Maybe I should wait one day.
thanks and ciao,
gabriele

Aug 22, 2015 at 10:47 AM
Hi Ron,

for me is clear the 5 minutes step. When I run the script:
/usr/local/bin/sbfspot.3/SBFspot -finq -v -ad90
5 minutes CSV files have been created by SBFspot starting from 19 June 2015, according to data available/stored in the inverter. Right two days before the script I did download the data from the inverter by means of Sunny Explorer and so I have 5 minutes CSV files from 17 June 2015. Only now I notice that CSV files of SBFspot (not the spot ones) are the same of CSV files from Sunny Explorer (unless the first row). Another difference between above script and Sunny Explorer is that the script did not created the monthly CSV file with daily summary kWh (whilst Sunny Explorer downloaded the monthly file with daily summary kWh of May and June), but maybe is just a question of settings. But this is not an issue for me, I can just copy from mac into raspberry pi the file downloaded with Sunny Explorer.

On the pvoutput.org I already uploaded monthly CSV data with daily kWh from 25th May till end of June. Now I want to upload 5 minutes CSV data to pvoutput.org starting from the first day in which I have such data, that means 17 June 2015.

My question now is: can I someway use SBFspot to upload past 5minutes data or should I do it manually on pvoutput.org?

thanks for bearing me,
gabriele
Aug 22, 2015 at 2:32 PM
Hi Gabriele,

This may help: http://www.pvoutput.org/help.html#load-intraday.

You should be able to upload your 5 minute data with the above facility.
73
Ron / W4MMP
On 8/22/2015 05:47, course66 wrote:

From: course66

Hi Ron,

for me is clear the 5 minutes step. When I run the script:
/usr/local/bin/sbfspot.3/SBFspot -finq -v -ad90
5 minutes CSV files have been created by SBFspot starting from 19 June 2015, according to data available/stored in the inverter. Right two days before the script I did download the data from the inverter by means of Sunny Explorer and so I have 5 minutes CSV files from 17 June 2015. Only now I notice that CSV files of SBFspot (not the spot ones) are the same of CSV files from Sunny Explorer (unless the first row). Another difference between above script and Sunny Explorer is that the script did not created the monthly CSV file with daily summary kWh (whilst Sunny Explorer downloaded the monthly file with daily summary kWh of May and June), but maybe is just a question of settings. But this is not an issue for me, I can just copy from mac into raspberry pi the file downloaded with Sunny Explorer.

On the pvoutput.org I already uploaded monthly CSV data with daily kWh from 25th May till end of June. Now I want to upload 5 minutes CSV data to pvoutput.org starting from the first day in which I have such data, that means 17 June 2015.

My question now is: can I someway use SBFspot to upload past 5minutes data or should I do it manually on pvoutput.org?

thanks for bearing me,
gabriele

Aug 24, 2015 at 12:09 PM
Clear,

thanks
Coordinator
Aug 24, 2015 at 1:40 PM
rlpatton wrote:
pvoutput will allow a user to store "past" data that is up to 90 days old. But only if you are in "donation mode", meaning you donated money to pvoutput.org.
This 14/90 day limit only applies to "live" (5min) data.
It is possible to upload daily generation (E-TODAY) without limitation. Not yet possible with SBFspot (it's my own feature request) but manually it can be done in bulk via http://pvoutput.org/load.jsp
Aug 25, 2015 at 12:11 PM
I realized; so I uploaded alomst 4y daily data back to pvoutput, and I'm uploading around 60 days of 5min data (remaining 30 already uploaded by SBFspot)

ciao,
gabriele