Help on recreating dict per house

[aleonnet]

Hi @ZhangRaymond,
I really tried to recreate this dict, but I could not handle nilmtk/REDD adequately I think :( I'm sorry.
Please help me. I'd really like to try your code out.

Thank you.
import pickle
from nilmtk import DataSet, TimeFrame
import pandas as pd

datafolder = '/Users/alessandro/Documents/data/'

redd = DataSet(datafolder + 'redd.h5')
metadata = dict(redd.metadata)

deviceName = set()
appName = set()
mains={}
appliances={}
for i in range(1,7):
mains[i] = {}

for house in range(1,7):
print('house_',house)
elec = redd.buildings[house].elec
appliancelist = elec.appliances
mainsvalues = elec.mains().power_series_all_data()
mains[house]=mainsvalues
for app in appliancelist:
label = app.label('unknown')
print(' '*3,label)
appName.add(label)
appliances[house][label] = elec[label].power_series(ac_type='active')
deviceName = list(appName)
with open('house_{}.pickle'.format(house), 'wb') as file:
pickle.dump([deviceName], file)
print('End')

[ZhangRaymond]

Hi, Aleonnet, I am too busy for my final thesis these days and I will help you to recode this when I am free. Forgive me for my busyness. Regards, Raymond

…

________________________________ 发件人: aleonnet <notifications@github.com> 发送时间: Saturday, April 20, 2019 11:38:36 AM 收件人: ZhangRaymond/Neural-NILM 抄送: Raymond, Zhang Yuanmeng; Mention 主题: [ZhangRaymond/Neural-NILM] Help on recreating dict per house (#2) Hi @ZhangRaymond<https://github.com/ZhangRaymond>, I really tried to recreate this dict, but I could not handle nilmtk/REDD adequately I think :( I'm sorry. Please help me. I'd really like to try your code out. [image]<https://user-images.githubusercontent.com/20618327/56451834-14b5f980-6304-11e9-9fc2-5ac7c6e2313f.png> Thank you. import pickle from nilmtk import DataSet, TimeFrame import pandas as pd datafolder = '/Users/alessandro/Documents/data/' redd = DataSet(datafolder + 'redd.h5') metadata = dict(redd.metadata) deviceName = set() appName = set() mains={} appliances={} for i in range(1,7): mains[i] = {} for house in range(1,7): print('house_',house) elec = redd.buildings[house].elec appliancelist = elec.appliances mainsvalues = elec.mains().power_series_all_data() mains[house]=mainsvalues for app in appliancelist: label = app.label('unknown') print(' '*3,label) appName.add(label) appliances[house][label] = elec[label].power_series(ac_type='active') deviceName = list(appName) with open('house_{}.pickle'.format(house), 'wb') as file: pickle.dump([deviceName], file) print('End') ― You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#2>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AD23A4QEUICH7QMPLOWFU4TPRKF3ZANCNFSM4HHI3CDQ>.

[aleonnet]

Hi Raymond, Sorry to trouble you, I perfectly undeserving. I’d really like to follow up your results on the article and code. Thank you. Regards, Alessandro

…

On Sat, 27 Apr 2019 at 02:48 Raymond Zhang ***@***.***> wrote: Hi, Aleonnet, I am too busy for my final thesis these days and I will help you to recode this when I am free. Forgive me for my busyness. Regards, Raymond ________________________________ 发件人: aleonnet ***@***.***> 发送时间: Saturday, April 20, 2019 11:38:36 AM 收件人: ZhangRaymond/Neural-NILM 抄送: Raymond, Zhang Yuanmeng; Mention 主题: [ZhangRaymond/Neural-NILM] Help on recreating dict per house (#2) Hi @ZhangRaymond<https://github.com/ZhangRaymond>, I really tried to recreate this dict, but I could not handle nilmtk/REDD adequately I think :( I'm sorry. Please help me. I'd really like to try your code out. [image]< https://user-images.githubusercontent.com/20618327/56451834-14b5f980-6304-11e9-9fc2-5ac7c6e2313f.png> Thank you. import pickle from nilmtk import DataSet, TimeFrame import pandas as pd datafolder = '/Users/alessandro/Documents/data/' redd = DataSet(datafolder + 'redd.h5') metadata = dict(redd.metadata) deviceName = set() appName = set() mains={} appliances={} for i in range(1,7): mains[i] = {} for house in range(1,7): print('house_',house) elec = redd.buildings[house].elec appliancelist = elec.appliances mainsvalues = elec.mains().power_series_all_data() mains[house]=mainsvalues for app in appliancelist: label = app.label('unknown') print(' '*3,label) appName.add(label) appliances[house][label] = elec[label].power_series(ac_type='active') deviceName = list(appName) with open('house_{}.pickle'.format(house), 'wb') as file: pickle.dump([deviceName], file) print('End') ― You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub< #2>, or mute the thread< https://github.com/notifications/unsubscribe-auth/AD23A4QEUICH7QMPLOWFU4TPRKF3ZANCNFSM4HHI3CDQ>. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AE5JYV7PDJ5LD766H37MGILPSPSJVANCNFSM4HHI3CDQ> .

[aleonnet]

Hi Raymond, I was able to recreate the house pickle structure with the following code: [image: image.png] [image: image.png] But a few other issues came up: 1- The 'deviceName' names refer to the nilmtk standard types. I've not found any documented method/metadata to get 'original_name' appliances from REDD. 2- Only house_2, shows the right structure according to the picture you sent me. For example, check house_1 below. Some appliances have more samples than others. How to solve that? [image: image.png] house_ 1 unknown (745878, 1) unknown (745878, 1) fridge (745878, 1) sockets (745878, 1) electric oven Loading data for meter ElecMeterID(instance=4, building=1, dataset='REDD') Done loading data all meters for this chunk. (1044688, 1) 3- The pickle file struture(keys) seems to be correct, but get_house_raw_data in lib.py is still raising an AttributeError: 'list' object has no attribute 'deviceName' and it has it as you can see above. [image: image.png] *Would you help me pls?* It took me a lot of time to get to this point. Thank you. Alessandro

…

On Sat, Apr 27, 2019 at 2:48 AM Raymond Zhang ***@***.***> wrote: Hi, Aleonnet, I am too busy for my final thesis these days and I will help you to recode this when I am free. Forgive me for my busyness. Regards, Raymond ________________________________ 发件人: aleonnet ***@***.***> 发送时间: Saturday, April 20, 2019 11:38:36 AM 收件人: ZhangRaymond/Neural-NILM 抄送: Raymond, Zhang Yuanmeng; Mention 主题: [ZhangRaymond/Neural-NILM] Help on recreating dict per house (#2) Hi @ZhangRaymond<https://github.com/ZhangRaymond>, I really tried to recreate this dict, but I could not handle nilmtk/REDD adequately I think :( I'm sorry. Please help me. I'd really like to try your code out. [image]< https://user-images.githubusercontent.com/20618327/56451834-14b5f980-6304-11e9-9fc2-5ac7c6e2313f.png> Thank you. import pickle from nilmtk import DataSet, TimeFrame import pandas as pd datafolder = '/Users/alessandro/Documents/data/' redd = DataSet(datafolder + 'redd.h5') metadata = dict(redd.metadata) deviceName = set() appName = set() mains={} appliances={} for i in range(1,7): mains[i] = {} for house in range(1,7): print('house_',house) elec = redd.buildings[house].elec appliancelist = elec.appliances mainsvalues = elec.mains().power_series_all_data() mains[house]=mainsvalues for app in appliancelist: label = app.label('unknown') print(' '*3,label) appName.add(label) appliances[house][label] = elec[label].power_series(ac_type='active') deviceName = list(appName) with open('house_{}.pickle'.format(house), 'wb') as file: pickle.dump([deviceName], file) print('End') ― You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub< #2>, or mute the thread< https://github.com/notifications/unsubscribe-auth/AD23A4QEUICH7QMPLOWFU4TPRKF3ZANCNFSM4HHI3CDQ>. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AE5JYV7PDJ5LD766H37MGILPSPSJVANCNFSM4HHI3CDQ> .

[aleonnet]

Hi On Sun, 2 Jun 2019 at 17:03 Alessandro Barbosa <barbosa.alessandro@gmail.com> wrote:

…

Hi Raymond, I was able to recreate the house pickle structure with the following code: [image: image.png] [image: image.png] But a few other issues came up: 1- The 'deviceName' names refer to the nilmtk standard types. I've not found any documented method/metadata to get 'original_name' appliances from REDD. 2- Only house_2, shows the right structure according to the picture you sent me. For example, check house_1 below. Some appliances have more samples than others. How to solve that? [image: image.png] house_ 1 unknown (745878, 1) unknown (745878, 1) fridge (745878, 1) sockets (745878, 1) electric oven Loading data for meter ElecMeterID(instance=4, building=1, dataset='REDD') Done loading data all meters for this chunk. (1044688, 1) 3- The pickle file struture(keys) seems to be correct, but get_house_raw_data in lib.py is still raising an AttributeError: 'list' object has no attribute 'deviceName' and it has it as you can see above. [image: image.png] *Would you help me pls?* It took me a lot of time to get to this point. Thank you. Alessandro On Sat, Apr 27, 2019 at 2:48 AM Raymond Zhang ***@***.***> wrote: > Hi, Aleonnet, > > I am too busy for my final thesis these days and I will help you to > recode this when I am free. Forgive me for my busyness. > > Regards, > Raymond > > ________________________________ > 发件人: aleonnet ***@***.***> > 发送时间: Saturday, April 20, 2019 11:38:36 AM > 收件人: ZhangRaymond/Neural-NILM > 抄送: Raymond, Zhang Yuanmeng; Mention > 主题: [ZhangRaymond/Neural-NILM] Help on recreating dict per house (#2) > > > Hi @ZhangRaymond<https://github.com/ZhangRaymond>, > I really tried to recreate this dict, but I could not handle nilmtk/REDD > adequately I think :( I'm sorry. > Please help me. I'd really like to try your code out. > > [image]< > https://user-images.githubusercontent.com/20618327/56451834-14b5f980-6304-11e9-9fc2-5ac7c6e2313f.png> > > > Thank you. > import pickle > from nilmtk import DataSet, TimeFrame > import pandas as pd > > datafolder = '/Users/alessandro/Documents/data/' > > redd = DataSet(datafolder + 'redd.h5') > metadata = dict(redd.metadata) > > deviceName = set() > appName = set() > mains={} > appliances={} > for i in range(1,7): > mains[i] = {} > > for house in range(1,7): > print('house_',house) > elec = redd.buildings[house].elec > appliancelist = elec.appliances > mainsvalues = elec.mains().power_series_all_data() > mains[house]=mainsvalues > for app in appliancelist: > label = app.label('unknown') > print(' '*3,label) > appName.add(label) > appliances[house][label] = elec[label].power_series(ac_type='active') > deviceName = list(appName) > with open('house_{}.pickle'.format(house), 'wb') as file: > pickle.dump([deviceName], file) > print('End') > > ― > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub< > #2>, or mute the > thread< > https://github.com/notifications/unsubscribe-auth/AD23A4QEUICH7QMPLOWFU4TPRKF3ZANCNFSM4HHI3CDQ>. > > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#2 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AE5JYV7PDJ5LD766H37MGILPSPSJVANCNFSM4HHI3CDQ> > . >

[ZhangRaymond]

Hi On Sun, 2 Jun 2019 at 17:03 Alessandro Barbosa barbosa.alessandro@gmail.com wrote:
…
Hi Raymond, I was able to recreate the house pickle structure with the following code: [image: image.png] [image: image.png] But a few other issues came up: 1- The 'deviceName' names refer to the nilmtk standard types. I've not found any documented method/metadata to get 'original_name' appliances from REDD. 2- Only house_2, shows the right structure according to the picture you sent me. For example, check house_1 below. Some appliances have more samples than others. How to solve that? [image: image.png] house_ 1 unknown (745878, 1) unknown (745878, 1) fridge (745878, 1) sockets (745878, 1) electric oven Loading data for meter ElecMeterID(instance=4, building=1, dataset='REDD') Done loading data all meters for this chunk. (1044688, 1) 3- The pickle file struture(keys) seems to be correct, but get_house_raw_data in lib.py is still raising an AttributeError: 'list' object has no attribute 'deviceName' and it has it as you can see above. [image: image.png] Would you help me pls? It took me a lot of time to get to this point. Thank you. Alessandro On Sat, Apr 27, 2019 at 2:48 AM Raymond Zhang @.> wrote: > Hi, Aleonnet, > > I am too busy for my final thesis these days and I will help you to > recode this when I am free. Forgive me for my busyness. > > Regards, > Raymond > > ________________________________ > 发件人: aleonnet @.> > 发送时间: Saturday, April 20, 2019 11:38:36 AM > 收件人: ZhangRaymond/Neural-NILM > 抄送: Raymond, Zhang Yuanmeng; Mention > 主题: [ZhangRaymond/Neural-NILM] Help on recreating dict per house (#2) > > > Hi @ZhangRaymondhttps://github.com/ZhangRaymond, > I really tried to recreate this dict, but I could not handle nilmtk/REDD > adequately I think :( I'm sorry. > Please help me. I'd really like to try your code out. > > [image]< > https://user-images.githubusercontent.com/20618327/56451834-14b5f980-6304-11e9-9fc2-5ac7c6e2313f.png> > > > Thank you. > import pickle > from nilmtk import DataSet, TimeFrame > import pandas as pd > > datafolder = '/Users/alessandro/Documents/data/' > > redd = DataSet(datafolder + 'redd.h5') > metadata = dict(redd.metadata) > > deviceName = set() > appName = set() > mains={} > appliances={} > for i in range(1,7): > mains[i] = {} > > for house in range(1,7): > print('house_',house) > elec = redd.buildings[house].elec > appliancelist = elec.appliances > mainsvalues = elec.mains().power_series_all_data() > mains[house]=mainsvalues > for app in appliancelist: > label = app.label('unknown') > print(' '*3,label) > appName.add(label) > appliances[house][label] = elec[label].power_series(ac_type='active') > deviceName = list(appName) > with open('house_{}.pickle'.format(house), 'wb') as file: > pickle.dump([deviceName], file) > print('End') > > ― > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub< > #2>, or mute the > thread< > https://github.com/notifications/unsubscribe-auth/AD23A4QEUICH7QMPLOWFU4TPRKF3ZANCNFSM4HHI3CDQ>. > > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#2 (comment)>, > or mute the thread > https://github.com/notifications/unsubscribe-auth/AE5JYV7PDJ5LD766H37MGILPSPSJVANCNFSM4HHI3CDQ > . >

Hi, Aleonnet,

Sorry for the late reply, I have just finished my thesis.
I can't load your figures so I don't know what's wrong for you code, but I will rewrite the code for you soon.

Regards,
Raymond

[ZhangRaymond]

Hi，@aleonnet，

I am currently using the extracted data, one file per house. It didn't uploaded to github before because these files are too large (341MB). Just now I am surprised to find that there is only 7MB after compression, so I uploaded the compressed package and committed just now, please unzip it and use it directly. I'll take the time to rewrite the generating code.

[aleonnet]

Hi Raymond, I’ve got the code working! I’ll send it to you. I’m surprised to learn that the training is taking so long about 31h per epoch. Is this correct? Regards,

…

On Mon, 10 Jun 2019 at 12:29 Raymond Zhang ***@***.***> wrote: ***@***.*** <https://github.com/aleonnet>， I am currently using the extracted data, one file per house. It didn't uploaded to github before because these files are too large (341MB). Just now I am surprised to find that there is only 7MB after compression, so I uploaded the compressed package and committed just now, please unzip it and use it directly. I'll take the time to rewrite the generating code. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2?email_source=notifications&email_token=AE5JYV355GHSMYQGPXYACHDPZZXORA5CNFSM4HHI3CD2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXKGUOA#issuecomment-500460088>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AE5JYV5VBXAL2WLQT7HDTVDPZZXORANCNFSM4HHI3CDQ> .

[ZhangRaymond]

Hi，@aleonnet，
Congrats for your code!
The training time depends on hardware and configurations of experiment especially the length of window size. I don't know what's wrong for your code or handware but I can share my training time:
20 ~ 120 mins/epoch on CPU (i7-7) or 1 ~ 20 mins/epoch on GPU (P40).

Good luck for you!

Regards,
Raymond

[aleonnet]

Hi @ZhangRaymond
Thank you!
Have you published your results (thesis or article)?
I'd like to check them out before moving forward (still 20min/epoch is plenty of power!)
What about some unsupervised or semi supervised approach?

Regards,

PS: Check my contributions at https://github.com/aleonnet/GSP_energy_disaggregator

[hanyan0705]

In line 25 of Seq2point.py (from packages import *), the run shows ModuleNotFoundError: No module named 'packages'.What caused this?

[Mars0207]

Hi @ZhangRaymond,
I'm sorry to trouble you, when I run the code 'main', I met a problem called 'get_modelPath() missing 1 required positional argument: 'dir_path'' , I can not find a file named 'model.h5'. May I have your help?
Thank you,
Ying Zhao

[WT19960917]

Hi Raymond, I’ve got the code working! I’ll send it to you. I’m surprised to learn that the training is taking so long about 31h per epoch. Is this correct? Regards,
…
On Mon, 10 Jun 2019 at 12:29 Raymond Zhang @.> wrote: @. <https://github.com/aleonnet>， I am currently using the extracted data, one file per house. It didn't uploaded to github before because these files are too large (341MB). Just now I am surprised to find that there is only 7MB after compression, so I uploaded the compressed package and committed just now, please unzip it and use it directly. I'll take the time to rewrite the generating code. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2?email_source=notifications&email_token=AE5JYV355GHSMYQGPXYACHDPZZXORA5CNFSM4HHI3CD2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXKGUOA#issuecomment-500460088>, or mute the thread https://github.com/notifications/unsubscribe-auth/AE5JYV5VBXAL2WLQT7HDTVDPZZXORANCNFSM4HHI3CDQ .

hi，aleonnet,
I want to know how you create the.pickle file for each house. Can you provide me with the code you wrote? Thank you.
Regards,
Wang Tao

[WT19960917]

Hi Raymond, I’ve got the code working! I’ll send it to you. I’m surprised to learn that the training is taking so long about 31h per epoch. Is this correct? Regards,
…
On Mon, 10 Jun 2019 at 12:29 Raymond Zhang @.> wrote: _@**.**_ <https://github.com/aleonnet>， I am currently using the extracted data, one file per house. It didn't uploaded to github before because these files are too large (341MB). Just now I am surprised to find that there is only 7MB after compression, so I uploaded the compressed package and committed just now, please unzip it and use it directly. I'll take the time to rewrite the generating code. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2?email_source=notifications&email_token=AE5JYV355GHSMYQGPXYACHDPZZXORA5CNFSM4HHI3CD2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXKGUOA#issuecomment-500460088>, or mute the thread https://github.com/notifications/unsubscribe-auth/AE5JYV5VBXAL2WLQT7HDTVDPZZXORANCNFSM4HHI3CDQ .

hi，aleonnet,
I want to know how you create the.pickle file for each house. Can you provide me with the code you wrote? Thank you.
Regards,
Wang Tao

Hi, aleonnet，

I have found the code for get_house_data.py in your repository. I am new to Github and thank you very much for providing the code.

Regards,
Wang Tao