| Computer Related > Using REST APIs | Miscellaneous |
Thread Author: smokie |
Replies: 13 |
|
New Thu 10 Jan 19 08:57
Using REST APIs - smokie
|
|
Many moons ago I made my living as a programmer (- then went through high level technical to project management). I've found out that some stats I'm looking for are available using a REST API and I'm wondering if there's something I could learn reasonably easily to get the stats, more or less just for the fun of it. I've always wanted to return to doing a bit of programming, for fun, and have three unused Raspberry Pi's (and a mass of related sensors, cameras etc) under my desk to prove it :-) Anyway, all I want is my half hourly usage stats from my electricity supplier, and I'd like it in CSV format to bung into Excel. Here's a link to the API and function I want developer.octopus.energy/docs/api/#example-usage , and the rest of the page talks about other usage. The problem is, I don't know where to start. I've read about REST APIs and understood a fair bit of it, and it seems it's somewhat language-agnostic, so I'm hoping someone here can recommend a simple way to implement it, and ideally point me towards some free online training materials. This is likely to be a long-term project which may never reach the top of the to-do list if it's too complicated, so no rush :-) Thanks |
|
New Thu 10 Jan 19 20:32
Using REST APIs - Kevin
|
|
I haven't used those APIs but looking at the examples on your link it appears that all you have to do is fetch the required data from the server with either http or curl and then parse the return with the jq command to extract and format the fields that you want using the join(",") command to store/print the fields in csv format. I'd start by installing httpie and experimenting with it manually to fetch the complete json output before deciding which particular fields you want and how to format them with jq and join. There are plenty of online tutorials available for http, curl, jq and join. |
|
New Fri 11 Jan 19 09:16
Using REST APIs - smokie
|
|
Thanks that's a really useful starting point, on that it's given me a bit of context and a pointer to just what I need to be researching. I have now installed Python, PIP and HTTPIE and will have a play with it all over the next week or so. |
|
New Fri 11 Jan 19 23:27
Using REST APIs - Kevin
|
|
Having thought about this again I'd use perl. jq is designed to handle everything from very simple to incredibly complex json files so the filter rules can be complicated and confusing. If you are working with a relatively simple json file you will find it easier (and more intuitive) to use perl regular expression pattern matching and then print the results to your csv file. You can call your http API request from within the perl script and parse & print the results directly. If you need help with perl let me know. |
|
New Sat 12 Jan 19 00:32
Using REST APIs - smokie
|
|
Having spent a small amount of time with this today it actually seems quite easy, if only I could get the syntax right. I installed HTTPIE and it's pre-reqs which took a while, but I can't find out how to actually fire up HTTPIE. So I tried by using their example in a CURL command (in a command window), which must have come with something I've installed over the years. I just piped the output to a text file and I had some viable data. Except I need to format it better to allow for a straightforward import, but that is, I presume, stage 2. I couldn't get it to accept any parameters though, like a from and to date. And apparently, according to SWMBO, I had more important things to do, like getting on with the decorating I am in the middle of. I'm away for the weekend but will find a bit of time for this next week I hope. This was the CURL command curl -u "sk_live_[[my metering info]]:" api.octopus.energy/v1/electricity-meter-points/[[my metering info]]/meters/18P5046815/consumption/ > d:d.txt which gave some output from the past three days but I when tried adding the parameters they mention after a , without a and other combos but it then couldn't find the host. Like this curl -u "sk_live_[[my metering info]]:" api.octopus.energy/v1/electricity-meter-points/[[my metering info]]/meters/18P5046815/consumption/ period_from=="2018-12-25T00:00" period_to=="2018-12-26T00:00" > d:c.txt Possibly I am mixing command types or something. The output came as one long coma separated string which I managed to get into Excel (easily!) but I could do with making it suitable to easily alter to a table after importing it. I could probably have done that with a macro but I'm trying to keep it simple. |
|
New Sat 12 Jan 19 09:03
Using REST APIs - smokie
|
|
I'm feeling I've confused myself somewhat on this, and maybe I'm making it much more complicated than it need be. Probably confusing you along the way... :-) I took the Curl command I mentioned above from another developer page, which is a customer-only page. Going back to their example linked above, it says "All API requests should use a base URL of: api.octopus.energy" then Authentication is required for most API requests. This is performed via HTTP Basic Auth. Provide your API key as the basic auth username value. You do not need to provide a password, eg: curl -u "$API_KEY:" api.octopus.energy/v1/accounts/ and then the actual function I need is a parameter (?terminology? - well, it's a GET Command)), as per here developer.octopus.energy/docs/api/#list-consumption-for-a-meter on which I want to use parameter of period_from and period_to. So on this page curl.haxx.se/docs/manpage.html it seems that the GET command is a parameter of the Curl command, so I can run it all in one go in my command prompt. Is that something I can do in PERL (which I've just installed) I feel I am probably pushing beyond reasonableness in my request here, given the difficulty I am having getting my head round this. So please say so if I am. |
|
New Sat 12 Jan 19 22:34
Using REST APIs - Kevin
|
|
>..but that is, I presume, stage 2. Correct. Stage 1 is to get the data you want from the server via the command line. Personally, I'd get http working since it's more user friendly and the API examples in your link all use http which makes it easier. As long as you have the directories where the python executable and http live in your PATH you should just be able to call http from the command line. When you call http from the command line does it fail with an error or just say that it is not a valid command? Here's a guide to installing it correctly tinyurl.com/ybe3l3jq With the latest versions of python I'm pretty sure that there's an option to add the correct directories to your PATH automatically. On my windows laptop the following is part of my PATH for python C:\Users\kevin\AppData\Local\Programs\Python\Python37-32\Scrip ts\;C:\Users\kevin\AppData\Local\Programs\Python\Python37-32\ ; >..it seems that the GET command is a parameter of the Curl command, so I can run it all in one go in my command prompt. GET is one of the basic parameters you need to give to http or curl. It tells them what funtion you want to perform (ie. GET data or PUT data etc.) Once you've got the http command doing what you want you can then either embed that command directly into the perl script (perl can call any command as if it had been called from a command line) or you can use perl itself to fetch the data from the server. If you want perl to fetch the data you will need to install the LWP::Simple or LWP::UserAgent module. Perl has thousands of 'modules' (add-on library fuctions that give extra functionality). I would recommend just embedding the working http command, a) because you know it's working and can execute it via command line to test it, and b) you won't need to learn the LWP:: syntax. In either case, the data returned is available within the script so you can manipulate and output it in any way you want. >I feel I am probably pushing beyond reasonableness in my request here, given the difficulty I am having getting >my head round this. So please say so if I am. If you've already got curl to connect, authenticate and return meaningful data you're halfway there. |
|
New Sun 13 Jan 19 18:57
Using REST APIs - smokie
|
|
Thanks. I seem to have HTTPie and CURL working OK, in that when I enter HTTP in my command prompt it comes back with the valid usage, as per that guide you linked. But how do I input the example he has? This one. C:> http POST localhost:50231/api/Contact name=scott age:=100 HTTP/1.1 200 OK Content-Length: 26 Content-Type: application/json; charset=utf-8 Date: Fri, 17 Aug 2012 21:59:51 GMT Server: Microsoft-HTTPAPI/2.0 { "age": 100, "name": "scott" } I'm thinking the second line onwards is the response so as the first line starts with c:> it looks like command prompt, in which case the same command for me gives http: error: ConnectionError: HTTPConnectionPool(host='localhost', port=50231): Max retries exceeded with url: /api/Contact (Caused by NewConnectionError(' Maybe it just needs something specific set up which I don't have. maybe my firewall has rejected it (on an internal connection request?) But more to the point I'm assuming I am now at the point where I'm going to start doing a PERL script, to make the connection to Octopus then pass the parameters - is that so? Last edited by: smokie on Sun 13 Jan 19 at 18:57
|
|
New Sun 13 Jan 19 20:51
Using REST APIs - Kevin
|
|
>But how do I input the example he has? This one. You don't have an http server listening on localhost so the connection is being refused. I wouldn't worry about it, I'd just get http connecting to octopus and retrieving the data you want. The second line onwards in the example command is indeed the response from the server and you should see the "200 OK" in all responses from octopus unless there is an error. You can check for the "200 OK" in your perl script to make sure that the response is valid. >But more to the point I'm assuming I am now at the point where I'm going to start doing a PERL script, to make >the connection to Octopus then pass the parameters - is that so? No. I would save the response from a valid http command into a local file and work with that to get perl parsing and outputing exactly what you want. Once you're happy with that you can just add the http command to your perl script and use live data. There's no point in querying octopus every time while you try to debug the parsing and formatting in your script. Also, be careful of being too specific/literal with your pattern matching in the script, for example the "HTTP/1.1" in the "200 OK" response could change and break your script if the server admins upgrade or make changes to the server. After the script is working with a fixed http command and you are comfortable with perl you can add more functionality as needed. |
|
New Fri 18 Jan 19 23:21
Using REST APIs - Kevin
|
| Any luck getting your head around perl smokie? |
|
New Sat 19 Jan 19 08:11
Using REST APIs - smokie
|
|
Yes and no - I was going to update you this weekend. With some of the help you gave me upthread I'd managed to get something out of the API and I think I was making very slow progress, but I was facing a bit of an uphill struggle to get exactly what I needed in an efficient manner, then to do anything useful with it- and therefore was losing interest a bit. Anyway I remembered that an old mate, with whom who I occasionally go motor racing at Le Mans and Sebring, was also a tech in a past life, and much more Unixy than me. We've worked more intensely than it deserves in putting together a PERL script. It's not been without problems but it's now nearly there. One of our hindrances was that running it under Windows PERL seems to bring limitations which do not happen on Unix, so he's been developing it on a Pi. (All sounds very grand but really there isn't much to it - which is what I expected but I just was floundering with no background in Perl!) Essentially the script uses more or less exactly the CURL command I was already using to get the data from the supplier, but the rest of the script is what was needed to do something with it. As I think you've alluded to, it is really quite simple (if you know it!) and I am now tweaking the script without any help. An inordinate amount of time has gone into this, by yourself and my mate, and I am really grateful to both of you. Dipping my toe back into programming has been really enjoyable and I could well be drawn into another similarly ultimately pointless little project sometime And I can still think of enhancements to this existing script which I'd like to make, not least sticking a GUI over the top, and making it run as a batch without my involvement, but again I have to learn some new product to do it. As I now have the data I want, and can get the updates (albeit clumsily) this now has no value whatsoever except as a hobby to pass a bit of time on a wet day. :-) Anyway I will come back an finalise this at some point but in the meantime Kevin, thanks for your encouragement and assistance - I'm sure I would have given up early doors without that!! And of course I may be back for more advice... :-) EDIT: Just found Perl/TK, which allows a GUI input. now the fun begins :-) EDITEDIT: Perl Tk is version specific, can't see a version for my version of Perl. Oh well... parked for now... Last edited by: smokie on Sat 19 Jan 19 at 09:17
|
|
New Sat 19 Jan 19 23:11
Using REST APIs - Kevin
|
|
Brilliant, enjoy. Perl is a very powerful and rewarding language to use once you get to know it. 99% of my perl has been on Unix/Linux but I haven't come across any big problems on Windows. I'd be interested to know what problems you encountered. If you want a "Unixy" environment on your Windows machine you could install a hypervisor (VMware workstation player is free for non-commercial use) or Cygwin. |
|
New Sat 19 Jan 19 23:46
Using REST APIs - smokie
|
| Thanks, I'll have a look at that. I did a bit with Unix many years back, support rather than programming, so it isn't all completely new to me! |
|
New Fri 8 Feb 19 08:15
Using REST APIs - smokie
|
|
As promised - an update. I spent best part of a week on and off, tying to get to grips with PERL. I found it quite hard learning a completely new language when the only resource I have is a mate on the end of a phone occasionally and the internet. Especially when he got extremely engaged with it and was streets ahead of me - it re-kindled his imagination and he's found it a thoroughly enjoyable project. Whereas I was struggling to get even fairly simple stuff going (though it makes it most satisfying when something does work!!). My logic was OK but implementing it was hard going. And anyway I now have a short span of attention and something else shiny came along. I got out my Raspberry Pi again to make a security camera (successfully - motion detection, emails of activation, files retained on NAS etc!!). This was partly because during my training stints with my mate I had actually got the raw data I wanted from my electricity supplier so really it's just a matter of keeping it up to date. Also he has now progressed way beyond what I considered a requirement, in that he is producing graph of usage plotted against cost and all sorts. So why would I re-invest the wheel? Lazy, that's me! :-) it did spark an interest again and I do have another related project - to use a bit of SQL to extract my solar generation and so I can look at it against my electricity usage. Also to extract my gas usage from teh supplier, though that is less interesting. My mate has also gone with Octopus more or less so he has his own data to fiddle with - he's really got the bug! On a sales note, they are one of the cheaper ones and if anyone wants to swap to them ask me as we both get £50. So again, thanks for your help. I'll probably come back to this stuff sometime in the future!! |
Forum Version 3.0 - By Khoosys
Car4play Forum
