Raspberry Pi lightning strikes detection station that tweets storm information

A few days ago I was browsing the world wide web, when suddenly I stumbled upon an article speaking about an IC sensor made by AMS, called the AS3935 Franklin Lightnin Sensor, which could detect the electrical signature of lightning strikes up to 40km and give the “strength” of the lightning as well as an estimation of the distance to the head of the storm. And all this in a tiny 4x4mm SMD circuit designed to be embedded in personal weather stations, watches, cars, etc.

My dark nerd side was tickled and I suddenly needed wanted to buy one and build a lightning detection station that could tweet about incoming storms, using my Raspberry Pi which is lying on a table doing nothing most of the time. And so I did it ! What I ended up with is a Twitter account on which lightning strikes info is tweeted in real time, with energy and distance estimation : https://twitter.com/toulouse_orages

After some research, I found that Embedded Adventure was selling a module with a pre-soldered sensor with the right antenna and components added to the board, ready to be used via an I2C interface. Some more research and I found that somebody already implemented a Python library to use such sensors on a Raspberry Pi. Perfect ! We have everything we need.

MOD-1016

Wiring the sensor to the Raspberry Pi

Here is the wiring diagram to connect the sensor to the Raspberry Pi (you can use a different GPIO pin for the IRQ, but remember to change code accordingly):

AS3935 PinMOD-1016 PinRaspberry Pi Pin
4 (GND)GND25 (Ground)
5 (VDD)VCC1 (3v3 Power)
10 (IRQ)IRQ11 (GPIO 17)
11 (I2CL)SCL5 (SCL)
13 (I2CD)SDA / MOSI3 (SDA)

Here is a picture of my setup. For now, it’s just a quick prototype set up on a breakboard :

Raspberry Pi + AS9535 Lightning sensor

Writing the code to make it work

The Python library is pretty well self-documented, so just browse through the code and it should be enough. If you want to understand exactly how each of the sensor parameter works (Watchdog Threshold, Signal Rejection, etc, even some of which are not implemented in the library), refer to the AS9535 datasheet.
It could be useful to change those parameters that are not directly exposed, and it’s pretty easy to do so by directly calling RPi_AS3935.set_byte(register, value) (just be sure to call RPi_AS3935.read_data() first, and get the current value of the register so you only change the bits you need in it). For the Twitter part of the script, I chose to use the Tweepy module.

I just wanted to write a script that would listen for lightning, then tweet when a storm is happening. So first, to avoid getting false positives, I set the “minimum strikes” parameter to 5 (the sensor waits for 5 strikes in a predefined period before it starts sending IRQs for each subsequent strike). Then I have a handler which is run each time an IRQ is received. If it’s a strike, we tweet about it. But to avoid spamming tons of tweets, I set up a 5 minutes delay : if more than one strike happen during those 5 minutes, I’ll just tweet the number of strikes that have been detected. After 30min with no activity from the sensor, we can declare that the storm is over and tweet people that they should be okay now.

Here is an example tweet (yes it tweets in French, because I live in France. Baguette.) :

The translation would be “/!\ 117 strikes detected in the last 5 minutes. Power of the last strike : 429714 – distance to the head of the storm : 1km“.
As you can see, it was a pretty violent storm, with tons of intra-cloud strikes happening and being detected almost every other second (which is close to the maximum the sensor can detect, since in the datasheet we can read it waits for 1,5s after a strike has been detected and analyzed, before it starts listening again).
The Twitter account is : https://twitter.com/toulouse_orages

Final words

I had the chance to set it up right before a period of storms, here in Toulouse. At first I thought it might be crazy because it detected hundreds of strikes every few minutes. But no… it’s been 4 days now and I saw no false positives. The sensor didn’t detect any lightning for 48 hours, and started going crazy anytime a storm was approaching ! So I’m really happy with the results, that are much better than I would have thought from a little sensor like it. Especially since I didn’t take any time to calibrate it properly and mainly used factory settings. I just used the antenna capacitor tuning value given by Embedded Adventure, but it can be a good idea calibrating it again since temperature and other factors can influence it. A way to calibrate the antenna by reading the oscillation speed from the IRQ pin using an Arduino is explained here : as you can see, the process could be easily automatized.

If you’re interested in building your own station, you can find the final Python code for this bot on Github : Github – Hexalyse / LightningTweeter

Feel free to fork it, modify it and hack it to your needs ! If you have any question, remark or observation, do not hesitate to post a comment.

Hope you enjoyed 🙂 Happy storm-hunting !

Python version of the Ask.fm API

In the previous articles How I reverse-engineered the Ask.fm API – Part 1 and Part 2 we saw how I proceeded to analyze and reverse engineer the Ask FM API so I could automate some tasks.

From there, I decided to implement a basic version of the API, with only a few functions I needed (a lot are missing, like liking an answer, following and unfollowing people, blocking people, etc.). After some time using this ugly code, some people asked me to share my work. I was reluctant to do so for multiple reasons :  First, I’m not sure about the law around reverse engineering, it’s always been a grey area to me. Also, I’m not sure the people who asked me for it would benefit from this code or these articles because they won’t understand anything and won’t know how to use it. And lastly, I didn’t want to share an easy “working as-is” executable, because it would allow script kiddies to spam on Ask.fm because of the absence of captchas when using this.

For these reasons (and for intellectual property matters), I voluntarily stripped off the API secret key from the Java code on the Github repository, otherwise Ask.fm might get angry at me for sharing such a thing. I might contact them and if they’re okay with it, I’ll add it back.

The code

Anyway, enough talk, here is the link to the github repository :

Hexalyse/pyAskFm – Basic python implementation of the Ask.fm API

Final word

As you can see, there are a lot of things that could be done to improve this code. If you’re interested in contributing, read the README.md file on github, and feel free to send a Pull Request.

How I reverse-engineered the Ask.fm API – Part 2

In the second part, we’ll take a look at the requests we intercepted thanks to Fiddler, and see if Aks.fm implemented some sort of security (tokens, checksums, etc)

Understanding the request headers and parameters

Here is the request we’ll analyze :

POST /authorize HTTP/1.1
X-Api-Version: 0.8
Host: api.ask.fm:443
X-Client-Type: android_3.8.1
Accept: application/json; charset=utf-8
X-Access-Token: .4WspnFnDpwQNevsbXIEExPDgJZDM
Accept-Encoding: identity
Authorization: HMAC a9f98b69f649e0c96240cc6e36980da96f308cea
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
User-Agent: Dalvik/2.1.0 (Linux; U; Android 6.0.1; GT-N7100 Build/MOB30R)
Connection: Keep-Alive
Content-Length: 186

json {"did":"84a2b70bfae4ae65","guid":"84a2b70bfae4ae65","pass":"password123lol","rt":"4","ts":"1471967146","uid":"JohnDoe"}

As we can see , there are lots of interesting headers here. I’ll not make you wait and explain directly what I deducted from my tests :

  • X-Client-Type: This header indicates the version number of the Ask.fm application used. Let’s just use the same as the one we captured.
  • X-Api-Version: This is the… API version, you guessed it. It is important because the parameters we need to send to the API change with each version and they are NOT retro-compatible ! This must be a nightmare to maintain for the developers.
  • X-Access-Token: Before you make any requests, you need to make a GET request to the address https://ask.fm/token, with a few get parameters (the device id, the current timestamp and a third parameter we’ll explain later). Then you use this token in your authorization POST request. The token you get back from this request will be the token that identify you and keeps trace of your “session”.
  • Authorization: Aaaaah, this header is the one who gave me a headache. We can see it sends an HMAC, which is just a glorified checksum, used to verify the request. The application basically generates this checksum (HMAC) from the URL  and the parameters of the request. Then, when the server receive your request, it will compute the HMAC and compare it to the one you sent. If it’s the same, the request is valid. Otherwise, the server will send you a nasty “Invalid request” error message.
  • Other headers are simple and classical HTTP headers

So… the three first headers are simple to reproduce. But this HMAC Authorization header is a barrier : we need to find how this HMAC is generated. And for this, we’ll have to dwelve into the AskFM source code. We’ll look at it later. For now, let’s look at the request parameters.

Every POST request has only one parameter, called “json”, containing an URL-encoded version of all the parameters in JSON. What is funny, is that I discovered you must format this JSON parameter exactly like the application does, or else the request will be considered invalid (I’m really wondering how they parse it). The correct formatting is : the keys must be alphabetically sorted, and there must be no spaces or new lines. Everything must be on a single line. OK, why not…

Also, besides the parameters specific to each request, there are four parameters that will always be present :

  • “did”: The Android device ID, in hexadecimal
  • “guid”: The same ID (can it be different ? I’m not an Android expert)
  • “rt”: I discovered this is a counter that is incremented at each request made. Maybe they use it for rate analytics ? Or for an ugly rate limiting ?
  • “ts”: This is just the current timestamp (in seconds)

Decompiling the apk to find the HMAC generation routine

So now we only need one last thing : to know how this HMAC is generated. For this we’ll need to decompile the apk. You can do it with tools like APKTool, but there are websites like https://www.apkdecompilers.com/ that allows you to send an APK and download the decompiled code. I used a software with a GUI but I don’t remember its name.

Anyway, now we have the decompiled “human readable”, somewhat-Java-ish code, we just need to go through the different packages name and find the ones that sounds useful. I did it months ago so I don’t exactly remember how I proceeded and how I ended up finding the code that was interesting. All I know is that the code we’re interested in is the package com.askfm.network.utils and the class is called Signature.

Here is the decompiled Java code : https://gist.github.com/Hexalyse/fe31295ba1ff6685397e3bee955350b0

As we can see, the code uses a private key (the HMAC secret key), which is stored somewhere else in the source code. I won’t tell you where since… I don’t remember. But it’s pretty easy to find.

UPDATE : I’ve been informed by a reader that the developers changed the way the HMAC secret key is generated in the newer versions of the app. It is no longer stored as plain-text in the code. It is instead generated via another Java class. If you want it, you can extract this class, import it in your own Android app project, compile it, run it, and you’ll have the key without spending too much time reversing the code.

From there, we can just replace this function call with the secret key and we have a Java class we can use to generate a valid HMAC 😀

Conclusion and source code

I will stop this article here, since all that is left is clean up the Java code or re-implement it in any language we want. Then we just have to spend a LOT of time analyzing the requests to each of the API endpoints we’re interested in, and implement it.

I chose to make a very basic implementation of the API with only the functions I needed for a few projects I had which needed some sort of automation (posting answer, sending questions, etc.), but I was too lazy to port the Java code to Python so I kept it in Java.

If you want to get the source code, you can get it on Github : Hexalyse/pyAskFm – Basic python implementation of the Ask.fm API

How I reverse-engineered the Ask.fm API – Part 1

In this first post, we’ll see how I managed to reverse engineer the Ask.fm API. In a subsequent post, I might give the (ugly) code I came up with, which implements some of the API features I needed for various bots and scripts.

If you don’t already know Ask.fm, it’s a social network where people can create profiles and can send each other questions… and answer it, obviously.

Unfortunately, their website has nasty limitations (Google reCAPTCHA) that prevented me to do what I wanted. But I noticed their application never ask for any captchas. So I guessed it was using a backend API, and if I can use it, I can code bots more easily to automate various tasks (like sending “Group Questions”, a widespread practice amongst ask.fm users). I searched if their API was documented and if they offered developers access to it : they didn’t. I searched the web for people who would have implemented it in any language, and found nothing.

So it was my job to reverse engineer and implement (a basic version of) their API in Python. I will retrace my journey, step by step, of how I finally got to the point of intercepting API calls made from the Android app, to the ask.fm servers.

Step 1 : Using Fiddler to sniff API calls made from the app

The first obvious step was to install an HTTP-proxy application to be able to intercept the HTTP requests and replies between the app and their servers. I decided to use Fiddler because it’s free and supports HTTPS traffic recording (via a MitM attack using a crafted SSL certificate, thus allowing to decode HTTPS data, then re-encrypt it and sending it to the other party). So I installed Fiddler, activated the HTTPS decoding option, and installed the root certificate they provide you on my Android phone. You need to do this step, or otherwise, the SSL certificate crafted by Fiddler won’t be accepted by your phone, since it won’t have the root certificate to validate the chain. Then you just have to configure your WiFi connection to use a proxy, and use the correct IP address and port to point to the computer where Fiddler is running.

Unfortunately, even after doing all this, the Ask.fm application didn’t seem to work when I was proxying the network connection. It kept telling me “No internet connection”. But the connection was good, because everything worked fine in a browser and I could even access the HTTPS Ask.fm mobile website in Firefox and see every requests made, decrypted in Fiddler.

So there was another problem. And suddenly it stroke me : they must be using some sort of certificate pinning. What this mean is, even if the certificate given by Fiddler to the ask.fm application is valid, the application is waiting for a very specific certificate. Every others would be rejected. This is a very good practice, because it prevents such Man-In-The-Middle attack from potentials attackers who could then steal your credentials. And it also prevents people from reversing their API.

But I wasn’t done already.

Step 2 : Installing Xposed framework to disable certificate pinning

Because there is a way to bypass certificate pinning. If you have the Xposed framework installed on your phone (requires root access), there are plugins for it which sole purpose is to bypass certificate pinning. There are multiple of them, actually :

Yaaay \o/ We just need to install one of this, and we’ll finally be able to intercept traffic from the app !

Step 3 : Intercepting API calls, then trying to figure how it works

Let’s try sniffing traffic again : It works !

Here is an example of a request (the one sent when we login in the app) :

POST /authorize HTTP/1.1
X-Api-Version: 0.8
Host: api.ask.fm:443
X-Client-Type: android_3.8.1
Accept: application/json; charset=utf-8
X-Access-Token: .4WspnFnDpwQNevsbXIEExPDgJZDM
Accept-Encoding: identity
Authorization: HMAC a9f98b69f649e0c96240cc6e36980da96f308cea
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
User-Agent: Dalvik/2.1.0 (Linux; U; Android 6.0.1; GT-N7100 Build/MOB30R)
Connection: Keep-Alive
Content-Length: 186

json {"did":"84a2b70bfae4ae65","guid":"84a2b70bfae4ae65","pass":"password123lol","rt":"4","ts":"1471967146","uid":"JohnDoe"}

So now we’ve captured traffic going between the AskFM app and their servers, we need to analyze it.

We’ll cover this in the part 2 of this article.

 

What’s this blog ?

Hello, you. I am Mr Nobody, aka Hexalyse, and I’m French. Welcome to the first post of this blog.

What will you find here ?

Well, good question. Lots of stuff, I guess. Or maybe not very much, if I’m too lazy to redact posts. But what I can say is that it will surely be a mix of these things :

  • Computer related technical shenanigans (code, reverse engineering, tips etc.)
  • Photography, retouching, CGI etc.
  • Various rants about things that annoy me
  • Etc.

So… have a good read !