Note that mitmproxy, which requires a Python install, is not necessary to monitor what is being sent out from your computing device.<p>The same results can be achieved using only socat and the openssl binary.<p>While I understand the terminology is popular, I would not call this "reverse-engineering"; to me this is simply viewing your own traffic.<p>I believe users have a right to see the traffic they (or the apps they use) are sending, and for security reasons alone they should monitor what is being sent. https plus third party CA usage complicates such transparency, making proxying techniques necessary.<p>I wish more users would view their own traffic.<p>Keep up the good work.
Another great mitm proxy is Charles Proxy <a href="http://www.charlesproxy.com/" rel="nofollow">http://www.charlesproxy.com/</a>.<p>It has a really nice UI for looking at JSON responses such as the Kayak. Sometimes a collapsible tree is invaluable in looking through a response.<p>The easy filtering and formatting is primarily why I like it so much. Here is how it handles SSL for various ways <a href="http://www.charlesproxy.com/documentation/using-charles/ssl-certificates/" rel="nofollow">http://www.charlesproxy.com/documentation/using-charles/ssl-...</a><p>Here is a screen shot of my iPhone Kayak app request for comparison <a href="http://imgur.com/gvKB6fr" rel="nofollow">http://imgur.com/gvKB6fr</a>
I don't know about Kayak's economics, but at least at Hipmunk we pay our data providers per search and it's really quite expensive. If they aren't offering an API anymore, it's probably because it was too pricey to operate.<p>You could easily cost a travel search company thousands of dollars very very quickly using an API they don't want you to use. I don't know if it's illegal or not, but it's certainly immoral.
> From that folder, get the mitmproxy-ca-cert.pem file onto your mobile device by emailing it to yourself, for example. Then follow certificate installation steps for iOS or Android.<p>You can just go to <a href="http://mitm.it" rel="nofollow">http://mitm.it</a> on the device. It's a 'magic domain' for the proxied host. See <a href="http://mitm.it/doc/certinstall/webapp.html" rel="nofollow">http://mitm.it/doc/certinstall/webapp.html</a>
Does anyone know the possible legal repercussions open-sourcing a web service's API when the company doesn't explicitly grant permission? This is really neat, but could also raise the ire of a service that doesn't offer an API for a reason.
Great writeup, there's definitely potential to do something cool, especially if it's possible to get around any tracking and the following price manipulation. I tried installing your demo client, however on running it I got the following error:<p><pre><code> root@kayak:~/kayak-mobile-client# python client.py
Departure airport code: LBG
Destination airport code: HAM
Departure date (MM/DD/YY): 12/26/14
Traceback (most recent call last):
File "client.py", line 56, in <module>
searchid = json.loads(r.text)["searchid"]
KeyError: 'searchid'</code></pre>
A useful technique for sure. The only technique I know to slow this down is to use certificate pinning, but it's probably pointless. Some of your users are probably extremely motivated (like ours [1]) and it's obvious to them that what they are doing is unsupported.<p>1. <a href="http://difm.eu/dox/" rel="nofollow">http://difm.eu/dox/</a>
Interesting! We've used a similar technique to reverse-engineering mobile apps from different banks in order to create a Ruby gem to fetch bank data (balance and transactions).<p><a href="https://github.com/ismaGNU/bankScrap" rel="nofollow">https://github.com/ismaGNU/bankScrap</a>