finctrl Goes Live

After working on many prototypes, setting up cgit, receiving permission from my employer to publish, presenting the idea at ConleyCon 2015, and finally switching from shell scripts to Python, I am pleased to present the first code commit to “finctrl”, my attempt at automating financial control. Please find the repository at http://212.47.237.237/git/finctrl/. It currently works by downloading email notifications of transactions from a “Cashflow” IMAP folder and parses them assuming the winter 2015 coastal24.com format. Support for additional input and output formats will hopefully follow soon. (I’ve got shell scripts for Chase; it should be quick and easy to port that to Python, but making a generic framework may take significant work.)

My Inter Planetary File System

I’ve long found peer-to-peer networking interesting. Some time ago, I realized that in the context of peer-to-peer networks, libre licensed content has a compelling advantage over proprietary content. Sharing libre licensed works on a peer-to-peer network is clearly legal, while the situation for proprietary content is murky at best.

So, to experiment with a new (at least to me) peer-to-peer network, IPFS, I’ve set up node on my recently rented Scaleway (ARM-based) private server. You can find a few libre licensed songs I like at the link below. If it works out, I’ll add more down the road.

Update: The server is crashing. I’m not sure if the IPFS code, libraries, ARM Go port or what is to blame. Until further notice, the following URL is probably unreachable.

http://gateway.ipfs.io/ipns/QmaFrTsJm6fs2JQnb34e52AdzCA9C2Kxn6gjZEx3RXBNzk

I also wonder how difficult it would be to resurrect or build upon the DebTorrent effort, this time using IPFS. While I must admit ignorance on the details of how IPFS compares to BitTorrent, from what little I’ve read, IPFS uses many of the good ideas from BitTorrent while providing better backwards compatibility with the ubiquitously supported HTTP.

Scripting Kolab Calendar Interactions

As mentioned previously, I would like to be able to do spreadsheet like calculations on my Kolab groupware calendar. Here is how I’m currently attempting that. There were a couple challenges that made this take longer than I had hoped. The first was making sure that, given two processes communicating via a pipe, closing the upstream process didn’t prematurely end the downstream process. This is what the funny redirection is for. The second issue that caused me grief was the carriage return character echo -en '\x0d', present in IMAP results. It caused bash -x output to be really confusing. An extra operation while sed’ing took care of it.

#!/bin/bash -x

username="me@mykolab.com"
password="secret-code"
lastsunday="22-Mar-2015"

rm -f out result
mkfifo out result

(
  echo "1 login ${username} ${password}"
  echo '2 select Cashflow'
  echo "3 search sentsince ${lastsunday}"
  echo '4 logout'
  sed -nr '/^\* search.*/I {
             s/^\* search +//I
             s/ *\r$//
             s/ /,/g
             p
           }' out > result
) | openssl s_client -starttls imap -connect imap.kolabnow.com:143 &> out &

msgs="$(cat result)"

(
  echo "1 login ${username} ${password}"
  echo '2 select Cashflow'
  echo "3 fetch ${msgs} body[2]"
  echo '4 logout'
  cat out > result
) | openssl s_client -starttls imap -connect imap.kolabnow.com:143 &> out &

cat result

#do math
#use imap append command to create new (or what else to modify?) entry

Next steps are to use string processing (probably sed) utilities to extract the icalendar XML and xmlstarlet to parse it (beware the namespace).

Watching the GETs, POSTs, and OKs

Having put in place a simple man-in-the-middle SSL/TLS stripping setup using socat, I used Wireshark to view the HTTP traffic to my bank, so I could replicate it in a more programmatic manner than manually using a web browser. I set Wireshark to capture on any device using the following filter.

tcp port 1337

Using a custom port (1337) slightly hindered automatic decoding so I right clicked on the data section of TCP packets with a data section and selected “Decode As…”, and selected transport tab for port 1337 to be decoded as HTTP. With this in place I put “http” in the filter box and removed a bunch of TCP noise. Now there was still a bunch of javascript fetches that I hopefully won’t have to deal with, but it was easy to locate the POST method that I was particularly interested in.

Taking the S out of HTTPS for Reverse Engineering

I’d like to be able to automatically query my checking account balance (and all my other accounts to, but checking is a good place to start). To my knowledge, my credit union does not provide an API like The Open Bank Project. So I must resort to screen scraping. To start, I’d like to observe my browser logging in. To do so, I’ll use socat.

socat \
  tcp4-listen:1337,fork \
  openssl:subdomain.domain.tld:443,cafile=/etc/ssl/certs/Appropriate_CA.pem

You may need to look closely at the URLs you’re dealing with. I found that my credit union used a different certificate for the account login subdomain than for the home page at the regular domain. With that in place I can navigate to http://127.0.0.1:1337/path/to/login in Firefox and observe the traffic unencrypted in Wireshark. (Apparently I could give Wireshark my private key as well, if it was built against GNU TLS, but I’ve yet to try that approach.)