finctrl Goes Live

After working on many prototypes, setting up cgit, receiving permission from my employer to publish, presenting the idea at ConleyCon 2015, and finally switching from shell scripts to Python, I am pleased to present the first code commit to “finctrl”, my attempt at automating financial control. Please find the repository at It currently works by downloading email notifications of transactions from a “Cashflow” IMAP folder and parses them assuming the winter 2015 format. Support for additional input and output formats will hopefully follow soon. (I’ve got shell scripts for Chase; it should be quick and easy to port that to Python, but making a generic framework may take significant work.)

Scripting Kolab Calendar Interactions

As mentioned previously, I would like to be able to do spreadsheet like calculations on my Kolab groupware calendar. Here is how I’m currently attempting that. There were a couple challenges that made this take longer than I had hoped. The first was making sure that, given two processes communicating via a pipe, closing the upstream process didn’t prematurely end the downstream process. This is what the funny redirection is for. The second issue that caused me grief was the carriage return character echo -en '\x0d', present in IMAP results. It caused bash -x output to be really confusing. An extra operation while sed’ing took care of it.

#!/bin/bash -x

username="[email protected]"

rm -f out result
mkfifo out result

  echo "1 login ${username} ${password}"
  echo '2 select Cashflow'
  echo "3 search sentsince ${lastsunday}"
  echo '4 logout'
  sed -nr '/^\* search.*/I {
             s/^\* search +//I
             s/ *\r$//
             s/ /,/g
           }' out > result
) | openssl s_client -starttls imap -connect &> out &

msgs="$(cat result)"

  echo "1 login ${username} ${password}"
  echo '2 select Cashflow'
  echo "3 fetch ${msgs} body[2]"
  echo '4 logout'
  cat out > result
) | openssl s_client -starttls imap -connect &> out &

cat result

#do math
#use imap append command to create new (or what else to modify?) entry

Next steps are to use string processing (probably sed) utilities to extract the icalendar XML and xmlstarlet to parse it (beware the namespace).

Cashflow Calendar

My wife and I recently regained sufficient interest in tracking and projecting our personal finances. One of the questions we wanted to be able to answer was how much money would we have each day for the next month or so. We receive direct deposited salaries every week or two, but we also have some irregular income like the employee stock purchase plan that my company offers and I participate in. So we made a cashflow calendar. Using a Google Docs spreadsheet, we put days of the week as column headers, allowed seven rows for transactions, and calculated a daily total and a running sum at the bottom. I used conditional formatting to highlight when the checking account would dip below a threshold.

Screenshot from 2015-02-14 08:09:22

I like this because at least within the time period you’re writing up, you can track cashflow that has happened and project cashflow you expect to happen at the natural frequencies. You don’t have to divide a once-a-month bill in half to fit into a 2 week budget or in fourths to fit in a weekly budget. The cashflow calendar seems like an obvious tool, and indeed it is a slight evolution of what I was taught in Engineering Economy at Virginia Tech, but it wasn’t how the Financial Peace University budget template was formatted, and we had been coasting off of that class and its methods for some time.

At the moment we’re only really tracking out checking account and doing it all manually. I hope that with some clever scripting I may be able to have my computer handle the data entry automatically, and all I have to do is predict the future.

Watching the GETs, POSTs, and OKs

Having put in place a simple man-in-the-middle SSL/TLS stripping setup using socat, I used Wireshark to view the HTTP traffic to my bank, so I could replicate it in a more programmatic manner than manually using a web browser. I set Wireshark to capture on any device using the following filter.

tcp port 1337

Using a custom port (1337) slightly hindered automatic decoding so I right clicked on the data section of TCP packets with a data section and selected “Decode As…”, and selected transport tab for port 1337 to be decoded as HTTP. With this in place I put “http” in the filter box and removed a bunch of TCP noise. Now there was still a bunch of javascript fetches that I hopefully won’t have to deal with, but it was easy to locate the POST method that I was particularly interested in.

Taking the S out of HTTPS for Reverse Engineering

I’d like to be able to automatically query my checking account balance (and all my other accounts to, but checking is a good place to start). To my knowledge, my credit union does not provide an API like The Open Bank Project. So I must resort to screen scraping. To start, I’d like to observe my browser logging in. To do so, I’ll use socat.

socat \
  tcp4-listen:1337,fork \

You may need to look closely at the URLs you’re dealing with. I found that my credit union used a different certificate for the account login subdomain than for the home page at the regular domain. With that in place I can navigate to in Firefox and observe the traffic unencrypted in Wireshark. (Apparently I could give Wireshark my private key as well, if it was built against GNU TLS, but I’ve yet to try that approach.)

Moving to Electrum

I think Bitcoin is cool. I bought one bitcoin for $20 back in 2011 and have been living off of it ever since. Just kidding. I’ve only bought a few donuts with it so far (at Rise), and I’ve also bought some additional milliBitcoins since then. I’m very happy with the Bitcoin Wallet application on my phone. When I got back from my first Bitcoin purchase, I thought to myself though, “Gee, won’t it be neat when I don’t have to carry a wallet around because I can pay for everything I want from my phone. But that might make losing my phone pretty disastrous.” So I don’t stash my entire Bitcoin savings on my phone.

Back in 2011 I used the mainline Bitcoin application on my Linux laptop and was happy. These days though, the blockchain is some 2GB and I don’t really feel like keeping that updated on my laptop’s solid state (faster but shorter lifetime) drive. So I looked at how the wallet application on my phone works, using Simplified Payment Verification, and looked around for something that could do the same without depending on Dalvik. Electrum came up in my search. In the meantime I had deleted the 2GB block chain from bitcoin-qt, so it wouldn’t start. So I used PyWallet to extract the private keys from my wallet file and import them into Electrum. Here was the command.

for k in $(python ../../pywallet/ --dumpwallet | sed -nr 's/.*"sec": "(.*)",/\1/p'); do
  ./electrum importprivkey $k