Caffeinated Simpleton

Flockr is slow. I would profile it, but profiling in Java is a pain, so I’m going to commit a cardinal sin and guess as to what is causing the slowdown. Every time a user loads their flockr page, a bunch of synchronous http requests go out to twitter, return, and the responses are rendered. This could easily happen in a parallel fashion, and my guess is that the slowdown is caused by doing these requests synchronously.

“The Right Way” to solve this problem is using non-blocking I/O. However, that doesn’t let me play with fancy clojure features. Clojure has Agents built in. Agents are bits of code that execute asynchronously and in a thread safe manner. They allow concurrency without any of the usual pains.

Internally, I believe agents are simply functions distributed to a thread pool. Since state is immutable in clojure, it’s pretty trivial to make that arrangement thread safe. However, this is not the optimal solution for my problem as it is limited by the capabilities of the thread pool. The highest throughput would be through non-blocking IO, but we’ll use agents for now for the sake of learning Clojure.

Ok. This might get complicated. However, Berlin Brown was very helpful in getting this all figured out.

First we need to construct the agents. Since there are a dynamic number of channels, we need to put them in a data structure. I just construct a single agent for every twitter channel in column 1 and stick those in a list (the output of map), and then the same thing for the second column.

Constructing the agents is pretty easy. All we need to do is delcare an agent with agent and its default value. If I were to read it right when I created it (by doing @<agent-ref>), I would get the empty string.

I then pass the freshly constructed agent to send-off. send-off takes an agent and the function that will modify the agent. The return value of the generic function that I pass in will become the new value of the agent at some unspecified time in the future. send-off itself returns a reference to its agent immediately.

After running those first two maps I have two lists of agents, which represent the two columns of content. I then need to wait for all that content to get filled in. To do that, I use await. await takes any number of agents that it will wait for before continuing. If I did not wait for the agents to finish, I would return a blank page to the user! Not wanting to do that, I take my two lists of agents, concatenate them, and them use them as the arguments to await using apply.

After that, it’s easy! I have all the rendered channels in my lists of agents, so I iterate through each one, stick them in their columns by dereferencing (@ch-agent) and then send the whole thing off.

The question is whether it really improves performance. Without understanding the internal implementation of the thread pool, we are still limited by the slowest response from Twitter. This is very unfortunate, and in the end, this should be handled on the client side with JavaScript. That’s no fun though, so we’ll just optimize the server side and see what kind of performance we can squeeze out of the thread pool (I’m pretty sure mine is still sub-optimal). Even this fairly straightforward default configuration did cut average response time in half, however, so that’s a pretty good start.

As always, the entirety of the code is available on github.

The nice thing about a bubble bursting is that you know you’re not in a bubble. Now is the time to look at how we should be investing for the long term, both on personal and governmental levels.

Who Can Help

The recently passed stimulus bill (not the currently debated spending bill) focused on a few main areas. The first was “shovel ready” government projects. These projects are mere stopgaps. They employ people for a short time and then they are finished. Building a bridge or a road, while nice, creates a minimum amount of additional value. Coming up with a single idea that transforms our energy industry, on the other hand, potentially creates huge amounts of additional value. It is these types of innovations that need to be funded, and that was what a part of the stimulus bill was aimed at. Funding “green” technology and IT companies that would transform healthcare by making it efficient.

The problem with these investments is that the markets are artificial, or at least unproven. The market for energy efficient vehicles and other appliances will be created, in large part, by another part of the stimulus bill. Raw incentives and tax cuts are being handed out to those who will purchase the products the stimulus funded the development of. It’s a market, perhaps, but the value is all what the government put into it. If there is actual demand for these energy solutions, they will succeed wihout subsidies (and be extraordinarily lucrative with the government subsidies).

Determining what energy solutions will actually create new markets with obscene profits and goodness for all mankind is not an easy task. Luckily, we have professionals that do this on a day to day basis. These people are called “investors”. There are many types of investors, but they all have one goal: to invest in securities that will increase in value. Since they share this common goal, they have come up with numerous ways of determining whether something will make money or not. They are not always right, but they are way better at it than anybody else.

Fixing Societies Woes

The ideal situation, then, is to somehow get these investors the capital and time necessary to find the corners of the market that will flourish given enough time and capital and allow the investors to give these markets time and capital. As you can see, this all requires time and capital. There is an institution in this country that has plenty of both, and that is the Social Security Administration.

The Social Security Administrations currently looks over a fund of nearly 2.5 trillion dollars. That is enough money to fund just about everything. What’s more, people don’t need it until they are at least 62. This gives approximately 40 years from the time people start investing until the fund needs to reach maturity. There has never, ever been a 20 year period, let alone a 40 year period, in which the stock market on average has not returned a better return on investment than any other security.

It is my feeling that privatizing social security provides us with the capital necessary to bust out of any economic crunch, this one included. However, capital alone is not sufficient.

Privatizing Social Security, Responsibly

When the housing bubble burst a year ago, people were stunned to learn that their retirement savings had been slashed by huge percentages. I can’t even imagine the emotional impact of helplessly watching your grand retirement being washed away in a matter of weeks. This cannot be allowed to happen to Social Security, and it’s my belief that it needn’t happen to either Social Security or 401k’s.

As Cass Sunstein and Richard Thaler argue in Nudge, people are very unlikely to change from default choices. By following some of the recommendations there, we can make privatizing Social Security safer than the 401k’s people use today.

An ideal system would offer a very good default with a limited number of good alternatives, while also allowing the reckless individual to invest however he pleases. The default option for a person’s Social Security should be a private fund that invests heavily in index funds with a small percentage set aside for risky venture and low risk bond investments. It should also be age aware. By the time the individual reaches age 55 (or whatever is deemed appropriate), they should be exposed to the minimum amount of risk. A stock market crash on your 62nd birthday should ruin your day, not your retirement.

Investing in the Game Changers

The final piece of the puzzle that is necessary for true economic stimulus is investing in the potentially explosive markets. Getting in on the next Google or Microsoft: the next big thing. These are the companies that will employ vast numbers of people in short periods of time, provided they have the capital to grow. Currently, this niche is filled almost exclusively by Venture Capitalists. I feel that while these players certainly have a role to fill, current legislation practically prohibits the American public from taking advantage of these opportunities.

Sarbanes-Oxley was a bill passed shortly after the Enron scandal. It imposes very strict accounting requirements on public companies which end up costing these companies millions of dollars a year. Before these regulations existed, having an IPO (Initial Public Offering) was a good way to raise a round of capital to expand. After Sarbanes-Oxley, only companies that have enough money to offset the prohibitive costs of entry have an IPO. This means no regular Americans get to invest in the little guys that are steadily hiring and trying to grow with as little money as possible. The irony is that Enron was successfully prosecuted on laws that were already on the books, not Sarbanes-Oxley. These laws had just been poorly enforced until the scandal broke.

Now I do not mean to suggest that we should throw our Social Security into the venture capital arena. What I do mean to suggest is that a well balanced fund should invest a small percentage in risky investments, and that small companies should be able to benefit from these investments. Two percent of 2.5 trillion is still 50 billion dollars, and that is enough to fund whatever the next big thing may be. If we are careful, it could pay back in spades.

Stimulus Through Prudent Investment

With the index funds at ten year lows, this is a buyers market. The only way to bring the markets up, along with the economy, is to invest in those companies that are poised for expansion. Nothing is more about investing in the future than Social Security, and it seems obvious that these two identical goals should be combined. By removing the obstacles to investment, providing the capital for investment, and putting it in the hands of professionals, we can get America’s economy back on track.

If you’ve been following, you know that I’m trying to build the Web 2.0iest site out there. In fact, this is so Web 2.0, I’m tempted to call it Web 2.1. I’m using only the hottest language (Clojure) and the coolest social networking APIs (twitter). Now I’m kicking it up a notch and using the newest player in the key/value database arena, Tokyo Cabinet.

Introduction

Tokyo Cabinet is a simple, small, fast key/value store. Similar to DBM, it’s a very basic database. If you combine it with Tokyo Tyrant, it becomes a very capable, scalable network database (like mysql or couchdb). However, we don’t really need those things right now, so I’ll just be using Tokyo Cabinet straight up. It comes with a Java interface, so I’ll be using that. I will be using Tokyo Cabinet to store and retrieve user preferences.

Setting up Tokyo Cabinet

Going with the latest and greatest does have its drawbacks. Tokyo Cabinet isn’t packaged up and easy to install yet, so there is some setup involved. In fact, I would recommend not using this for anything beyond your own satisfaction. The project is out of Japan, and the support in English isn’t really there yet. Luckily, since it’s so small and simple, it’s really not that bad.

• Install Tokyo Cabinet
  • Download the source from sourceforge.
  • Unpackage
  • Make sure you have the development headers for zlib and bzip2

  • ./configure

  • make

  • make check

  • make install

• Install Java bindings
  • Download the source from sourceforge
  • Make sure you have jni.h. I didn’t and installed the ubuntu package “default-jdk-builddep”. That in turn installed most of the current open source software in existence, which seemed to work.
  • Run the configure script. Mine screwed up the paths to all my java utilities, so I manually edited the Makefile to get it to work. Perhaps that’s not “The Right Way” to do things, but it worked.
  • Make sure tokyocabinet.jar ends up in your classpath
  • Make sure the libjtokyocabinet.*  is in your java.library.path. I haven’t figured out a good system for configuring Java yet, so I just have it defined in my “compojure” bash script.

Storing Data

The next challenge for me (but luckily not for you) was to write a nice clojure wrapper over the Java interface for Tokyo Cabinet. This introduced me to all sorts of new concepts in clojure, and I wouldn’t have survived if it weren’t for hiredman and others on the #clojure IRC channel. Those guys are great!

I wanted the API to be simple and clear, and this is what I came up with:

To accomplish this, I ended up with the wrapper below.

Whew. Pretty intense stuff. There’s all sorts of new stuff here for the beginning lisper, so let’s step through this line by line, though we’ll skip a few of the less interesting lines.

 (declare *db*)

This is pretty straightforward. It declares *db* in the tokyo-cabinet namespace. *<var name>* is the convention for declaring globals in lisp.

(defmacro use [filename & body]

Macros are what lispers tend to rave about, and this is my first one. Macros in lisp are basically the same concept as in C, you can substitute whatever you like into the place where it’s used at compile time. The difference is that lisp’s macro system is part of the language itself, so you can do absolutely anything. They do have their drawbacks, however, in that they’re exceptionally difficult to debug. After all, the whole thing is being substituted into your original source, so errors come up as if they had happened inline.

The “&” symbol might also be new to some of you. It allows for the macro to be passed an arbitrary number of arguments after the name of the database that we’re operating on. In our case, we’ll be executing the passed expressions in the context of the open database. Therefore any calls to put and/or get will operate on that particular database.

`(with-open [hdb# (HDB.)]

Oh boy. This line is basically magic. The backtick (`) is a form quote macro. This means that everything after from the backtick until the following expression is closed is the source that will be substituted into the caller’s source. There are two forms of form quotes in clojure, backtick (`) and quote (‘). They have one important difference. The backtick version namespaces all things declared with the macro to the current namespace. So in this case, anything declared would be namespaced to tokyo-cabinet. The quote version does not do this.

The # macro is the next confusing thing. When you have a macro, there’s a chance that you will be overwriting a symbol in the original source. If the caller of “use” already had “hdb” defined, I would overwrite it. The # macro automatically generates a unique name, so all references to #hdb will actually refer to something like hdb_1829_auto. Finally we’re using with-open. This macro binds hdb# to the HDB() instance for all the expressions passed to it. It then calls .close on hdb# when it exits. It basically does exactly what we want.

(.open hdb# ~filename (bit-or HDB/OWRITER HDB/OCREAT))

The new thing here is the “~”. The tilde is the unquote macro. Since we’re in a quoted form (due to the backtick on the previous line), we can’t actually access filename. filename would need to be defined in our caller. The tilde pops us out of the quoting for a second to pull in the passed filename. After that, we continue to be quoted.

We still have a problem here. “get” and “put” both refer to tokyo-cabinet/*db*, but *db* is not defined. Luckily, we can easily fix that.

(binding [*db* hdb#]

This binds hdb# to *db* for whatever expressions are passed in. Since we’re using the backtick instead of the quote, this is automatically converted to tokyo-cabinet/*db*, which is what we want.

(do ~@body))))

“~@” is the last bit of magic. Basically this is the same as ~, except you can pass a sequence and it will apply ~ to every member. This is what we want, for every expression passed to be executed in the context we’ve defined.

As time goes on, I will add more of the Java API into this wrapper. You can follow the project on github if you’re interested in the updates.

That’s all for now! We can now store and retrieve things. After I spend some time building that into the site, we’ll be largely done with the Clojure bits. I’m sure a couple more things will come up though!

Things are cruising right along now in creating my awesome twitter portal in clojure. So far we have gotten set up with compojure, started using the twitter API to grab data, and built some forms to make sure the data is relevant to the logged in user. The next little chore is to find URLs in tweets and make them into actual, clickable links. I want to keep this simple for now, so we’ll just find http:// or https:// and link that.

The Code

It turns out that the code to do this is really simple. Clojure just uses Java’s regular expression engine, but integrates it into the language a bit cleaner than Java does. A big thanks to Fatvat for basically walking me through it.

Nothing too complicated here, but there is an interesting new concept. For the first time ever, Clojure doesn’t do everything we want and we talk to Java. This is one of the most powerful attributes of Clojure. Even though it’s a young language, it’s built on a mature platform that does basically everything you need. In this case, we wanted to mutate the “text” string. This isn’t exactly kosher in a functional language, but I didn’t want to slice and dice the text when there was a perfectly usable Java method that would do the replacement for me.

Anyway, how does this work? “.replaceAll” is a method of java.util.regex.Matcher. What we’re trying to express in Java is:

In clojure, re-matcher returns matches constructed out of applying a Pattern instance to a string (“text”). So, we’re applying the .replaceAll method to the object returned by re-matcher, which is a Matcher instance created out of a Pattern (indicated by the “#” macro). This is exactly what we want, expressed in a nice, functional style. After the instance that we’re operating on, we can pass additional arguments to the method. In this case we pass the replacement string.

Another thing you might notice is the string in the urlize function definition. Clojure has extensive support for metadata, which is something that I’ve largely ignored. In it’s simplest form, you can pass a string to defn as I have done, and that will be included as the docstring. The language also includes introspection features to pull these things out, but I have yet to investigate them in depth.

Again, pretty straightforward, and now we’re starting to do some real damage. I think I’m going to dive into JavaScript and CSS for a while, but I’ll be back soon with static storage. It should be fun! As always, all the code is on github.

There has been a lot of debate in the past few months about what the government should do about all these “toxic assets” that various financial institutions have on their books. My less than educated opinion is that it doesn’t matter, but the government has to make its intentions clear, and it has to make them clear fast.

These “toxic assets”, which are largely comprised of bad mortgages to people who will never be able to pay them back, have an undefined value. Nobody is really sure how much they are worth, if they’re worth anything at all. Once people realized that these assets were of questionable value, and certainly worth a lot less than previously thought, the total value of the financial institution plummets down. This is an overly simplified version of what makes these institutions “insolvent”.

The customary thing for a bank to do when a mortgage cannot be paid back is to foreclose on the home. This allows the bank to sell it for something close to its actual value and recoup the money they lost on the bad loan. This is what happened en mass at the beginning of the financial crisis. However, it’s my belief that this is only continuing because of governmental indecision.

Houses are as close to worthless now as they have been in anybody’s lifetime. We suddenly find ourselves in a situation where there are far more houses than there are people with the means to purchase them. This means that foreclosure no longer makes sense. If a bank cannot sell the property they foreclose on, then it is in their best interest, by far, to renegotiate the mortgage. In many cases, the people most willing to pay for a house are the people who don’t want to lose their home, and they’ll pay what they can.

The government sees this, of course, which is why they want to help. By purchasing all these bad assets, or so the argument goes, they can get what value is there without throwing people out on the streets. However, they have not made their intentions clear. The government’s failure to do something or say that they will not do anything is the single worst thing they could do. As it is now, financial institutions have no incentive to renegotiate mortgages because there is a sporting chance they will get a better deal from the government. If the bad mortgages look worthless, so much the better, the government has to act to save these institutions that appear insolvent and are “too big to fail”, and the government will look at those “worthless” assets, see that they are not worthless, and see a potential money making opportunity (especially after the savings and loan crisis). In essence, the financial institutions can’t lose, as long as they keep these toxic assets looking worthless.

I have my personal opinions about what the government should do, but anything is better than straddling the fence, debating whether or not to do something or not. These institutions will be forced to work out their problems, or they will have them taken off their hands. Either outcome is preferable to the current  predicament of millions of people in default. This isn’t the banks’ fault, it’s the government’s, and it’s crucial that they do something about it.