Caffeinated Simpleton

Things are cruising right along now in creating my awesome twitter portal in clojure. So far we have gotten set up with compojure, started using the twitter API to grab data, and built some forms to make sure the data is relevant to the logged in user. The next little chore is to find URLs in tweets and make them into actual, clickable links. I want to keep this simple for now, so we’ll just find http:// or https:// and link that.

The Code

It turns out that the code to do this is really simple. Clojure just uses Java’s regular expression engine, but integrates it into the language a bit cleaner than Java does. A big thanks to Fatvat for basically walking me through it.

Nothing too complicated here, but there is an interesting new concept. For the first time ever, Clojure doesn’t do everything we want and we talk to Java. This is one of the most powerful attributes of Clojure. Even though it’s a young language, it’s built on a mature platform that does basically everything you need. In this case, we wanted to mutate the “text” string. This isn’t exactly kosher in a functional language, but I didn’t want to slice and dice the text when there was a perfectly usable Java method that would do the replacement for me.

Anyway, how does this work? “.replaceAll” is a method of java.util.regex.Matcher. What we’re trying to express in Java is:

In clojure, re-matcher returns matches constructed out of applying a Pattern instance to a string (“text”). So, we’re applying the .replaceAll method to the object returned by re-matcher, which is a Matcher instance created out of a Pattern (indicated by the “#” macro). This is exactly what we want, expressed in a nice, functional style. After the instance that we’re operating on, we can pass additional arguments to the method. In this case we pass the replacement string.

Another thing you might notice is the string in the urlize function definition. Clojure has extensive support for metadata, which is something that I’ve largely ignored. In it’s simplest form, you can pass a string to defn as I have done, and that will be included as the docstring. The language also includes introspection features to pull these things out, but I have yet to investigate them in depth.

Again, pretty straightforward, and now we’re starting to do some real damage. I think I’m going to dive into JavaScript and CSS for a while, but I’ll be back soon with static storage. It should be fun! As always, all the code is on github.

There has been a lot of debate in the past few months about what the government should do about all these “toxic assets” that various financial institutions have on their books. My less than educated opinion is that it doesn’t matter, but the government has to make its intentions clear, and it has to make them clear fast.

These “toxic assets”, which are largely comprised of bad mortgages to people who will never be able to pay them back, have an undefined value. Nobody is really sure how much they are worth, if they’re worth anything at all. Once people realized that these assets were of questionable value, and certainly worth a lot less than previously thought, the total value of the financial institution plummets down. This is an overly simplified version of what makes these institutions “insolvent”.

The customary thing for a bank to do when a mortgage cannot be paid back is to foreclose on the home. This allows the bank to sell it for something close to its actual value and recoup the money they lost on the bad loan. This is what happened en mass at the beginning of the financial crisis. However, it’s my belief that this is only continuing because of governmental indecision.

Houses are as close to worthless now as they have been in anybody’s lifetime. We suddenly find ourselves in a situation where there are far more houses than there are people with the means to purchase them. This means that foreclosure no longer makes sense. If a bank cannot sell the property they foreclose on, then it is in their best interest, by far, to renegotiate the mortgage. In many cases, the people most willing to pay for a house are the people who don’t want to lose their home, and they’ll pay what they can.

The government sees this, of course, which is why they want to help. By purchasing all these bad assets, or so the argument goes, they can get what value is there without throwing people out on the streets. However, they have not made their intentions clear. The government’s failure to do something or say that they will not do anything is the single worst thing they could do. As it is now, financial institutions have no incentive to renegotiate mortgages because there is a sporting chance they will get a better deal from the government. If the bad mortgages look worthless, so much the better, the government has to act to save these institutions that appear insolvent and are “too big to fail”, and the government will look at those “worthless” assets, see that they are not worthless, and see a potential money making opportunity (especially after the savings and loan crisis). In essence, the financial institutions can’t lose, as long as they keep these toxic assets looking worthless.

I have my personal opinions about what the government should do, but anything is better than straddling the fence, debating whether or not to do something or not. These institutions will be forced to work out their problems, or they will have them taken off their hands. Either outcome is preferable to the current  predicament of millions of people in default. This isn’t the banks’ fault, it’s the government’s, and it’s crucial that they do something about it.

After the first two posts in this series, we have a site that displays the twitter public timeline and twitter searches. Now let’s customize this by allowing users to access their own twitter accounts.

To do that, we need to allow users to log in to their twitter accounts. Twitter hasn’t rolled out their OAuth solution yet, so we’ll have to ask users to trust us with their twitter names and passwords. For now, we won’t store the password, so all we need to do is get it out of a form and store it in a session.

Processing Forms in Compojure

This is pretty easy. Compojure provides a “params” hash-map that has the parameters of the servlet, which we will pass to a “login” controller. There are several things provided to the servlet. I won’t bore you with details of creating the form itself, so here’s the server side code that processes “twitter-user” and “twitter-password”.

As you can see, we have some new syntactic goodies to learn about. “session” is a map provided by Compojure. It is thread safe, which in clojure, is done through transactional memory. Dealing with that transaction is what most of this code does. dosync is a macro that allows many expressions to be executed in one transaction. alter takes a reference (session), a function that will alter the reference (assoc) and the arguments for the altering function. We finish our dosync call, which executes the transaction, and our session is all set.

After we set up our session, we redirect to the url of the user. We pull the user name out of the session. The “@” symbol is a macro that refers to the deref function, which allows us to get values out of references. If no name was provided, we kick back to the home page.

Pretty clean and painless. Clojure isn’t so bad.

After yesterday’s introduction to the syntax of clojure, now I’m going to start organizing my code a bit more and pulling in real data.

Templating

The first thing I want to do is abstract out a base page so that I don’t need to worry about headers, JS libraries, standard CSS, and all that when going from page to page. To do this, I’ll create a “page” function in a new namespace.

Then I add this into our original page.

Now I can call the “page” function, pass it a title and some contents, and it outputs a full page. Perhaps later on we’ll make it so we can derive templates from other templates, but I have no need for that yet.

Getting My Twitter Feed

Let’s make it so that getting “/justin_tulloss” will give me my twitter feed. To do that, I add a handler to our servlet.

Now when I go to http://localhost:8080/justin_tulloss, it will say “Welcome justin_tulloss”. Pretty high tech stuff.

Now let’s see if I can change that justin_tulloss parameter into my actual twitter data. First, I head over to the twitter API docs. The public timeline doesn’t require any authentication, so let’s pull that down first. The public timeline is a list of “status” messages, which we will map to html representations of the same. To do this, I used the clj-http-client library by Mark McGranaghan. It was a pain to get installed because I don’t understand Java JAR files fully, but once I got it figured out, it worked great.

Hopefully this code isn’t hard to follow, but let’s take a look. The “Welcome” part is the same. Then we map everything that we read from the twitter URL into a twitter-status function, which just grabs some data out of the statuses and displays it.

Referring back to the non-templating code, the “read-json-string” function is in clojure.contrib.json.read, which you can get here. The *twitter-url* is a global constant. Apparently it’s a lisp thing to surround globals with asterisks. I wonder if that also applies to global constants.

Well, that’s all for tonight. Next time we’ll fetch some more relevant data from twitter and see what we can do with it. As always, the complete code is on github.

I’ve decided I want to learn clojure. Clojure is a lisp dialect that runs on the Java Virtual Machine. It’s all the rage these days, and I want to be cool. I’m going to post my trials and tribulations here for others to read and hopefully learn from. If you are a better hacker than I and see problems with what I’ve posted, please comment! This is going to be a multi-day process, and a slow one as I don’t have much time, but I hope to finish a project within the next six months.

There are a couple problems with Clojure and I. First of all, I’ve never programmed in any lisp dialect. Second of all, I’ve never programmed in Java. Clearly I do not have the tools to get into this, but I’m going to go for it anyway.

The Project

I’ve been thinking a lot about Twitter recently, and there are some really cool applications of Twitter that I find kind of exciting. However, Twitter’s signal to noise ratio can quickly become unbearable. So, I thought I’d come up with a Twitter feed app that will organize people you care about, conversations you want to monitor, tweets around where you are, and the regular twitter feed into a sort of Twitter portal. It should be fun and easy, and it’s not the project that’s important. It’s learning Clojure. I’m going to call it Flockr for maximum Web 2.0 street cred. You can find the source on github.

Getting Started

I’m using Compojure as my web framework. I like web frameworks, especially for little projects like this that don’t need any specialized plumbing. Compojure isn’t terribly mature, but it should be good enough to get me off the ground.

Installing compojure is easy. Just download, compile with ant, and put the resulting jar file in your classpath. Being a vim guy, I wanted clojure syntax highlighting and auto-indenting. There’s a vim package available to do that stuff, and I recommend it.

The First Page

This is basically copied straight off the compojure “Getting Started” page. However, this is the first time I’ve really looked at lisp, so let’s see what this does.

Clojure Basics (or at least, what I’ve inferred about Clojure thus far)

Parentheses indicate a list. I’m a big C/Python/JavaScript guy, so this a bit strange, but I can deal. Evaluation is simply evaluating every list that is passed into the REPL. This first program has three things that are evaluated. The first is the namespace. The second is the servlet, and the third is the server. Let’s that a closer look at the first line.

Returns the result of the “ns” function, which creates a new namespace called “index” (in this case). Within this namespace, we are dependent on the “compojure” namespace. We pass the “:use” key to indicate this.

The colon syntax is something I’m still getting used to. It means a key, which makes sense in a map, but it would appear that you can also put keys in other data structures.

The only other things you need to know to understand the rest of the code is “[]” and “{}”. “[]” is a vector. I’m not certain I understand when to use vectors versus lists except for performance. “{}” is a map. A colon indicates a key (IE :port), the thing following it is the value (IE 8080). Commas are optional, and don’t seem to be used in examples at all.

Doing something real

Ok, so we’ve got something up and running. That’s good. Now let’s grab some twitter statuses. Unfortuneatly, it’s past 4 in the morning. This will have to wait for another day.