Friday, April 1, 2011

Writing Application Security Specifications with Geb and Spock in Groovy

I've had a chance to play with the excellent Geb and Spock frameworks for Groovy a little more lately and can say my technique has greatly improved.  I think getting a firm grasp on Geb's Page Object Pattern has really improved the expressiveness of the specifications.  For example, here's my latest security spec for an app I have been working on.



I bet non-technical folks can read that and understand what the behavior around security roles are by reading this specification. In fact this is an artifact I can share with the product owner to verify we are on the right track.

He's another that verifies some very specific UI logic. Certain users only see certain menu options.



Remarkably expressive in my opinion and dead simple to write.

Sunday, March 20, 2011

About Relational Database Management Systems

Unless you have your head under a rock you've heard about the NoSQL movement.  In general, a NoSQL database is not based on the classic, relational database model, but rather uses a document-oriented approach.  As the name implies, the big difference is that you no longer use SQL it insert/update/delete data.

The biggest problem I have with relational databases is that storing a well defined domain model is hard to do in a relational database.  The fact is that in order for a relational database to scale you need third-normal form and a normalized database is hard to map to a well defined domain model.

I have experience with scaling databases.  I have worked at Overstock.com and Discovery Channel building consumer-facing web sites backed by Oracle databases.  I can not recall a time when a performance problem was ever due to problems in the middle tier.  Generally speaking I most often encounter performance problems in the database tier.

A lot of people say "relational databases don't scale!"  I'm not saying that.  I've served 5 milllion page views per day with an Oracle back-end.  The problem I see is that for RDBM's to scale you have to be a witchdoctor.  You have to ruthlessly normalize and tune.  Usually, the tuning that you have to do to achieve performance comes at the cost of maintainable, reusable application code.  Often I have had to alter a clean, reusable object model in order to get performance out of the database.

So the promise of NoSQL databases like MongoDB is that the details of persisting data is obscured from the user.  Rather than worrying about SQL and performance you can define your object model as you see fit and just save it.  My initial experience using MongoDB with Grails has been very positive.  I have a robust domain model and I can just call save() to store it.  Querying syntax is much easier and makes more sense to an object-oriented developer, in my opinion.

Does it scale to millions of hits?  I'm not sure.  More on that later.  However, for new development my money is on MongoDB.

Thanks for reading!

Saturday, March 12, 2011

Generating Encrypted User Keys in Python

There is no doubt that if you want your web application to scale it needs to be stateless. As soon as you start maintaining session state on the server you have a problem. More users means more memory required. That is very bad if you hope to serve millions of concurrent users.

One of my pet projects at the moment is a consumer facing web site related to movies. Early on I made the decision that the site would be absolutely stateless. In order to achieve this and still maintain the illusion of statefulness I simply pass a unique user ID with every call. One important element to consider with this approach is the potential you have of opening yourself up to attackers who hi-jack other user's ID's. If you make the user's unique identifier an integer you are hosed. Attackers will send requests using other people's ID's and make a mess of your site quickly. It's a rookie mistake.

The answer to this problem is to generate encrypted tokens for each user. The token must have strong enough encryption that no reasonable person would have the time or resources to crack it. I actually took a multi-layered approach to generating the tokens. Below is the code I used to generate my tokens.



In the encode_user_id() method you send in an integer which is the user's unique ID generated by the database. First we setup the AES encrypter with a crazy stupid hard key. Using a key like "key" is stupid. Use an insane combination of alapha, special characters, and numbers. This ensures the encryption is hard to crack. Then we pad the ID with characters so we have a 32-bit key to encode. This also adds to the complexity of the encryption. Finally, so we can pass the token around in URL's and in cookies we need to Base64 and URL encode it.

Here is a simple unit test that verifies the behavior of the API.



As you can see we simply send in an integer and verify that it's encrypted and encoded correctly. Then we feed the token back into the API to transform it back to an integer. With the user's integer Id in had we can easily look up the user in the database.

Yes, i could have made each user's primary key some unqique hash string. However, I need to scale to millions of users (hopefully!) and doing queries on 32-bit strings is not nearly as efficient as on integers.

Sunday, March 6, 2011

Polyglot Programming :: Using Groovy, Spock, and Geb to Test Python Applications

I have a real problem with programmers who say "I'm a language "X" programmer.." When you claim to only work in one language what you are really saying is that you don't care enough about your craft to learn better ways to solve problems. The idea that any one language can efficiently solve all problems is silly. Languages have strengths and weaknesses. An important skill in software development is being able to bring a particular languages strengths to bear on a particular problem.

In a previous post I talked about using Python to do web scraping. You can use virtually any high-level language to do this but Python (and the lxml.html library) make very quick, efficient work of it.

I have been recently working on a pet project that involves movies. I chose to write the website in Python, largely because Python does not have a lot of the FUD you see in the Java community these days (that's good for another blog post entirely). Regardless of why I wrote the web site in Python, the fact is that Python does not have great behavior-driven development (BDD) tools nor does it have great integration test tools.

Groovy, on the other hand, has outstanding BDD test tools as well as many terrific integration test tools to choose from. For this project I decided to use the most excellent Spock framework for BDD and Geb for web browser automation. I chose Spock based on a very non-scientific query of Twitter folks. Most people in the Groovy community seem more interested in Spock. I have used easyb in the past and it is a fine BDD tool. However, prevailing wisdom seems to be steering everyone towards Spock. I have been very happy with it.

Another stand-out in the Groovy community is a new framework called Geb which is based on the excellent WebDriver framework. Geb provides a very clean DSL-like syntax that makes quick work of doing browser automation testing. Geb also has a jQuery-like syntax for selecting elements on a page. I'm a huge fan of the jQuery selector syntax and Geb's support for it is a major win in my book.

Here is a simple test I created early on for my movie site which uses Spock and Geb:



These tests actually fire up a browser and run the web site and verify behavior. The first specification is: "Load the home page correctly." For that we simply need to go to the root of the server and verify that a few elements exist on the page. The second specification says: "Verify the movie view loads correctly." This one involves verifying that when a user clicks on a movie icon that the movie's details load correctly. Due to the jQuery syntax specifying elements on the page to interact with is simple and concise (maybe not so simple if you don't know jQuery.)

The Groovy/Spock/Geb combination is extremely powerful. I can't imagine building web applications today without using it. This is an excellent example of how a specific language (and accompanying tools) are very well suited for a particular task even when the rest of the application may be written in something else.

Power of Python :: Web Scraping 5 Lines or Less

Python is an extremely flexible and productive scripting language. I have been using it for a lot of tasks lately not only because of it's flexibility but also because it is powerful and extremely stable. There are ports for every major operating system but I prefer to use it on Linux (I prefer to use everything on Linux, actually.).

One place where Python really shines is in web scraping. For those who don't know, web scraping is the act of programmatically pulling values of the pages of other web sites. When you use the lxml.html library this becomes trivial (as long as you have good XPath fung-foo).

Here's a sample method using the IMDB movie web site as an example. This method loads the specified IMDB url and pulls out any IMDB id's found on the page.




Here is a simple unit test to demonstrate the behavior:


As you can see from the unit test we are loading the IMDB top 250 page and expecting 250 id's in return. All of the work inside ImdbService.get_ids_from_page() is being handled by lxml.html's parse method and some basic XPath.

Monday, January 24, 2011

Groovy's XmlSlurper :: Superior XML Processing For Java

I am a huge fan of Groovy.  Groovy is a dynamic language for the Java Virtual Machine that compiles to Java byte-code.  One of the great features of Groovy is that fact that any library available to Java is also available to Groovy so you can leverage all of your existing knowledge of the Java stack in Groovy.  Groovy also has a nice, dynamic syntax that is fun to use.  It supports many unique language features, most importantly closures!

As we will see in this post one of the great things that Groovy offers is a class called XmlSlurper.  Since Groovy is a dynamic language it offers a unique way of processing XML using a dot notation syntax.  Let's take a look shall we?

First, here is the XML we are going to parse:



It's a pretty simple XML document with one complex type Person with a two nested complex types called Address. Let's see how Groovy's XmlSlurper makes quick work of it!



The beauty of XmlSlurper is in the fact that it takes only one line of code to parse the XML document and return an object representation of it. Using dot notation syntax you can easily get value out of the document. Notice how we can reference the address using either the array syntax or using the find() method with a closure. The closure method is a little more "groovy."

To high-light the elegance of this solution let's look at the equivalent in Java using the standard JAXB library. The first thing we have to do is define an XML Schema for the document since JAXB works on Schema. That, in and of itself, is lame. What if we just need a quick XML document format to throw around for a few weeks then throw away? You still need a schema. Once you have a schema you have to run the xjc compiler on it to generate the stub code. Finally, after you have defined your Schema *and* have generated the JAXB code, only then can you write some Java code. Here is the Java equivalent.:



First note the added ceremony initializing the JAXBConext and Unmarshaller objects. After which you still then have to call the unmarshall() method passing in a java.io.File object. Finally you get a reference to Person where you can then use similar Java-style getters to get values out of the document.

I don't care for the JAXB solution simply because I have to first run code generators if I want to use a clean dot notation syntax to get values out of the document. It only adds another step in the build process but invariably, over time, it is forgotten about and is prone to breakage. I also think the JAXBContext and Unmarshaller objects don't make sense. I still have to reference the documentation when I use JAXB because I never remember the syntax the parse the document.

The great thing about XmlSlurper, and Groovy in general, is you don't have to replace all of you code just to get the benefits of it. If you are using Spring it is dead-simple to call Groovy classes from Java code.  You can write your XML parsing code quickly in Groovy and then call the code from your existing Java application.

Groovy also has superior file-handling capabilites by way of its File object.  If you mix XmlSlurper with Groovy's File object you have the ability to write extremely clean code for processing hugh amounts of XML.  We'll talk about that some other time!

Thanks for reading!

Sunday, January 23, 2011

JavaScript Design Patterns :: Managing Asynchronous Calls

About a month ago I started a small project that I call SocialMapper, which is a fun app for rendering your social graph on to a map.  It is a Google Maps/Facebook mashup that further expresses my affinity for service-oriented architecture.  The real reason I wanted to do SocialMapper is to do some advanced JavaScript programming.  This is not my first forray into JavaScript programming.  I built some web apps in early 2000 that had some cool DHTML interfaces.  JavaScript back then was horrible.  You essentially had to have two code-bases, one for Internet Explorer and one for Netscape.  It was a nightmare to maintain.  Today you have great libraries like Dojo, Prototype, and my personal favorite, jQuery.  All of these libraries do two important things:  First, they hide implementation details specific to each browser.  Second, the let you do really complicated things with less code.  For example, if you want to do a simple Ajax call to your server for a snippet of HTML and load it into a DIV container all you have to do with jQuery is:



Sweet! To do that in raw JavaScript to support the major browsers you have to write about 10 lines of code.  If you stay within the confines of jQuery life is very easy.

As I discovered while writing SocialMapper, however, you run into some major problems when you have to rely on multiple libraries to get your work done.  To illistrate let's look at the problem in some over-simplified JavaScript:



This code is mostly using the Facebook JavaScript library to first go out and pull in your personal list of friends. Then, using the jQuery .each() function we iterate on your list of friends. For each friend in your list we again use the Facebook API to pull down the friend's details. Depending on how many friends you have this could mean 1,000's of calls to Facebook for friend details. More importantly, the call to FB.api() is asynchronous which means we can't inline the code and call renderFriends() immediately after the first FB.api() call because all of the callbacks have not completed yet. Are you still with me? Good!

What we need here is a way of delaying the call to renderFriends() until we know all the callbacks have completed. If we were staying strictly in the jQuery library, i.e., using $.ajax() methods this would be an easy. We could register a global event hanlder for .ajaxComplete(). But since we are not using jQuery for these calls that does not work.

The solution I found was to use a global variable to keep count of asynchronous calls in conjunction with a JavaScript timer. Before you execute the first asynchronous call you create a timer and assign the ID to a global variable. Then as you execute asyncronous calls you increment your call counter. In the callback of each call you decrement the timer. While your calls are being executed the timer calling your renderFriends() method every 500 milliseconds. Inside your renderFriends() method you are first checking to see if your call counter is zero. To illustrate here is our example from before now enhanced:





As we can see the first thing we did is add two new global variables, callCounter and timerId. Before kicking off our remote calls to FB.api() we create a JavaScript timer which calls renderFriends() every 500 milliseconds. Then as we call FB.api() we increment our counter. In our callbacks we decrement it. All the while the timer has been calling renderFriends(). In renderFriends() we are first checking to see if the counter is 0 and if it is not we exit. Only when the counter is zero is it "safe" to proceed to the rendering logic.

This works but feels like a bit of a hack to me. The reliance on global variables gets my spider senses tingling. What if some other method inadvertently increments that counter by just one? Our code breaks because counter will stay at 1 forever. Boo. I found this solution on Stack Overflow (I can't find the article right now) so other people are doing it like this. Maybe I would feel better if the counter and timer variables were encapsulated in an object?