Saturday
Jul312010

NoSQL on the Cloud: Our First Application

Banner Cloud formation on the Matterhorn

We're ready to get started making NoSQL apps on "the cloud", so let's review our cloud base image, log into it, and get going!   We'll use "describe-instances" to get its ID, and then use SSH to log in.



So far, so good.   Now that we're in our cloud application we can install Mongo (or any other NoSQL data store), create our data store, and front-end it with a new application.



INSTALLING MONGODB



First, we've got some building and utility programs to install.   From the command line we enter:



$ apt-get -y install tcsh git-core scons g++

$ apt-get -y install libpcre++-dev libboost-dev libreadline-dev xulrunner-1.9.1-dev
$ apt-get -y install libboost-program-options-dev libboost-thread-dev libboost-filesystem-dev libboost-date-time-dev



Next, we'll import the 10gen signing key (10Gen is the company that makes mongodb)

$ apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10



And we'll add the 10gen repo to the apt sources.list



$ sudo vi /etc/apt/sources.list



add the following line for ubuntu 10.10:



deb http://downloads.mongodb.org/distros/ubuntu 10.10 10gen



save and exit.



Finally, we'll install mongo-db.   Linux has a lot of nice installation utilites, and a series of apt-get install commands makes this easy.



$ apt-get -y install mongodb



Then check the installation

$ mongo
MongoDB shell version: 1.4.4
url: test
connecting to: test
type "exit" to exit
type "help" for help
>



NOW WE CAN CREATE A MONGO TEST APPLICATION

The important thing here in creating our application in Rails is to avoid loading ActiveRecord, which we won't need or use with Mongo as our data store. One way to do this is with the --skip-activerecord switch.  We'll call this application mongomapper, because it should give us a nice quick overview of Mongo and the mongo_mapper gem that gives us active-record-like access in Mongo.   You can create your app skeleton like so:



$ cd /var/www/apps
$ rails new mongomapper --skip-active-record

$ echo "rvm --create use 'ruby-1.8.7-p302@example_app'" > ~/projects/example_app/.rvmrc



Ruby 1.9.2 appears to be about 2X faster than the old standby 1.8.7, but lots of older code isn't 1.9-compatible, so for now we'll stick with Ruby 1.8.7. We'll edit our Gemfile next, to give it the gems we'll need for this Mongo demo. We edit the Gemfile to bundle your application's dependencies, and the ability to manage gem versions is one of the terrific advances that Rails3 makes over Rails2.   Here's how we edit the Gemfile -- add the following code for the gems and dependencies we'll see with our Mongo front-end application:  



require 'rubygems'

require 'mongo'

source 'http://gemcutter.org'

gem "rails" 
gem "mongo_mapper"

gem "ruby-debug"



Next we'll add our test environment. We won't use this for this quick demo, but these are terrific tools that we'll want available for all of our projects. Add the following as well to our Gemfile:



group :development, :test do
gem "rspec-rails", ">= 2.0.0"

gem "cucumber-rails", ">= 0.3.2"

gem "webrat", ">= 0.7.2"

end



RSpec and Cucumber are probably the most popular test packages for Rails applications, and it's great to have them at-the-ready here.   I actually prefer "capybara" for Rails integration testing, but I haven't worked with it enough yet to make it part of my standard app install package.   Now we're ready to create and install our bundle of ruby gems specific to this application.   "Bundle install" I've found to be my answer to most unplanned startup problems with Rails3 apps -- a good bundle install here and we're rolling.



$ bundle install


Your bundle is complete! Use `bundle show [gemname]` to see where a bundled gem is installed.



INITIALIZING MONGO



The last thing we need to do is to create an initializer in our Rails app to connect to MongoDB. Create a Ruby file in config/initializers. We could call it any name we like; but here let's stick to convention and call it config/initializers/mongo.rb:

MongoMapper.connection = Mongo::Connection.new('localhost', 27017) 
MongoMapper.database = "#myapp-#{Rails.env}"
    if defined?(PhusionPassenger) 
        PhusionPassenger.on_event(:starting_worker_process) do |forked|
        MongoMapper.connection.connect_to_master if forked
    end
end



I've created a simple Rails app for accessing Mongo, the files of which are shown below.   We have Mongo installed and ready, so if we fire Mongo up we can then fire up our Rails app and work with our NoSQL data store.  We'll make sure Mongo has a place to store its data, and then lauch Mongo with the run command:



$ mkdir -p /data/db     #the -p option creates a /data directory if there isn't one yet as well

$ mongod run &



We can also bring up a browser, and enter: http://ec2-174-129-81-163.compute-1.amazonaws.com:28017/ and see the Admin screen for our MongoDB data store in all its glory!   (Note:  I've done a bit more bookkeeping with nginx, thin, and dynamic DNS that I'll cover in my next posting)
Even better, we can now start our Rails test application, and play with our Mongo store in the cloud.   We've created the application in the conventional /var/www/apps/mongomapper, so from that directory we can start our application with:

$ rails s thin  #Note -- I've got nginx fronting thin on port 80, but either way start Rails here...

We can then enter our URL in a web browser, and we can see our Rails/Mongo/Amazon cloud application:

http://ec2-174-129-87-241.compute-1.amazonaws.com/notes/  # OR, with dyndns:  http://jkr-blog-dyndns.org

And this command takes us to our Mongomapper application home page:





We can then click on "Add Notes" and add a note that will be stored in MongoDB:




And when we click on the Save button, our new record is saved, like so:



We can now go back to our home page and we'll see our first note added to the list.   We can add more notes, edit them and delete them in a simple little introduction to Rails' and Mongo's simplicity, and maybe indication of a hint of the power of the Amazon (or similar) cloud, Rails, and Mongo.   

This is, admittedly, still just a little toy app, but we can create it's bigger brothers and sisters with ease -- just by adding more data and then focusing our work on the analytics that follow the data store.  This is just a beginning, but if we can do simple things quickly, impossible thing should take only a bit longer!



More to follow:

Wednesday
Jun302010

The Software That Links It All

Not long ago, when you kicked off a new software project, perceived advances in technologies and methodologies made it seem desirable to start with almost EVERYTHING new. Each project selected from among the latest frameworks, toolsets, and languages, and data modeling, tiering, and presentation approaches. With so many unknowns and so many decisions to make the resulting projects started on shaky ground, and many fairly promptly resulted in failure.

Hope was on the horizon, but it's taken some time getting here. Back in 1996 Ann Winblad spotted the way out. Brad Cox's ideas of a software factory had some truth to them, and Winblad spoke of a future when she wrote:

Small developers have become specialty parts suppliers to the growing population of software assembly-line workers.

She was singing the praises of "object oriented" programming, and with time the great truth here has only gotten truer. The key was not "object-oriented programming" over its predecessor "procedural programming", but a design doctrine that is far more profound: Convention over Configuration.

The right conventions get things rolling sooner and more surely, and let development teams be more starfish than spiders. As approaches and toolkits become conventions there are far fewer niggling decisions that need to be made, and everything can move a lot faster.

A couple of years ago, hiring a good team, getting set-up, writing a decent spec and delivering salable code for a modestly-scoped project might be done in 6 months. Today the same tasks, with still-greater use of conventions, might be done in half the time. This for projects that might have taken years back before the Internet era.

There are more tools and conventions than I can write about in a single post, but to give you some idea of the basic toolkit -- all tools that can be chosen conventionally and adopted -- just among the FOSS toolsets we have

  • SCM tools (Subversion, Git...)
  • Cloud providers (Amazon, RackSpace, Google...)
  • Virtualization environments (VMWare, Xen...)
  • Conventional Relational databases (mySQL, PostgresSQL...)
  • "NoSQL" databases (MongoDB, CouchDB, SimpleDB, Cassandra...)
  • Mid-tier frameworks (Rails, Django, Sinatra, Merb (now merging into Rails)
  • Languages (Ruby, php, perl...)
  • Presentation frameworks (JQuery, Prototype...)
  • Testing frameworks (RSpec, Cucumber, Selenium...)

There are lots more, but the remarkable thing about the design of modern frameworks is that each layer is self-contained and with the progress of the codebases it's possible to swap components in and out while maintaining the core of a working application. In each of the major projects I've worked in over the past three years, we've swapped framework components without affecting the overall schedule or milestones.

Rich frameworks and solid conventions count for more than you might think. In 2010 the question is no longer "can we build it?" but rather "what is most worth building?"

The software that links it all is ushering us all into a new age....

Saturday
Jun122010

June 11 - The Undersea World at 100

I miss the world I grew up in. Every age has its angels and devils, but this was a magical time when you had to believe that the angels were winning! I remember the Apollo space program, and things like summer camp on July 20, 1969 and the newspaper headline that day as if it were yesterday. These were dynamic times and scientific times, and we were proud of our scientists and our science. From July 20, 1969 to freshman year at MIT 10 years later was a short step for me.

Jacques-Yves Cousteau

In that world Jacques Cousteau was one of my heroes. He'd have celebrated his 100th birthday yesterday, and I still marvel at the "undersea world" he created in the very short time since his birth in 1910. I grew up with "scuba gear" as a cool idea, when he grew up such gear wasn't even invented yet.

He first did the inventing (the air regulator was the breakthrough of the "self-contained underwater breathing apparatus"), and then he used his inventions to share his love of the sea with the whole world. Everyone my age remembers his TV shows - so interesting, so passionate, so scientific, and so French. He had the best voice I ever heard on TV, and no one could have conveyed the science, the beauty, and the love of the sea as he did. Our world is a better place because he lived, born 100 years ago yesterday.

He would be heartbroken at the Deepwater Horizon disaster, and mystified at how we could have lost respect for the sea he loved so much. If there is science in the solution, that science might well have come from him. I admire his world, his life and how he lived it, and I hope at 100 his spirit still lives on with us.

Aye Calypso the places you've been to
the things that you've shown us,
the stories you tell.
Aye Calypso I sing to your spirit
the men who have served you so long and so well...

100 Years. Happy Birthday, Captain Cousteau.

Here's a final bit - John Denver singing live at his 75th birthday, and Cousteau's mission for the rest of his time with us. Bon anniversaire, Commandant!

Saturday
Jun122010

WHAT COMES NEXT: THE PAD, THE CLOUD, AND THE SOFTWARE THAT LINKS IT ALL

It is the Tale, not He who tells it. - Stephen King "Breathing Method"

Part 2. The Cloud

I love cloud computing, and I've found it to be a real blessing to anyone who may have lots of ideas but doesn't have a lot of money. It's also an untold blessing to anyone with a phobia for filling out hardware and rack space purchase orders. Before we dive into the wonders of the cloud, I have to add a bit more of a cautionary tale, from Stephen King again:

"There is nothing there, in the dark, that isn't there in the light..."

I'm often asked questions like: "I think the 'cloud' is cool ... Can we make a cloud-based software offering?" The answer is "of course," but this is where King kicks in -- there's nothing intrinsic to a "cloud offering" that adds value -- a lame offering will still be lame in exactly the same ways when you put it on the cloud! The key to the cloud (as the key to horror fiction) lies in the invention of things in the dark that could never exist in the light. Now THAT is worth talking about.

I've closely followed some cloud-based offerings in the past few years -- Sales Sonar at Chemidex, and a sample online application that I'll include here just for contrast. These two examples highlight just what the cloud can and cannot do.

Chemidex describes Sales Sonar as revolutionary, and I believe that assessment is justified. With Sales Sonar the cloud is used to link companies, products, buyers and sellers - with terrific graphics and the kind of analytics that only the cloud can provide. Sales Sonar is remarkable because it builds a community, and it was designed to build it better because of "the cloud."

Our contrast application fits in another camp. It's possible to build a nice application, but without differentiation the cloud doesn't add much sparkle. Putting an indistinct app on the cloud doesn't provide anything more than an indistinct app. That's the rule here: if your application isn't fundamentally different because of what it gets from the cloud, then you wasted your time putting it there.

So, what can the cloud provide? The magic does not lie in the parts: practically all the components you find used for cloud apps (e.g. Xen, Linux, MySQL, NoSQL, nginx, Passenger, Ruby, Rails, etc...) run fine and offer the same benefits in a cloudless world. Here it really is ...The tale, not he who tells it. Suppose you have an application ready for pilot, but it'll take 100 servers to run. Here the cloud can come to your rescue -- spin up the app on AWS on 100 servers over a long weekend, and your life-changing pilot will come in at under $1000. Try comparing that to the cost of renting data center space for 100 servers, buying or leasing all the computer and network hardware, setting it all up, running the test, and then tearing it all down again in a long weekend.

"You see this? This is this. This ain't somethin' else. This is this!" Robert DeNiro "The Deer Hunter"

What makes the cloud magical is its flexibility -- and when combined with web standards and tools and modern development practices, it's possible to "solve" the IT equation -- specific rich solutions generated quickly and reliably, and built from standard parts that are familiar to any IT shop. That's where Pikasoft is headed, but to understand the rest of the puzzle, you have to understand the current state of software.

So up next ... software for the new millenium

Wednesday
Apr072010

What Comes Next: The Pad, The Cloud, and The Software that links it all

Part 1: The Pad

The pad has to come first.   With all the frenzy about Apple's recently released iPad, it's easy to miss the main point here, and the principal means by which the iPad is an innovation:   Everyone who uses a "computer" already has one!    The world of computing that Apple defined and Microsoft conquered 35 years ago really hasn't changed since -- the machines have gotten more powerful, but the basic means of human interaction with them hasn't changed.   It hasn't changed since Douglas Engelbart's invention of the computer mouse, and it really hasn't changed since Ivan Sutherland's PhD thesis Sketchpad (both 1963).

Alan Kay had the right idea with his Dynabook, and he may have been the first to realize the need for changes in both the interaction model and the programming model for computers. Programmers of that earlier era he referred to as

High priests of the low cult

That terminology could have been used to describe most computer users through the ages; but the "Pad Computer" can change that. The applications are there at the touch of a finger. Hand gestures (not code) are what the user uses to interact with the machine. It has been said that you "don't interact with the applications, you live in them."

This really is Alan Kay's Dynabook made real. Kay's own estimation of such a device, made years ago, put it all in perspective:

“When the Mac first came out, Newsweek asked me what I [thought] of it. I said: Well, it’s the first personal computer worth criticizing. So at the end of the presentation, Steve came up to me and said: Is the iPhone worth criticizing? And I said: Make the screen five inches by eight inches, and you’ll rule the world.”

The iPad and similar devices may not be the full Dynabook yet, but you can see it from there.