Tuesday, February 26, 2008

Analogies, Construction Workers and How to Derail Your Own Technology Presentation

Analogies are great for explaining complex subjects in everyday terms. A great example is the analogy that "Software Development is like building a house." That analogy comes up all the time because it seems to so aptly describe how software is written to people outside of software development. Analogies are supposed to show similarities, however, some listeners take it to be more like a concrete description.

I was once pitching a product that kept track of which features users were running in the field and reported the information back to the development team. The developer could see which features were most heavily used, which bugs would impact the most users, where to put QA and where to focus future development dollars. The message wasn’t coming across very well so I made the colossal mistake of saying it was like a code coverage tool for features in that it told the developer how many times a feature was used, by how many users and so on.

"Code Coverage" has a very specific set of features, like method and line number execution counts. I immediately started getting questions like "Why don't we just use Clover?" You can safely insert every code coverage product you ever heard of in place of Clover and I can assure you it got mentioned.

I spent the next twenty minutes explaining how the application was not actually code coverage but like a code coverage product. Everyone fixated on the term "code coverage" and consequently the product was only seen as my spin on code coverage. Various insights on how other code coverage products would be hard to compete against, the various features of great code coverage tools, whether it would integrate into Eclipse, etc.

I left the meeting knowing that it was a complete failure. My jaw was clenched so tightly that I thought I was about to bite through my teeth. I had used an analogy that completely derailed my presentation on a technology that could accurately detect feature use in a deployed application. I now avoid using analogies like a rat avoids a cat.

I recently had an odd situation where someone went back to the old analogy of "software development is like building a house." This analogy is pretty safe when on a golf course trying to make polite conversation with someone completely outside of technology. It is such a good analogy from 50,000 foot level and it really does explain how many people can be involved in a project and their roles.

When it creeps back into the office is how the milk gets spoiled. First, software developers are not construction workers. I'm not intending anything negative toward construction workers at all. The process for becoming a skilled carpenter, electrician, plumber and several other trades, is managed through unions, guilds, licensing and other regulatory systems for ensuring that a given trades person is certifiably skilled in a particular field and can actually be found liable in cases of negligence. An electrician, for example, is actually a skilled carpenter too. A carpenter could unseat and seat a toilet if it wasn't that flood damage, and other less enjoyable aspects of the operation, make the process less appealing. Nonetheless, trades people often have a big range of skills.

Software developers are different from construction workers in that we can still go off and write components well outside our specific niche. A hibernate developer can still write a JSP and not be in any risk. We are encouraged to show our complete range of skills and exercise them daily.

The idea of designs, requirements and even the term architect appear to be taken directly from the construction realm. The concept of plans and requirements are similar but far more complex in software development as a design must taken into account a wide range of users and manage a variety of input sources while a house generally has a limited number of users and consequently an entirely different bug domain. Perhaps software development is more like building a stadium or an airport?

Production problems between houses and software development are entirely different. You are not often going to run into a house that has an error which may require a major redesign. Again, think stadium type issues, such as a drainage problem, could require a massive amount of work to resolve, so may an application's database run into scalability issues and have an equally lengthy process too.

The most negative aspect, to me, about the analogy of "software is like building a house" is when management starts to believe it. While the analogy encourages planning, it also encourages micromanagement (as some home buyers will watch the building process like hawks and sometimes with good reason) and can creates odd expectations on production delays.

This scenario can be especially troubling when a person outside of normal technology spaces decides to try to get a grip on the subject. Consider Senator Ted Steve's analogy of the Internet being a series of tubes as an example of how detrimental an analogy can be.

The core of the problem is that analogies are often misleading and far too open for interpretation. While they can be a great tool for explaining a complex topic, they are also an excellent mechanism which a listener’s mind can lock-on to and never release. This obviously perverts any message that you may have had all thanks to a well intentioned tool to help the audience. Yep, one bad analogy and soon your presentation is like a chicken with its head cut off- your message is out of control and is making one hell of a mess.

Metaphors and similes can be just as confusing, overused and mixed-up.

Saturday, February 23, 2008

The End-To-End Developer

The end-to-end web developer is a rare find though it would seem like she should be all too common. The term refers to developer that can write quality code at every tier of a web application. For a Java developer, that means being able to write good HTML, Javascript, CSS, JSP, Servlets (though they appear in many forms), EJB (and their various kinds), JDBC (or Hibernate), SQL and a little system administration too. It sounds like a common set of skills and many employers claim to require them, but often these skills are difficult to gain and some jobs make it difficult to practice.

Before I dive into the ethereal land of opinionated-qualification-make-believe, please consider how your company divides its teams. Many companies have their developers work on specific tiers of the application. Some developers work on the UI, others write EJB services and some focus on database development. Employers may want to see a wide range of qualifications on a resume, but may quickly lock a developer into a specific tier or role that doesn't take advantage of those other skills. This organizational style also makes it difficult to grow skills outside of a developer’s immediate role.

Another style operates on a functional slice basis. The concept is that a developer is responsible for one or more features and writes the UI, middle and back tiers. Obviously, this organizational style takes advantage of the developer's entire range of skills. This style does not preclude specialists, such as expert UI and database engineers. The specialists may do fit-and-finish work, perform design reviews and set the direction for the particular tier. A developer in this organizational style is able to practice a wide range of skills daily.

The first approach encourages specialization and the second is generalization. An end-to-end developer coming from the first approach may have some skills but was afforded limited opportunity to grow them. The multidisciplinary requirements of the second approach requires a lot of upfront knowledge that is difficult to attain without being in such an environment previously.

Back to being an end-to-end developer. Why are end-to-end developers valuable? Startup and small companies often have a wide range of development needs but may lack a significant budget to hire specialists for each tier. Generalists fill those gaps though they can command a higher initial salary, they are still less expensive than hiring two or more specialist developers.

Are generalists more skilled than specialists? This is an extremely difficult question to answer. Specialists are focused on a given subject and should be more knowledgeable on that subject. Yet a generalist may have actually worked with that technology for longer and at a greater depth than the specialist. An example of this is a UI specialist with two years of Tapestry experience compared to a generalist with four years experience in the same technology.

Years of experience is not necessarily an accurate gauge of the developer's depth of knowledge though. The only way to tell if a developer has the experience necessary to do the work is by probing through the interview process. This is necessary when dealing with generalists and specialists alike.

The generalist vs specialist argument is one of personal choice and there is no right or wrong answer. There are advantages to both and the rule of staying current with evolutions in technology remains the same.

The concept of the end-to-end developer is fascinating as the developer should be able to write an entire web application on her own. Such a developer is proven when she has actually built something on her own and made it publicly available. It doesn’t need to be earth shattering or massively successful, it just needs to solve a problem and be accessible to the world.

But there are other, less obvious skills being learned too, like self-determiniation, time management and follow-through. Many job descriptions look for a self-starter: some go-getter developer that can work with little supervision, lead a team and finish what they agreed to in a reasonable time. If a developer spends the time to learn to every tier of web application development, builds something and makes it public then she meets pretty much all the listed requirements except for leading a team. Thankfully, team leadership is a learned skill.

One item that should be clarified is the difference between the hobbyist and end-to-end developer. There is a reluctance to hire a developer that learned specific skills through a hobby project rather than a commercial project. There are some good reasons for this, such as hobby projects usually don’t require a developer to focus and develop a project for eight, or more, hours a work-day. It’s hard for a developer to gain breadth and depth, in a vacuum, working on a hobby only a few hours a day or week. The big difference is that a hobbyist doesn’t have to finnish, the end-to-end developer does and then works harder on it and makes it into something big.

It would be reasonable for someone to look at this and say “he’s just trying to justify hobbyists.” There might be some truth in that. Yet look out at the successful sites and think about how many of them emerged from little more than hobbyist developers. How many blogs are little more than hobbies and yet we glean great information from them. A big difference between the hobbyist and the end-to-end developer is that the former is interested in learning something but has no real goal beyond that while the latter has a goal, finishes and makes it public.

Finishing makes the difference.

Monday, February 18, 2008

Building a Web App on $30 a Month and an Hour a Day

What can you do with $30 a month and an hour a day? You can build a darn nice web application on the cheap. It’s common to see all the time, money, servers, developers and IT staff necessary to build and manage a web application and think that one person just can’t build anything of merit. The reality is that one person can build a lot in his spare time, alone, make it publicly available and with very few expenses.

The old way of building an application required tremendous planning and figure out pretty much everything ahead of time. From features to design, everything was laid out and reviewed and approved. Agile development taught us that you still need some focus and some structure, but being able to change direction and priorities to meet changing customer needs was vastly more valuable. Building your own application still requires some focus and an idea of features, but doesn’t require all the rigor. There are a few things one should know about managing a project to ensure that it gets finished and solves some problem.

It’s often useful to focus on developing a personal project quickly and exposing it to the world. It often happens that applications fail in the marketplace while a few succeed. The simple fact is that you cannot hope to succeed until you start building a product. One key factor is the balance between building the application precisely right vs getting the application done in a timely way.

Some people get it in their head to write the application correctly right from the beginning. It’s true that you should build quality into the product from day one, but it isn’t necessarily true for performance or design purity. You have no idea if anyone will want to use your new application. A safer approach is to make design compromises to get the application out there and in users hands. Then go back and make incremental improvements as well as rolling in user suggestions.

It’s rare in a traditional product development cycle that developers actually have the time to go back and fix things. However, you don’t have to follow this trend because this is your project and you can run it the way you like. You can write relatively miserable code that “works” and get it on-line and then fix it later. You can also build your architectural dream, but you may not finish.

Developing your project quickly is important. You probably see it all the time where people start a “garage band” software company and lose interest after a month. It wasn’t a bad idea (unless it really was) but it may be too big to stay interesting long enough to finish it. Find that one or two critical features and get them online. Then work on the next one and so on. The idea is to rapidly finish components and make them live. This often helps to motivate people to continue working on the project until its finished as you see progress and get validation.

Third-party libraries are a huge help. You have a product to build and you should take advantage of what is already out there. My own experience with this has to do with charts. I almost always have charts in my applications and I often have a hard time finding a cheap or OSS chart library that works well. I once spent a month updating Plotkit, to get better chart labels, only to drop it in favor of a very inexpensive static image chart library called ChartDirector that did exactly what I wanted. I did enjoy changing Plotkit, but it was a huge distraction from the application.

Bringing it on-line is incredibly simple and inexpensive depending upon the application. I generally build Javascript-Java applications, however, I now often build hybrid Javascript-PHP-Java applications. Javascript and static HTML is obviously for the browser. PHP serves up those pages, as well as acting as a JSON RPC layer for responding to ajax operations, and the database layer. Finally, I use Java to manage any background or external tasks.

I’m about to make some hosting suggestions that may seem silly. Remember that my motive is that you are able to go live with a small number of users while you are in development. By small I mean less than a thousand. The goal is to go live, get feedback and learn about the scalability issues. You can migrate to a more significant server configuration, and its costs, when you think you can make some money with it.

The cost for the above Javascript-PHP-Java setup can be as little as about $30 a month. That includes PHP and database hosting as well as a VPS for running any Java processes. PHP and Database hosting can be very inexpensive thanks to Yahoo and GoDaddy. Their web page hosting is often fine for doing most of your development work and allowing a small number of beta users. You could spend anywhere from $4 to $15 per month to go live.

Most web applications need background or external tasks. Adding this functionality is again, not difficult nor expensive. A virtual private server (VPS) is often all you need to get started. Rent a virtual server that gives you a linux share that you are pretty much free to do with as you like. From there develop, in the language of your choice, a series of tools that fire off at specific times to run your background tasks on the PHP server. These command line applications generally call a URL on the PHP server, which then does the real work. The client’s only purpose is to start the process while PHP does the real work.

I often like doing timed analytics of external data sources. My command-line clients come live for a few minutes at set intervals (cron jobs), query their remote data sources for information, summarize the data and dispatch it back to the PHP server where the data is stored and viewed.

VPS servers have a wide range of pricing and service. Cheaper servers often have less system resources and are heavily split with other customers like yourself. I found one called VPSLand that charges around $15 for their lowest end server: a machine with 256 MB ram and 6GB of disk space. It doesn’t sound like much, but how much more memory do you need to run a command line application?

There are plenty of issues with this arrangement but there is the expectation that this is a development setup and you will migrate on to something with greater horsepower later on. With that you should keep in mind a few design considerations.

Database design is probably the most important thing to keep in the front of your mind. Some web hosting companies set you up with highly reliable, but old version of database software, like version 4.x of MySQL when 5 has been out for years. Their goal is to give you something really reliable and inexpensive. Reliance on some features, like stored procedures, may not be present and so you simply design around those issues.

A very common problem with database design, in this environment, are table primary keys. Sometimes it is reasonable to use auto-increment integers for a row ID. However, it is very difficult to migrate this data later because the new database’s IDs may not work exactly the same. You can do three things to help with migration. First, you can turn off the auto-increment feature during import so that you can use the old IDs more easily. Second, you can migrate the data programmatically, which will recreate the data more reliably. Third, you can use GUID values for your IDs. The third option is often less appealing, but GUID IDs work across machine boundaries and make migration much easier. Granted, GUIDs are not the most efficient way to lookup a specific row.

Another common database problem during migration is the size of the resulting export script. Its often huge and can take a very long time to generate. An RPC interface that can grab a few hundred rows is one way to work around the issue. You can help to ensure success by always keeping track of the row number you last imported incase an error occurs so that you can resume more quickly. Exposing one RPC API per table to simplify the import process is very helpful too. Of course, you must turn constraints off on the destination database until the import process is done as you may be importing disjointed data. This migration process can take hours. It is also helpful to take the service off-line during while migrating as you don’t want to continually grow already migrated tables.

I mentioned the necessity of an RPC mechanism. This is a real requirement for the PHP side beyond just migration. Background tasks will need a way to communicate with the server and may need to transfer large amounts of data. Sometimes it is as simple as transferring a file from the VPS server to the PHP server which then gets parsed. Better ways to do the same thing are done by implementing SOAP, JSON-RPC or REST interfaces. I am partial to JSON-RPC and REST because I can send larger chunks of data that can be easily parsed and its handy to be able to test the API calls from a browser.

The development challenges of building a system on the cheap are obviously very easy for most developers. I think the hardest is time however. Most developers are very busy between work and personal life. Finding time to work on a project can be very difficult and it is also difficult to stick with a personal project to the end. Working as little as an hour a day may seem inconsequential, as it is easy to spend a day working on a particular feature in a regular project, yet an hour a day is better than nothing. The real point of working an hour a day is to keep your head in the project until you can make real strides, such as a weekend or when TV is really dull.

Is this a strategy for everyone? No. Projects with extremely large databases and scalability issues under the best of circumstances cannot hope to work under these conditions. They are just doomed to large servers and equally large costs.

Software development on the cheap forces us to think about the long term, the complete solution, in terms of what can be accomplished and what will be accepted. Consider that users can be fickle and that projects are a commitment of your money, time and passion. It’s a hard thing to face when users are not so enthusiastic about your dream and solution. Building applications on the cheap provides the opportunity to build something different, on your own terms, using new technologies, and the chance to explore the process of building a business. By keeping costs down and the development schedule short, you get to see the fruits of your labor sooner and maybe contribute something really valuable to the world.

This is part 1 of a series. I’ll shortly bring up an application showing the above development style and the code that goes along with it.

Saturday, February 16, 2008

The Weekly Demo

I've been involved in a few Agile projects and have formed a few opinions some positive and some negative. For example, I don't think that Agile methods work very well for established mass market desktop products while I think it works superbly for IT projects, web applications and projects that have a relatively small number of customers. I especially like the ability to come to a stopping point quickly and show the changes to the team and customers, which is what I'm going to talk about today.

I love weekly or bi-weekly demonstrations with the team and monthly customer meetings because I can get feedback and adjust my product rather than committing to something very uncertain. This demonstration schedule does not necessarily match the iteration cycle either and can run on just a developer's desktop. The point is to show the latest changes from one demo to the next, give credit where it is due, make minor adjustments to design and requirements and give a clear indication of the velocity of progress. My favorite benefit is the amount of sleep I get at night.

Yes, sleep is my favorite benefit. I don't worry about whether the customer is going to like what I am building by the time I hit delivery date. I don't have to worry about the work-flow because the team and product management have tried it a dozen or more times. I don't have to worry about major silly bugs because the demonstrations have usually exposed them already. As we know, all software crashes at the most embarrassing time: during a demo in front of a crowd. By ship date I can rest assured that anything glaring, bug or requirement, has been discovered and solved.

Building demonstrable code is not easy on a one or two week schedule. The goal is to stop building new functionality early enough to integrate and stabilize the code for demo. I like having only one week between demonstrations because the code base has not moved terribly far forward, which makes integration easy. I usually spend a portion of the day prior just for testing the changes and making fit and finish modifications. However, I can't do very much in the way of major changes. I work to get to the next demo and so I either must finish my changes entirely, disable the new code or demo around the incomplete code. Planning the demo is obviously important.

It often happens that the conceptualization of a work-flow turns out to be somewhat flawed. One can argue during the design phase that the flow is wrong but it is difficult to dissuade some individuals without having them actually try it. A demo illustrates the work-flow issues with perfect clarity. The results are the requirements, UI, back-end and test plans may change, but they changed early enough in the development cycle to be corrected investing a major development effort.

Regular customer demonstrations may not be an option. Some customers find demos annoying while others are happy to see progress. Regardless, showing them the progress made and the work flow at regular intervals is incredibly valuable in ensuring that the delivered product is what they want. The changes can be as simple as labeling issues where the customer realizes that the label is Ok, but they really call it X or Y. Its a 2 minute fix that makes the customer happy. Other changes can be very invasive, but you caught it early so there is hopefully time to solve it.

Making weekly team demonstrations effective is a straightforward process. First, have goals for what should be accomplished from one week to the next. Second, plan when to stop development and focus on integrating, testing and bug fixing. Third, during the demo describe the changes from the previous demo, giving credit where it is due. Fourth, modify the code after the demo to include suggestions and comments.

The weekly demo is incredibly valuable for finding flaws in requirements, work flow and implementation. In the best case, it validates that you were right all along. In the worst cases, it gives you the opportunity to change directions earlier rather than shortly before a delivery date. In any event, you know that by the time you deliver the final product that you are aligned with customer expectations.

Wednesday, February 13, 2008

Entrepeneur Wannabe: Choosing a Domain Name

I’m beating a dead horse with this one, but I recently saw someone refer to me as “the savvy duck” and felt like I needed to set the record straight about my method for selecting a domain name. I’m not giving any rules, best practices or anything like that. This is story time.

I was about to start writing a data tracking tool and was looking for a domain name that would fit with the product. I went with an adjective and noun combination and registered my domain. My adjective was the word empirical. The platform I was building was for developers and I figured that most of them knew the word empirical and would hopefully work out well. This was not the case at all. I had only slightly better results with empirical than if I chose the Hawaiian word for trigger fish: Humuhumunukunukuapua'a.

I didn’t realize my error until the day I called Dell about buying a laptop. I had to give my email address to the sales rep so he could send me a quote. I rattled off the email address and we ended the call. I was getting quotes from three other vendors so I made a few other calls.

One sales rep didn’t know the word at all. Ok, some developers slam sales guys as not being the brightest folk but good sales people are actually pretty darn smart. I spelled out empirical and moved on to my next call. This new guy just asked me to spell it out. Apparently he was tired of trying to figure out spellings for domain names awhile ago and made his request in a way that indicated he does this a million times a day and tomorrow he’ll do it a million more times.

The email from the Dell rep didn’t arrive. It had been an hour. I logged into another email account and sent my empirical account an email. My message arrived within a few seconds so I knew the account was working. I called the rep and he told me the message bounced. He spelled empirical wrong.

I had to talk to a person with the state about registering my business. There too I had to define the word and how to spell it. These people deal with business names all day and I live in a state on the French-Canadian border. Our towns are packed with business names that have pronunciations that seem only loosely related to their spelling. I knew I was screwed when I realized that the state employees couldn’t spell the word empirical either.

I wanted to start a new analytics application last year. I was talking with an odd CEO about figuring out a domain name and he gave me some interesting advice. He basically said that “many companies change their focus in the first few years so pick a name that can be anything.” The idea is that the company can reinvent itself and the domain name is still applicable.

He also suggested using a cute animal for the domain or have a really attractive woman on the home page.

My wife and I were sitting around trying to think of cute animal domain names. I wanted an adjective that described the animal as being intelligent. The animal’s name had to be something easy to spell. We churned out names like “wisesquirrel”, “trickybadger” but most people don’t write animal names very often. How often do you type the word wombat, chipmunk or wolverine?

Finally, we came up with savvyduck.com. Sure, not everyone knows the double-v in savvy but everyone knows duck. A lot of people like ducks too. It sounded cool and it meant nothing. The best bit was that a product could be called savvy duck, but it seemed to apply better to the user. The user is the savvy duck. The user is the smart one. That’s true of the users of most products too.

Picking savvyduck.com worked out really well because I did end up switching from a product to a blog and it is memorable. I, however, am more of an opinionated schmuck rather than a savvy duck.

Monday, February 11, 2008

Javascript: Mixing Ext’s Grid with JQuery’s Flot

Data visualization through charts and graphs is incredibly common and yet Ext doesn’t have a readily apparent charting library! JQuery has a great little charting library called Flot and getting it to work within an Ext application turned out to be pretty easy. I’ll show how I integrated the two and the problems I ran into. I’ll also go over some common issues in choosing a charting library.

Charts, whether static or dynamic, are a common requirement in almost every application with a grid. I say almost because some users could care less for a chart, they just want the data and can visualize the data just fine on their own. Many users can’t imagine the data on their own and need some kind of chart sitting near any table or grid. The volume of data in a grid may also be so large that a user simply can’t imagine the breakdown without some automated visualization.

I mentioned two kinds of charts and they are worth discussing briefly. Static charts are usually non- or minimally interactive images generated on a server and displayed on the browser. They often have image maps that let a user click on a line or a point and drill-down into the data. Then the browser loads the next chart image and the user can continue drilling down. The refresh rate is obviously slow, because there is a roundtrip between the browser and the server when redrawing the chart, and the interaction is limited.

Dynamic charts are usually flash or Javascript based and are excellent for displaying fast moving data. These kinds of charts are usually more interactive: allowing for zoom, filtering and rapid updates. They can provide a better user experience depending upon on the volume of data to draw.

Choosing one method over another is dependent on product requirements and volume of data. Consider that you want the ability to cut-and-paste a chart from a web application into an email. Static images work great, but javascript and flash charts usually need some kind of external capture utility like SnagIt(PC) or Grab (Mac).

Images work very well when you have huge amounts of data to plot. A scatter plot with 5000 points runs well on a server. Trying that with a javascript control incurs both the rather large download of the data and the long processing time, which can sometimes stretch long enough for the browser to popup the dialog asking the user if he wants to continue running the script. The image again can be cached on the server, so the creation time is incurred only when the data changes, which is hopefully only once every couple of minutes. Images with image maps still have a long download though.

Javascript and Flash charts can interact seamlessly with a Javascript application. The chart’s look and feel is often far more polished and its responsiveness is often surprising to users.

Again, product requirements and volume of data are the most important drivers for a technology than the technology itself. It often happens that a product manager will want a flash chart when a static image chart is actually more effective. It also happens that someone wants a static image chart to support a rarely used requirement rather than a dynamic chart to show rapidly changing data, which is a more often used feature.

I’ve yammered on about charting technologies long enough. I started by saying that I needed to integrate a Javascript chart library into an Ext based grid application. I’ve taken a previous example of the EditorGridPanel and extended it with charts and a few other minor embellishments. I went along with JQuery’s Flot because I only needed bar charts, but it also supports line and scatter plots too.

Ext has an unusual feature that allows you to swap out its default base library and replace it with one that acts as a facade on other libraries, like JQuery. I am not going to take advantage of this feature for two reasons. First, Flot uses a more recent version of JQuery than what is distributed with Ext. Second, I am more interested in seeing if Ext will continue to operate with a new library messing around with the DOM. The second reason is especially important if we later want to use a library that supports pie charts or another chart not supported by flot.

The integration process is very easy and didn’t run into any problems that couldn’t be worked around. Flot needs only a few things to build a chart. It obviously needs data which I will supply via a Store. We need some event support to update the charts when data changes. We need a container to display the charts within the Ext application. Finally, it would be really snazzy if the charts updated as the data changes. Let’s look at the code to see how these were accomplished. The following description will refer to the red numbered comments. Also note that the code is just an example and not meant to be used in production.

<html>

<head>

<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">

<title>Ext EditorGridPanel with JQuery’s Flot Example</title>

<link rel="stylesheet" type="text/css"

href="./ext-2.0.1/resources/css/ext-all.css" />

<script type="text/javascript" src="./ext-2.0.1/adapter/ext/ext-base.js"></script>

<script type="text/javascript" src="./ext-2.0.1/ext-all-debug.js"></script>

<script type="text/javascript" src="./flot/jquery.js"></script>

<!-- FOR IE <script type="text/javascript" src="./flot/excanvas.js"></script> -->

<script type="text/javascript" src="./flot/jquery.flot.js"></script>

<script>

var initComplete = false;


function main() {

// Setup the store and its data

var sampleData = [

[ 'Bob', 20, 13.45 ],

[ 'Bill', 40, 12.92],

[ 'Mike', 45, 23.02]

];


var store = new Ext.data.SimpleStore({

fields : [

{ name: 'name', type: 'string' },

{ name: 'hours', type: 'int' },

{ name: 'rate', type: 'float' },

{ name: 'cost', type: 'float' }

],

data: sampleData

});


// Initialize the grid control

var simpleGrid = new Ext.grid.EditorGridPanel({

store: store,

columns: [{

header: 'Name', // Field cannot be edited.

width: 160,

sortable: false,

dataIndex: 'name'

},

{

header: 'Hours', // Field can be edited

width: 75,

sortable: false,

dataIndex: 'hours',

editor: new Ext.form.NumberField({

allowBlank: false,

allowDecimals: false

})

},

{

header: 'Rate', // Field can be edited

width: 75,

sortable: false,

dataIndex: 'rate',

editor: new Ext.form.NumberField({

allowBlank: false,

allowDecimals: true

})

},

{

header: 'Cost', // Calculated field

width: 75,

sortable: false,

renderer: formatCost,

dataIndex: 'cost'

}

],

clicksToEdit: 1,

stripeRows: true,

height: 100,

enableHdMenu: false,

region: 'south'

});

var commonChartStyle = {"margin": "10px 10px 10px 10px"};

// 1: Set up the containers for the charts

var rateContainer = new Ext.Panel({

height: 200,

width: 200,

style: commonChartStyle,

id: 'rateChartContainer'

});

var hoursContainer = new Ext.Panel({

height: 200,

width: 200,

style: commonChartStyle,

id: 'hoursChartContainer'

});

var costContainer = new Ext.Panel({

height: 200,

width: 200,

style: commonChartStyle,

id: 'costChartContainer'

});

var viewPort = new Ext.Panel({

title: 'Chart-Grid Example',

frame: true,

layout: 'border',

renderTo: Ext.getBody(),

height: 400,

width: 674,

items: [{

region: 'center',

border: true,

bodyStyle: 'background-color: white; border: 1px solid #99BBE8',

layout: 'column',

items: [{

title: 'Hours',

width: 220,

items: hoursContainer

},{

title: 'Rate',

width: 220,

items: rateContainer

},{

title: 'Cost',

width: 220,

items: costContainer

}

]

},

simpleGrid]

});

drawCharts(store);

initComplete = true;

}


function drawCharts(store) {

drawChart(store, 'rate', 'rateChartContainer');

drawChart(store, 'hours', 'hoursChartContainer');

drawChart(store, 'cost', 'costChartContainer');

}


function getData(store, nameColumn, dataColumn) {

var dataResults = new Array();

var tickResults = new Array();

// 2: get the chart data

for( var recordIndex = 0; recordIndex < store.getCount(); recordIndex++ ) {

var record = store.getAt(recordIndex);

var tmpData = [(recordIndex+1)*2, record.get(dataColumn)];

var series = {

bars: { show: true },

label: record.get(nameColumn),

data: [ tmpData, tmpData ], // workaround

color: recordIndex

}

dataResults.push( series );

tickResults.push([ ((recordIndex+1)*2)+1, record.get(nameColumn) ]);

}

return {

data:dataResults,

ticks:tickResults

};

}


function drawChart(store, column, chart) {

var chartInfo = getData(store, 'name', column);

// 3: draw the chart in the container

$.plot($("#"+chart),

chartInfo.data,

{

xaxis: {

autoscaleMargin: .25,

ticks: chartInfo.ticks

}

}

);

}


// Formatting function for the cost column

function formatCost(value, metadata, record, rowIndex, colIndex, store) {

var results = getCost(record);

record.set( 'cost', results);

if ( initComplete ) {

// 4: redraw the charts when the data changes

drawCharts(store);

}

return Ext.util.Format.usMoney(results);

}


// Calculates the cost

function getCost(record) {

var hours = record.get('hours');

var rate = record.get('rate');

var results = hours * rate;

return results;

}


</script>

<style>

p {

margin:5px;

}

</style>

</head>

<body style="margin: 10px 10px;">

<script>

Ext.onReady( main );

</script>

</body>

</html>

First, we create three nearly identical Ext.Panel(1) objects to contain the chart. They all have hard coded sizes and an explicit id field. Flot will calculate the size of the chart based upon the size of the container. If you let Ext automatically set the height or width, Flot may see the value as being 0 and make a very small chart. The user can resize the panel, but you must ensure that the height and width are valid.

The id field is necessary so that we can point JQuery to the right DOM object to modify. You can use this technique with other charting libraries too. Ext will generate a div with the id you specify. Once generated, it is easy to point any other compatible library at the div and generate whatever you want.

Second, we load the data for the chart from the Store(2). Flot’s data series has the format of [ [x axis value, y axis value]] . A line chart would have multiple x-y value pairs.

We will also replace the normal tick values with a person’s name, which is done by substituting an axis value with a label. Each axis value can represent a range from N to N+1. The 0 position of the X axis is flush to the Y axis. Having a label on the 0 location makes the label look strange and somewhat unreadable. The goal of the tmpData assignment line is to move the X axis value away from the 0 position.

The code then creates a series object that sets the chart type to bar, the label for the series, the color and the data. The data line is really odd because there is a bug in the Flot library for bar charts. Flot requires more than 1 data point or it doesn’t render the series. The workaround listed in the bug report is to have two identical values in the series. That’s why tmpData is used twice.

Another interesting aspect of Flot is that each series can specify a different chart type. You can mix line, scatter and bar charts in a single chart. Generally it is unwise to do so as it makes for a cool looking chart that doesn’t necessarily make the data more readable. Google Edward Tufte and you will find plenty of documentation on good charting techniques.

The last interesting line deals with the ticks. The tick values are using a different index than what was used earlier. This was done so that the label would be on the trailing edge of the bar rather than the leading edge and looks marginally better.

Third, we render the chart(3). The plot method gets called with three parameters: the id for the container, an array of series data, and the chart options. The container id reference finds the DOM object and adds a couple of canvas and div tags to the Panel. Flot uses the array of series data to draw the chart. Finally, the options parameter modifies the X axis to change margins to be more readable and replaces the ticks with the values mentioned earlier.

Fourth, we modify the formatter to update the charts when the grid changes(4). The effect is that a user can modify either the rate or hours columns and the charts immediately update.

You will see a few odd sections of code wrapped in if(initComplete) statements. JQuery cannot go to work unless all the Ext objects are rendered, which may not occur until later. The if statement ensures that the UI is fully rendered before trying to run Flot code.

The goal of this example was to show that it is possible to integrate Ext with other major Javascript frameworks to supplement Ext’s features. In this case we added a chart to an existing project. The selection of the chart library was almost arbitrary, though I tried a few others and thought that Flot made a really nice looking bar chart. Now you can go off and try mixing libraries using the techniques shown here.

N00B!

Steve writes about being a software developer n00b. It’s a great read and I agree with him on so much. Check it out.

Thursday, February 7, 2008

Javascript: Introduction to the Ext Grid Object

Grid controls get used in lots of user interfaces and thankfully, Ext makes it pretty easy to use a normally cumbersome component. It takes only a few relatively simple lines to create the control and most of them are configuration parameters that leave little room for bugs to appear later. To illustrate the point, let’s take a look at a small grid that has three populated data columns and one calculated column.

We have four columns: name, hours, rate and cost. The cost column is calculated by multiplying the rate and hours columns. This is pretty simple stuff. We’ll make it more interesting by letting you change the values of the rate and hour columns and automatically update the cost column.

Let’s first look at the grid and its children. The GridPanel class allows you to show data only. The EditorGridPanel allows you to configure a selectively editable grid control. What I mean is that you can configure a specific editor for each column in the grid. You can use any of the form package’s Field objects as an editor.The PropertyGrid is derived from the EditorGridPanel and let’s you manage a grid of key/value pairs. We will focus on the GridPanel and EditorGridPanel in this discussion.

We can see from the class diagram that a GridPanel has a ColumnModel field which is a collection of column configurations. The ColumnModel.renderer member lets you change the default rendering behavior of a column. The ColumnModel.editor member is an Ext.form.Field object, such as TextField or NumberField. GridPanel ignores the editor member, while EditorGridPanel uses the editor to allow the user to modify a cell’s contents.

So how does a user edit a cell? She clicks on it one or more times. You can set the number of clicks (single, double, etc.) during the configuration of the EditorGridPanel by setting the clicksToEdit field. The default number of clicks is two.

A GridPanel is backed by an Ext.data.Store object, which contains the data for the rows of the grid. There are several different kinds of stores for loading and managing various data sources. The relationship between a GridPanel and a Store is very tight: modify a value from the either the grid or the store and the other will immediately update.

This intimacy between the two objects is very helpful when managing calculated fields. Consider for a moment that the data is being loaded remotely, as it would be in a real application, and that the rate and hours columns are populated by the server and locally calculating the cost column when either the hours or rate columns change. There are several ways to accomplish this.

The first method is to register for the afteredit event, which fires after a cell is edited. Your event handler calculates the values of your calculated fields and saves them to the store. Remember, the grid will reflect the changes to the store so the grid will display the new cost value. A subtlety is that the store can load the cost column already populated by the server.

The second method is that you define a renderer for a calculated field and not bother to update the store. A row gets rendered whenever a cell is modified. I use this method in the example, however, I believe the first method is more useful.

Why write an example that uses an odd implementation? Both work well. The second implementation doesn’t require the calculated field to be populated by the server. I can simply calculate its value on the fly. It will perform more poorly if I have a lot of rows though. It also implies that the server doesn’t store the calculated field. The first method obviously implies that the server has a representation of the calculated field and that the grid should render more quickly because it has fewer calculations to perform.

The last thing I will mention about the Store class is that it has a kind of transaction management. You can make a series of changes and commit them by calling Store.commitChanges() or reject them by calling Store.rejectChanges(). The rejectChanges() method is handy for undoing mistakes.

Let’s look at the example application’s code now. The interesting sections have a red number in the comments and following explanation will refer to them.

<html>

<head>

<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">

<title>Simple Ext EditorGrid Example</title>

<link rel="stylesheet" type="text/css" href="./ext-2.0.1/resources/css/ext-all.css"/>

<script type="text/javascript" src="./ext-2.0.1/adapter/ext/ext-base.js"></script>

<script type="text/javascript" src="./ext-2.0.1/ext-all-debug.js"></script>

<script>

function main() {

// 1. Create some dummy data data

var sampleData = [

[ 'Bob', 20, 13.45 ],

[ 'Bill', 40, 12.92],

[ 'Mike', 45, 23.02]

];


// 2. Create the Store

var store = new Ext.data.SimpleStore({

fields : [

{ name: 'name', type: 'string' },

{ name: 'hours', type: 'int' },

{ name: 'rate', type: 'float' },

{ name: 'cost', type: 'float' }

],

data: sampleData

});


// 3. Initialize the grid control

var simpleGrid = new Ext.grid.EditorGridPanel({

store: store,

columns: [{

header: 'Name', // 4. Field cannot be edited.

width: 160,

sortable: false,

dataIndex: 'name'

},

{

header: 'Hours', // 5. Field can be edited

width: 75,

sortable: false,

dataIndex: 'hours',

editor: new Ext.form.NumberField({

allowBlank: false,

allowDecimals: false

})

},

{

header: 'Rate', // 5. Field can be edited

width: 75,

sortable: false,

dataIndex: 'rate',

editor: new Ext.form.NumberField({

allowBlank: false,

allowDecimals: true

})

},

{

header: 'Cost', // 6. Calculated field

width: 75,

sortable: false,

renderer: formatCost,

dataIndex: 'cost'

}

],

clicksToEdit: 1,

stripeRows: true,

autoHeight:true,

width:500,

enableHdMenu: false, // 7. Disable the context menu

title:'Editor Grid',

renderTo: 'formDiv'

});

}


//8. Formatting function for the cost column

function formatCost(value, metadata, record, rowIndex, colIndex, store) {

var results = getCost(record);

return Ext.util.Format.usMoney(results);

}


//9. Calculates the cost

function getCost(record) {

var hours = record.get('hours');

var rate = record.get('rate');

var results = hours * rate;

return results;

}

</script>

</head>

<body>

<div style="margin: 10px 10px;">

<div id="formDiv"/>

</div>

<script>

Ext.onReady( main );

</script>

</body>

</html>

The code creates some sample array data(1) and a Store(2) to map the array into name-value pairs for each row. Then we create a new EditorGridPanel(3) with four columns: 1 non-editable(4), 2 editable(5) and 1 calculated(6). Note the dataIndex member in each column configuration that maps to a name in the Store’s configuration. This mapping allows the grid to lookup and update values in the store. We also set a formatter for the cost column. The formatCost(8)method does the calculation by calling getCost(9) and then changes its formatting using the Ext.utilFormat.usMoney() method. The formatter gets called whenever the row gets updated, including during its initial population.

It may seem like I glossed over a few important points, however, there isn’t very much to tell for this example. Most of the code is for the configuration of the EditorGridPanel and there are very few odd gotcha type things. Ok, there is one odd gotcha, but I haven’t tried it recently and don’t know if Ext 2.01 even suffers from it. The problem was that a grid would sometimes take up the entire width of the browser regardless of its container. Hardcoding the width gets around that problem.

That concludes our discussion of the Ext grid component for today. We’ll talk about it more though in the coming weeks with more interesting examples.

Sunday, February 3, 2008

Criteria for Selecting Technologies

Every developer starts out simply being told which technologies he will use, but it doesn’t take long before he can choose his own destiny. Selecting technologies for a project is surprisingly difficult. Some technologies are incredibly good for a specific set of tasks but become equally cumbersome or even unusable when you go beyond those boundaries. Sometimes a developer knows that a technology is flaky but it promises such a huge benefit that it’s worth the risk. The criteria for selecting a technology is what I’ll talk about today.

A number of factors come into play whenever evaluating any technology. It can be software or hardware, you still have to answer a set of questions about whether its a good idea. Let’s think about a few of the common ones.

  • Does it really solve the problem or do I just want to use it?
  • How long will it take to integrate and build from?
  • Does anyone at the company have any experience using it?
  • How much does it cost and are support agreements available?
  • What is the licensing scheme?
  • How big is the internet community?
  • What are the posts like in the support forum (if there is one)?
  • How long has it been available?
  • How does it impact the schedule?

This list is certainly not exhaustive nor in any particular order. It’s arguably short and other engineers may extend it out further too. It sounds odd, but I do carefully consider each of these questions before selecting a new library and hardware. Let’s go over each question.

Does it really solve the problem or do I just want to use it? What a dreadful question to ask? It is one that requires some introspection as to your own personal motives. Let’s say you are presented with a choice of TopLink and Hibernate and you haven’t worked with either but you’ve heard of both. Which one do you choose and why? Sometimes we inadvertently pick a technology that is going to help our resumes later. Face it, Hibernate is probably worth more to your career than TopLink. But what if TopLink is actually better for the scenario and can finish the project faster and more reliably?

This is a difficult challenge as you must remember that a new project is an opportunity to grow professionally and personally. There is also the ability to be forward thinking and realize that a given technology is growing and so more people are interested in learning it and want to work for places using it. Nonetheless, the first issue to resolve is whether the technology will really solve the problem. If the answer is no, then it is your obligation to move on in the selection process.

How long will it take to integrate and build from? You’ll spend some time trying to figure out how to get it to build in your application, packaging and deployment. Some technologies require one download after another to get all the dependencies and can be painful to get setup. Then figure out how to replicate it across all the developer’s desktops. You can put it in source control or someplace else, but the trick is putting it in a place that requires the least amount of individual developer effort to get it.

There is also a learning curve implied here. You will have to spend some time learning how to use it and write code to take advantage of it. Some technologies are poorly document and very complex. The early documentation days of Struts and Hibernate were only vaguely useful. They have evolved and become much easier, but back then the learning curve was steep.

Does anyone at the company have any experience using it? It is really helpful to hear the experiences of those that have used a technology. First, you get their experiences and good or bad, it is valuable. Second, you may learn something about your teammates and show respect by asking for their opinions. You only have to ask yourself if you like it when a developer asks you for your opinion on a design or implementation to be sure if this is a good idea.

How much does it cost and are support agreements available? Some libraries are open source but have commercial licensing requirements. In those cases you really need to consider the costs of a technology and sometimes you rule it out based on that alone. It is fairly common that some libraries require an OEM license, so you end up buying a license for every copy of your product. This is an annoying and time consuming process. The cost can sometimes be steep or completely inconsequential, but the time spent having to buy licenses and configure customers with those licenses is considerable. This may not be an issue if you send a person to install the product at a customer’s site or if you have a web service that you host.

Support agreements can be incredibly valuable if you use them. It happens often that people buy a support agreement and then never call. It’s a lot like a guy with OnStar whose lost but won’t ask for directions. He bought the service but he just won’t use it. Lots of companies buy service agreements for tens of thousands of dollars a year that they never use. Depending on the agreement you may have unlimited tech support and they can often answer a question a lot faster and more accurately than you can research it.

What is the licensing scheme? Some open source licenses are restrictive in how you may use and distribute the component. Some require you to open your source tree to the public. Some require you to only publish your modifications. Finally, some let you do whatever you want with no restrictions.

I believe that this is the most important question to answer. It is not worth even prototyping code if the license agreement is not compatible product’s sales model. It happens often that a developer picks library with the wrong kind of license and several months later he spends weeks removing the code because he can’t sell the product because of the license.

How big is the internet community? The size of the internet community tells you how popular it really is. In particular, pay attention to the number of blogs talking about it and whether it is positive. Look for people talking about their experiences rather than tutorials. Experiences tell you what it is like to use the technology six months or a year from now, how long it takes to learn and the various gotchas. The bigger the community, the more likely it has been successfully used in other applications. If they could do it successfully then you can too.

There is the risk of zealots. These are the guys that couldn’t imagine using anything else ever. I am careful to take a zealots opinion as a grain of salt. I’ve run into too many zealots that want to use fringe languages, strange protocols and really odd design patterns to solve very common problems. I become especially paranoid of these guys when they are involved in language development as they will recommend that your Java or PHP web application be rewritten in something odd and complicated, like Ada.

What are the posts like in the support forum (if there is one)? Forums can be extremely valuable, except when they are accused of censoring. The value is that you see the kinds of problems that other adopters are experiencing and how they solved them. The common problems are often easy to see from titles alone. The difficulties people are having can also be figured out by the number of readers for a given post.

No forum, or a censored forum, leaves you with very few places to turn for help. Even the simplest library has bugs and has corners that are poorly documented so a developer cannot assume that he will never need help. No forum means very little help.

How long has it been available? This one is a little tricky. Some new technologies are just so incredibly valuable that their age is almost irrelevant. Take Adobe AIR and Google Gears as examples of amazing new technologies that only came on the scene a short time ago.

Consider that you are looking at two different Javascript libraries where one is two years old and the other is only three months. The three month old library may not be as well tested, may not have the community, but more importantly, it hasn’t proven that the developers supporting it are going to be there for the long haul. Just imagine getting into a year long development cycle and finding out that the OSS developers for your Javascript layer have decided to move on and now your stuck with an unsupported third-party Javascript library.

How does it impact the schedule? Some will argue that this shouldn’t factor into the decision making process at all. However, we are expected to produce a lot of code usually at an aggressive pace. Scheduling becomes a very important factor because we are expected to continue producing code and product, not researching something genuinely interesting.

The schedule impact can be broken down in several ways:

  • Time to learn
  • Time to prototype (optional and recommended)
  • Time to implement
  • Time to learn how to debug
  • Time to integrate into the build system
  • Time to deploy
  • Time to test

You then compare the schedule impacts of using a different, equally applicable, technology. Let’s say you have experience in Struts and you want to switch to Tapestry because you’ve heard so many good things. You figure that the Struts implementation may take three weeks to integrate, build, and deploy, but all your reading says the Tapestry implementation will take only a week, maybe two at the most. The difference is the learning curve. Will it take a week or two to figure out the right implementation? How does that affect the schedule now?

Using a new technology requires a lot of schedule padding. The bigger the framework, the more padding. It may work really well and pull in your schedule initially but an unforeseen glitch, like lack of experience with the technology, in your design may use the technology in an odd way and ultimately delay your release by a month or more.

The other scheduling issue, especially when presenting the plan to a project manager, is if the tried-and-true way has nearly the same schedule as the new technology way. Its very hard to argue the merits of a new technology when an old technology takes just as long to implement.

We have looked at some criteria for choosing technologies. To me, the only criterion that is an actual gating condition deals with the licensing terms. The others need to be balanced against each other to see if the overall picture of the technology’s impact is positive enough to warrant the various risks. If all things are equal, then you are left with one last item to consider: your reputation.

Your reputation within an organization is incredibly valuable. Every time you make a great choice your reputation increases. The better your reputation, the better assignments later and a greater likelihood of promotion, pay raises and leadership opportunities. So when choosing a technology, be sure that it’s something you are very sure will work, as success opens doors.

Friday, February 1, 2008

Thanks for January

I’ve spent the last two weeks writing about my opinions and beliefs about somewhat negative topics: recessions, crying wolf and ill-advised refactoring. My plan for the February is to write more about Ext and a couple of web application design issues.

Ext became my favorite Javascript library pretty quickly because it let me design an entire UI that really worked well with a tiny amount of effort. Turns out that I get several hits day from people looking for help using buttons and I’ve spent the last month putting together around 12, or more, different examples of various Ext buttons, their configurations and event management. I was pretty surprised about how much there was to write about.

I am also working on a grid control tutorial and that’s been a blast. Its far from complete and will be ready towards the end of February.

I can’t resist writing about some other team, design and management issues either. I have a few half written entries that will appear throughout the month.

Thanks for reading!