The end-to-end web developer is a rare find though it would seem like she should be all too common. The term refers to developer that can write quality code at every tier of a web application. For a Java developer, that means being able to write good HTML, Javascript, CSS, JSP, Servlets (though they appear in many forms), EJB (and their various kinds), JDBC (or Hibernate), SQL and a little system administration too. It sounds like a common set of skills and many employers claim to require them, but often these skills are difficult to gain and some jobs make it difficult to practice.

Before I dive into the ethereal land of opinionated-qualification-make-believe, please consider how your company divides its teams. Many companies have their developers work on specific tiers of the application. Some developers work on the UI, others write EJB services and some focus on database development. Employers may want to see a wide range of qualifications on a resume, but may quickly lock a developer into a specific tier or role that doesn't take advantage of those other skills. This organizational style also makes it difficult to grow skills outside of a developer’s immediate role.

Another style operates on a functional slice basis. The concept is that a developer is responsible for one or more features and writes the UI, middle and back tiers. Obviously, this organizational style takes advantage of the developer's entire range of skills. This style does not preclude specialists, such as expert UI and database engineers. The specialists may do fit-and-finish work, perform design reviews and set the direction for the particular tier. A developer in this organizational style is able to practice a wide range of skills daily.

The first approach encourages specialization and the second is generalization. An end-to-end developer coming from the first approach may have some skills but was afforded limited opportunity to grow them. The multidisciplinary requirements of the second approach requires a lot of upfront knowledge that is difficult to attain without being in such an environment previously.

Back to being an end-to-end developer. Why are end-to-end developers valuable? Startup and small companies often have a wide range of development needs but may lack a significant budget to hire specialists for each tier. Generalists fill those gaps though they can command a higher initial salary, they are still less expensive than hiring two or more specialist developers.

Are generalists more skilled than specialists? This is an extremely difficult question to answer. Specialists are focused on a given subject and should be more knowledgeable on that subject. Yet a generalist may have actually worked with that technology for longer and at a greater depth than the specialist. An example of this is a UI specialist with two years of Tapestry experience compared to a generalist with four years experience in the same technology.

Years of experience is not necessarily an accurate gauge of the developer's depth of knowledge though. The only way to tell if a developer has the experience necessary to do the work is by probing through the interview process. This is necessary when dealing with generalists and specialists alike.

The generalist vs specialist argument is one of personal choice and there is no right or wrong answer. There are advantages to both and the rule of staying current with evolutions in technology remains the same.

The concept of the end-to-end developer is fascinating as the developer should be able to write an entire web application on her own. Such a developer is proven when she has actually built something on her own and made it publicly available. It doesn’t need to be earth shattering or massively successful, it just needs to solve a problem and be accessible to the world.

But there are other, less obvious skills being learned too, like self-determiniation, time management and follow-through. Many job descriptions look for a self-starter: some go-getter developer that can work with little supervision, lead a team and finish what they agreed to in a reasonable time. If a developer spends the time to learn to every tier of web application development, builds something and makes it public then she meets pretty much all the listed requirements except for leading a team. Thankfully, team leadership is a learned skill.

One item that should be clarified is the difference between the hobbyist and end-to-end developer. There is a reluctance to hire a developer that learned specific skills through a hobby project rather than a commercial project. There are some good reasons for this, such as hobby projects usually don’t require a developer to focus and develop a project for eight, or more, hours a work-day. It’s hard for a developer to gain breadth and depth, in a vacuum, working on a hobby only a few hours a day or week. The big difference is that a hobbyist doesn’t have to finnish, the end-to-end developer does and then works harder on it and makes it into something big.

It would be reasonable for someone to look at this and say “he’s just trying to justify hobbyists.” There might be some truth in that. Yet look out at the successful sites and think about how many of them emerged from little more than hobbyist developers. How many blogs are little more than hobbies and yet we glean great information from them. A big difference between the hobbyist and the end-to-end developer is that the former is interested in learning something but has no real goal beyond that while the latter has a goal, finishes and makes it public.

Finishing makes the difference.

I've been involved in a few Agile projects and have formed a few opinions some positive and some negative. For example, I don't think that Agile methods work very well for established mass market desktop products while I think it works superbly for IT projects, web applications and projects that have a relatively small number of customers. I especially like the ability to come to a stopping point quickly and show the changes to the team and customers, which is what I'm going to talk about today.

I love weekly or bi-weekly demonstrations with the team and monthly customer meetings because I can get feedback and adjust my product rather than committing to something very uncertain. This demonstration schedule does not necessarily match the iteration cycle either and can run on just a developer's desktop. The point is to show the latest changes from one demo to the next, give credit where it is due, make minor adjustments to design and requirements and give a clear indication of the velocity of progress. My favorite benefit is the amount of sleep I get at night.

Yes, sleep is my favorite benefit. I don't worry about whether the customer is going to like what I am building by the time I hit delivery date. I don't have to worry about the work-flow because the team and product management have tried it a dozen or more times. I don't have to worry about major silly bugs because the demonstrations have usually exposed them already. As we know, all software crashes at the most embarrassing time: during a demo in front of a crowd. By ship date I can rest assured that anything glaring, bug or requirement, has been discovered and solved.

Building demonstrable code is not easy on a one or two week schedule. The goal is to stop building new functionality early enough to integrate and stabilize the code for demo. I like having only one week between demonstrations because the code base has not moved terribly far forward, which makes integration easy. I usually spend a portion of the day prior just for testing the changes and making fit and finish modifications. However, I can't do very much in the way of major changes. I work to get to the next demo and so I either must finish my changes entirely, disable the new code or demo around the incomplete code. Planning the demo is obviously important.

It often happens that the conceptualization of a work-flow turns out to be somewhat flawed. One can argue during the design phase that the flow is wrong but it is difficult to dissuade some individuals without having them actually try it. A demo illustrates the work-flow issues with perfect clarity. The results are the requirements, UI, back-end and test plans may change, but they changed early enough in the development cycle to be corrected investing a major development effort.

Regular customer demonstrations may not be an option. Some customers find demos annoying while others are happy to see progress. Regardless, showing them the progress made and the work flow at regular intervals is incredibly valuable in ensuring that the delivered product is what they want. The changes can be as simple as labeling issues where the customer realizes that the label is Ok, but they really call it X or Y. Its a 2 minute fix that makes the customer happy. Other changes can be very invasive, but you caught it early so there is hopefully time to solve it.

Making weekly team demonstrations effective is a straightforward process. First, have goals for what should be accomplished from one week to the next. Second, plan when to stop development and focus on integrating, testing and bug fixing. Third, during the demo describe the changes from the previous demo, giving credit where it is due. Fourth, modify the code after the demo to include suggestions and comments.

The weekly demo is incredibly valuable for finding flaws in requirements, work flow and implementation. In the best case, it validates that you were right all along. In the worst cases, it gives you the opportunity to change directions earlier rather than shortly before a delivery date. In any event, you know that by the time you deliver the final product that you are aligned with customer expectations.

I’m beating a dead horse with this one, but I recently saw someone refer to me as “the savvy duck” and felt like I needed to set the record straight about my method for selecting a domain name. I’m not giving any rules, best practices or anything like that. This is story time.

I was about to start writing a data tracking tool and was looking for a domain name that would fit with the product. I went with an adjective and noun combination and registered my domain. My adjective was the word empirical. The platform I was building was for developers and I figured that most of them knew the word empiricaland would hopefully work out well. This was not the case at all. I had only slightly better results with empirical than if I chose the Hawaiian word for trigger fish: Humuhumunukunukuapua'a.

I didn’t realize my error until the day I called Dell about buying a laptop. I had to give my email address to the sales rep so he could send me a quote. I rattled off the email address and we ended the call. I was getting quotes from three other vendors so I made a few other calls.

One sales rep didn’t know the word at all. Ok, some developers slam sales guys as not being the brightest folk but good sales people are actually pretty darn smart. I spelled out empirical and moved on to my next call. This new guy just asked me to spell it out. Apparently he was tired of trying to figure out spellings for domain names awhile ago and made his request in a way that indicated he does this a million times a day and tomorrow he’ll do it a million more times.

The email from the Dell rep didn’t arrive. It had been an hour. I logged into another email account and sent my empirical account an email. My message arrived within a few seconds so I knew the account was working. I called the rep and he told me the message bounced. He spelled empirical wrong.

I had to talk to a person with the state about registering my business. There too I had to define the word and how to spell it. These people deal with business names all day and I live in a state on the French-Canadian border. Our towns are packed with business names that have pronunciations that seem only loosely related to their spelling. I knew I was screwed when I realized that the state employees couldn’t spell the word empirical either.

I wanted to start a new analytics application last year. I was talking with an odd CEO about figuring out a domain name and he gave me some interesting advice. He basically said that “many companies change their focus in the first few years so pick a name that can be anything.” The idea is that the company can reinvent itself and the domain name is still applicable.

He also suggested using a cute animal for the domain or have a really attractive woman on the home page.

My wife and I were sitting around trying to think of cute animal domain names. I wanted an adjective that described the animal as being intelligent. The animal’s name had to be something easy to spell. We churned out names like “wisesquirrel”, “trickybadger” but most people don’t write animal names very often. How often do you type the word wombat, chipmunk or wolverine?

Finally, we came up with savvyduck.com. Sure, not everyone knows the double-v in savvy but everyone knows duck. A lot of people like ducks too. It sounded cool and it meant nothing. The best bit was that a product could be called savvy duck, but it seemed to apply better to the user. The user is the savvy duck. The user is the smart one. That’s true of the users of most products too.

Picking savvyduck.com worked out really well because I did end up switching from a product to a blog and it is memorable. I, however, am more of an opinionated schmuck rather than a savvy duck.

Data visualization through charts and graphs is incredibly common and yet Ext doesn’t have a readily apparent charting library! JQuery has a great little charting library called Flot and getting it to work within an Ext application turned out to be pretty easy. I’ll show how I integrated the two and the problems I ran into. I’ll also go over some common issues in choosing a charting library.

Charts, whether static or dynamic, are a common requirement in almost every application with a grid. I say almost because some users could care less for a chart, they just want the data and can visualize the data just fine on their own. Many users can’t imagine the data on their own and need some kind of chart sitting near any table or grid. The volume of data in a grid may also be so large that a user simply can’t imagine the breakdown without some automated visualization.

I mentioned two kinds of charts and they are worth discussing briefly. Static charts are usually non- or minimally interactive images generated on a server and displayed on the browser. They often have image maps that let a user click on a line or a point and drill-down into the data. Then the browser loads the next chart image and the user can continue drilling down. The refresh rate is obviously slow, because there is a roundtrip between the browser and the server when redrawing the chart, and the interaction is limited.

Dynamic charts are usually flash or Javascript based and are excellent for displaying fast moving data. These kinds of charts are usually more interactive: allowing for zoom, filtering and rapid updates. They can provide a better user experience depending upon on the volume of data to draw.

Choosing one method over another is dependent on product requirements and volume of data. Consider that you want the ability to cut-and-paste a chart from a web application into an email. Static images work great, but javascript and flash charts usually need some kind of external capture utility like SnagIt(PC) or Grab (Mac).

Images work very well when you have huge amounts of data to plot. A scatter plot with 5000 points runs well on a server. Trying that with a javascript control incurs both the rather large download of the data and the long processing time, which can sometimes stretch long enough for the browser to popup the dialog asking the user if he wants to continue running the script. The image again can be cached on the server, so the creation time is incurred only when the data changes, which is hopefully only once every couple of minutes. Images with image maps still have a long download though.

Javascript and Flash charts can interact seamlessly with a Javascript application. The chart’s look and feel is often far more polished and its responsiveness is often surprising to users.

Again, product requirements and volume of data are the most important drivers for a technology than the technology itself. It often happens that a product manager will want a flash chart when a static image chart is actually more effective. It also happens that someone wants a static image chart to support a rarely used requirement rather than a dynamic chart to show rapidly changing data, which is a more often used feature.

I’ve yammered on about charting technologies long enough. I started by saying that I needed to integrate a Javascript chart library into an Ext based grid application. I’ve taken a previous example of the EditorGridPanel and extended it with charts and a few other minor embellishments. I went along with JQuery’sFlot because I only needed bar charts, but it also supports line and scatter plots too.

Ext has an unusual feature that allows you to swap out its default base library and replace it with one that acts as a facade on other libraries, like JQuery. I am not going to take advantage of this feature for two reasons. First, Flot uses a more recent version of JQuery than what is distributed with Ext. Second, I am more interested in seeing if Ext will continue to operate with a new library messing around with the DOM. The second reason is especially important if we later want to use a library that supports pie charts or another chart not supported by flot.

The integration process is very easy and didn’t run into any problems that couldn’t be worked around. Flot needs only a few things to build a chart. It obviously needs data which I will supply via a Store. We need some event support to update the charts when data changes. We need a container to display the charts within the Ext application. Finally, it would be really snazzy if the charts updated as the data changes. Let’s look at the code to see how these were accomplished. The following description will refer to the red numbered comments. Also note that the code is just an example and not meant to be used in production.

<html>

<head>

<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">

<title>Ext EditorGridPanel with JQuery’s Flot Example</title>

<link rel="stylesheet" type="text/css"

href="./ext-2.0.1/resources/css/ext-all.css" />

<script type="text/javascript" src="./ext-2.0.1/adapter/ext/ext-base.js"></script>

<script type="text/javascript" src="./ext-2.0.1/ext-all-debug.js"></script>

<script type="text/javascript" src="./flot/jquery.js"></script>

<script type="text/javascript" src="./flot/jquery.flot.js"></script>

<script>

var initComplete = false;

function main() {

// Setup the store and its data

var sampleData = [

[ 'Bob', 20, 13.45 ],

[ 'Bill', 40, 12.92],

[ 'Mike', 45, 23.02]

];

var store = new Ext.data.SimpleStore({

fields : [

{ name: 'name', type: 'string' },

{ name: 'hours', type: 'int' },

{ name: 'rate', type: 'float' },

{ name: 'cost', type: 'float' }

],

data: sampleData

});

// Initialize the grid control

var simpleGrid = new Ext.grid.EditorGridPanel({

store: store,

columns: [{

header: 'Name', // Field cannot be edited.

width: 160,

sortable: false,

dataIndex: 'name'

},

{

header: 'Hours', // Field can be edited

width: 75,

sortable: false,

dataIndex: 'hours',

editor: new Ext.form.NumberField({

allowBlank: false,

allowDecimals: false

})

},

{

header: 'Rate', // Field can be edited

width: 75,

sortable: false,

dataIndex: 'rate',

editor: new Ext.form.NumberField({

allowBlank: false,

allowDecimals: true

})

},

{

header: 'Cost', // Calculated field

width: 75,

sortable: false,

renderer: formatCost,

dataIndex: 'cost'

}

],

clicksToEdit: 1,

stripeRows: true,

height: 100,

enableHdMenu: false,

region: 'south'

});

var commonChartStyle = {"margin": "10px 10px 10px 10px"};

// 1: Set up the containers for the charts

var rateContainer = new Ext.Panel({

height: 200,

width: 200,

style: commonChartStyle,

id: 'rateChartContainer'

});

var hoursContainer = new Ext.Panel({

height: 200,

width: 200,

style: commonChartStyle,

id: 'hoursChartContainer'

});

var costContainer = new Ext.Panel({

height: 200,

width: 200,

style: commonChartStyle,

id: 'costChartContainer'

});

var viewPort = new Ext.Panel({

title: 'Chart-Grid Example',

frame: true,

layout: 'border',

renderTo: Ext.getBody(),

height: 400,

width: 674,

items: [{

  region: 'center',

border: true,

bodyStyle: 'background-color: white; border: 1px solid #99BBE8',

layout: 'column',

items: [{

title: 'Hours',

width: 220,

items: hoursContainer

},{

title: 'Rate',

width: 220,

items: rateContainer

},{

title: 'Cost',

width: 220,

items: costContainer

}

]

},

simpleGrid]

});

drawCharts(store);

initComplete = true;

}

function drawCharts(store) {

drawChart(store, 'rate', 'rateChartContainer');

drawChart(store, 'hours', 'hoursChartContainer');

drawChart(store, 'cost', 'costChartContainer');

}

function getData(store, nameColumn, dataColumn) {

var dataResults = new Array();

var tickResults = new Array();

// 2: get the chart data

for( var recordIndex = 0; recordIndex < store.getCount(); recordIndex++ ) {

var record = store.getAt(recordIndex);

var tmpData = [(recordIndex+1)*2, record.get(dataColumn)];

var series = {

bars: { show: true },

label: record.get(nameColumn),

data: [ tmpData, tmpData ], // workaround

color: recordIndex

}

dataResults.push( series );

tickResults.push([ ((recordIndex+1)*2)+1, record.get(nameColumn) ]);

}

return {

data:dataResults,

ticks:tickResults

};

}

function drawChart(store, column, chart) {

var chartInfo = getData(store, 'name', column);

// 3: draw the chart in the container

$.plot($("#"+chart),

chartInfo.data,

{

xaxis: {

autoscaleMargin: .25,

ticks: chartInfo.ticks

}

}

);

}

// Formatting function for the cost column

function formatCost(value, metadata, record, rowIndex, colIndex, store) {

var results = getCost(record);

record.set( 'cost', results);

if ( initComplete ) {

// 4: redraw the charts when the data changes

drawCharts(store);

}

return Ext.util.Format.usMoney(results);

}

// Calculates the cost

function getCost(record) {

var hours = record.get('hours');

var rate = record.get('rate');

var results = hours * rate;

return results;

}

</script>

<style>

p {

margin:5px;

}

</style>

</head>

<body style="margin: 10px 10px;">

<script>

Ext.onReady( main );

</script>

</body>

</html>

First, we create three nearly identical Ext.Panel(1) objects to contain the chart. They all have hard coded sizes and an explicit id field. Flot will calculate the size of the chart based upon the size of the container. If you let Ext automatically set the height or width, Flot may see the value as being 0 and make a very small chart. The user can resize the panel, but you must ensure that the height and width are valid.

The id field is necessary so that we can point JQuery to the right DOM object to modify. You can use this technique with other charting libraries too. Ext will generate a div with the id you specify. Once generated, it is easy to point any other compatible library at the div and generate whatever you want.

Second, we load the data for the chart from the Store(2). Flot’s data series has the format of [ [x axis value, y axis value]] . A line chart would have multiple x-y value pairs.

We will also replace the normal tick values with a person’s name, which is done by substituting an axis value with a label. Each axis value can represent a range from N to N+1. The 0 position of the X axis is flush to the Y axis. Having a label on the 0 location makes the label look strange and somewhat unreadable.The goal of the tmpData assignment line is to move the X axis value away from the 0 position. 

The code then creates a series object that sets the chart type to bar, the label for the series, the color and the data. The data line is really odd because there is a bug in the Flot library for bar charts. Flot requires more than 1 data point or it doesn’t render the series. The workaround listed in the bug report is to have two identical values in the series. That’s why tmpData is used twice.

Another interesting aspect of Flot is that each series can specify a different chart type. You can mix line, scatter and bar charts in a single chart. Generally it is unwise to do so as it makes for a cool looking chart that doesn’t necessarily make the data more readable. Google Edward Tufte and you will find plenty of documentation on good charting techniques.

The last interesting line deals with the ticks. The tick values are using a different index than what was used earlier. This was done so that the label would be on the trailing edge of the bar rather than the leading edge and looks marginally better.

Third, we render the chart(3). The plot method gets called with three parameters: the id for the container, an array of series data, and the chart options. The container id reference finds the DOM object and adds a couple ofcanvas and div tags to the Panel. Flot uses the array of series data to draw the chart. Finally, the options parameter modifies the X axis to change margins to be more readable and replaces the ticks with the values mentioned earlier.

Fourth, we modify the formatter to update the charts when the grid changes(4). The effect is that a user can modify either the rate or hours columns and the charts immediately update.

You will see a few odd sections of code wrapped in if(initComplete)statements. JQuery cannot go to work unless all the Ext objects are rendered, which may not occur until later. The if statement ensures that the UI is fully rendered before trying to run Flot code.

The goal of this example was to show that it is possible to integrate Ext with other major Javascript frameworks to supplement Ext’s features. In this case we added a chart to an existing project. The selection of the chart library was almost arbitrary, though I tried a few others and thought that Flot made a really nice looking bar chart. Now you can go off and try mixing libraries using the techniques shown here.