• Seilevel Team

    Here’s the Team

    Welcome to our Seilevel Photo Op.

Software Requirements Tip: On Average, Our Applications Perform About Average

“Then there is the man who drowned crossing a stream with an average depth of six inches.”
– W.I.E. Gates

This pretty much sums up the problem I have seen with how Performance and Capacity requirements are far too often determined and defined.

 

As analysts, we are usually told things like this by our stakeholders:

  • “We have 100 salespeople who create on average about 2,000 orders a day.”
  • “Our financial analysts typically process about 25 price requests a day each.”
  • “Our Accounts Receivables team processes an average of 500 payments a day.”

If the “Average” measurement is used as the basis on which Performance and Capacity requirements are calculated and defined for an application, then we will almost certainly “drown attempting to cross a stream with an average depth of six inches.” Having made this mistake myself and seen it repeated frequently by other analysts, here is a simple lesson I learned from the school of hard knocks:

The ONLY measure that is meaningful and relevant when it comes to defining the Performance and Capacity needs of an application is the MAXIMUM.

Repeat it to yourself. Commit it to memory. Then repeat it again to yourself.

Determine the MAXIMUM number of Quotations, Orders, Calls, Refunds, Charges, etc., the application will be expected to create, transform, handle or otherwise successfully navigate at any given time. Use this value and ONLY this value for determining the Performance and Capacity requirements of the application.

Simply put, what volume, throughput, capacity (or performance measure appropriate to your situation) do the Users expect the application to handle on the busiest hour (minute, second, millisecond…) of the busiest day of the year? This number should be used to estimate and define the appropriate Performance and Capacity requirements for your application.

This MAXIMUM value can be significantly different from the Average. For example, your company could process an Average of 2,000 Orders a day but spike on Black Friday and hit an annual Peak (Maximum) of 20,000 Orders. If you defined your Performance and Capacity requirements around the Average transaction volume of 2,000 per day, it is almost certain that come Black Friday, your application will “give up the ghost and sleep with the fishes.”

Sadly, applications that meet untimely ends seldom “sleep ‘alone’ with the fishes.” Like the pharaohs of ages past, they are buried with their favorite baubles and those who were nearest and dearest to them while they still walked in the land of the living. And that is how one ends up drowning trying to cross a stream with an “average” depth of six inches.

Remind yourself of the story of the man who drowned crossing the stream with an average depth of six inches of water every time you see step across a puddle. You will thank me, one day.

2 Responses to Software Requirements Tip: On Average, Our Applications Perform About Average

  1. Robert van Lieshout August 16, 2012 at 12:52 am #

    Good post, and I love the quote! Quality aspects such as performance and capacity are often ignored, can’t stress enough how important they are.

    Your post does raise a few questions:
    – does it have to be the MAXIMUM? I quite like the idea of different levels which Tom Gilb puts forward, or thresholds as you see in the QUPER model. If I understand those correctly, it isn’t neccessary to agree to the maximum (particulary if cost/feasibility is prohibitive), but it is important to capture the information as a reference point.
    – at what level of detail should you specify? Typically I find that stakeholders make broad statements which seem to apply to the whole application, but you can find strong variations for different parts. For example, the response time for the ‘print’ functionality is usually not nearly as critical as the response time for the ‘enquire’ functionality.

    Would love to hear your thoughts on this.

  2. Ajay Badri August 17, 2012 at 2:47 pm #

    Hi Robert:

    Thanks a lot for the kind words. Let me take a crack at your questions.

    1. The reason I insist on specifying the MAXIMUM performance needs in the Requirements is that this is the point at which failure typically occurs. Depending on the magnitude of the difference between the average and MAXIMUM, failure can occur long before you get close to the MAXIMUM. The idea behind specifying the MAXIMUM is not to set some completely unrealistic target for the Development team to hit. Instead it is to convey to them the worst case scenario they need to plan, design and develop for. It goes without saying that this should NOT be a fantastical number that is some kind of an ideal but rather it should be as realistic an estimate of expected peak performance that needs to be supported.

    A very important point to keep in mind is that the MAXIMUM value does NOT have to be delivered by the Development team. We are only specifying the Requirements and this can (and often will) vary quite significantly from what is finally delivered. IF, for any reason, it is determined that the MAXIMUM cannot be satisfied, then Business and Development teams can work together to creatively solve the problem. But, IF this value is never specified, then the risk of catastrophic failure goes up exponentially because no one is planning for the MAXIMUM load on the system.

    Consider for example you are defining Requirements for a call center application. You discover that peak call volume is about 100,000 calls in an 8 hour period. This typically happens about thrice a year. The average call volume is about 10,000 calls per 8 hours. So, we have a situation where there is a tenfold difference between the MAXIMUM and the AVERAGE capacity needs. After considering the cost and effort to develop and maintain a solution that supports the MAXIMUM call volume, both Business and Development may decide it is not worth the expenditure. However, the REALITY of having to somehow handle 100,000 calls DOES NOT GO AWAY just because it is a difficult or very expensive problem to solve. So, both teams may decide to tackle the issue by developing a robust queuing system and renting hardware capacity during expected peak load times to handle the MAXIMUM call volume.

    In this example, the teams have NOT created a system that scales to 100,000 calls but managed around it. But, IF the MAXIMUM had not been specified as a Requirement, then the application would almost certainly have failed as the peak load hit. It would simply have attempted to scale linearly and failed miserably since it was never designed to handle a load ten times the expected average.

    2. Most business systems have key outputs that they produce. For example, an Online Order Entry system will create Carts, Quotes and Orders. I will specify the MAXIMUM of these key outputs that the application needs to support.

    For ancillary capabilities like ‘Print Speed’, I evaluate these in the context of supporting the key outputs as opposed to considering them as absolutes. If we extend the example of the Online Ordering system, ‘Print Speed’ is likely irrelevant for generating a print out of Carts, Quotes and Orders since these are rarely, if ever consumed as hard copy. But ‘Print Speed’ is a very relevant metric for the back end of this system where paper invoices are printed and sent out for the Online Orders entered by the Sales Team.

    In a nutshell, I think in terms of the overall throughput of key outputs generated by the system and evaluate the performance of any single feature in terms of their ability to support this throughput. It is quite easy to look at a system and quickly identify processes or specific pieces of functionality that are likely to be bottlenecks. Pay close attention to the performance metrics of the bottlenecks and you should be fine.

Leave a Reply

Your email address will not be published. Required fields are marked *