When I have discussions with my peers about the metrics they collect to run their operations, or at least measure how well their operations perform, the most confused and controversial metric is… On-Time-Delivery (OTD). Many operations feel that they have ODT under control because their monthly management reports say that they are meeting their goals, usually between 90% and 95%. However, in almost every single case when I’ve gone through the OTD numbers with a center’s manager it becomes very clear that the number that they are reporting is wrong and is over reporting the success of on-time-delivery.  Centers that report 95% or better OTD are usually performing at between 60% and 80%, sometimes even lower. How can a service put a measurement system in place, test it and report data for some time and produce numbers that are this far off? The answer isn’t that a single mistake has been made. Far more often, a series of errors and misunderstandings has crept into the reporting process. Let’s take a look at each: 

1. Standard estimates: In order for deadlines to make any sense, you need to have a standard method of estimating how  it takes to do work. If three different people bring you the exact same work on three different shifts, or on three different days, it should take the same amount of work time. Yes there are nother time segments that go into the deadline calculation (more on that below) but the estimation of work time cannot vary between shifts. Imagine going to a restaurant with 20 tables, served by 4 waitresses and only 2 tables are occupied. On your first visit it takes 10 minutes for the waitress to take your order, and bring your meal. The next time it takes an hour. The time after that, 30 minutes. You would think, “Why can’t they get their act together and serve me in a predictable way?” Different results every time you use a service is a frustrating and negative experience for  clients. Standardize your estimates.          
2. Standard queue: Aside from work time, there are other factors for the deadline: how much work is ahead of you, how busy the service is today, how many production workers are available, etc. However you estimate the queue, do all shifts estimates according to the same instructions? If not, this is another reason why clients are experiencing variable satisfaction. Follow a standard.    
3. Variability: Yesterday we discussed variation (“Variations On A Theme: Why Do I Need All This Math?”)? OTD is one of the most variable metrics, because it is directly affected by changes in volume, and voume always changes… from day to day and from hour to hour. Services have “rush hours” where the majority of the work is performed in an hour or two. This variability shades the client’s opinion of OTD.   
4. Standard time: Where does the delivery time come from? If it is automatically generated and entered into your reporting system, great! For most operations this number is entered manually. Which is a big problem. Do I use my watch or a clock on the wall; does everyone follow the same procedures or do they interpret; do I enter data to the second or do I round to the nearest 5 minutes; did I enter the time immediately or estimate after the fact? Depending on how this is done, there could be a lot of variations. So, if you can use a standard and automated measure. For example, if you send an email to the client to inform them that the work is done, use the time generated by the email. If the email is not reporting the right time… tell your IT department to fix it! If they won’t and it remains 2 minutes off, adjust your estimation system by 2 minutes. If you can’t enter time automatically, put ONE clock on the wall  (preferably digital, so there is no interpretation about the exact minute), tell the timekeepers (whoever is enters time info) to enter the minutes and ignore the seconds (let’s not bother adjusting 12:31 to 12:32 if it’s 31 seconds past the minute). And NO adjustments, alterations, or formulas in the management report to adjust the time!
5. Renegotiation: This is a big one! When you set up your metric definitions (or perhaps in a revision that you may or may not have been aware of), OTD MUST reflect the originally agreed to deadline. However, every tracking system I’ve ever seen starts out by using the last renegotiated deadline. Most change this back to the original OTD at some point, but these renegotiations almost always get out of hand, and are a key reason why the client’s opinion of your service can severely vary from what your management reports show. Yes there are reasons why deadlines are missed, but using the renegotiated deadline hides the data, preventing you from seeing or fixing the underlying problems. You may want to retain both the original and final deadlines (to see just how long a deadline can be moved) or even who many renegotiations have happened (if a lot of jobs are renegotiated 3 times, how much time is being wasted with renegotiations rather than killing the cause of renegotiations?).
6. Overnight: Another adjustment is frequently made on overnight or weekend shifts. During the day clients want a specific deadline. But when they head home at night they may say… “just get it done by morning” (or by Monday, if it’s late on a Friday). For these jobs work may not get a deadline, or the deadline is the next morning or Monday morning. Work that should have a deadline of 2 hours is given a deadline of 12 hours or 48 hours. This paints a falsely positive view of weekend and late night OTD, and weakens management skills on these shifts. On-time delivery of work is a responsibility of each shift manager. If you take that away, should you also reduce the pay of weekend and late night managers, since they have fewer responsibilities. Make deadline work the same way on all shifts… although you might want to track these “special jobs”. If someone gives you two days to do two hours of work, even if it is a seemingly minor piece of work, expect the client to be outraged if it is just a few minutes late or has errors.
7. ASAP: This is the same issues as overnight. ASAP is a comment, not a deadline. If work needs special treatment (such as more production workers than normal), or is at high risk for problems, note and track that separately from the deadline. Not surprisingly, you may find that ASAP (and overnight) generates a disproportionate number of client complaints. Trtack ASAP, but don’t alter the deadline.
8. Calibration: Do you regularly verify that ALL intake staff follows the same estimation instructions? This is especially important if you have multiple shifts, where more variations may be affecting estimates. To ensure conssitent estimates, calibrate the individuals who provide estimates. Take some work samples, write out a hypothetical state of the service for the estimator (how busy, how much staff available, any special conditions, etc.) and let everyone estimate work and queue times. If you’ve never done this before, expect a wide range of estimates. Correct whatever appears to be a mistake and have outliers repeat, until their estimates are within 5% of each other. Do this monthly for 2-3 months, then move to quarterly, and possibly to semi-annually after that. You will be amazed by how much of an improvement this will bring.  
9. Consistently early: This one is a shocker. If you are consistently early, if you consistently beat the deadline… you need to be penalized. At first, this doesn’t seem to make sense. Think of it as the other side of variation. Work can be late or it can be early. Either is a missed deadline, or an inaccurate estimation. If you are frequently early, rather than indicating great performance it indicates an  estimation system with an overly large buffer built into the process. This inflates OTD success, and it infuriates clients. Why? Because clients often have tightly scheduled days; knowing that work is likely to arrive early, I might have moved other  items around on my schedule so that I coud read/use/edit etc. this work. Instead, I have something sitting on my desk that I cannot  work on just now (or schedule a meeting, prepare to send to a client, etc.)  How do you fix this? Allow for some early delivery (say, up to 25% early, or 30 minutes on a 2 hour deadline), and reduce the percentage of early deliver over time. Anything earlier than that is report in your monthly management report.

As you can see the cumulative effect of all of these adjustments and assumptions can be huge. Clean these up and you will have much greater visibility into your operations, and much greater alignment with the opinions of your clients. And that’s my Niccolls worth for today!