Header Ads

Testing and Quality Assurance

Check out this SlideShare Presentation:

Project Metrics & Measures - Presentation Transcript
Project Metrics & Measures Roadmap to gain agreement on key metrics vital to measure project success Anand Subramaniam

“If Columbus had turned back, no one would have blamed him. Of course, no one would have remembered him either.”
- Source Unknown
Project Metric & Measures - Overview
Project Metric & Measures - Roadmap
Project Process Metric - Model & Guidance
Project Metrics
Project Measures
Monitor & Control
Earned Value
Project Metric & Measures - Overview
Definition Metrics Provides measuring units to depict values, thresholds, constraints, scope, duration, maximums and minimums, averages, etc. Measures Represent information used to establish a common understanding of status, condition, and position of something
Definition Metrics A precise description of what is measured including: numerator, denominator, inclusions and exclusions Targets The agreed upon metric value the organisation desires to achieve
Metric - Challenges Metric Challenges No defined approach to measure return on investment No data validation for accuracy Lack of resources to perform function No formal approach to establish measures and metrics (program & projects) Lack data for reviewing metrics pre & post-live reporting No top management support or participation for metric definition & management No timeline for refining and reporting metrics in line with implementation activities Lack of organisation buy-in towards a single methodology and standardisation of data elements Lack continuous reinforcement, correlation of metrics to project outcomes Lack training / buy in
Metrics – Key Success Factors Key Success Factors Clear charter, scope, roles & responsibilities and direction vital to measure project success Consistent reinforcement of the terminology and metric definition / cal Gain agreement on key metrics for projects & program Governance process defined for metric management Metrics & measures clear and concise Support and ownership from business Dashboard” and summary reports developed pre & post implementation Gain agreement on reporting and ongoing monitoring Develop metric program early in the project Defining metrics at the beginning of the project
Project Metric & Measures - Roadmap
Metrics & Measures - Roadmap #1 Scope & Governance #2 Metric Identification #3 Baseline Measurement #4 Industry Benchmarks #5 Targets #6 Establish Reporting Process #7 Implement #8 (Post) Implement Review / Audit
#1 Governance & Scope
Define scope – In / Out
Define governance process
Identify key individuals
Define authority and accountability levels
Define participants and reporting structure
Lack organisational understanding of scope
Lack participation
Lack commitment from steering committee team
Lack clear definition of authority (who can make decisions)
Metrics – Scope Validation Metrics – Identification & Validation
Involve stakeholders
Correlate to business goals and objectives
Develop a manageable list of metrics
Draft calculations and definitions
Perform industry research and recommend improvement
Set appropriate targets (lag & lead)
Create plans to achieve goals
Monitor and communicate results
Periodic review of improvement ranges
Recommend metric owners
Collect baseline data
Develop tools & techniques
Prepare comparative reporting tool
Generate & distribute report / changes
Operating Processes Management Process
#2 Metric Identification
Define terminology
Basic definitions
Identify guidelines
Define what to consider
Ensure guidelines support project’s objectives
Define approach, metrics and calculations
Define metric criteria
Identify anticipated project impacts
Not gaining agreement on terminology and definitions
Not knowing what to measure
Lack standardisation on the approach
Lack well defined guidelines
Metric Identification - Criteria
Effort and cost required to obtain the metric target
Financial quantification associated with the metric
Availability of industry information to benchmark the metric
Whether or not the metric would have a negative impact on employees and culture
Reliability (believability) of the baseline data supporting the metric
#3 Baseline Measurements
Measurement Criteria
Identify required sample size
Measurement period
Audit review process
Statistically validity
Identify how to obtain
Ensure consistency pre- and post-system implementation
No resources or man hours to collect baselines
No standardisation of data definition, source and / or the collection process
Measurement period not defined
No audit review process defined
#4 & 5 Industry Benchmarks & Targets
Review terminology
Perform industry research
Obtain benchmark data
Define improvement ranges and tolerances
Establish targets
Identify Metric Owners
Gain approval
No time commitment for research
Improvement ranges lack clear definition
Tolerances not defined
No target defined and / or lack understanding
Targets not approved
#6 Establish Metric Reporting Process
Define reporting approach
Define responsibilities as to who will collect, compile and validate data
Define executive reporting needs, training, frequency and distribution
Define data repository required
Define what will be done with the results
Lack ownership nor time commitment
Lack agreement on data sources
Executive reporting not defined and / or does not make sense
No training or education given on the reports
No data repository in place
#7 & 8 (Post)Implementation Review
Implement metrics and compare target to actual metric reports
Review for consistency
Review for executive leader ownership and action plans
Activate internal audit validation
Lack agreement on source of variance and action plans
Lack resources for monitoring and reporting
Not a metric driven implementation project
Project Process Metric - Model & Guidance
Project Process Metrics Model Output Input Process Project Metrics Productivity Rate Test Defect Cost / Schedule Variance Change request
Resources & People Deliverables Time - Schedule Risk Risk Risk Risk Initiate Plan Design Build Test Control Close Operate
Guidance on Metrics
Set clearly defined goals
Train and allocate adequate resources
Focus on project issues (risk management)
Provide timely feedback on metrics analysis
Begin by only collecting a few key measurements
Obtain top management commitment and “buy-in”
Automate the collection process, as much as possible
Understand that metrics are not an end to themselves
Metrics cannot identify, explain or predict everything
Most issues / problems require more than one measurement to characterise and understand
Metrics should augment, not replace, good management judgment
Guidance on Measurement Program
Encourage feedback from participants
Must be institutionalised and repeatable
Develop a “Project / Program Measurement Plan”
Reward participation in the measurement program
Measurements - not analysed & acted upon is useless
Should be part of an overall process improvement effort
Plan for changes on the measurements types collected
Measurement results should remain anonymous
Do not measure individuals
Focus on the process and the product
Project Dashboard 20/07/2009 27/07/2009
Project Metrics
Metrics – Process Steps 1. Identify Metrics Customers 3. Ask Questions 4. Select Metrics 5. Standar-dise Definitions 6. Choose a Model 2. Target Goals 7. Establish Counting Criteria 9. Define Reporting Mechanisms 10. Determine Additional Qualifiers 11. Collect Data 12. Consider Human Factors 8. Decide On Decision Criteria
Focus - Project Metrics & Measures Schedule Quality Risk Cost / Budget Contracts Resources Focus - Metrics & Measure Status Stakeholder Involvement Scope Change Manage-ment
Example – Project Metrics
Completion dates
Response times
Capacity / load
Forecast / Actual Usage
Schedule, Cost, Resources, Deliverables
Example – Project Metrics Description Metric # of training hours per project team member. Training Hours # of project team members who quit or terminated. Turnover # of resources assigned to more than one task. Over allocated Resources Budgeted Cost of Work Performed (BCWP) Earned Value # of tasks that have cost more to complete than expected Over budgeted tasks # of task that should have started but have been delayed Tasks # of tasks that were started but not finished on time Overdue tasks # of scope changes that were approved Scope Change Approvals # of scope changes requested by the client or sponsor Scope Change Requests
Metrics – Do & Don’t
Obtain “buy-in”
Provide feedback
Select metrics based on goals
Focus on processes, products & services
Ignore the data
Use only one metric
Measure individuals
Use metrics as a “carrot” & “stick”
Project Measures
Measure - Purpose Processes, Products & Services Understand Predict Control Evaluate
Measurement Process Tailor Measures Apply Measures Implement Process Stakeholder Needs Issues, Process Characteristics Measurement Plan External Constraints Measurement Needs Data Actions Project Boundary
Monitor & Control
Monitor & Control - Purpose
To track progress
To take corrective action
To detect variance from plan
Monitor & Control – Tracking
Periodic project status meetings & progress reports
Formal project milestones review
Compare actual vs. planned start date for each project task
Earned Value Management
Pitfalls to Avoid
Not assessing significant variances or unfavorable trends
Not monitoring status frequently and / or all aspects of the project
Not monitoring the effects of corrective actions applied
Not addressing the cause of a variance, but addressing the effects
Not enough interim milestones or measurement points to objectively monitor status and when to apply backup plans
Not considering potential impacts of corrective actions – for example, actions taken to address schedule variances will generally affect cost
Earned Value
Planned Value (PV) or Budgeted Cost of Work Scheduled (BCWS)
Planned expenditure cash flows based on the completion of tasks in accordance with the project’s budget and schedule
Actual Cost (AC) or Actual Cost of Work Performed (ACWP)
Actual Project Expense based on completed tasks
Earned Value (EV) or Budgeted Cost of Work Performed (BCWP)
The amount of the budget that we should have spent for a given amount of work completed
Example - Budget, Schedule, Tasks
Project Budget = $40,000
Schedule = 4 months
Tasks = 20 Tasks (evenly divided over 4 months)
$2,000 per task
5 tasks per month
Planned Value or BCWS = $10,000 / month
Example - Budget, Schedule, Tasks
Invoice – 1 st Month Payment = $8,000 Actual Cost or ACWP
Work Completed – 1 st Month, 3 tasks costing= $8,000 as against $6,000 ($2,000 x 3)
Earned Value or BCWP = $6,000
Comparison – PV, AC & EV
Cost Performance Indicators 1 month $1000 Scheduled / Budgeted to do $10,000 work over 5 tasks in a month window BCWS = $10,000 PV / BCWS 1 month $6000 $4000 EV / BCWP Schedule slippage permits only 3 tasks/$6,000 work to be performed BCWP = $6,000 Schedule variance = $4,000 1 month $8000 $2000 AC / ACWP Actual cost of work performed = $8,000 ACWP = $8,000 Actual cost variance = $2,000
Cost Metrics
Cost Variance (CV) - difference between a task’s estimated cost and its actual cost
Negative Value = over budget and / or behind schedule
Positive Value = under budget and / or ahead of schedule
Cost Performance Index (CPI) - percentage of work completed per dollar spent
ratio > 1 = ahead of schedule and / or under budget
ratio < 1 = behind schedule and / or over budget Schedule Metrics Schedule Variance (SV) – the difference in terms of cost between the current progress and our originally scheduled progress SV = EV – PV or BCWP – BCWS Schedule Performance Index (SPI) – a ratio of the work performed to the work scheduled SPI = EV  PL or BCWP  BCWS ratio > 1 = ahead of schedule and / or under budget
ratio < 1 = behind schedule and / or over budget
Earned Value Metrics
Minimum Funds Needed if things do not get worse
Minimum funds = Original total budget  CPI
Funds Needed if things continue to get worse at the same level of slippage
Funds Needed = Original total budget  (CPI x SPI)

“The fellow who does things that count, doesn&apos;t usually stop to count them.”
- Source Unknown
Powered by Blogger.