Software Architecture (3)

Some useful links to Software Architecture:

Software Architectural Styles: http://hebb.cis.uoguelph.ca/~dave/27320/new/architec.html
A class website: http://courses.cs.tamu.edu/cpsc689/hohin/fall00/


Chapter 4 Shared Information System


4.1 Shared Information System

• Separate programs for separate subtasks;
• Multiple individual steps is composed to larger tasks by passing data in a known format; (not flexible, not interactive)
• Shared data store; (data stores with different representation)
• Shared information system evolution pattern.

4.2 Database integration

2 strategies to handle data diversity: unified schema and multidatabases.

4.2.1 Batch sequential

• Transaction (commit(), rollback())

4.2.2 Simple repository

• interactive tech provides the opportunity and demand for online update adn query.
• Increasing transaction and modification.

4.2.3 Virtual repository

• Use schemas to provide a logic view of the multiple distributed databases.

4.2.4. Hierarchical layers

4.2.5. Evolution of shared information system in business data processing

• Batch processing
• Interactive processing
• unified schemas
• Multidatabase (layered hierarchy with client server interaction).

4.3. Integration in software development environment

• Batch sequential
• Repository
• Hierarchical layer

4.4. Building design

• Repository
• Intelligent control

4.5 Architectural structures for SIS

Teamwork, team structure, and motivations

From chapter 11, 12, 13 of "Rapid Development" by Steve McConnell.

1. Motivation

Developers are more introvertious than the general population. Achievement, possibility for growth, work itself, and technical supervision opportunity are among the top five factors motivating them. On the contrary, managers are more excited by the responsibility, and recognition.

For the long term, reward is important. However, sometimes the reward itself may undermine the quality of work, since the passion for the work itself is more important. Therefore, the rewards should be presented as a gesture of appreciation rather than an incentive. The more external the reward is, the less interested the work will be.

The Hawthorne effect is gold to project managers. Run every software project as an experiment, as a pilot project. Try something new, and make sure every team member knows it. If get some conclusive result, use it for the later projects, otherwise you get the better productivity at least. Note that: don't manipulate.

Performance review. The effect can be negative or positive. a 2 blades sword.

Morale

2. Teamwork

The characterisitics of a productive team:

1. A shared, elevating vision or goal;
2. A sense of team identity;
3. A results-driven structure(roles must be cleared and accountable, an effective communication system, monitoring individual performance, decision made on facts);
4. Competent team members (specific tech skills, desire to contribute, collaboration skills, mixed roles);
5. A commitment to the team;
6. Mutual trust (honesty, openness, consistency, and respect);
7. Interdependence among team members;
8. Effective communications;
9. A sense of autonomy;
10. A sense of empowerment;
11. Small team size;
12. A high level of enjoyment.

Why teams fail?

The contrary of any one of above.

To point out one thing: who first, then what. To keep a unqualified person on the bus hurts the team most. If someone is not qualified, the firing decision has to be made as early as possible before it hurts the project.

Long term team building: don't dismiss the team because of short peroid of idleness. The cost of rebuilding a good team is way much more than the pay of idleness. The risk is increasing as well.

A summary of teamwork:

1. for team leaders

1. Avoid compromising team goals with political issues;
2. Exhibit personal commitment to the team's goal;
3. Not to give too many priorities;
4. Be fair and impartial to the members;
5. Confront the inadequency of any team member;
6. Open to new ideas and information from the member.

2. for team member

1. Demonstrate a realistic understanding of my role and accountability;
2. Demonstrate objective and fact based judgement;
3. Collaborate effectively with other members;
4. Make the team goal at higher priority;
5. Demonstrate a willingness to devote whatever effort to achieve team success;
6. Be willing to share and feedback appropriately;
7. Provide help;
8. Demonstrate high standards of excellence;
9. Support team decision;
10. Demonstrate courage of conviction by directly confronting important issues;
11. Demonstrate leadership in ways that contribute to the team success;
12. Respond constructively to feedbacks.

3. Team structure

Organize the team according to the objectives:

1. Problem resolution---trust;
2. Creativity---autonomy;
3. Tactical execution--clarity.

Team models:

1. business team: expert lead + equal members, the most common model;
2. chief-programmer team(surgical team): surgen+backup programmer+administrator+toolsmith,e tc; but rare programmers have the expertise to serve as surgen;
3. skunkworks team: a group of talents, black-box to the outsider;
4. Feature team;
5. Search-and-rescue team;
6. SWAT team: a group of experts, each are good at one tool;
7. Professional athletic team;
8. Theater team;
9. Large team;

Managers and tech leaders

Customer oriented development

Notes from chapter 10 of "Rapid development" by Steve McConnell

1. Customers' importance to rapid development

Top 3 reasons of delayed projects:

1. Lack of user input;
2. Incomplete requirements specifications;
3. Changing requirements and specifications.

2 main reasons that pay attention to customer relations:

1. Good relations with customers improve the development speed;
1. Improved efficiency
2. Less rework
3. Reduced risk
4. Lack friction
2. Good releations with customers improve preceived development speed.

2. Customer-oriented practices

1. Planning
1. Select an appropriate lifecycle model, keep the customer with steady, tangible progress update.
2. Identify the real customer.
3. Establish an efficient method for interacting with the customer (single point of contact).
4. Create a win-win project.
5. Manage risks.
2. Requirement analysis
1. The challenge is to collect the real requirements;
2. sometimes real requirements conflict with collected requirments;
3. More often, they are simply missing;
4. Customer tends to interpret the requirement broadly, and the developers is opposite.
5. Case: **visual c++ (lower the switch cost to c++) against borland c++(expert c++)
3. Design
1. Lack of design flexibility was the weakness of the design
2. Employ design practices that allow customers to change their minds occasionally.
4. Construction
1. Write maintenable and modifiable code in order to responed to user's change;
2. Progress monitoring practices;
3. Select a lifecycle model allow customers with steady, tangible progress update.

3. Managing customer expectations

Many problems in software development, especially speed, arise from unstated, unrealistic expections.

1. Customers assumes that the development is as easy as they preceive, such as drag and drop documents from one application to the other on win 3.1, it is easy to them, what about the implementation?
2. Customers assumes that you understand all of their requirements without having to explain to you.
3. One important task is to educate the customers during the development.
4. With inflated expectation, developers look like losers even when they do a good job. Who inflate expectations damage their credibilit, undermine their working relationships with customers.

Further reading:

Naomi Karten, Managing expectations

Some interesting posts

http://www.problogger.net/archives/2005/09/23/problogger-turns-1/

A Digital Cryptography Survival Guide

http://digitalcryptography.blogspot.com/

Software development fundatmentals

Chapter 4 of "Rapid Development" by Steve McConnell

3. Quality assurance fundamentals

Pay attention to quality assurance fundamentals from day 1.

Facts:

• Most of the delayed software projects are due to the defects;
• The products will lowest defect rate were also the ones with shortest schedules;
• 20% modules occupy 80% defects;
• The longer the defect is not solved, the more it costs;

3.1. Error-prone modules

Identify 20% error prone modules, and fix/redesign them.

3.2. Testing

Testing effectiveness varies enormously. Typically unit test find 10~50% defects, system test has 20~60%, but both of them can only catch less than 60% defects.

Testing typically slows the development speed, but it has to be done. And it has to be done as early as possible. Develop a good test design in the first place is the key to all module and system developments.

3.3. Technical reviews

• Walkthroughs;
• Code reading;
• Inspections;
• Peer review.

3.4. Following the instructions

Especially SOP.

Risk Management

Notes for Chapter 5 of "Rapid Development" by Steve McConnell

1. Elements of risk management

Task of risk management is to identify, address, and eliminate sources of risk before they threat the software project completion.

5 levels of risk management:

1. Crisis management;
2. Fix on failure;
3. Risk mitigation;
4. Prevention;
5. Elimination of root causes

The purpose of risk management is to address the software schedule risk at level 4 and 5, instead of 1 to 3, in which you have already lost the battle.

Risk management is composed of 2 elements:

1. Risk assessment


• Risk identification: produce a list of potential risks;
• Risk analysis: the likihood and impact of the risks, and the risk levels of alternative practices;
• Risk prioritization: a list of risks prioritized by impact, a basis for risk control.


2. Risk control


• Risk management planning: a consistent plan to handle each significant risk
• Risk resolution: execution of the plan
• Risk minitoring: monitor the progress toward resolving each risk item, and identify new risk and feed them back to risk management process.

2. Risk identification

3 general risks the software development may face are the classic mistakes listed in common risks, the ignorance of development fundamentals, and lack of risk management.

Most common risks:

1. Feature creep
2. Requirements or developer gold-plating
3. Shortchanged quality
4. Overly optimistic schedules
5. Inadequate design
6. Silver bullet syndrome
7. Research oriented development
8. Weak personnel
9. Contractor failure
10. Friction between developers and customers

There are also some other risks, for example, the development team has less power on the schedule, resources, and product definition, no effective top management sponsor, layoffs, market changes, budget cut, tools, etc.

3. Risk analysis

Risk analysis is based on risk exposure or risk impact, which equals to the probability of the unexpected loss X the size of loss. Example:

1. Overly optimistic schule with 25% probability to extend 4 weeks, the impact is 0.25 *4 = 1.
2. New requirement with 50% probability to extend 2 weeks, the impact is 0.5*2=1.

They are of the same risk.

1. Estimate the size of loss: drill down the detail loss and combine them together;
2. Estimate the probability: typically ask the person familiar with the system to estimate the probability, or use group delphi meeting.

4. Risk prioritization

Use the risk impact or exposure to set the priority of the risks, focus on the top 20% risks.

On other hand, some risks rarely happen, if it happens, the loss is big, therefore need to bring those risks to the top as well.

5. Risk control

Riskmanagement plan:

• Who, what, when, where, why, and how for each risk management;
• How to monitor the risk, the status;
• Identifiy the emerging risks after this closes out.

Risk resolution

• Avoid the risk;
• Transfer the risk from one to another part;
• Buy(investigate) information about the risk;
• Eliminate the root cause of the risk;
• Assume the risk;
• Publicize the risk;
• Control the risk: develop contingency plan to handle it if you can't resolve it;
• Remember the risk;

Risk monitoring

• Top 10 risk list;
• Interim postmortems;
• Risk officer

6. Risk, high risk, and gambling

Software Architecture (2)

2. Architectural styles

A list of common architectural styles:

1. Dataflow systems
• Batch sequential
• Pipes and filters
2. Call and return systems
• Main program and subroutine
• OO systems
• Hierachical layers
3. Independent components
• Communication processes
• Event systems
4. Vitural machines
• Interpreters
• Rule based systems
5. Data centered systems
• Databases
• Hypertext systems
• Blackboards

2.2 Pipes and filters

In this style, each component has a set of inputs and a set of outputs. The style is composed of 2 components: filter to apply transformation between input and output, pipe to connect filters.

Properties:

• Filters must be independent, should not share state with other filters;
• Filters do not know their upstream and downstream filters;
• The specifications of input pipes and output pipes are restricted, but not the end of pipes: filter.
• The correctness of filter-pipe network should not depend on the order the filters apprears.

One variant of pipe and filter style is batch sequetial in which each filter processes all of its input data as a single entity.

Example: Unix shell

Pros:

• Easy to understand of the system behavior;
• Support reuse;
• Easy to maintain and enhance;
• permit special analysis, such as throughput, dead lock etc.

Cons:

• Lead to a batch organizaion of processing;
• Not good at interactive operations;
• Have to maintain correspondences between 2 separate but related streams;
• May force a lowest common denominator on data transmission, bad performance and complexity.

2.3 Data abstraction and OO organization

In this style, each component has a set of inputs and a set of outputs.

2.4 Event based implicit invocation

In this style, each component has a set of inputs and a set of outputs.

2.5 Layered systems

In this style, each component has a set of inputs and a set of outputs.

2.6 Repositories

In this style, each component has a set of inputs and a set of outputs.

2.7 Interpreters

In this style, each component has a set of inputs and a set of outputs.

2.8 Process control

In this style, each component has a set of inputs and a set of outputs.

2.9 Other architectures

In this style, each component has a set of inputs and a set of outputs.

2.10 Heterogeneous architectures

Software Architecture (1)

Preface

Start from today, I will spend the rest of week working on "Software Architecture" by Mary Shaw. One objective is to review the book so as to get some background information for the presentation I am going to give; the other intention is to leave myself some trace what I have read. My plan is to read the first 5 chapters in this weeek, after each one, I will summarize the content here.

The first chapter is introduction, giving the definition of engineering and software architecture; The second chapter mainly gives seven popular architectures; Some case studies are presented in chapter 3 and the different designs are discussed and compared; Chapter 4 is on the shared information systems, which is really about databases to my understanding; and chapter 5 provides architectural design guidelines. The book also provides some formal tools (chapter 6 to 9), however they are too academic to be useful to the introduction of this topic, therefore are excluded.

1. Introduction

1.1 What is software Architecture (SA)?

Software architecture involves the description of elements from which systems are built, interactions among those elements, patterns that guide their composition, and constraints on those patterns. In short, it is related to the system structure instead of algorithm and data structure.

2 reaons that SA is of substantive value to SW engineers:

• Engineers have evolved over time a collection of idioms, patterns, and styles of SW organization that serves as a shared, semantically rich vocabulary;
• Although architecture are abstract in relation to the actual computation details, those structures provide a natural framework for understanding broader, system level concerns.

1.2 An engineering discipline for software

Engineering refers to the disciplined application of scientific knowledge to resolve conflicting contraints and requirements for problems of immediate, practical significance in the service of mankind.

• cost-effective solutions;
• practical problems;
• apply scientific knowledge;
• build things;
• in the service of mankind.

Engineering design are of 2 kinds:

• Routine design: familiar problems;
• Innovaive design: unfamiliar problems.

Software is an innovative design, but we try to make it more routine (component, routine libraries?)

Evolution of engineering discipline:

Production

> Commercial

Craft > Professional Engineering

Science

Engineering exploration starts from craftmanship, later it becomes widely accepted, therefore routine, then science comes in to describe the routine process, which leads to the engineering discipline.

1.3 The status of SA

It is still immature, but made good progress:

• architectural description language;
• codification of architectural expertise;
• frameworks for specific domains;
• formal underpinnings for architecture.

Estimation

Notes for Chapter 8 of Rapid Development by Steve McConnell.

1. Tell a story why estimation is difficult

1.a Software and contruction.
It is difficult to know whether u can build the product that the customer wants in the desired time frame until u have a detailed understanding of what he wants.

1.b Software development as a process of refinement
Until each feature is understood in detail, you can't estimate the cost of a program precisely. Software development is a process of making increasingly detailed decisions: product concept->statements of requirment->preliminary design->detailed design->working code. (the uncertainty principle)

At different stage the estimation error is different.

Phase Effort and Size Schedule
Initial Product Concept (0.25, 4.00) (0.60, 1.60)
ApprovedProduct Concept (0.50, 2.00) (0.80, 1.25)
Requirment specification (0.67, 1.50) (0.85, 1.15)
Product design specification (0.80, 1.25) (0.90, 1.10)
Detailed design specification (0.90, 1.10) (0.95, 1.05)

1.c Estimation vs. control

The features and resources have to meet each other, sometimes one bends to meet the other, sometimes both bend to meet in the middle. Like building a house, you can build to budget, or build your dream home.

1.d. Cooperation

Ask the customers the estimation information you need, and give them the update as always. Estimates can be refined over the course of a project. Promise customers that u will provide more refined estimates at each stage.

1.e. Convergence of the estimate and reality
Shortest actual schedule comes from the most acturate estimate, either too optimistic or pessimestic estimate ends up with more cost.

Note: accuracy and precision
40 man-months is accurate, 45 months is more procise, but may not be accurate at all.

2. Estimation process overview

2.a Estimate the size of the product (number of lines of code or function points);
2.b Estimate the effort (man-months)
2.c Estimate the schedule(calendar month)
2.d Provide estimates in ranges and periodically refine the ranges to increase precision as the project progresses.

3. Size estimation

3 ways: algorithm approach, software tools, use a similar project (nearest neighbor?)

Some tips:
1. Avoid off-the-cuff estimates. don't give any estimate when you are not familar with the task.
2. All time for the estimate and plan it.
3. Use data from previous projects.
4. Use developer-based estimates.
5. Estimate by walking-through. Ask team members to estimate pieces of the project individually, and have a walk-through meeting to discuss the each piece until reaching the consensus.
6. Estimate by categories.
7. Estimate at a low level of detail.
8. Don't omit common tasks, such as demo, data conversion, installation, customization, meetings, maintenance, defect corrections, QA, user documentation, reviewing, etc.
9. Use tools;
10. Use several different techniques, and compare them.

Presentation of the estimate
1. Plus/minus qualifiers: 6 +- 3month; (con: reviewer may only look at 6 months)
2. Ranges [3, 9] month
3. Risk quantification

Estimate 6 month +3 month, -2 months
+1 for late delivery of formating system;
+1 for new tools not working;
+0.5 for staff sickness;
+0.5 month for ...
-1 less delay on hiring effort;
-1 tools better than expected.

4. Cases: best case, worse case, planned case, and current case.
5. Coarse dates and time periods
6. Confidence factors
Apr 1: 5%
May 1: 50%
June 1: 95%

4. Effort estimate
Based on size estimation, do the effort estimate:
4.a. Use tools;
4.b. Look up table;
4.c. Historical data;
4.d. Algorithm approach.

5. Schedule estimation

Optimal Schedule in months = 3.0 * power(man-months, 1/3).
Optimal number of members = man-months/optimal scheduled months.

Commitment based scheduling

Pro:

• increase the developers' buy-in to schedules;
• High morale for the period immediately following the commitment;
• Elicit a lot of voluntary overtime from the developers;

Con:

• Developers usually 20~30% optimistic, commitment based scheduling encourages this optimism and create larger estimation errors;
• Commitment based planning is good only if the it is an accurate one, otherwise it may depress the members.

6. Ballpack schedule estimates

Background

3 types of projects:

• System software (not include firmware, real-time embedded software, process control etc)
• Business software
• Shrink-wrap software

Use the combination of those 3 to get the schedule.

I.e: a sw is composed of 60% system sw(5 month), 40% business sw (5 month), total = 0.6*5+5*0.4=5 months.

Schedules (calendar year)

Efforts (man-months)

System sizes

Small projects

The project less than 10K lines is typically assigned to one person, the schedule depends on the person's capability, who is the best to estimate the effort.

Line of code

Accuracy of the estimates

Assumptions to consider in the schedule

• staffing,
• management,
• tool support,
• methodology,
• compression

Two facts of life

• There is a shortest possible schedule, and you cannot beat it;
• Costs increase rapidly when you shorten the schedule blow the norminal.

2 schedules (require different assumptions and cost):

• Shortest schedule;
• Efficinet schedule;
• Norminal schedule

7 Estimate refinement

The accuracy of software estimate depends on the the leve of refinement of the software definition. U have to do the estimate refinement in the work of software project development.

Single point estimation contains tremendous imprecision.

Recalibration

If the milestone is missed, recalibrate the schedule. Options:

1. Make up the lost time;
2. Add the amount of time to the total schedule
3. Multiply the whole schedule by the magnitude factor of the slip.

Some researches show:

1. Projects hardly ever make up lost time.
2. It is rarely true that only the one slips is inaccurate, typically all estimates need to be calibrated.
3. Use option #3

Scheduling

Notes for chapter 9 of "Rapid Development" by Steve McConnell.

What's does scheduling have to do with Rapid Development? It has to do with creating an environment that isn't characterized by hasty, error-prone decision making and is conducive to effective planning, thoughtful design, and time-saving quality assurance.

Note: time planning is not to find a short-cut solution to the task, which always leads to a nasty outcome embrassing yourself, instead, it is to look for a effective way to the task with better quality.

1. Overly optimistic scheduling

This is a tradition in software development.

WinWord example: Bill Gates wanted it to be done in 12 months, it ended up with 5 years. The aggressive schedule came out with 4 project lead, 3 quitted because of schedule pressure, and 12 months to stabilize the product(normally 3 month).

Reasons for overly optimistic scheduling:

• External deadline;
• Manager or customers' refusal;
• Manager/developer underestimate the oost;
• Management/sales underestimate;
• New features piled up;
• Poorly estimation

Effects:

• Schedule accuracy;
• Quality of project planning;
• Adherence to plan;
• Underscoping the project;
• Project focus;
• Customer relations;
• Premature convergence;

Excessive schedule pressures on manager and developers.

• Motivation;
• Gambling;
• Creativity;
• Burnout;
• Turnover;
• Long-term rapid development;
• Relationship between developers and managers;

The bottom line: It just doesn't work.

2. Beating schedule pressure

Schedule pressure produces damaging. short-term thinking on 2 levels:

• Local level, short-cut-taking on specific projects and damage them;
• Global level, fire-fighting mentality about schedule pressure itself.

3 factors causing the pressure:

• Wishful thinking
• Little awareness of the software estimation tory or the real effect of overly optimistic scheduling
• Poor negotiating skill: developers are good at estimation, but poor at defending their estimate.

How to negotiate effectively? principled negotiation: face the brutal facts, and take win-win alternatives, keep the schedule, cost, and product in balance.

• Separate the People from the Problem: improve the relationship with ur manager/customer and be cooperative;
• Focus on interests, not positions;
• Appeal to true development speed: be firmly connected to reality and realistic schedule
• Appeal to increasing the chance of success;
• Invoke your organization's track record: not to make same mistakes again?!
• Invent options for mutual gain;
• Product: +/- features
• Resources: +staffing, tools, expense
• Schedule: coarse estimates, set goad instead of deadline, staged delivery.;
• Management: overtime pay, bonus, etc
• Insist on using objective criteria
• don't negotiate the estimate itself: talk about the input,.
• insist that the estimate be prepared by a qualified party.
• insist on a rational estimation process: nail down the details, don't provide unrealistic precision, reestimate after changes.
• weather the storm: endure the thunderstorm of a unwelcome estimate earlier, the better.