Summary. This entry takes up where my earlier entry on Needs Based (or Goal Free) Evaluation [at the general level] left off. In this one I've decided to conduct a 'mental experiment' by beginning to describe a needs based evaluation of our schools.

In order to procede with some intellectual order that we'll start with the following definition:

"A student is having educational needs met when:

• a) she/he is receiving appropriate instruction for her/his "zone of proximal development" (roughly synonymous with "readiness level" and first articulated by L. Vygotsky);
• b) this condition (a above) will be bet in each of 14 distinct areas of development (see below for a list);
• c) the conditions indicated in a and b above are also delivered at a rate that is determined by maximum comfortable rate of learning in each area of development for each person.*"

---

Then we get busy.

For starters: Conduct a needs based (goal free) assessment in one tenth of the school districts of one state. Get pre and post data over one calendar year. Use criterion and curriculum referenced testing in each of 14 areas of human endeavor. Add a sampling at the beginning of summer to enable an estimate ofthe amount and sorts of learning that occur outside of school.

In order to design school improvement efforts we must know what to improve. Thus we will have to profile the benefits (i.e., the learning) presently derived from schools. Such a profile will allow us to portray the rate, breadth and depth of school learning. This information cannot stop with the basics, i.e., 'reading, writing and arithmetic'. We are concerned that our developing citizens are capable of engaging with deeper issues, such as active citizenship, an understanding and care for the natural environment, etc. We are also concerned with development of individual potential wherever it is headed (as long as it is not hostile to or destructive of the general social order). In order to test our present delivery (via home and school interventions) of such a breadth of skills we'll have to test quite broadly. We will also --- in order to be sensitive to rate of learning as an indicator of potential-- we'll also have to be able to determine the degree to which the average rate of learning of such skills is close to optimal (i.e., a sufficient realization of what is possible with "state of the art" instruction).

Statewide Measures of Developmental Knowledge: Sixth Grade
Data Timing or Analysis-------------> Type of Skill	Test at Beginning of School Year (v1**)	Test at End of School Year (v2)	Test At Beginning of Next School Year (v3)	School Year Growth per Week (v4)	Summer Growth Per Week (v5)	Annual Growth Per Week(v6)	National Avg. at Beginning of Next School Year (v7)	National Avg. Growth Per Week (v8)
General Cultural Information
Receptive Communication (Listening, Signing, etc)
Expressive Communication (Speaking, Signing)
Written Expression
Numerical and Logical Understanding
Bodily Coordination
Eye-Hand Coordination
Social Interaction (Including Self Control)
Introspective Knowledge
Ethical/Moral Understanding and Behavior
Musical Appreciation and Expression
Visual and Multidimensional Appreciation and Expression
Mechanical/Scientific Appreciation and Expression
Appreciation and Interaction with Natural (Living and NonLiving) Systems

This would give us a basis for deciding what we would like to do given what we presently [have been proven to] accomplish with our schools.

---

* This implies that the instructional system will be providing instructional materials, lessons and tests at a pace that is dictated by individual rate of learning as opposed to a fixed schedule. Thus, at any given time, students will be working on different objectives, with different materials, for differing lengths of time. It should be obvious that, while I have stated needs to be, roughly,"to grow as much as possible in areas of activity that are valued by most human societies", this does not overlap with subject coverage in most schools.
** All variables are calculated averages deriving from statewide grade level sampling of actual instruction. Another variable, Potential Annual Growth per Week, could be created through use proven alternative curricula on a cross sectional subsample of students in representative regions of the state . *Edited 10/20/05

Summary: Must we always evaluate only because a funder or service provider mandates or requests the evaluation? For example, must we only evaluate a school as required by, say, the school board or the school's faculty or administration? Must the local hybrid gas-electric engine plant be evaluated only as corporate offices dictate? The answer to all is no! In this entry I work to unpack the ideas using thoughts from Michael Scriven, eminient philosopher and evaluator. As you probably can guess, I will fold in some thoughts and interests of my own.

---

Scriven's Description:

"... the evaluator is not told the purpose of the program but does the evaluation with the purpose of finding out what the program is actually doing without being cued as to what the program is trying to do. [....] Merit is determined by relating program effects to the relevant needs of the impacted population, rather than to the goals of the program (whether the goals of the agency, the citizenry, the legislature, or the manager) for the target (intended) population. It could equally well be called "needs-based evaluation" or "consumer-oriented evaluation" by contrast with goal-based (or "manager-oriented") evaluation. It does not substitute the evaluator's goals nor the goals of the consumer for the program's goals, contrary to a common criticism; the valuation must justify (via the needs assessment) all assignments of merit. The report should be completely transparent with respect to the evaluator's goals.

One of the major arguments for the pure form is that it is the only systematic or design procedure for improving the detection of side-effects. Evaluators who do not know what the program is supposed to be doing loom more thoroughly for what it isdoing. Other arguments for it include:

• (i)it avoids the often expensive, always speculative, and time-consuming problems involved in determining true current goals and true original goals, reconciling and weighting them;
• (ii) it is less intrusive into program activities than GBE [Goal Based Evaluation];
• (iii) it is fully adaptable to midstream goal or need shifts;
• (iv) it is less prone to social, perceptual, and cognitive bias because of reduced interaction with program staff; and
• (v) it is 'reversible', that is, one can begin an evaluation goal-free and switch to goal-based after a preliminary investigation thereby garnering the preceding benefits (whereas if you begin goal-based, you can't reverse);
• (vi) it is less subject to bias arising from the desire to please the client because it's less clear what the client was trying to do.

[...]

GFE is somewhat analogous to double-blind design in medical research; even if the evaluator would like to give a favorable report (e.g., because of being paid by the program, or hoping for future work from them) it is not (generally) easy to tell how to 'cheat' under GFE conditions. The fact that the risk of failure by the evaluator is greater in GFE is desirable since it increases effort, identifies incompetence, and improves the balance of power.

Doing GFE is a notably different and enlightening experience from doing the usaul kind of evaluation. there is a very strong sense of social isolation, and one comes to be extremely conscious of the extent to which GBE evaluations are not reallly 'independent evaluations' even when they are called that; they are cooperative efforts, and hence easily co-opted efforts. Ones is also conscious of the possibility of enormous blunders. It is good practice to use a metaevaluator and very desirable to use a team.

[GFE is not a method in the same sense of other evaluation methods ... in that it can be combined with any one of them, except a goal-based evaluation, and that only for a part of the investigation. i.e., start multimethod goal free and after having reaped all desired benefits switch to goal based and start working more closely with program personnel.] [Evaluation Thesaurus,4th edition, 1991, Sage p 180-182]

---

It is clear from what MS has argued that he believes that one can evaluate without being program driven. He has made clear that the GFE does revolve around some form of needs analysis which puts limits on the breadth and depth of inquiry. Thus there is some agenda, with argument and political subscription of some degree, which will serve as a driving set of values from which the needs analysis will be derived. You have to start somewhere however, and, if you are doing other than putzing around, you better have some large group of stakeholders invested in the needs assessment that you use in your GFE in order for it to be deemed worthwhile.

Let's say, for the sake of argument, that we have built our needs analysis [an analysis of some portion of human existence which leads to a description of needs which are entailed and a working definition of what "meeting" those needs amounts to]. Our GFE would then concern itself with determining the causal effects that the chosen program or programs, has had upon the client population.

If, say, one were inquiring about safety from various forms of fire, there would be a set of needs that, if addressed effectively, would result in a minimal average risk of injury or loss due to fire. It seems clear that since absolute safety (no fires, no damage, no form of injury, ever) is impossible, some statistical goals will serve as standards for excellence, satisfactory and unsatisfactory service. All of this having been said, the fire system of, say, Dogpatch, could be directly observed and statistically weighed against those statistical standards without any consideration of what programs are being offered by the Fire Department as a whole, by the separate fire houses or by the individual firemen. This, then, would be GFE of fire protection in Dogpatch.

Could we also do a Needs Based Evaluation of education in Dogpatch or its distant, metropolitan neighbor, Erehwon? Yes! I believe we could.

The basic outline would be the same: List the needs, educational needs in this case, locate and translate standards for excellent, satisfactory, unsatisfactory and dangerously low levels of meeting of those standards and sample, measure and estimate until a confident estimate of true standard adherence has been created. However, it would be more difficult with education.

First, education's mission differs, at least at the outer boundaries, from town to town. Further, even given a common general agreement as to mission, to develop each child to her/his maximal individual potential, for example, the difficulty will derive from great number of possible practical translations. Given this considerable ambiguity what I will next say can only be taken as illustrative. Whatever the standards and however chosen they must be a potent, useful, assessable, yet at the time acceptable to the stakeholding recipients of the GFE Education report.

Let's say that criterion referenced developmental sequences of objectives in each of Howard Gardner's distinct developmental areas are chosen as basis for testing. Further, we've found an authority who's already developed and applied those sequences in K-15 in multiple regions of country. Part of that translation has involved the development of criterion-referenced, group administratable, tests which reliably place each person within their "zone of proximal development" (ala Vygotsky). Finally the same tests have been reliably used to assess learning rate (objectives mastered) for each individual in each area of development, given appropriate instruction at proximal level of development.

Given this much we will be able to proceed to some fruitful needs-related assessment.

[Go to my next entry for further details.]