Negotiation for Automated Generation of Temporal Multimedia Presentations

"Negotiation for Automated Generation of Temporal Multimedia Presentations," in Proceedings of ACM Multimedia '96 55 (1996) (with Mukesh Dalal, Steven Feiner, Kathleen McKeown et al.)

"Negotiation for Automated Generation of Temporal Multimedia Presentations," in *Proceedings of ACM Multimedia '96* 55 (1996) (with Mukesh Dalal, Steven Feiner, Kathleen McKeown et al.)

Mukesh Dalal et al., Negotiation for Automated Generation of Temporal Multimedia Presentations, in Proceedings of ACM Multimedia '96 55 (1996)

Mukesh Dalal
Steven Feiner
Kathleen McKeown
Shimei Pan
Michelle Zhou
Tobias Höllerer
James Shaw
Yong Feng
Jeanne C. Fromer

Multimedia presentations can provide an extremely effective way to communicate information. However, designing highquality multimedia presentations by hand is a difficult and time-consuming task, even for skilled authors. This is especially true when the presentation involves temporal media, such as speech and animation, because the order and duration of actions that occur in different temporal media must be coordinated so that the presentation's goals are communicated coherently. To address the multimedia authoring problem, we are developing a testbed system that uses AI techniques to automatically design temporal multimedia presentations that are customized to a specific situation. Our focus in this paper is on the problem of coordinating how and when information is conveyed by different temporal media to create a coherent and effective presentation. Our approach is embodied in a testbed system, MAGIC (Multimedia Abstract Generation for Intensive Care) [6], which automatically generates multimedia briefings that describe the postoperative status of a patient undergoing Coronary Artery Bypass Graft (CABG) surgery. MAGIC uses the computerized information infrastructure already present in operating rooms at Columbia Presbyterian Medical Center, which provides a detailed online record of the patient's status before, during, and on completion of surgery. Among the available data are vital signs, administered drugs, intravenous lines, information about devices such as a pacemaker or balloon pump, echocardiogram data, and severity assessments. Our system is designed to tailor briefings for different caregivers, including Intensive Care Unit (ICU) nurses, cardiologists, and residents, all of whom need updates on patient status in the critical hour following surgery. MAGIC is a distributed system whose components use knowledge-based techniques for planning and generating briefings in written text, speech, and graphics. One of its main contributions is a negotiation process for coordinating the order and duration of actions across different media. Order and duration information are both represented using temporal constraints that are generated dynamically by individual media-specific components. To avoid the expensive replanning that is required when the different components' constraints are mutually inconsistent, each media-specific component in MAGIC produces a prioritized list of partial orderings of its actions. A coordination component negotiates with the media-specific components to arrive at a global partial (or even total) ordering of actions that extends some high-priority partial ordering in each medium. Compatibility among orderings is ensured by explicitly representing the relations between conceptual objects and the media actions that refer to them. Durations are coordinated only after obtaining a compatible ordering of actions. As described below, our approach is different from most earlier work on coordinating multiple media, in which temporal relations are either defined by hand or exist a priori (e.g., as stored video with associated audio). Because MAGIC automatically generates the content and form of all media, tem-In: Proceedings ACM Multimedia, Boston, MA, Nov. 1996, pp. 55-64 poral relations between media objects are determined only at run-time. After a discussion of previous work in multimedia coordination, we provide an overview of MAGIC's architecture and introduce a set of examples that will be used to describe the generation and coordination process. We then describe the language-generation and graphics-generation components, and explain how coordination is achieved through negotiation between the generation components and a coordination component. Related work in multimedia systems falls into three broad categories: low level synchronization of stored multimedia, flexible synchronization using hand-specified temporal constraints, and dynamic generation of multimedia. In their unified overview of low-level multimedia synchronization, Steinmetz and Nahrstedt [24] present a four-layer synchronization reference model, an overview of multimedia synchronization approaches, and an account of multimedia presentation requirements; they focus on multimedia synchronization issues for stored multimedia systems. Research on network and operating system synchronization issues, such as feedback techniques and protocols for intermedia synchronization given network jitter (e.g., [18,19,20]), also falls within this category. In MAGIC, conceptual objects are generated dynamically, so object duration information and inter-object synchronization points can rarely be stored or computed prior to the generation of a presentation plan. Although research in the development of multimedia authoring tools also addresses the problem of automatic synchronization, coordination constraints are explicitly stated by the presentation designer and scheduled at the media object level. The media objects could be audio, video segments, or graphics animations. For example, in [12], each media object is associated with a triple: maximum, minimum, and optimum length. The system is able to provide an optimal cost solution that can satisfy all the temporal constraints with fairness in distributing necessary stretching or shrinking across the media objects. Others (e.g., [4], [13]) incorporate unpredictable temporal behaviors in their temporal relation models. Their algorithms can adjust to compensate for the behavior of indeterministic objects at run-time. In contrast, most of MAGIC's temporal coordination and synchronization constraints are dynamically generated, and they are specified at a more detailed level of representation. For example, temporal constraints are specified among words and phrases in speech and among displaying and highlighting in graphics. Media synchronization is controlled by the system at run-time with much finer granularity. Although [12,13] allow both qualitative and quantitative temporal constraints, they do not allow disjunctions among those constraints. While the constraint solver in [12], which is based on linear programming, is exponential in the worst case, the solver in [13], which allows flexible quantitative constraints, sometimes provides "false inconsistent" results. In contrast, MAGIC allows disjunctive qualitative constraints using a language that extends Allen's interval algebra [1] by allowing more than two intervals in the same disjunct. MAGIC also allows simple quantitative constraints and uses an efficient but incomplete constraint solver. Dynamic multimedia generation systems include SAGE [21,14], COMET [9], and WIP [3], which are knowledge-based multimedia generation systems that coordinate written text with static graphics. Weitzman and Wittenburg [27] also handle media coordination in their work on generating multimedia presentations. They construct a simple network of spatial and temporal constraints for each grammar to accommodate dynamic relationships among presentation elements, but do not support the replanning capabilities we provide in MAGIC. Temporal media introduce additional complexity, and none of these systems support them. CUBRICON [15] is one of the earliest systems to dynamically generate coordinated speech and graphics. It combines speech with simple deictic gestures and 2D map graphics to guide the user's visual focus of attention. Both speech and graphics are incorporated into a single unified language generator, so there is no need for formal communication between the generation components. More recently, a system has been developed at University of Pennsylvania that automatically generates conversations between two human-like agents through synchronized speech, facial expressions, and gestures [5,16,17]. This system synchronizes 3D graphics with speech by using the timing information generated by the speech generator to decide when and how facial expressions and gestures are produced. For example, if insufficient time is available to generate a gesture, the system may abort the gesture. Since speech controls the actions of the other generators, no negotiation among the generators is required. In contrast, the language and graphics components in MAGIC have equal status, so some mechanism is required to negotiate the order and duration of actions in the presentation. As shown in Figure 1, MAGIC consists of a set of components that work concurrently and cooperatively. MAGIC's source of information about a patient's condition is data derived from several online medical databases. Our current data server prototype uses a static knowledge source that contains information previously downloaded from these databases and "sanitized" to allow publication. The data filter component selects relevant parts of this data and infers new information from it. Using the domain-independent concept hierarchy and the domain-dependent domain hierarchy, the data filter creates a knowledge representation of the particular patient, which we call the instance hierarchy. These hierarchies are all accessible to the other components in MAGIC. The concept hierarchy is a general domain-independent ontology providing a taxonomy for generic information, such as the fact that reals and integers are both numbers (similar to those found in [2,22]). The domain hierarchy represents domaindependent medical information using the same classifications used in the clinical information system at Columbia Presbyterian Medical Center. Based on the user model (e.g., of nurses or cardiologists), plan library, and the instance hierarchy, the general content planner creates a media-independent plan that expresses the high-level communicative goals to be accomplished by the multimedia briefing. This presentation plan is represented as a directed acyclic graph (the presentation graph) and serves as the main data structure for exchanging information among MAGIC's components. The media allocator specifies one or more media to express each communicative goal by annotating whether it should appear in one or more media. (Our current implementation uses a simple algorithm based on semantic properties alone.) The full set of annotated communicative goals is then handed over to the media-specific content planners and generators. A single system-wide media coordinator ensures that the media-specific content planners and generators all work out a consistent and synchronized presentation, by allowing media-specific components to access and update the overall presentation plan. Though there are many interesting issues related to the general content planner, the media allocator, and the media-specific content planners, we will not discuss these components in further detail in order to focus on the theme of the paper, temporal coordination. Several temporal and spatial constraints must be satisfied in the presentation plan. For example, certain tabular information may have to be displayed graphically in a specific spatial order. The media coordinator uses a constraint solver component to detect inconsistencies among the different components' plans. The media coordinator handles any inconsistencies by negotiating alternatives with the media-specific content planners. Currently only temporal constraints are explicitly represented, by using qualitative relations based on Allen's interval algebra [1] and simple quantitative information. The media-specific content planners and generators engage in fine-grained collaboration to develop a detailed plan for each communicative goal. When the presentation for a goal is agreed upon, it is ready for display. The media conductor gives the media-specific generators a ready signal and that part of the presentation is played for the user. We will rely on two examples, shown in Figures 2 and3, to illustrate MAGIC's multimedia coordination process. In each example, the goal is to use speech and graphics to communicate information about a patient's demographics, with each spoken reference to visual material synchronized with graphical highlighting. The figures show actual tabular text, speech, and graphics created by MAGIC for the beginnings of two presentations for two different patients. Each figure contains a series of images with the associated speech that is spoken while each is displayed. Two kinds of highlighting are shown: a yellow block that appears around the demographics information to emphasize it, and temporary use of red instead of grey text. (In the black-and-white halftone reproductions included here, the yellow block is rendered as light grey and the highlighted red text is rendered as black.) The graphics are organized in a standardized spatial layout. Although different data is displayed in both examples, consistent spatial layout of similar information is desirable, since it enables a user to easily find information across multiple patients. Since in each example all the graphics shown are presented before speech starts, synchronized highlighting is used to emphasize parts of the display as they are mentioned in speech. The decision to use a stylized representation of the patient's body with surrounding information is based on both the information characteristics (e.g., a patient is a physical entity) and the situation/user model (e.g., of the ICU nurses, for whom this particular presentation layout is designed, prefer to see this information arranged relative to the body, but do not need a detailed body model) [31]. Although additional graphics and speech are presented in later parts of both patients' briefings, they are not shown or discussed here. In this paper, we concentrate only on the generation and coordination of speech Each medium has its own criteria that must be satisfied to produce an ordering of the objects that it presents. The speech components give preference to more concise presentations (e.g., using fewer words). The graphics components give preference to highlighting in a regular, spatially ordered sequence, without jumping around, in a style that highlights in units of an entire column of attribute-value pairs. If speech and graphics were to be planned independently for these examples, speech would generate exactly the sentences shown, and graphics would use a highlighting sequence that preceded from left to right, one column at a time. Although these presentations may be individually satisfactory, when presented together, spoken references and highlighting would differ in order and duration. Thus, to achieve coordination, it is necessary for each medium to constrain the other. In Figure 2, the ordering of information in speech forces graphics to abandon its preferred ordering and use an alternative instead: the leftmost two attribute-pair columns are highlighted as a block as the words "Ms. Jones is an eightyyear-old, diabetic, hypertensive, female patient" are spoken. The right column is then highlighted as the remainder of the sentence is spoken. No other regular (e.g., left-to-right) highlighting sequence would be coordinated with speech, since the reference to medical history (middle column) is embedded between references to age and gender (left column). The same left-to-right highlighting would produce an uncoordinated presentation in Figure 3, since the surgeon's name is spoken before the medical history. Instead, graphics must use a less desirable sequence, first highlighting the left column, then the right, and finally, the middle column. (Highlighting the middle and right columns simultaneously is ruled out because the first reference in them is to the right column.) Alternatively, as shown in Figure 4, if the media coordinator requests speech to use a different ordering, speech can find an alternative phrasing that also meets its own constraints in this case. Given this wording, graphics can use its preferred left-to-right ordering, one column at a time. While in Figure 3, speech constrains graphics choices, in Figure 4, graphics constrains the ordering used by speech, following negotiation. Our work to date gives MAGIC the capacity to generate either of the sequences shown in Figures 3 and4. (Through user studies we will determine which sequence is more desirable for which users in which circumstances. This will ultimately allow MAGIC to select the appropriate option for the specific situation.) The temporal constraints used for coordinating speech and graphics are represented using relations in Allen's interval algebra [1]. For example, the qualitative constraint (< name (* age gender)) among speech objects indicates that the reference to name should be spoken before (<) the references to age and gender, which in turn may be spoken in any order (*). In general, any subset of the thirteen basic relations defined by Allen can be used in place of and *. (We use * as an abbreviation for the disjunction consisting of all thirteen basic relations.) We also allow quantitative temporal constraints that specify starting and stopping times of individual intervals in seconds, relative to the start of the entire presentation; for example: Speech is an inherently temporal medium: words and phrases occur in strict temporal sequence, one after the other. The ordering of words and phrases is constrained both by grammatical properties of the language and by communicative goals. The speech components must perform two main tasks to accomplish coordination. First, they must determine possible orderings of spoken references to objects in the accompanying graphics that meet grammatical and communicative constraints. Second, they must determine the duration of each such spoken reference. These tasks are complicated by the fact that the complete ordering of words is not usually known until all grammatical constraints have been applied and one or more sentences produced. Clearly, negotiating a compatible ordering with graphics at this late stage is undesirable because MAGIC would have to backtrack through the entire process of producing sentences should an incompatibility be detected. Furthermore, given just a string of words, the correspondence is missing between phrases and the objects they reference. To address these problems, we exploit paraphrasing and determination of partial orderings midway through the generation process, by representing and reasoning about possible choices for speech. In this process, MAGIC retains a representation of objects referenced for each phrase, computing the duration of each phrase. The speech components are capable of realizing the same content in different ways, each of which encodes a different ordering of references. This makes it possible for the media coordinator to select a sentence that matches the ordering requirements from the other media. Input to the speech components includes objects to be communicated in speech, such as a patient's name, age, gender, medical history, operation and surgeon's name in the patient's demographics section, each of which is represented by a unique identifier. Based on this input, the speech components design several ways to communicate this information. For example, the information given by speech in Figure 2(bc) can be conveyed in a single sentence or several sentences: 1. Ms. Jones is an eighty-year-old, diabetic, hypertensive, female patient of Dr. Smith undergoing CABG. 2. Ms. Jones is an eighty-year-old female patient of Dr. Smith undergoing CABG. She has a history of diabetes and hypertension. 3. Ms. Jones is an eighty-year-old female. She has a history of diabetes and hypertension. She is a patient of Dr. Smith undergoing CABG. Additionally, within one sentence, different sentence structures can be produced by changing the word order or paraphrasing. For example, by reordering the words "diabetic" and "hypertensive" in sentence 1 above, we have: 1a. Ms. Jones is an eighty-year-old, hypertensive, diabetic, female patient of Dr. Smith undergoing CABG. For each sentence shown above, a preference value is produced to indicate how preferable the sentence is from the point of view of the speech components. The speech components are provided with the communicative goal of being concise. Therefore, they prefer one sentence over several, adjectives over prepositional phrases, and prepositional phrases over relative clauses. Since several candidate orders are available to the media coordinator at the same time, this can significantly reduce the number of re-negotiations that are needed when coordination fails. The object ordering needed for negotiation is produced after planning the sentence and before generating its surface structure. The speech components consist of a speech content planner and a speech generator. The speech content planner determines how much information goes into a sentence. The speech generator includes a lexical chooser that determines overall sentence structure and the words to use, and a sentence generator that uses a grammar to enforce grammatical constraints. The lexical chooser uses unification to generate all possible sentence structures and word choices. The resulting object orders are expressed as partial-order constraints, where the partial orders are used to represent variations that may happen in the final stage of generating the sentence. Thus, each object order may correspond to several sentences that can be generated. For example, when the speech components do not care about the difference in orderings between them, as in sentences 1 and 1a above, they specify the order between "diabetic" and "hypertensive" as (* diabetes hypertension), where * indicates that the order of the following objects does not matter. Since ordering is determined at the point of lexical choice, the speech components have available both the object and the words selected to refer to the object. The speech components keep track of the mapping between the words or phrases in a sentence and their object identifiers. For example, the speech components maintain a representation indicating that "eighty-year-old" represents age in the speech input. The candidate object orders, 1 which correspond to example sentences 1-3 above, produced for the demographics information in the speech output for Figure 2 are: Note that the speech content planner and lexical chooser determine that diabetes and hypertension can be referred to using adjectives and thus folded into one sentence in a concise way. Thus, Option 1 is much more preferable than either Option 2 or 3, both of which involve separate sentences and consequently, a number of additional words. Option 1 is given the highest possible weight (10 on a scale of 1-10, shown in square brackets) and Options 2 and 3 are given lower weights, where Option 2 (weight 5) is marginally better than Option 3 (weight 4). In contrast, for Figure 3, the medical history ("transient ischemic attacks, pulmonary hypertension, and peptic ulcer") cannot be realized in adjectival form; therefore, a separate sentence for the medical history must be generated, which, using the most natural wording, follows references to all information in the left and right columns. Thus, only the last two partial orders above would be generated for Figure 3 (with Smith's medical history items replacing diabetes and 1 Although shown here as orderings over objects, speech actually produces orderings over spoken references to objects. For example, instead of name, the constraint is over (refer name). This was done to simplify the presentation here. hypertension). Again, Option 2 is ranked marginally better than Option 3. After a compatible total order has been found by the media coordinator, the speech generator generates the surface structure of the set of one or more sentences that meet those constraints. The speech generator then computes duration information for each of the objects referenced in the sentences. The duration, start time, and stop time of each phoneme in the sentences are generated by the speech synthesizer. From this information, the speech generator derives the start and stop time of each word and finally of each object in the given input. As a result, each object identifier referred to in speech is tagged with time information. Note that because durations are provided at the word and phrase level, highlighting can be synchronized to occur for the exact duration of the spoken reference as opposed to the entire sentence, thus yielding a fine level of coordination. The time information produced by the speech generator for Figure 2, and used by the media coordinator, is: ((name 0.5) (age 1.46) (diabetes 2.41) (hypertension 3.17) (gender 4.15) (surgeon 5.25) (operation 6.29) (end-time 7.66)) MAGIC's graphics content planner and graphics generator handle both time-independent and time-dependent graphics. An example of the former is a static tabular data layout, while examples of the latter include interactive highlighting and more general animation. There is a very high degree of flexibility in how to present several pieces of information graphically in this fashion. Given as input a set of general communicative goals, the challenge is to compute efficiently a course of graphical actions that best fulfills these goals and that at the same time can be smoothly coordinated with the actions performed in other media. To this end the graphics content planner uses a hierarchical-decomposition partial-order planning component [30] that selects visual goals to be accomplished. We have adopted the visual goal categorization developed by Wehrend and Lewis [26]. The planning component computes a partially ordered (and, therefore, flexible) set of graphical actions. While the speech components order spoken references to objects to obtain possible object orders, the graphics components partially order graphical actions that imply possible object orders. Note that the spatial arrangement (spatial ordering) of graphical items on the screen is implicitly determined by the graphical actions. The graphics content planner's input has the same structure as that of the language content planner. For example, when handling the patient's demographics information, the presentation plan advises the graphics content planner to convey information about the patient's age, gender, medical record number (MRN), medical history, operation, and surgeon's name. (Based on domain knowledge provided by our medical experts, the media allocator determines that the MRN is to be conveyed only by graphics.) Given the general communicative goal to "emphasize a patient's demographics," the graphics content planner proposes lower-level visual goals. The graphics generator has a set of visual operators that are based on standard graphic design techniques, such as highlighting [31]. The graphics components utilize a set of visual policies to determine what visual operators to use and how to use them to achieve visual goals. In our examples, demographics are displayed in a textual table and a visual policy states that highlighting is appropriate for distinguishing textual table objects. Therefore the highlight operator can be selected to accomplish the visual task distinguish demographics. Furthermore, the graphics generator uses the action subhighlight to further distinguish subparts of the demographics. Thus, a set of actions are proposed to accomplish the current task: Action1: (highlight (demographics)) The graphics generator communicates with the media coordinator to negotiate the ordering of these actions. A compatible order of the objects involved is produced, utilizing their individual unique object IDs. What is actually communicated to the media coordinator is a set of partially ordered graphical actions. For example, the set of partial orders specified for the actions listed above are: Here, di specifies the relationship contains (also known as during-inverse), indicating that Action1 starts before and ends after all the other actions. The relation m, called meet, indicates that the stop time of Action2 is the same as the start time of Action3, and that the stop time of Action3 is the same as the start time of Action4. The list of operators < and m specifies a disjunction. Thus, each action in the list of actions ends either before or at the same time that the next action in the list starts. The first, highly-weighted, partial order highlights information from left to right, while the second indicates equal preference for all other orders. For Figure 2, after receiving negotiation requests from the media generators, the media coordinator returns to the graphics components a compatible order specified in terms of the objects; in this case: (di demographics ((< m) mrn age medhistory gender surgeon operation)) A complete order of graphical actions is constructed by adapting this compatible object order. If the object order does not agree with the current structure of graphical actions, then the graphics generator needs to fix the current plan. For example, in the order shown above, medhistory comes between age and gender, effectively breaking the structure of Action2. In this case, the graphics generator can merge Action2 and Action3 together to resolve the conflicts. Thus, a new action is generated to replace Action2 and Action3: Action5: (subhighlight (mrn age gender medhistory)) For Figure 3, the compatible order returned by the media coordinator already agrees with the graphical action structure rated 7 above: (di demographics ((< m) mrn age gender surgeon operation medhistory)) When a satisfactory order has been found, the execution time for each graphical action is estimated within the graphics generator and graphical time constraints are sent to the media coordinator. The time constraints for each graphical action are represented as a time interval specified by its start time and stop time in seconds. Currently, the speech component dictates the time constraints for graphical actions. The graphics generator gets the time constraints from the media coordinator, and then assigns these constraints to each graphical action. Once a total order of the graphical actions with their time duration constraints has been constructed, the graphics generator sends a ready signal to the media coordinator. Upon the return of a ready signal from the media coordinator, all graphical actions are sent to the rendering component to be realized with the specified timings. The media coordinator currently coordinates both the order and duration of graphical and speech objects. This is done in two separate negotiation phases: a total ordering of media objects is determined first, and then the object durations are synchronized. Computing durations only after agreeing on a total ordering improves the efficiency of the system because durations (and therefore complete generation) of alternative orderings are not computed at all. Since MAGIC uses automatic generation, when there are incompatibilities in duration, additional material can often be generated to fill the gap. An important task of the media coordinator is to relate the media objects generated by different media components. For example, the speech generator provides the media coordinator with temporal constraints over spoken references such as "Ms. Jones" and "eighty-year-old." These constraints are represented using actions that specify the underlying conceptual objects, in this case (refer name) and (refer age), respectively. Constraints from the graphics generator refer to actions such as (subhighlight medhistory). Note that this becomes complicated because of the hierarchy of objects; for example, in Jones's case, medhistory refers to diabetes and hypertension. The media coordinator correlates the actions provided by speech with the actions provided by graphics, while using subsumption in the hierarchy of conceptual objects. Because multimedia objects are organized hierarchically and generated automatically, it is important that the constraints provided by the speech generator and the graphics generator be flexible. Regularly backtracking to the planning level because of overly restrictive temporal constraints would be costly in a real-time environment because new object hierarchies may have to be generated. Thus, both the speech and graphics generators provide a list of partial temporal constraints; since they are partial, the temporal constraints can be relaxed if needed. This approach greatly facilitates negotiation between the graphics generator and the speech generator. For coordinating order, each media generator provides a weighted list of possible partial orders of media actions, ranked according to the generator's preferences. These constraints are expressed in an interval-based model [1] for representing qualitative constraints among temporal intervals. The media coordinator determines a total ordering compatible with the highest-ranked ordering of each medium. If this fails, it negotiates among the speech and graphics preferences until it finds a compatible ordering by systematically traversing the lists. When determined, the compatible ordering is then passed back to the speech generator and the graphics generator. For Jones's demographics (Figure 2), the highest-ranked constraints for graphics and speech are not compatible. The media coordinator then considers the next ranked graphics constraints, since they have a much higher weight than the next ranked speech constraints. The compatible ordering produced is: (di demographics ((< m) name mrn age diabetes hypertension gender surgeon operation)) For Smith's demographics (Figures 3 and4), recall that the speech constraint corresponding to that ranked highest for Jones is not generated. While Smith's highest-ranked speech and graphics constraints are still not compatible, the differences between the weights of the top two speech constraints and the top two graphics constraints are similar. Therefore, there is not necessarily a clear choice between orders that use the highest-ranked speech constraint and ones that use the highest-ranked graphics constraint. Figure 3 was generated by using the highest-ranked speech constraint (corresponding to Jones's second-ranked speech constraint) and the second-ranked graphics constraint. In contrast, Figure 4 was generated by using the second-ranked speech constraint (corresponding to Jones's third-ranked speech constraint) and the highest-ranked graphics constraint. After a global total ordering has been agreed upon, each individual media generator computes the duration constraints of its conceptual objects. The media generators provide these duration constraints to the media coordinator, which again uses the constraint solver to determine compatible global start and stop times for each media object. Since graphics does not currently provide any duration constraints, the durations of all objects are computed directly from the speech duration constraints. In particular, the duration constraints generated by speech are used to compute the start and stop times for each of the graphics generator's composite objects. The start time for a composite object c of the graphics generator is computed to be the start time of the first object in the speech generator's object duration information that overlaps with c. The stop time is similarly computed to be the end time of the last object in the speech generator's object duration information that overlaps with c. In When the media generators are ready to begin the presentation, they each request a ready signal from the media conductor, which waits for both the requests in order to synchronize their start times. MAGIC's modules are written in Common Lisp, C, C++, and CLIPS. The entire system is integrated using the ILU (Inter-Language Unification) package [10]. ILU makes it possible for the different components to share data structures and to communicate efficiently across different hardware platforms and implementation languages. The hierarchies and presentation graph described in the section on system architecture are implemented as two multithreaded ILU server modules, one for the three hierarchies and one for the presentation graph. The modules along the main pipeline of MAGIC act as ILU servers as well as ILU clients, so they offer functionality to other modules and also call functions provided by other modules. The media coordinator currently uses a constraint solver based on Metric/Allen Time System (MATS) [11] for solving the temporal constraints and determining a total global ordering. The constraint solver can handle both qualitative constraints for expressing ordering and quantitative constraints for expressing durations. It works by computing the transitive closure over qualitative constraints and then using constraint satisfaction for point-based metric reasoning [29]. The media coordinator, which is implemented in C, runs on a Sun Ultra-Sparc workstation. Currently the general content planner and the media allocator are implemented as one module, using Allegro Common Lisp. The speech content planner and generator are implemented in Allego Common Lisp and the FUF/Surge package [8]. FUF (Functional Unification Formalism) is a unificationbased natural language generator program, consisting of a unifier and a linearizer. Surge is a large, robust unification grammar of English. Both were developed at Columbia University and have been used in a wide range of applications at Columbia and elsewhere. The speech synthesizer is Lucent Bell Laboratories' Text to Speech system. The speech content planner and generator run on a Sun UltraSparc workstation. The graphics content planner and generator are written in C++ and the CLIPS production system language [23]. We have implemented the knowledge-based design component in CLIPS, and the rendering component using the SGI Open Inventor 3D graphics toolkit [25]. We use a hierarchical decompositional partial order planner [32] that is based in part on the research of [28,30]. The graphics content planner and generator run on a 250 MHZ R4400 SGI Indigo Maximum Impact. We have described our first steps in the automated generation of multimedia presentations in which multiple media-specific generators negotiate on an equal basis to determine the order and duration of the material presented. This allows MAGIC to automatically determine which media objects should be synchronized and their position, in addition to standard synchronization. This is achieved in an efficient manner through the development of media-specific content planners and generators that can reason about possible orders for presentation and a media coordinator that negotiates between their statements of preferred orderings using temporal reasoning to determine compatibility. Although the media supported by MAGIC currently include only generated speech, text, and graphics, its negotiation approach can accommodate additional media. In particular, we are working on supporting static images and video to allow the inclusion of material such as x-rays and echocardiograms. We are also interested in making it possible for users to interrupt MAGIC's presentations; for example, to change the topic being presented or how it is presented. Since our negotiation approach does not rely on the specifics of the medical domain, it should extend to other domains as well. Our constraint solver is currently based on MATS, which is efficient but not complete. That is, there are certain temporal constraints that cannot be expressed (e.g., disjunctions of conjunctions of temporal relations). We are exploring using a new anytime algorithm for reasoning with temporal temporal constraint networks that is based on propositional satisfiability [7]. In addition to our focus thus far on representing and reasoning about temporal constraints, we will also explore using similar approaches to handle spatial constraints on multimedia objects. This will allow MAGIC to reason about the consistency of spatial constraints, the spatial lay-out of displayed objects, and the interactions among these objects. Currently, the media coordinator and the media conductor return their results to the speech generator and the graphics generator via ILU. We plan on extending the presentation graph to serve as declarative blackboard representation through which temporal ordering and duration results would be posted. Ultimately, media-specific content planners and generators will also post their constraints, preferences and decisions in the presentation graph so that we have a single representation where different components can inspect the decisions of others. Finally, we are working on a more sophisticated version of the media conductor, which can handle dynamic synchronization between media. Our work on the presentation graph will help in this task. For example, once duration constraints are represented declaratively in the graph, MAGIC can both interrupt and replay presentations simply by reading the graph. We will also extend negotiation to synchronize durations, if no compatible timings can be determined initially. Furthermore, we will investigate the incorporation of established techniques to alleviate lags that arise from network delays [24].

2. Ms. Jones is an eighty-year-old female patient of Dr. Smith undergoing CABG. She has a history of diabetes and hypertension.

3. Ms. Jones is an eighty-year-old female. She has a history of diabetes and hypertension. She is a patient of Dr. Smith undergoing CABG.

The time information produced by the speech generator for Figure 2, and used by the media coordinator, is:

((name 0.5) (age 1.46) (diabetes 2.41) (hypertension 3.17) (gender 4.15) (surgeon 5.25) (operation 6.29) (end-time 7.66))

Action4: (subhighlight (surgeon operation))

(di demographics ((< m) mrn age medhistory gender surgeon operation))

In this case, the graphics generator can merge Action2 and Action3 together to resolve the conflicts. Thus, a new action is generated to replace Action2 and Action3:

Action5: (subhighlight (mrn age gender medhistory))

For Figure 3, the compatible order returned by the media coordinator already agrees with the graphical action structure rated 7 above:

(di demographics ((< m) mrn age gender surgeon operation medhistory))

(di demographics ((< m) name mrn age diabetes hypertension gender surgeon operation))


" Net Neutrality Is Great, but It Won’t Make Broadband Cheaper ," New Yorker , June 21, 2016

"Net Neutrality Is Great, but It Won’t Make Broadband Cheaper," *New Yorker*, June 21, 2016

Christopher Jon Sprigman, Net Neutrality Is Great, but It Won’t Make Broadband Cheaper, New Yorker (June 21, 2016), https://www.newyorker.com/business/currency/net-neutrality-is-great-but-it-wont-make-broadband-cheaper

Say you’ve just rented an apartment in Manhattan. One of the first things you’ll likely want to do, after turning on the gas and electricity, is shop for broadband-Internet access. Depending on the location of your apartment, you’ll have two choices, Time Warner Cable and Verizon’s FiOS, or just one, Time Warner. Neither service will be cheap, both charging about sixty dollars a month for moderately fast (but not blazing fast) broadband service. Had you, instead, moved to London, you might have been choosing between as many as four big providers, BT, Sky, Virgin, and TalkTalk, plus a number of smaller players with names like Demon and Zen Internet, and even the postal service. The competition among those providers means you could be paying less than half what you’d pay in New York for comparable service. That more competition leads to lower prices is basic economics, but the complacency in Washington about this monopolistic (or, at best, duopolistic) situation is long-standing and, until last week, seemed unlikely to shift. But then, on June 14th, the federal appellate court in D.C. issued a ruling that gave the Federal Communications Commission powers to regulate cable companies more aggressively. If the F.C.C. chooses to use those powers, it could increase competition among broadband providers. And if competition works as it’s supposed to, then market-opening regulation could slash the cost and raise the quality of broadband services. The front-burner issue in the court’s ruling was not competition but “net neutrality.” The court upheld F.C.C. rules from 2015 that were designed to prevent broadband companies, such as Comcast, Charter, and Verizon, from discriminating in favor of services they own. So, for example, Comcast cannot push subscribers toward its own Xfinity video-on-demand service by slowing down Netflix’s streaming speeds. Or, as John Oliver put it when he went on a pro-net-neutrality rant on his show “Last Week Tonight,” in June, 2014, an Internet without net neutrality “would allow big companies to buy their way into the fast lane, leaving everyone else in the slow lane.” Oliver’s bit went viral, and the F.C.C. was inundated with letters, nearly four million of them, almost all in favor of the policy. A rule against broadband discrimination makes it more difficult for the cable giants to abuse their power—which is why the cable industry and its allies have spent more than nine hundred million dollars since 2003 lobbying Congress to stop the F.C.C. from imposing net neutrality. That said, net neutrality has its limits. Telling the few giant corporations that control access to the Internet that they have to play nice doesn’t change the fact that there are only a few giant corporations that control access to the Internet. There was a moment when it seemed DSL—digital subscriber line service, which is provided by telephone companies over copper wires—would compete with cable companies for broadband services. But because DSL doesn’t provide service fast enough for smooth video streaming on multiple devices, which is what users now demand, its prospects faded. Satellite Internet, with its sluggish speeds and tight limits on data usage, is relevant only for people with no other choice—those who live in rural areas where there is no cable service. Wireless-telephone providers, such as Sprint and T-Mobile, lack access to enough wireless spectrum to provide Internet access at true broadband speeds (although the F.C.C. is trying to remedy that). Other technologies once considered promising, like providing broadband over power lines, have proved unworkable in practice. Net neutrality won’t fix any of that. In fact, net neutrality is built on the expectation that broadband service will be a monopoly for a long time to come; where cable companies continue to rule, the best we can do is to stop the giants from extending their monopoly power to new services. The appellate-court ruling, however, gives the F.C.C. a chance to escape that narrow formulation and address competition head-on. In upholding the F.C.C.’s net-neutrality rules, the D.C. appellate court accepted the government’s view that broadband access is as essential to American consumers as telephone or electricity service. The court affirmed the F.C.C.’s decision to reclassify broadband access as a “telecommunications service,” a category over which U.S. law grants the F.C.C. broad regulatory power. Before the reclassification, the F.C.C. had considered broadband access to be an “information service,” a category over which the F.C.C. exercises much narrower authority. This shift in legal classification may sound like lawyers quibbling, but it has real-world consequences. It gives the F.C.C. the same kind of regulatory power over cable companies that it had over the Baby Bells’ regional telephone monopolies. The F.C.C. could use this power in all sorts of ways to improve broadband price and service. One crucial step would be to mandate what telecom geeks refer to as “local-loop unbundling”—an awkward term of art that essentially means forcing cable companies to lease access, for a price and on terms set by the F.C.C., to the copper and fiber-optic cables, switches, and local offices that connect the main arteries of the Internet to individual homes and buildings. (If you’ve heard the term “last mile,” that’s another name for the “local loop” part of the network that would be subject to “unbundling.”) If that happened, new companies would arise to connect to the cable giants’ networks and vie to provide broadband access. That new competition would push down prices, improve service, spark innovation, and also ease the concerns about discrimination that provoked the F.C.C.’s net-neutrality mandate in the first place. In a competitive market, if Comcast puts Netflix at a disadvantage, then a customer can call one of Comcast’s competitors. This is not easy to do. If the F.C.C. sets the price for connection too low, current providers would have less incentive to improve their networks; if it sets the price too high, prospective new entrants wouldn’t think it worth their while. And, above all, the F.C.C. must write rules governing competitors’ right to connect that are simple, clear, and enforceable without enormous expense and years of delay. Otherwise, the cable companies will respond to regulation as the Baby Bells once did—by fighting endlessly over the terms on which competitors may connect to their networks, in the process driving many of their would-be competitors out of business. The experience in Europe offers a useful example. Broadband prices in Europe are lower in countries that make more use of local-loop unbundling. And, really, there aren’t many other options to increase competition. Building a new broadband network to compete with cable is too costly, as Verizon’s experience with its FiOS network shows. Verizon has halted FiOS expansion because the colossal expense of tearing up streets to lay fiber-optic cable, or even stringing cable on utility poles, made replicating the cable company’s local network uneconomic except in the densest urban areas. Even mighty Google has made slow progress. Google Fiber has built out competing networks in just three medium-sized cities thus far. More are planned, but it will be many years, at least, before Google Fiber provides meaningful competition across the nation. The court’s approval, however, doesn’t mean the F.C.C. will use its regulatory power to the fullest. When the F.C.C. voted last year to implement net-neutrality rules, the Commission’s chairman, Tom Wheeler, said that the F.C.C. will engage in only “light touch” regulation, and that there will be “no rate regulation … and no network unbundling.” Wheeler won’t likely be in his job for much longer, and in a recent Op-Ed, Hillary Clinton criticized the absence of competition in a number of industries, singling out broadband access as a particular problem. Of course, no matter who becomes President, the cable companies won’t give up their monopoly power without a titanic fight. And prices for broadband in a Manhattan apartment—or in most of the country, for that matter—aren’t likely to come down soon. Christopher Jon Sprigman is a professor at the New York University School of Law and co-director of the Engelberg Center on Innovation Law and Policy. He wrote a brief submitted to the U.S. Court of Appeals for the D.C. Circuit in the open-Internet case on behalf of thirty members of Congress who are proponents of net neutrality.


" Nonobviousness: A Comment on Three Learned Papers ," 12 Lewis & Clark L. Rev. 431 (2008) Abstract This Article, a comment on the contributions of John Duffy, Rebecca Eisenberg, and Gregory Mandel, addresses three areas where improvements could be made in the law on nonobviousness. First, the quantum of inventiveness required for patentability should reflect the capabilities of the ordinary artisan. Second, the asymmetry in the error rate of nonobviousness determinations should be taken into account in setting the standard of nonobviousness. Third, the concept of nonobviousness—or, better, inventive step—should be operationalized by considering the opportunities, risks, and nonpatent incentives the inventor faced at the time of the innovation.

"Nonobviousness: A Comment on Three Learned Papers," 12 *Lewis & Clark L. Rev.* 431 (2008)

									Abstract

This Article, a comment on the contributions of John Duffy, Rebecca Eisenberg, and Gregory Mandel, addresses three areas where improvements could be made in the law on nonobviousness. First, the quantum of inventiveness required for patentability should reflect the capabilities of the ordinary artisan. Second, the asymmetry in the error rate of nonobviousness determinations should be taken into account in setting the standard of nonobviousness. Third, the concept of nonobviousness—or, better, inventive step—should be operationalized by considering the opportunities, risks, and nonpatent incentives the inventor faced at the time of the innovation.

Rochelle C. Dreyfuss, Nonobviousness: A Comment on Three Learned Papers, 12 Lewis & Clark L. Rev. 431 (2008)

LEWIS & CLARK LAW REVIEW [Vol. 12:2 Within much of the literature and policy initiatives, these three problems are highly intertwined and often conflated. For example, section 103, the statutory requirement for nonobviousness, is actually a procedure for determining it. 4 Suggesting that the methodology was introduced into the statute to reduce indeterminacy, Mandel notes that the failure to define the quantum of inventiveness is the provision's most spectacular flaw. 5 The Federal Circuit, which was also largely created to reduce the indeterminacy of nonobviousness decisions, 6 has chosen to police the requirement through the stringent application of procedural rules. As Eisenberg illustrates with the example of DNA sequencing, 7 the Federal Circuit rarely discusses the only part of section 103 that sets a standard, which is the level of skill in the art. Further, the court's favored procedure-the "'teaching, suggestion, or motivation' test," which was adopted as a check on indeterminacy, has been applied so rigidly it effectively lowers the standard. 8 And, as Duffy points out, none of these initiatives moves the law to the theoretically optimal standard of patentability, which would grant "patents only for inventions that need the patent incentive." 9 This Comment attempts to unpack these three issues by discussing each in turn. As noted above, the level of ordinary skill in the art is the only part of the nonobviousness analysis that determines how large an advance is needed to merit patent protection. All three authors comment on it. However, they make somewhat contradictory points. Eisenberg worries that the Court fails to consider the "changing expectations [of success] of those working in a field" and "ignore[s their] problem-solving approach[es]." 10 In other writing, she has criticized the Federal Circuit's failure to acknowledge that the people who self-select as scientists and technologists do so because they have some creative capacity. 11 Mandel highlights the paucity of case law on the question of the level of ordinary skill in the art; he also notes that little attention is paid to whether NONOBVIOUSNESS: A COMMENT ON THREE PAPERS 433 ordinary artisans have the resources available to make the invention. In other writing, he has stressed the hindsight bias that he says can lead decision-makers to assume that ordinary artisans have the same level of skill as the inventor. 12 These observations point in opposing directions because Eisenberg's argument is, essentially, that the quantum of inventiveness is too low (real artisans are more creative than assumed, so some advances that ordinary artisans could generate are nonetheless patented), whereas Mandel's observations about hindsight suggest that the quantum of inventiveness is too high (that in the real world, artisans are less skilled than assumed, and so could not have generated the advance disclosed in the patent application). My own intuition is that, as a general matter, the level of inventiveness is set too low. Indeed, I read the central message of KSR as adopting that view as well, and I give Professor Eisenberg's earlier work credit for two key lines in the opinion: the Court's discussion of the person of ordinary skill as "a person of ordinary creativity, not an automaton" and its admonition that consideration be given to the "creative steps that a person of ordinary skill in the art would employ." 13 Nor do I believe this view to be controversial. I take it that it is also the main thrust of Duffy's claim that the current standard of nonobviousness is an imperfect proxy for the "patent-induced" standard. In a sense, the above description of Mandel's position is not entirely fair. While Eisenberg and Duffy are concerned with the standard of inventiveness, Mandel's main focus is on the question of indeterminacy. Thus, his view is that the threshold of nonobviousness is set correctly, but that errors in making the determination are throwing a monkey wrench into the system. These include type I errors-false negatives-inventions that are nonobvious which are nonetheless considered obvious, as well as type II errors-false positives-advances that are obvious which are nonetheless considered nonobvious. He supports his argument with his earlier experimental work, where he demonstrated the cognitive impossibility of putting oneself into the position of thinking about whether an invention one knows has been made was nonobvious when it was made. I, however, also disagree with the more subtle version of the argument. Mandel is clearly right that the incidence of error cannot be judged by the raw numbers. As he perceptively explains, because there are so many more marginal advances than major ones made, the same level of uncertainty on both sides of the threshold will produce more wrongly granted patents than wrongly denied patents. As a result of all the wrong issuances, it will look to observers as though the standard of inventiveness is set too low even when it is set in the right place. Still, it is hard to believe that there is real symmetry here, for there are structural features in the patent system that systematically create type II, and only type II, errors. Examiners operate under resource constraints. They have imperfect information because patentees can be less than candid, because prior art searches can be difficult to conduct, 15 and because there is art that is secret which nonetheless counts for nonobviousness purposes. 16 In addition, examiners have notoriously little time to think through the relationship between the known prior art and the advance claimed and, as Robert Merges has suggested, the incentive structure in the Patent and Trademark Office (PTO) favors allowances. 17 As Mandel notes, a vicious cycle can take hold: the more indeterminate the decision, the more patent applications are filed; the more applications, the more work for the PTO; the more work, the more grants-and the more indeterminate the standard will appear, leading to even more applications. In litigation, there are analogous sources of type II errors. The "clear and convincing evidence" standard is difficult to overcome and applies regardless of whether the PTO reviewed the prior art presented at trial. Aside from the hindsight bias issue, jury trials are particularly likely to skew in favor of patentability. Kimberly Moore's work shows that juries 15 NONOBVIOUSNESS: A COMMENT ON THREE PAPERS 435 tend to favor American inventors over infringers. 20 In high-technology cases, the "wow" factor-the tendency of juries to be awed by technological complexity-can be very strong. Additionally, practitioners often complain that jurors uninterested in technology (or in jury duty) tend to stop listening as soon as they hear about the presumption of validity. 21 To put this another way, cognitive problems arise only where there is real cognition-hindsight bias is an issue only for decisionmakers who have a reasonable opportunity to make decisions. To the extent that the indeterminacy problem is insolvable (and its persistence throughout the history of the patent system suggests it is insolvable), one way to reduce the number of type II errors is to shift Professor Mandel's nonobviousness threshold to the right-to require advances to display much more ingenuity than that possessed by the ordinary artisan. 22 Of course, the move would not be costless, for the number of type I errors would rise. In my view, however, the distortions produced by erroneous grants vastly outweigh the cost of erroneous denials. 23 Type II errors take material out of the public domain, increase patent thickets and transaction costs, act as barriers to entry and to cumulative research, and encourage trolling. But while type I errors may be bad for the inventor, they can be very advantageous to society. Without a patent, users get important inventions for free. The Kohler and Milstein principle for making monoclonal antibodies, described by Duffy, is an example. 24 A foundational development, its free availability meant that the biotechnology could quickly advance and influence "virtually all aspects of basic research in immunology, cell biology, biochemistry, and medicine." 25 Admittedly, as the risk of not acquiring a patent increases, incentives to invent could decrease. However, the problem is likely to be significant only if the shift in patentability is very dramatic. After all, there are reasons to innovate that are not dependent on patents. Kohler and Milstein were awarded the Nobel Prize for Medicine in 1984. 26 Less dramatically, competition and predictable obsolescence produce their LEWIS & CLARK LAW REVIEW [Vol. 12:2 own dynamics of innovation. 27 As important, there are methods outside the patent system for appropriating returns on inventive investment. Thus, it is worth noting that when the Supreme Court interpreted the nonobviousness requirement in 1966 in Graham v. John Deere Co., 28 it was writing against a backdrop of a very aggressive preemption rule: Sears/Compco, which held state unfair competition law preempted by federal patent policy, had been decided two years earlier. 29 The Court did not retreat from its skeptical view of state protectionism until 1974, when it decided Kewanee Oil Co. v. Bicron Corp. and upheld state trade secrecy law. 30 Since that time, courts have also enforced contractual restrictions that promote the capture of returns from innovation. 31 Furthermore, in many modern technologies, lock in and network effects can vastly prolong first-mover advantages. 32 Patent rights are, in short, no longer the only game in town. Greater reliance on non-patent incentives could interfere with the disclosure function of patents. But even here, the shift in patentability would have to be fairly drastic before it had real bite. In many fields (including research science), there are strong norms of sharing. Besides, the litigation literature and experience with Rule 11 indicate that lawyers tend to believe in their cases. 34 Thus, just as many unmeritorious cases are pursued to trial, so, too, most of the same patent applications will be prosecuted. And again, the law has changed since Graham was penned; where patent applications were once kept confidential unless the patent issued, current rules require disclosure of most pending applications eighteen months after they are filed. 35 Finally, 27 See, e.g. 2000). Those who file only in the United States can seek an exemption, but most important inventions are also patented abroad and cannot make use of the exemption. NONOBVIOUSNESS: A COMMENT ON THREE PAPERS 437 even if material is kept secret, the technologies of reverse engineering have improved, making early disclosure more likely. I read the Supreme Court as largely in agreement on the question of the relative cost of type I and type II errors. In KSR International Co. v. Teleflex, Inc., the Court emphasized that "[g]ranting patent protection to advances that would occur in the ordinary course without real innovation retards progress and may, for patents combining previously known elements, deprive prior inventions of their value or utility." 37 And in his dissent from the dismissal of the petition of certiorari in Laboratory Corps v. Metabolite Laboratories, Justice Breyer warned that "too much patent protection can impede rather than 'promote the Progress of Science and useful Arts.'" 38 None of the recent cases discusses type I errors. All three of these authors largely agree-correctly in my view-that as currently portrayed by commentators, section 103 cannot be the right way to measure inventiveness. Eisenberg demonstrates quite convincingly that the provision is not applied as consistently as is often contended and that the Federal Circuit is actually fairly sensitive to context. It varies its approach, depending on whether it is deciding the invention is obvious or nonobvious. Chemical cases have their own standard (which involves a different use of hindsight), and for pharmaceuticals, the court is more likely to uphold a patent on a new therapeutic agent than to allow the patentee to engage in "evergreening." 39 Mandel even provides an explanation for the court's behavior: if the level of skill in the art were accurately identified and utilized, the consequences would be perverse, for the more sophisticated a field, the higher the risk that advances in it would not be regarded as patentable. Taken to the extreme, there would be insufficient investment in the fields where society has the greatest interest (as measured by the resources devoted to it and the level of training within it). 40 Presumably, the opposite would also hold, and the ease of acquiring patents in "stupid" arts could draw investment to areas in which society has very little interest. It is no surprise then that the court would (as Eisenberg demonstrates) be resistant to updating the level of skill in biotechnology at the time when the field was in its period of early growth. 41 artisans were taken as understanding the genomic vocabulary, patent incentives would have been significantly compromised at a time when they may, in fact, have been necessary to spur the production of knowledge. That said, I am skeptical about the substitute analysis Mandel proposes. While I think Duffy's approach has much to recommend it, I have some concerns about it as well. Mandel would focus the decision-maker's attention on the point in the inventive process that required ingenuity and would ask the probability that a person of ordinary skill would have made the same advance. As to the probability issue, the question is when. In the fullness of time, it is highly likely that every invention will be made; to a large extent, the real goal of patent law is not to induce invention, but instead to induce it sooner rather than later. Or as Duffy suggested in earlier work, patent law's aim is to move the invention from the drawing board through the patent system and out into the public domain. 42 If the ordinary artisan is given forever (or even a long time) to come up with the invention, this goal would be defeated. Mandel's discussion of the simultaneity of invention as a defense to infringement suggests that "when" actually means at the exact same time as the inventor. If so, then his proposal has a great deal of resonance with Duffy's notion of examining how the patent race is affected by external factors, including demand-side and supply-side changes, because these changes would also tend to produce the simultaneous development of obvious advances. The problem with this approach is that it, too, may lead to perverse consequences. If inventors monitor other researchers and react strategically to their progress-as many commentators posit 43 then a simultaneity bar may lead researchers to abandon their efforts once they learn that others are working on the same problem. In his paper, Duffy slips rather quickly past his own observation that inventors are often racing for a patent. 44 If the patent opportunity is eliminated, there is a question whether they will race at all. In fact, KSR suffers from the same problem. According to the Court, if the market would induce the invention, there is no need for a patent. 45 But strong market demand will also induce rapid imitation. Without significant lead time and the opportunity to develop networks (or lock in), there is a real danger that in some fields, appropriability will be compromised and the incentives to innovate reduced. Duffy's reliance on 42 4 35 U.S.C. § 103(a) (2000); Id. at 17-18. 5 Mandel, supra note 2, at 335. 6 11 Rebecca S. Eisenberg, Obvious to Whom? Evaluating Inventions from the Perspective of PHOSITA, 19 BERKELEY TECH. L.J. 885, 891 (2004). 16 See, e.g. Oddzon Products, Inc. v. Just Toys, Inc., 122 F.3d 1396, 1401-02 (Fed. 35 35 U.S.C. § 122 ( 45 KSR Int'l Co. v. Teleflex Inc., 127 S. Ct. 1727, 1740 (2007).

35 35 U.S.C. § 122 (