Patent Citation Networks Revisited: Signs of a Twenty-First Century Change?

SIGNS OF A TWENTY-FIRST CENTURY CHANGET KATHERINE J. STRANDBURG, GABOR CSARDI, JAN TOBOCHNIK, PtTER tRDI, AND LASZLO ZALANYI**

This Article reports an empirical study of the network composed of patent "nodes" and citation "links" between them. It builds on an earlier study in which we argued that trends in the growth of the patent citation network provide evidence that the explosive growth in patenting in the late twentieth-century was due at least in part to the issuance of increasingly trivial patents. We defined a measure of patent stratification based on comparative probability of citation; an increase in this measure suggests that the United States Patent and Trademark Office is issuing patents of comparatively less technological significance. Provocatively, we found that stratification increased in the 1990s during the "patent explosion." Here we report a further study indicating that the trend toward increasing stratification leveled off beginning around 2000. This observation suggests that there was a de facto tightening of patentability standards well before the doctrinal shifts reflected in the Supreme Court's flurry of patent activity beginning around 2005. We also investigate the possibility that changes in our measure of stratification are due to something other than changes in patentability standards. While not conclusive, our results suggest that neither shifts in predominance of technological areas nor changes in citation practice account for our observations. We have thus identified an apparent puzzle: What happened around 2000 to cause a de facto tightening of patentability standards at the USPTO?

In our 2006 article, Law and the Science of Networks: An Overview and an Application to the "Patent Explosion," ' we argued that the emerging interdisciplinary field of network science, which seeks to understand the structure, growth, and behavior of a variety of network systems, would provide a useful conceptual and analytical perspective on many questions of interest to legal scholars 2 because of the ubiquity of networks, such as social networks, transportation networks, and communication networks, as objects of legal doctrine.

We then applied the network approach to the patent citation network-a network composed of patent "nodes" and citation "links" between them.' We argued that trends in the growth of this network provide insights into the nature of the patent explosion of the past few decades.' Most provocatively, our study demonstrated that a measure of patent stratification decreased in the 1980s and then increased in the 1990s, suggesting that comparatively more trivial patents were being issued throughout the 1990s. ' We hypothesized that the increasing stratification during the 1990s might be related to changes in legal doctrine, such as a weakening of the nonobviousness requirement. 6 In this Article, we look more closely at the period beginning around 2000 and find that the trend toward increasing stratification appears to have leveled off, though stratification remains greater than it was in the 1980s. ' The number of patents issued by the United States Patent and Trademark Office ("USPTO") per year also began to level off around 2000, despite the fact that the number of applications continued to climb steeply.' These observations suggest that there was a de facto tightening of patent standards well before the doctrinal shifts reflected in the Supreme Court's flurry of patent activity beginning around 2005.1 Though we consider some possible 3. Id. at 1329-40. In Part I.B, infra, we reprise parts of that discussion as necessary to make our present arguments. (finding that a patent does not automatically confer market power); Merck KGaA v. In our studies, the "nodes" of the network are U.S. patents and the "links" are citations of one patent by another. ° In analyzing the patent citation network, we have used the Hall, Jaffe, and Trajtenberg dataset, which includes approximately sixteen million citations made by the more than two million patents issued by the USPTO from 1975 through 1999," supplemented with updated citation data obtained from the USPTO that extends through 2006. 12 We briefly review our earlier results, which suggested that the standard for patentability loosened during the 1990s, in Part I. In Parts II.A and II.B, we describe our updated stratification study, which suggests that the patentability standard has tightened up again since around 2000. In Parts II.C and II.D, we consider alternative explanations for our stratification parameter observations. In Part II.C, we investigate whether the observed changes in average stratification can be explained by changes in the mixture of technical categories among issued patents. To study this question, we employ the division into six technical categories used in the Hall, Jaffe, and Trajtenberg dataset." 3 Though there are some differences between categories in the evolution of the stratification parameter, the overall pictures are similar. Patent stratification increased in all categories during the 1990s. We also see the beginnings of a plateau around 2000 in all categories, though our data for the years after 2000 have not been categorized. 4 In Part II.D, we consider the possibility that changes in citation practices, rather than changes in patent characteristics, were compounds in preclinical studies is protected" and is not infringement in most circumstances). 10. U.S. patents also cite scientific literature and foreign patents. We did not categorize the data we obtained directly from the USPTO. We used the available categorization from the National Bureau of Economic Research ("NBER") database, which ran only through 1999. See id. at 407. responsible for the changes in stratification that we observe. Our investigation is preliminary and cannot rule out this possibility, but finds no evidence that citation practice has changed so as to account for the changes in stratification.

We conclude our analysis by discussing some issues for further research, including the need to determine why stratification reached a plateau beginning around the turn of this century and the possibility that recent tightening of the legal standards for patentable subject matter and nonobviousness will result in further decreases in stratification.

During the 1980s and 1990s, there was a major upsurge in patenting (see Figure 1) and an expansion of patent eligibility in the United States, with the courts ruling in favor of the patentability of living things, software, and business methods. 15 During the same period, the U.S. Court of Appeals for the Federal Circuit was established to hear the vast majority of patent appeals in the United States. 16 This expansion in patenting led to widespread debate about patent quality and overpatenting. 17 To accomplish its 17. For a sample of the many discussions about patent quality and overpatenting, see constitutionally-mandated objective of promoting technological progress, 18 patent protection must be carefully tailored to balance its benefits against its costs. The benefits may include providing incentives to invent, functioning as a signal of technical competence, and facilitating a market for intangible knowledge. 9 However, because a patent provides exclusive rights to practice the patented technology, patents impose costs on society that may include not only supra-competitive pricing of patented products but also increased barriers to building upon existing technology. Empirical investigation of the patent system can play an important role in understanding how to maintain the appropriate balance. For example, empirical studies make it possible to observe and quantify the effects of technological and doctrinal changes.

In our earlier work, we found that, in addition to burgeoning growth in patent numbers during the 1990s, there was an increasing stratification of patent "citability, '21 by which we mean the probability that a patent with particular characteristics will be the object of a citation in a subsequent patent application. In this Part, we review some of those results. With that context in place, we then describe our updated study of patent stratification in Part II. 21. Strandburg et al., supra note 1, at 1329-51. We define citability by the probability that a given patent will be the object of the next citation depending on its current age and number of previous citations. Note that citability is normalized to back out the fact that there may be more patents of one value of (kl) than of another. When citability is larger for one set of parameter values than for another, it means that the probability of being cited is greater for each individual patent with those characteristics.

The USPTO issues patents after examining applications to determine, among other things, whether the patent claims meet the legal requirements of novelty and nonobviousness 2 Patent claims are specific statements of the scope of the legal coverage of a patent. As noted above, the legal effect of a patent is to provide the patentee a right to exclude others from using the claimed technology without a license, as detailed in the infringement provisions of the patent statute. 3 In the course of the examination of a patent application for novelty and nonobviousness, patent claims are compared against potential prior art, consisting in large part of prior patents and other publications in relevant technical fields. Applicants, their patent attorneys, and the official patent examiners all identify potential prior art. To qualify for a patent, the claimed invention must be novel, meaning there is no prior patent or other prior art that is identical to what is claimed. 4 More importantly, the claimed invention must be nonobvious, meaning that at the time it was invented, the invention would not have been obvious to a person having ordinary skill in the art in the field of the invention. Seeking out prior art patents (and other sources of prior art) is key to determining both novelty and nonobviousness.

An issued patent cites another patent if the cited patent's technical relationship to the claimed technology is close enough that it is relevant to determining whether the claimed technology is new and nonobvious. 2 6 A citation from one patent to another may indicate either that the later patent builds upon the technology of the earlier patent or simply that the earlier technology was closely enough related to be material to determining whether the later patent should be issued." Good evidence shows that, at least on average, the number of citations received by patents is indicative of their technical value. 8 Recent studies show that patent examiners provide a large fraction of the cited references. During the period from 2001 to 2003, for example, examiners provided sixty-three percent of all citations. 2 9 Indeed, in forty percent of the patents granted, examiners provided all citations. 3 ' Because such a large fraction of references are provided by patent examiners and common patent prosecution practice suggests that another large group is provided by patent attorneys, citations do not generally indicate a direct flow of knowledge.

We therefore treat them only as indications of technological relationships and, on average, as proxies for technical value. One can thus view the patent citation network as a kind of map of the space of patented technology, indicating the technical relationships between various pieces of "property" in that space. 31 In the next Section, we describe our study of how that map has evolved over time and interpret that evolution.

In our earlier paper, we demonstrated that patents have become increasingly stratified in their citability since the late 1980s. 32 Our approach was motivated by statistical physics studies of a diverse range of other growing networks. patent citation network in terms of the probability that a particular patent will be cited by the next citation made. We studied the dependence of this "citability" function on a patent's age and the number of times it has been cited previously. 34 We used the number of times a patent is cited as an approximate proxy for its technical value."

Looking at the way in which citability depends on the number of citations received, we determined a stratification parameter that measures the extent to which the citability of the most citable patents exceeds that of the least citable patents. 36 We interpreted this parameter as an indicator of the degree of stratification of patent technical value. 37 Because a case-by-case evaluation of the underlying reasons that one patent might cite another is impossible for a large network of citations, we used a random attachment model and analyzed the network in terms of objective characteristics that we expected to be reflective, on average, of the underlying citation process. 3 " Of course, the fact that we extracted a citation "probability" is not meant to suggest that the particular citation choices made by patent examiners or applicants are actually random. In reality, though there is only one U.S. patent citation network, and it did not evolve randomly. Instead, the network reflects the citation choices that examiners and applicants made over time. Our approach depended only on the fact that, cumulatively, those individual citation decisions result in a likelihood of citation which depends in interesting ways on characteristics that we can observe. In our initial study, we assumed that the probability that a particular patent will be cited at a given time depends primarily on two characteristics: its age, which we called 1 and measured in terms of numbers of patents issued; and the number of times it has already been cited, which we called k. 39 The citability function, which we denoted A(kl), is proportional to the average likelihood that a patent with k previous citations of age 1 will be the object of the next citation. 4 " Our assumption that the probability that a patent will be cited depends on its age requires little explanation-technology tends to become obsolete.

Our expectation that citability would depend on the number of previous citations arises out of studies of other networks. In many networks, nodes with large numbers of neighbors (here, citations) tend to accrue even more neighbors as the network grows. 41 This phenomenon, known as "preferential attachment," is plausibly expected in the patent citation network because of at least two factors. First, considerable statistical evidence suggests that highly cited patents are more valuable and may be of greater technological merit than less frequently cited patents. 4 2 We expect more meritorious patents to accumulate more and more citations. Second, technology has its own "popular crowd," depending on what field is "hot" at a particular time. In our analysis, we thus assumed that the citability varied with the number of previous citations to a patent, k.

To find the likelihood of citation for given k and 1, we developed a novel iterative technique, which we used to extract the citability function from the patent citation data. Our technical publications describe the approach used in that previous article in detail. 43 In that analysis, we did not assume a particular functional form for 39. For a more complete description of our data analysis procedure in that earlier citability-we derived the functional form directly from the data. In principle, however, our choice of variables (age and previous number of citations in our previous article) and the specifics of how we extract the citability function might have influenced our results. We return to these points below when we discuss our updated results."

In our previous study, we determined that the dependence of citability on age, 1, and number of previous citations received, k, is approximately separable into a factor depending on 1 and a factor depending on k. Even though the probability of being cited depends on age, the way in which it depends on k is more or less the same for patents of any age. In other words, 45 This separability is illustrated in Figure 2, which shows that the dependence of citability on previous citations received, Ak(k), is essentially the same for a range of different patent ages.

. As Figure 2 demonstrates, the more often a patent has already been cited (the higher its value of k), the more likely it is to be cited again (the higher its value of A(k))-thus demonstrating preferential attachment. Preferential attachment is cumulative-because highly cited patents are more likely to be cited, they become even more highly cited and thus even more likely to be cited, and so forth. 46 The pattern of citability shown in Figure 2 thus eventually leads to the extremely skewed distribution of citations received by patents shown in Figure 3.47 Most patents are hardly cited at all, while a few patents become citation "billionaires" (well, "hundredaires," really). This general picture of a highly skewed patent citability reflecting highly skewed patent value is now well known. 48

Our network analysis allowed us to go beyond this general observation, however, to ask just how stratified patent value is. We measured stratification by determining how much more citable the most citable patents are as compared to the least citable patents. We also determined how the extent of stratification has evolved over time. If more patents were being issued simply as a result of faster or broader technological progress, we would expect the degree of 46. This is sometimes called the "rich get richer" phenomenon or the "Matthew effect." See Newman, supra note 33, at 213. stratification to remain about the same over time. 49 On the other hand, if the patentability standard were lowered, there would be not only more patents issued but a higher proportion of them would be more technically trivial-the degree to which highly citable patents dominate trivial patents would increase. Looking at Figure 2, we can quantify the rate at which citability increases with previous citations by noting that Ak(k) is closely fit by the form Ak(k)k'. The parameter ax is a measure of the extent to which highly cited patents are preferred, or what we call patent stratification.

We measured the evolution over time of the degree of stratification by calculating a using only the patents within a 500,000patent sliding time window and calculating a value for x after every 100,000 patents.

The value of ax, and hence the degree of stratification of patent citability, has varied over time. Figure 4 shows the results of our previous analysis. As shown in Figure 4, the stratification of citability, as reflected in the value of (x, has evolved nonmonotonically. Following a period of decreasing stratification during the early 1980s, it began to rise in the late 1980s. 5 1 By contrast, Figure 1 shows that the number of patents issued annually has been rising essentially since the inauguration of the patent system and rose very rapidly beginning in the early 1980s. Thus, during a period throughout which the absolute number of patents issued per year was rising rapidly, the relationships between those patents were changing. Patent citability became first less and then increasingly stratified. In the next Part, we discuss new results for patent stratification since 2000 and then consider some alternative explanations for the evolution of the stratification parameter.

In this update to our earlier study, we take a closer look at the behavior of the stratification parameter using a more sophisticated maximum likelihood approach to finding the citability function and fitting the citability to determine the stratification parameter. 5 2 Part II.A discusses the methodological improvements. As discussed in Part II.B, the new method reproduces the declining stratification during the 1980s and increasing stratification during the 1990s that we reported in our earlier article. However, the improved method surprisingly reveals that, rather than increasing from 2000 to 2006, stratification leveled off around 2000. We also use the maximum likelihood approach to test the robustness of our results to the inclusion of some additional parameters in the citability function. Our basic observations about the evolution of the stratification parameter are unchanged by including those additional parameters. 3 As we discussed in our earlier article, there are several ways to interpret the increasing stratification of citability observed between the late 1980s and 2000.1' We argued that, consistent with societal concern with "low quality" patents, the most likely possibility was that the patentability standard was decreasing, resulting in the issuance of a larger fraction of more trivial-and hence less citable-51. The first data point in Figure 4 patents. 55 Another possibility was that the change in stratification parameter merely reflected changes in the subject matter mix of patented technologies. 56 We address that possibility in Part II.C, finding that the stratification parameter trends are consistent across technology classes. Finally, it is possible that the change in citability stratification reflects some kind of change in citation practices, rather than a change in patent characteristics. 7 We investigate citation practice in Part II.D.

In this Section, we give a brief impressionistic description of the differences between our methodological approaches in this and our previous article. Our basic approach in both studies is to look for the probability distribution in a random attachment model that is most likely to have produced the observed patent citation network. In other words, we seek to "reverse engineer" the citability function given the actual patent citation network. Our ultimate goal is to extract certain interesting features, such as the stratification parameter, from the citability function we have reverse engineered.

1) Select variables (such as age, previous number of citations, and technical category) for the citability function, A (e).

2) Use a "reverse engineering" method to approximate the citability, A(*), as a function of those variables given the empirically observed citation network.

3) Determine the stratification parameter, ct, by fitting the dependence of A(.) on number of previous citations, k, to a power law formk.

In our earlier article, we selected the variables we believed would be most likely to influence citability: age and number of citations previously received. 59 We then performed parts 2) and 3) sequentially. We first used a simple weighted counting method to reverse engineer the citability function A(kl).

Essentially, this method approximates the most likely probability distribution by simply counting the types of links that actually occur in the network. 6 ' In other words, if patents with a particular value of k and I are cited many times, A(kl) should be relatively large. We adjusted the count so as to assign a higher score for an edge that was added at a later time when there were more types of nodes available to cite. Also, if a patent type (particular value of k and 1) is very common in the network, then a citation to it should not add as much to the score as citing a more rare type. Applying these basic ideas, we obtained an approximation to the citability function.

However, the function we obtained is entirely numerical-it is just a list of values of A for each pair of k and 1 values. To make sense of the citability function, we needed to fit it to some functional form. To do this, we assumed, based on the observations shown in Figure 2, that the dependence of citability on number of citations previously received was independent of the age of the cited patent. In other words, A(k,l) = A,(l)Ak(k) and took a power law form, Ak(k) -V + a, where at is the stratification parameter. 61 We then attempted to fit the data to this form. For a variety of reasons, fitting power law forms to empirical data is often tricky, especially when, as in our case, the statistical errors for some data points are much larger than for others. 62 After experimenting with various approaches, we adopted a method in which we assumed that the value of a was equal to 1. 63 That approach gave us the values of stratification parameter shown in our earlier article and reproduced in Figure 4 above. However, the approximation that a equals 1 is not quite correct and particularly affects our results for the stratification parameter after 2000.

To improve the accuracy of our results, we developed a more sophisticated fitting method. We used that method to obtain the more accurate values of the stratification parameter, ax, and the intercept, a, which we report in this Article. We also use the new 63. The actual probability of citation is given by A(k,l)IS(t), where S(t) is a normalization factor obtained by adding up all values of A(kl). For this reason, we can (and did) arbitrarily set A(O,1) = 1. Setting a= 1 is equivalent to assuming that the approximate separation A(k,l) = A,(l)Ak(k) is exactly true. method to compare different choices of variables for A(.). Our original approach gave us a good numerical approximation to the citability function, A(.), but we had to apply a rather ad hoc method of fitting that function to obtain the stratification parameter. The maximum likelihood method that we employ here permits us to obtain the stratification parameter directly by fitting the original network data rather than extracting A(k,1) and then fitting it to obtain c. The catch is that in order to do that we have to assume a functional form for A(#); here it is reasonable to assume that Ak(k)k' + a. We might worry that any particular functional form we choose does not do a good job of approximating the citability function. To determine whether we have chosen a good functional form, we can generate two versions of A(*): a parameterized version in which we assume a particular functional form for A(.) and a free form version in which, as with the counting method, we make no such assumptions.' 4 If the parameterized citability function fits the data almost as well as the free form version, then we can conclude that the assumed functional form accurately represents the network.

To facilitate this comparison, we devised a method for measuring the "goodness of fit" of citability functions A(.) that use different variables and different functional forms to describe the network. Goodness of fit measures how similar a simulated network generated using our approximate citability function would be to the real patent citation network. 65 Essentially, the goodness of fit measures the probability that a random network generated using a given citability function would match the real citation network.

For example, to evaluate how well the power law form, Ak(k)k ' + a, fits the data, we can measure its goodness of fit and compare it to the goodness of fit we obtain for a free form citability function. If the goodness of fit is similar in the two cases, then we know that the power law form is a reasonably accurate representation of the citability function. We can then rely on values of c and a that we obtain from the maximum likelihood fit using the power law form. In our updated study, we obtain very similar values of goodness of fit 64. The maximum likelihood method can also be used to calculate a free form version of A (kl) similar to what we get from the counting method (essentially a table of numerical values for various choices of k and /). The main advantage to us of the maximum likelihood method, however, is that it allows us to extract parameters such as a without using an ad hoc fitting method.

Problem, supra note 39, at 4-5. Basically, the goodness of fit measure permits us to compare how well various parameterizations of the citability function represent the observed patent citation network. from the free form version and the power law form. 66 This means that we can reliably use the maximum likelihood method to obtain values for ax and a.

We can also use the goodness of fit measure to examine how important it is to take into account the dependence of citability on other variables, such as age, number of citations received, and technical category. This means, for example, that we can compare a model in which citability only depends on number of citations previously received, A(#) = A(k), to a model in which citability depends on both number of citations received and age, A(*) = A(k,l). While adding more variables always gives a more accurate fit, the extent to which the goodness of fit improves when we add a particular variable allows us to determine whether taking account of that variable is important. Using this improved approach to fitting the data, we can simultaneously obtain more accurate values of parameters such as the stratification parameter, ax, and test whether we are accounting for the most important variables.

In this Section, we present our results for the stratification parameter obtained using the improved method described in Part II.A. Figure 5 shows the time dependence of the stratification parameter a for two different sets of variables: 1) using k only (on the top of the figure); and 2) using k and 1 (on the bottom of the figure). As we would expect, the precise value of a depends on the variables we take into account. Notably, however, the general trends are the same for both sets of variables and are robust to incorporating various other variables, as we discuss below. The results confirm our earlier observation that stratification decreased during the 1980s and rose during the 1990s. Beginning around the year 2000, the stratification peaked and then either leveled off or began a slight decline. While patents issued beginning in 2000 remained more stratified than patents issued during the 1980s, the stratification stopped increasing. As Figure 6 shows, the number of patents issued annually by the USPTO also leveled off somewhat around 2000, despite the fact that the number of applications continued to climb steeply. 67 The most natural interpretation of these observations is that the USPTO is no longer lowering the bar to patent issuance (and may even be raising it).

What might have caused such a tightening of patentability? Though we cannot answer that question based on our empirical analysis, a few speculations are possible. One possible explanation would be a change in legal doctrine that made it harder to obtain patents. The legal doctrine most directly related to the tightness of 67. The comparison of the number of patents issued in a certain year to the number of applications submitted during that year is not a measure of the fraction of patent applications that eventually issue as patents. Patent examination generally takes more than a year, so the numbers relate to different patents. The comparison is also complicated by the fact that the USPTO has a growing backlog of applications in at least some fields. The comparison here is intended only to indicate that something seems to have happened around 2000 to make it more difficult to get a patent.

68. Data is from the USPTO website. U.S. Patent and Trademark Office, U.S. Patent Activity, Calendar Years 1790 to the Present, http://www.uspto.gov/web/offices/ac/ido/ oeip/taffh-counts.pdf (last visited Apr. 24, 2009). the patentability standard is the nonobviousness requirement. In our earlier article, we suggested that the increasing stratification we observed during the 1980s might reflect increasing reliance by the USPTO on the Federal Circuit's "teaching, suggestion, or motivation to combine" test for obviousness. 69 Though the fraction of cases employing that test at the Federal Circuit also leveled off around 2000 (see Figure 7), there does not seem to have been any major doctrinal change in the nonobviousness standard that would explain a retrenchment from the 1990s trend toward looser patentability standards. , supra note 1, at 1338. Under the Federal Circuit's test, a purported invention combining earlier technology could be deemed unpatentably obvious only if there were evidence of a specific "teaching, suggestion, or motivation" to combine the prior art. See KSR Int'l Co. v. Telefex, Inc., 550 U.S. 398, 407 (2007). Such a "rigid" test lowers the bar to patentability, see id. at 419, and thus might be associated with increased issuance of more trivial-and less citable-patents. If the disparity between most and least citable patents increases, the stratification parameter would also be expected to increase. 70. These numbers were obtained using LEXIS searches for "(suggestion or motivation or teaching) w/s combine" (to count references to the suggestion test) and either a reference to "obvious" in the headnotes (dashes) or at least 5 uses of the word "obvious" in the case (diamonds). These two methods of counting yield the same qualitative results, showing an increase in use of the suggestion test throughout the 1990s.

A second possibility is that the plateau in stratification reflects a de facto change in the patentability standard at the USPTO based on a more aggressive application of then-current legal doctrine. In 2000, for example, the USPTO began its "Second Pair of Eyes Review" of business method patents, probably tightening up the standard for that category of patents. 7 ' As we will see below, though, the time dependence of the stratification parameter is qualitatively similar across technical categories, at least up through the beginning of the flattening out of the stratification parameter in 2000.72 Although we did not isolate business method patents in our analysis, this uniformity across categories makes it seem unlikely that the Second Pair of Eyes Review per se was responsible for the change. More plausibly, the Second Pair of Eyes Review is indicative of a broader USPTO reaction to public outcry about patent quality, resulting in tightening of patenting standards in ways that cannot be pinned to any specific change in doctrine. In any event, the question of what happened in the early 2000s that apparently stabilized (and perhaps tightened up) the patentability standard is worthy of further study.

One can also speculate as to what these observations might suggest about the relationship between the USPTO and the courts in setting patent doctrine. While the standard story would be that the USPTO follows the doctrine set by the Federal Circuit and the Supreme Court, 73 no doubt the real push and pull between the courts and the agency is more complex. The fact that the stratification plateau precedes major doctrinal shake-ups in the courts suggests that perhaps the USPTO plays a proactive role in setting the stage for doctrinal change through its de facto interpretations of legal doctrine.

In the future, the stratification parameter may give us clues about the effects of recent major changes in the doctrines of nonobviousness and patentable subject matter that should make it harder to obtain patents. In KSR v. Teleflex, 74 the Supreme Court overturned the Federal Circuit's "rigid" reliance on the suggestion test and articulated a stricter nonobviousness threshold. 75 In In re Bilski, 76 77 making it harder to obtain certain kinds of business method patents, which many have argued are of suspect merit. 78 We might expect these decisions to result in a decreasing stratification parameter a few years from now. Thus, it will be worthwhile to track the evolution of the stratification parameter in the wake of this doctrinal change.

To test the robustness of our observations of the time evolution of the stratification parameter, we considered various changes to our selection of variables. Most importantly, we were concerned about the possibility, which we raised in our earlier article, 79 that the average stratification parameter ct was changing not because of changes in the threshold for patentability but because the subject matter mix of patented technologies was changing.

Patent "importance" might be inherently more stratified in one field of technology than in another because of a difference in the importance of "pioneer" patents, for example. If so, an increasing prevalence of patents in that field could change the average degree of stratification. As shown in Figure 8, the distribution of patents across technological areas has evolved throughout the period of our study. To rule out the possibility that the change in stratification was an artifact of changing predominance of different types of patented technology, we repeated our analysis allowing the citability function to depend on the technical categories of the cited and citing patents. To investigate the dependence of the citability on the technological category of the citing patent, on the other hand, we must define and fit a separate citability function for each of the six categories. The citability function for each citing patent category is approximately of the form A ,,ctej = CcitedA k c(k)A (1), where Ak c ' i (k) is of the power law preferential attachment form, A citing (k) k', from which we can extract a value of a for each citing category. We can account for the cited category by a multiplicative constant because the functional dependence of citability on citations previously received does not vary significantly based on the category of the cited patent.

When we fit the resulting citability functions to obtain the stratification parameters corresponding to each category of citing patent, we obtain the results shown in Figure 9. To probe the possibility that the time evolution of the stratification parameter that we observed in our earlier work is due to a changing mix of technologies among issued patents, we investigated the time dependence of the stratification parameters for each category of citing patent separately. The results are shown in Figure 10.

As Figure 10 shows, though there were differences in the magnitude of the change over time, the general trend of declining, then increasing, stratification is common in all categories. Because we used the categorizations from the NBER dataset, which extends only through 1999," 3 we could not investigate fully whether the turn of the century plateau occurs in all citing patent categories. However, the beginning of the plateau appears to be visible in all categories in Figure 10. Of course, because these technological categories are very broad, it is still possible that a targeted investigation into more specific categories would reveal greater differences between technological fields. However, the fact that both "Drugs and Medical" (symbolized by 'Y' on the graph) and "Computers and Communications" (symbolized by "A") exhibit the same qualitative time dependence is an additional indication that the decrease, subsequent increase, and flattening of the stratification parameter is likely robust across technologies.

It is also possible that the evolution in citability stratification reflects a change in citation practice rather than a change in underlying patent characteristics. Wte skeptical of this possibility, 83. Hall et al., supra note 11, at 407. however, because trends in patent citation practice, including most notably the increased ease of computerized searching for prior art, seem unlikely to have caused the observed change from decreasing aX to increasing a during the late 1980s, followed by a leveling off beginning in 2000. While computerized searching undoubtedly became more prevalent during the late 1980s and 1990s, for example, there is no reason to expect that it would have such a nonmonotonic effect. Nonetheless, we would like to find means to account for citation practice more directly in our analysis. Here we report two preliminary steps in this direction.

The first is a direct comparison of the evolution of the stratification parameter for patent citations to the evolution of the stratification parameter for citation practice in a different, but somewhat related, arena-scientific journals. This comparison gives us two pieces of useful information. First, it allows us to test the robustness of the stratification parameter itself. If the time evolution of the stratification parameter for scientific journal citations looks sensible, then we may be more confident in attributing meaning to its behavior in the patent citation context. Second, the comparison allows us to test whether the changes in stratification we observe in the patent citation network are due to some universal phenomenon, such as increased computerized searching, that affects all similar citation practices. If the trends in stratification parameter were the same for patent citations and journal citations, we could rule out a patent-specific explanation. However, if, as we show below, they are different, we cannot completely rule out an explanation based on patent citation practice, but we can rule out explanations based on broad trends in citation practice.

The second step reported here is a comparison of characteristics of the patent citation network to those expected from a simple general model of citation practice based on a common sense notion of how scholars find references to cite in their articles. This simple model is known as the "forest fire model." The fact that the patent citation network is consistent with the forest fire model bolsters the reasonableness of the comparison to scientific journal citation networks and makes it less likely that the observed behavior of patent citation stratification is due to some unusual mechanism by which patent citations are selected. The demonstration of consistency with this simple model lays the groundwork for future attempts to relate variations in the stratification parameter to the parameters of the model.

One way to check for the influence of search technology is to compare the behavior of the U.S. patent citation network with that of other citation networks, such as the European patent citation network or the network of citations in scientific journals. Differences between the behavior of one network and the behavior of others would indicate that there is some idiosyncratic, underlying cause of the behavior of the first network. So, for example, if the stratification parameters for citation networks other than the patent network do not evolve in a similar way to the patent network stratification parameter, it is unlikely that the trends observed in the patent citation network are due to general trends in search technology.

We have made a start in this direction by studying the network of citations to the premier physics journals, the various forms of the Physical Review published by the American Physical Society ("APS"). 8 ' The network of APS citations contains 378,077 papers and 3,615,892 citations. 85 To compare the evolution of citation behavior in this network to that in the patent citation network, we performed the same type of fits for the APS network that we performed for the patent network. The general behavior of citability that we observed in the APS network was similar to that which we observed in the patent citation network, with the citability function separating approximately into age-dependent and degree-dependent parts.

Just as we did for the patent citation network, we investigated the time evolution of the stratification parameter for the physics journal network. The results were quite different. The stratification parameters for the two systems are shown on the same graph in Figure 11 for comparison.

84. We obtained the data for this study from the American Institute of Physics, which is an umbrella organization of which the APS is a member. Although the a values for the physics journal citation network varied a bit over time and there is somewhat more noise in the journal network data, no systematic changes akin to those observed in the patent network occurred. During the entire period for which data is available, the stratification parameter for the physics journal citation network remained roughly constant following a slight decrease before 1975. This evolution is in sharp contrast to that of the patent citation network. We know that physicists are early adopters of computer technology; for example, they established an online preprint archive in 1991.86 If the general availability of computerized searching were responsible for the changes in stratification parameter we see in the patent citation network, one might expect to see some effect on the stratification of the physics journal citation network as well. No such effect is observed.

Citation practice likely differs between physics journals and patents, so this comparison goes only so far in determining whether the stratification changes in the patent network are due to substantive changes in issued patents, rather than to changes in patent citation practice. We certainly cannot yet reject the possibility that the changes in stratification for patent citations are due to some change in patent citation practice other than a general increase in computerized searching. More research is needed to gain insight into this question. For example, we might compare the evolution of the U.S. patent citation network with that of other citation networks, such as the European patent citation networks.

86. See arXiv.org, The Physics Archive (8/91), http://arxiv.org/new/physics.html (last visited Apr. 10, 2009).

Another approach to understanding the interplay between citation practice and stratification is to consider a more detailed model of the underlying citation practice. Such a model can be used to probe the underlying citation mechanism for the network evolution and thus can provide insight into what might be causing the global network features that we observe. If the observed network features are consistent with the predictions of an intuitive model of citation practice, for example, one can try to use that model to probe the observed network more deeply. If the observed features are inconsistent with a particular intuitive model, we may be able to deduce something about actual patent citation practice by determining why this is so.

Here we demonstrate that many observed features of the patent citation network are consistent with a simple, intuitive model of citing behavior, which was inspired by scholarly journal practice. The "forest fire model," proposed by Leskovec et al., 87 is based on a common sense search strategy in which relevant citations are found by following citation links recursively so as to cite documents that have been cited by or cite a known relevant document. The model is called the forest fire model because each document is only investigated once; thus, the links spread in a way similar to the way in which fire spreads between neighboring trees in a forest, burning trees as it goes. 88 In the application to a citation network, a completely new "fire" is associated with each new document added to the network. 89 While, in reality, authors select documents to cite according to their relevance to the subject matter of the citing document, this model approximates the citation process using a random probability of citing any document that is cited by or cites the most recently cited document. 9 ° More specifically, the model is as follows: when a new patent is added to the citation network, it first cites a fixed number of randomly selected other patents; then it checks each of the incoming and outgoing links of these original patents and with some probability cites those, too. Once a patent has been cited, it is considered "burned" and cannot be selected again as a jumping-off point. 9 1 The steps are repeated for the newly cited patents until the process "burns itself out." ' The model thus imitates how an inventor or patent examiner might find relevant patents based on the citation list and the "cited-by" list of a patent that has already been cited.

The predictions of the model are consistent with several features observed in the patent citation network. First, the model predicts that the frequency distributions of both number of citations received and number of citations made will be heavy-tailed, decaying approximately according to a power law at large numbers of citations. 93 This is exactly what we see in the patent citation network, as shown in Figure 12. The power law tail, which appears linear in the logarithmic graphs in Figure 12, is particularly evident in the distribution of number of citations made. 94 91. Id. 92. Id. 93. Id. The frequency distribution for citations received is a graph of the number of patents in the network that were cited k times as a function of k. Similarly, the frequency distribution for citations made is a graph of the number of patents that made m citations as a function of m. If a distribution has a "power law tail," the number of patents, N(k), with a particular number of citations, k, behaves as N(k) -k I for large k. For a more detailed discussion of such distribution functions, see Strandburg et al., supra note 1, at 1302-05.

94. The patent citation network distribution of citations received is not quite a power law at high numbers of citations because of the effects of aging, which are not taken into account in the Leskovec model. See id. at 1340 (describing the "long, slow decay" of the power law form). The heavy-tailed distribution for the number of citations received is intuitively understandable as a result of preferential attachment and is also a feature of any number of different models for network growth. 5 The extremely good approximation to a power law tail observed in the distribution of citations made, along with the fact that it becomes increasingly heavy-tailed over time, is more difficult to understand at first blush and is not a feature of most other network growth models. 96 The forest fire model provides an intuitive underpinning for this somewhat surprising observation.

The heavy tail on the distribution of number of citations received is due to preferential attachment. Because preferential attachment leads to a "rich get richer" effect, it results in a long tail in which some patents receive large numbers of citations. 97 The long tail on the distribution of number of citations made cannot be due to a preferential attachment mechanism because the citations made by a given patent are selected once and for all at the time the patent issues. It is thus somewhat surprising that the distribution of citations made does not follow a normal, bell-shaped distribution. 9 "

The recursive mechanism for selecting citations in the forest fire model provides a link between the preferential attachment in citations received and the distribution of numbers of citations made. Once a patent happens to cite an already heavily cited patent, it will tend to "burn through" a correspondingly large number of other citations, which it will add to its own list of citations with some probability. Thus, a heavy tail in citations received will be reflected in a heavy tail in citations made.

Other features characteristic of the forest fire model are also evident in the patent citation network. The number of citations made per patent increases over time, as shown in Figure 13. Meanwhile, the network "diameter," which is defined as the longest "shortest path" between two patents that are connected by "hopping" along citation links, decreases, as shown in Figure 14 We can use our method for extracting citability functions to perform an additional check on the forest fire model for the patent citation network. We do this by extracting citability in terms of a parameter that indicates whether or not a cited patent is a neighbor of a patent already cited by the citing patent. We then look to see whether these neighbors have higher citability than non-neighbors do. ° ' The calculation shows a strong preference for citing neighbors. Indeed, a neighbor of an already cited patent has more than twelve thousand times higher probability to be cited by the original citing patent than a non-neighbor patent. This model also has a very high goodness of fit. 02 The citability functions for neighbors and nonneighbors are shown in Figure 15. 101. There is one complication in our estimation of the citability function: we do not know the order of the citations in a given patent. Examiners and patent applicants do not record which citations are made first, second, etc. To avoid biasing our results in favor of neighbors or non-neighbors, we assume a random ordering of the citations made by a real patent. By generating synthetic networks, we found that this procedure yields an unbiased result, and on average the correct citability function is measured. Csirdi et al., Inverse Problem, supra note 39, at 24-25. 102. Id.; see supra note 65 and accompanying text.

These observations beg the question whether patent examiners and applicants actually use this recursive approach in determining which patents to cite. While the mechanism seems intuitively plausible-and something is needed to explain the observed features of the patent citation network-the model flies somewhat in the face of common understanding of the way patent examiners search for prior art using patent classification numbers and keyword searches. 3 Patent applicants also frequently employ patent search firms, which presumably do at least some of their searching on the basis of patent classification and keywords." 4 There are a number of possible explanations for the apparently good fit between the forest fire model and the characteristics of the patent citation network. One likely possibility is that those searching for prior art use a combination of search techniques, beginning with keyword or USPTO classifications and then following citations recursively. The forest fire model is consistent with this possibility because it does not account for the choice of the first patent cited but simply approximates that choice randomly. Another possibility is that keyword and USPTO classification searching lead to a citation network that is very similar to that which would have been obtained through a recursive citation approach. Perhaps, for example, the recursive approach produces a good fit because patents that cite one another use similar terminology or because keyword searching is itself recursive, as one keyword search may suggest further keywords to try.

To make further progress in understanding the underlying citation practice and its effects on citation networks, we will have to expand our approach. We need a better understanding of the practices of patent examiners and applicants. We also need to study various models of citation practice to determine whether the characteristic features of the forest fire model-power law "citations made" frequency distribution, increasing numbers of citations made per patents, and shrinking network "diameter"-are also observed in other models.

One intriguing clue to citation practice is already evident in our current analysis. In 2001, the USPTO began indicating on the face of issued patents which citations were made by examiners and which were made by patent applicants. 105 Thus, we can compare the frequency distribution of citations made by examiners with the distribution of citations made by patent applicants beginning at that time. Figure 16 illustrates this difference. As one can see in Figure 16, patent applicants are responsible for most of the occasions on which patents make large numbers of citations. Perhaps patent applicants rely more heavily than patent examiners on a recursive, citation-based method of looking for prior art. Of course, it is also possible that patent examiner citations are truncated because of time constraints that do not affect patent applicants. 0 6 The large number of citations made in some patents is not a simple result of the behavior of companies with large patent portfolios. The frequency distribution of number of citations made is shown in Figure 17 for companies with various sizes of patent portfolios. There does not appear to be any significant dependence on portfolio size.

We began this Section with the question of whether the observed evolution of the stratification parameter over time might be explained by some change in citation practice rather than by a change in the characteristics of issued patents. The comparison with the physics journal citation network suggested that the changes in stratification observed in the patent system were probably not the result of some universally-applicable change in search technology. The journal citation results also suggested that the systematic changes in stratification parameter observed in the patent citation network were not just "typical" variation for citation networks. In contrast to the patent citation network, the stratification parameter for the physics journal citation network was relatively stable over very long periods of time. Nonetheless, this one comparison is only a hint. It is still possible, for example, that citation networks for other journals or other patent systems reflect similar changes, which would undermine a conclusion that changes in the network of U.S. patent citations reflect changes in U.S. patent law.

The forest fire model plausibly explains some observed characteristics of the patent citation network, such as the power law frequency distribution of number of citations made, the ongoing increase in citations made per patent, and decrease in citation network diameter. Nothing we have observed so far suggests that this basic agreement with the forest fire model changed as the citation network evolved. 0 7 Thus, we have no indication of a basic change in citation practice that would explain the evolution of the stratification parameter over time. Nonetheless, a considerably better understanding of both the real world citation practices of examiners and applicants and the relationships between various models of citation practice and the stratification parameter would be needed to provide a definitive answer to the question of whether citation practice can explain the observed changes in stratification over time.

In this brief update to our network science study of the patent citation network, we have extended and confirmed our earlier observation that the citability of patents became increasingly 107. See, for example, Figure 12, which shows how the distribution of citations made has evolved over time.

stratified beginning in the late 1980s through the 1990s, suggesting that the least citable patents were increasingly trivial compared to the most citable patents. A plausible explanation for the observed increasing stratification is that the threshold for patentability decreased during that time. We then deployed an improved calculation methodology which uncovered an unexpected result: the stratification of citability hit a plateau around 2000, suggesting that the threshold for patentability may also have stabilized around that time. If this explanation for the behavior of the stratification parameter is correct, we are left with an open question as to the reason for this stabilization, which far preceded recent changes in legal doctrine that arguably tightened patentability standards. Notably, the plateau coincides with a change in patent office propensity to issue patents; the number of patents issued also leveled off at around the turn of the century, while the number of applications continued a meteoric rise.

We also investigated two alternative explanations for the evolution of the stratification parameter: (1) changes in the predominance of inventions from different technological fields; and

(2) changes in citation practice, perhaps as a result of digital search technology. While we cannot yet conclusively rule out these factors as explanations of the observed changes of stratification over time, our investigations so far do not lend support to either.

We thus consider it most likely that the evolution of the stratification parameter reflects changes in the threshold for patent issuance. If that is the case, it will be important to understand what happened around 2000 to account for the observed plateau. In the future, we can also use the stratification parameter to investigate the aftermath of important doctrinal changes, such as the Supreme Court's revision of the obviousness standard in KSR v. Teleflex 1°8 and the Federal Circuit's recent en banc restatement of the standard for patentable subject matter in In re Bilski.