5.1. Design Philosophy of Business Experiments
There are obviously many very broad classes of business questions that experimental methodology can provide answers. There is not enough room to cover them all. Instead, this chapter focuses on the low hanging fruit: areas that experimental economics has contributed immediately and significantly.
Contracting between manufacturer and retailers is one such area. Manufacturers operating in the contemporary market for technology products face a daunting task in designing effective incentives for their retailers. Channels of distribution are diverse, with new channels emerging, and demand fluctuations, market exposure, advertising, stocking, and product life-cycles are uncertain. The behavior of retailers is a critical element in whether a manufacturer achieves its business goals. Experimental economics offered a unique method to study and predict such behavior. This chapter described three studies, each was used to evaluate policies, in the following areas: return, minimum advertised price (Charness & Chen, 2002) and MDF. HP Consumer Business Organization, which was in charge of a $18 billion business, changed its contract policies based on the results of these experiments. These studies not only show how experiments can be used to help business decision making, but also illustrate how experimental designs and goals were affected by business needs.
In this class of problems, experimental method is perhaps the only reasonable alternative to field test. The complexity of a real retailer business environment makes a full theoretical analysis impractical. Even discounting the importance of scale (in the number of products, the number of retailers, and the number of manufacturers competing), there are non-trivial interaction between numerous economics structure and forces. For example, features in a typical retailer model used in one of these studies include retailers' heterogeneity in multiple dimensions, a consumer demand environment where the retailers competed in, an advertising model interacting with the underlying consumer model, product life-cycles, supply chain structures and multiple interacting policies. Although theoretical analysis of a simplified model can be useful in providing some insight for policy design, it is obviously not practical to create a theory model to encompass all the features that are deemed important. The major value of the experimental methodology is the ability to study the interactions of these features.
HP Labs developed in-house experimental economics capabilities instead of relying on academic institutions for consultants because business considerations make such consultation impractical. Business decisions must be made in a timely fashion, even if they are made with less than perfect information. HP typically develops its potential business in 3 to 6 months, depending on the cycle of contract and policy decisions. Experiments are often designed with the expectation that redesign and repetitions are unlikely, except in the most critical situations. Academic researchers generally want to establish statistical significance, necessitating replications and increasing the turnaround time.
Furthermore, time limitations often mean that exploring the parametric space fully is impractical. As a result, complexity of the field environment is preserved in many projects. For example, in the retailer experiments discussed above, many features such as stochastic supply, demand and delivery times, residual advertising effectiveness, and price reputation are included. The experimental environment was therefore quite complex. This design philosophy runs counter to standard academic experimental practice, where researchers prefer the simplest design that can encompass the modeling issues at hand. However, the goals of such studies are also modest. Although desirable, full identification of causes and effects are not required. It is more important to know whether a policy works than to know why it works. It is also important to identify possible exploitation of policies. From a business point of view, identifying such exploitation is unquestionably useful. It is less of a concern whether or not it is the equilibrium strategy for a retailer. In effect, we are employing subjects to find flaws in proposed policies.
This research strategy replies upon the accuracy of experimental models. Multiple safe-guards are built into the process to ensure experiments that are accurate reflection of the real business environment. Business experts are asked to evaluate all the features and assumptions included in the experimental models. Real business data, if possible, are used for calibration. All the models also go through a validation process in which business experts will play the game and offer feedback.
The reader needs to keep in mind that experimental results will at best provide an accurate evaluation of what will happen if something is done. They will be of limited utility as guidance of what to try. In the past, most policy alternatives were created in meetings where intuition and reasoning were the primary tools. An additional research strategy was developed to address this issue. When possible, we will develop a simpler model, theoretical, simulation-based and/or experimental, to provide some insights of what policies would be effective. These policies would then be evaluated along side with policies created by other means in full experimental models. This tandem strategy has only been applied on a limited basis due to resource and time constraints. The study of durable goods markets described below is a good example of how multiple methodologies were used to address the same problem.
It is not due to chance that all of the internal HP applications are dealing with issues of contracting between supply chain partners. There are very compelling reasons, business and intellectual, that make contracting a "sweet spot'' for experimental economics work. The importance of these policy decisions is the most obvious one. It is worth noting that these decisions are important because both the size of the business is big and that behavior can be substantially changed by them. As importantly, experimental results, such as the exploitation identified in the MAP experiments, can lead to actionable recommendations because a large manufacturer such as HP has substantial control over its contracts. A third reason is that while the environment is complex, it is not beyond the scope of a reasonable experiment design. In this issue, we are blessed by the fact that important characteristics of the environment can be captured by several rules that can be taught to subjects in a short period of time. For example, although retailers were heterogeneous, the number of types was quite manageable. All the models only considered the immediate business of the retailers surrounding several key products. It would be much more difficult to consider the interaction of different businesses (such as printers *and* PCs which have a different set of competitors) or the interactions of multiple points (e.g. including suppliers) in the supply chain. Furthermore, contracting is also an intellectually interesting area where game theory, micro economics theory, marketing science, behavioral economics, and even psychology can play a major role. It is more satisfying to researchers at HP Labs because this research theme was not planned but just emerged naturally driven by a set of consistent business needs. At present, experimental work is continuing to help HP explore contracting options with its reseller partners in multiple business areas.
There are other obvious application areas. Aside from the primary focus of contracting, HP Labs have also developed applications in other areas. The most noteworthy example is a study of durable goods market, collaboration between HP and another Fortune 20 company. This was an attempt to ascertain whether experimental methodologies could be successfully applied to a different industry. Thus, we selected a problem that was as different to the retailer studies as possible. Instead of focusing on game theory and behavioral effects, such as how a retailer can manipulate the rules of the contract, we focused on aggregate market behavior brought on by many small players. As a result, the durable goods market experiments (20+ subjects per experiment) are substantially larger than the retailer experiments (5-7 subjects per experiment). In Chen and Huang (forthcoming), an experimental model was developed to study the behavior of the secondary market for durable goods. This research created a general framework to address some of the unique issues in automobile marketing. Based on this work, experiments were developed to study whether the additional of a new channel of sales to the used-goods market would be beneficial or not? This manufacturer sold around 1 million used vehicles to dealers annually through life auctions. There was significant transportation costs associated with life auctions. Furthermore, life auctions did not necessarily capture all the potential demand. A new marketing program was designed to address these issues. Most of these used vehicles were "returns" from both consumers and commercial customers, such as rental companies, declining to purchase at the end of lease. The usual process was for the customer to return his end-of-lease vehicle to a nearby dealer. This dealer would then notify the manufacturer and send the unit to the auctions. The proposed marketing program works in the following manner. When a dealer notified the manufacturer the return of an end-of-lease vehicle, the manufacturer, based on some pre-established decision rules, may offer to sell this vehicle to the dealer at a dynamically generated fixed price. If the dealer declined, the vehicle would be shipped off to auction.
The proposal was to use auction prices of the previous month to determine the fixed prices for the current month. Previous internal research of this manufacturer has identified variables, such as mileage, color and an established way of measuring the condition of a vehicles, which are good predictors of auction prices under a specific regression model. This model, referred to as the floor price model, formed the basis of the pricing process. For any vehicle that might be offered, an "average" price was calculated based on the floor price model estimated from auction data of the previous month. The final fixed price offered to the dealer was the average price with a fixed percentage marked-up or marked-down. This process created a feedback loop. Yesterday's auction determined today's fixed prices. These prices would affect the acceptance rate of the offers. The acceptance rate would change the supply, as well as the demand, of today's auctions since the vehicles that were rejected would be sent to the auctions. Today's auctions would then again decide the pricing of tomorrow's offers. And the cycle continued.
This particular pricing feedback structure, in conjunction with the new marketing program, produced a significant more revenue to the manufacturers in the experiments. As a result, this manufacturer has implemented this new program.
Another example is the evaluation of the software procurement agent, AutONA (Byde, Chen, Bartolini, & Yearworth, 2003). Technology advancements have offered dramatically new ways of doing business. However, these technologies are often thrust into the business world with little understanding of how the economics would be affected. Sometimes, the effect can be wonderful (i.e. eBay) and sometimes it is a waste of capital (eToys, Webvan...). In situations where there is very little past experience, experimental economics offer a way to provide some guidance of the economics effects of applying new technologies. AutONA is a software that HP has developed for multiple one-to-one negotiations in a procurement setting. AutONA is a rule-based system that negotiates price and quantities with multiple suppliers on a one-to-one basis. The belief is that it will reduce the operational aspects of procurement costs by automating a significant part of the procurement functions. HP procurement was seriously considering turning over its negotiations to this software. The Achilles' heel of this argument is that it may not be true that this system could provide deals that are comparable to those made by human negotiators. Laboratory experiments were conducted to test whether human suppliers can take advantage of the robot buyer. Results show that although the robot buyer passed a simple Turing test, that is no human player could identify who the robot player was, it exhibited significant behavioral biases that resulted in worse prices when compared to those negotiated by human subjects. As a result, HP procurement decided not to deploy this system. It would be disastrous for HP if this system is deployed across an organization that procures around $4 billion worth of DRAM.
5.3. Implementation Issues: Software and Experimental Procedures
The carefully constructed research strategy will come crashing down if experiments cannot be implemented and modified in a timely basis. In the early days before the arrival of mass computing, simple games such as the prisoners' dilemma and simple auctions were implemented with pen and paper. With the advent of personal computers and networking technologies, more sophisticated games such as smart markets, combinatorial auctions, and information markets are possible. The complex nature of business experiments takes this evolution one step further. The needs for fast implementation and modification of economics scenarios have to be addressed if business decisions are to be made in a timely fashion. The strategy is to create a software tool that simplifies and automates as much of the implementation and experimental procedures as possible.
HP Labs developed a software platform, called MUMS, for the purpose of implementing economics experiments (Chen, 2003). The design called for two guiding principles. First, it needs to support many types of games, since there are many different economics models and business processes that are of interests. It is more cost-effective to invest upfront in a more sophisticated system, which can be used to support a wider range of models than to incur the costs of programming for each individual project from scratch. Second, the programming interface needs be simple so that researchers with little programming experience would be able to use it effectively. The ease of programming is the determining factor governing how efficient researchers can implement their models and execute their experiments. The main design challenge is to maintain the balance between ease of use and the flexibility of the system. A very flexible system such as the C++ programming language would be a terrible choice for someone without the right computing experience. At the other extreme, we can develop systems for particular games, such as auctions, but would lose the ability to use the system for any other types of games. We have decided on the approach of a script-language based system. The design of the language allows the user to define a game as a collection of high-level concepts: a set of players, inputs and outputs from players, and sequential logic that govern the rules of the game. Basic computing functions such as elements of interface design, networking, and database functions are taken out of the hands of the users.
The idea of script languages for particular games is not new. In chess, for example, there are several languages, which have been developed to simplify the knowledge acquiring process and to help creating better AI. (George et al., 1990; Donninger, 1996). The MUMS script language is a general purpose language, and have the common features found in other languages, such as data types, multi-dimension arrays, variables, functions, control statements, and so on. It has its root in the C programming language, although many complications have been eliminated. The syntax is similar although the lower level functions such as pointers are completely eliminated. The language hides all the details of the physical computing environment, such as a distributed network, where the game will be executed. For example, ''players'' are defined as elements in a script. A special global array variable ''Player'' is used to reference players. All input and output functions are called with reference to a player. The language treats players universally in definitions of games. During actual execution, the system determined dynamically how to map ''players'' to physical computers, which can be local or remote. It is even possible to map a "player" to a software agent.
Here are some examples of the types of games that have been implemented: retailers (oligopoly) games with business policies, multiple interacting durable goods markets, various types of auctions (one and two-sided), multiple one-to-one bilateral negotiation, information markets with Arrow-Debreu securities. All the research projects described below were also implemented in this system.
Although the primary focus of this tool is to support business experimental research, its design lends itself to research in other fields. This philosophy is not very different from goals of the Berkeley Xlab, which was created to serve multiple disciplines that study human behavior.
As a result, our development on computer technologies has gone beyond the issue of implementation of economics games. Scientists in both fields have started to exploit the synergies in economics and computer AI to ask the question of whether computers can be good economics agents either in place of or as support tools of human beings. At the focal point of these fields, which are different in nature but similar in goal, is an obvious need of a generalized platform to support games and agents. The MUMS system, which designed primarily to run economics experiments, has an additional feature that accommodates artificial behavior. The game scripts are agnostic to whether a player is going to be human. During run time, a human or a robot can be assigned to any role in a game. Switching between all human, all robot or partially human, partially robot experiments requires very little work.
In addition to developing software to implement economics scenarios, effort has been made to streamline other aspects of experimental operations. A standard set of procedures was developed from the point when recruitment of subjects is initiated to when the subjects are paid and escorted out of the laboratory. These procedures ensure continuity and consistency in laboratory operations when there are personnel changes in laboratory administration.
Training was also integrated into the procedures. Written instructions are posted on a special website created for the HP experimental economics program at a minimum three days before an experiment. All subjects are recruited either through an email list or notices posted on electronic bulletin boards. Subjects who are invited to participate are required to pass a web-based quiz, usually consists of multiple choice questions, before they are allowed to take part in experiments. It is particularly important in business experiments to recruit subjects that understand the mechanics of the games.
Web-based training and quiz helps to ensure the qualities of the subjects in this aspect. This process raises the issue of self selection because potential subjects can choose to sign up for experiments that they believe they can do well. This may be an issue for behavioral experiments that require a representative sample from the general population. However, self-selection usually is desirable for business experiments where we want the subjects to do well. Additional reference material and training time will be provided in the beginning of experiments.
Standard spreadsheet based database is used to keep track of the subject's profile, payment information and the history of participation. These data allow us to control samples of subjects as the need arise. A special checking account was created and all the subjects were paid by checks. In addition, a financial framework, internal to HP, was set up enabling the charging of experimental expenses to the proper accounts, since multiple business organizations are engaging in collaborative projects with the experimental economics program.
These mundane considerations, often ignored by researchers, can be determining factors of whether a business project is successful or not. This system ensures a consistent capacity of producing experimental data. In a world where timeliness and predictability can be as important as the validity of modeling assumptions, to be able to predict, plan and deliver experiments are crucial to whether the research will be useful and create significant impact in businesses.
There is an obvious parallel to be drawn between physical sciences and social sciences. If you want to build a quantum computer, you hire theoretical physicists to work out the underlying science. The experimentalists test whether the theoretical models are accurate in a laboratory. The engineers then create prototypes based on the experimental results. Economics, and in particular experimental methodologies, in my opinion, has developed to a point where a similar process of engineering has become possible for business processes. Furthermore, the advancement of software technologies will start to blur the line between an economics experiment and a business prototype in the future. In most of the research projects discussed above, business executives participated in mock experiments to get a feel for the experimental environment so that they could offer feedback. In some ongoing studies, plans have been made to go one step further and to use the experimental model as a war-game for business executives to test and develop their skills in an interacting environment. The day will come that software used in laboratory experiments can be scaled up and become prototypes of actual business processes.
There are two extremes of business engineering. The first is the tinkering of existing well-known business processes. This is akin to upgrading the design of an existing automobile. The engineers already know all the major elements to be included in a car. The work is to tweak each element so that they can work better together. There also may be an upgrade of a certain component such as the engine. The objective is incremental improvement or validating existing methods. The retailer experiments fall into this category. There might be a proposal of policy changes in one or more areas, but the fundamental way of doing business remained the same. The focus is to find out what works and what does not. The author would claim that there is a great need to use rigorous scientific experiments for this purpose. The need is driven by the way of how all these business processes came to existence. Rarely, they were designed from scientific understandings or even intuition. Most of them emerge from a kind of business evolution where any process that keeps a business from being unprofitable will survive. This is a far cry from optimality. In addition, experimental analysis is valuable to business decision-makers even in the cases where it validates the optimality of the status quo. Simply knowing there is no need to change saves all the costs of new marketing program and field tests. The author believes that this kind of business engineering, tinkering with existing businesses, will blossom in the next few years.
The second extreme is similar to trying to build nano machines, something completely new to the world. Information aggregation technologies such as prediction markets, combinatorial auctions, and quantum economics fall into this category. A cottage industry surrounding prediction markets has already mushroomed overnight.
Laboratory experiment is obviously not the only scientific method to answer policy questions. Theoretical (game theory or market theory) analysis and software simulation are the other two methods. These methods have advantages and disadvantages. Theory can provide invaluable understanding and guideline for actions but it becomes intractable quickly in a complex environment. Furthermore, theory based on rational assumptions sometimes is not the best predictor of human behavior. Computer simulation can scale up to large complex system easily, but their applicability depends on assumptions about the decisions human agents make in the field. There are obvious complementarities among these methodologies. My belief is that there will be a convergence of all these methodologies, including experimental economics, as arrows in a quiver to target business decision-making problems in the future.
The research discussed in this chapter has only begun to scratch the surface of potential business applications. Policy analysis, obviously not the only application area, can be extremely lucrative. Since most of the decisions made with the help of experimental economics scaled with the size of the business, the monetary impact of these projects was enormous. For example, the MAP policy designed based on experimental results has become the standard policy in a $12 (now $18) billion retail business for the last 4 years. Before the project in 1999, it was widely known inside HP businesses that one incident connected with the failure of the MAP policy at that time has cost HP around ten million dollars.3 The belief is that if experimental methodology was available prior to this incident, HP would have been able to avoid it. Obviously, it cannot be known how many more of this types of problems were avoided after the new policy was implemented. Similarly, the MDF study stopped the implementation of a new policy that distributed on the order of $300 million worth of funds. These numbers seem to indicate a very bright future for experimental business research.
The nature question is how HP can develop a coherent strategy to integrate experimental economics into its decision-making process. On this issue, HP suffers slightly from its size, which ironically, also makes experimental economics very compelling. HP can be viewed as a loose federation of business units with semi-autonomy.4 HP Labs is the central research arm of HP, which often acts as an internal consultant to the businesses. Despite all the past successes, it remains an ongoing process to educate and communicate to the businesses about the potential applications. In the past, all the projects were initiated because specific business needs, for example, the need to change the MAP policy, was brought to the attention of HP Labs. Initially, these contacts were brought about by regular HP Labs to HP business information exchange. HP Labs obviously will accept or reject based on its assessment of the importance of the problem as well as its scientific value. Due to the increasing visibility of the program in the past few years, more and more business problems are brought to us because one has heard that experimental economics may offer a new solution. This also means that, unlike in the beginning of the experimental economics program, substantial effort was no longer necessary to "market" the technology. Nevertheless, the resulting choices of experimental projects may not intentionally adhere to a strong theme and can scatter across many different types of problems and businesses.
On the other hand, as mentioned earlier, the ''sweet spot'' of these applications seems to be in the area of managing contracts with supply chain partners, particularly downstream resellers. HP Labs is developing a more efficient strategy for the application of experimental economics given the fact that (a) managing the reseller channel is important (b) contracts and policies need regular updating due to the changing nature of business environment, and (c) HP Labs experimental economics has the best track record in this area. The idea is to integrate regular laboratory testing of policies into selected business units, which are responsible for contracts and reseller policies. The experimental process will need to synchronize with the typically annual cycle of contract updating and runs in parallel to the real business. Experimental models will be updated according to changing business environment. Potential policy changes, or just the status quo policy, should be tested in the regularly updated environment before implementation. A system like this will not only be reactive to the policy problems that HP businesses identify and want to fix but also preventive to potential problems. To institutionalize experimental economics into business processes will be a major endeavor. At present, HP Labs is still at a very early stage of exploration although several business units have expressed interests in such a vision. Finally, HP Labs is also working with HP consulting to explore whether it is possible to create a consulting business. It is obvious that if such research is valuable to HP, it will also be valuable to many other companies.
Was this article helpful?
How To Get That Car With Little Or No Credit. Think you can't get a car because you have no credit? You need to read this report today. Are you in need of a car? The cost of a new car nowadays equals about what my parents paid for their first home. It's not a purchase to be taken lightly. You want to do everything possible to ensure you get a great deal.