WO2001048666A1 - System, method and business operating model optimizing the performance of advertisements or messages in interactive measurable mediums - Google Patents

Abstract

System and method for improving the performance of marketing campaigns in which advertisements or other messages are distributed over an interactive measurable medium such as the Internet (106). System for autonomously selecting attributes of a message to improve message performance in an interactive measurable medium. System includes a database storing message attribute configuration information, including future message attribute configuration information, and past message performance report information; an optimization engine (102) receiving the past performance information and making decisions based on the received past performance information and predetermined rules to generate future message attribute configuration information; and a message server proxy (112) that interacts with an external message delivery system over a communication link to configure the external system to deliver the future message based on the future message attribute configuration information. Method for improving the stage-to-stage performance of a message. Method includes: reading prior stage message which includes a cumulative number of trials and successes for a message from the prior stage; reading message performance results; computing a current message; and generating a current message allocation. Computer program and product are also provided.

Description

SYSTEM, METHOD AND BUSINESS OPERATING MODEL OPTIMIZING THE PERFORMANCE OF ADVERTISEMENTS OR MESSAGES IN INTERACTIVE MEASURABLE MEDIUMS

Related Applications

U.S. Patent Application Serial No. __/___, (Attorney Docket No. A- 69258/RMA), filed 2000 and entitled Method, Algorithm, and Computer

Program for Optimizing the Performance of Messages Including Advertisements in an Interactive Measurable Medium; U.S. Patent Application Serial No. __/__,

(Attorney Docket No. A-68760/RMA/LXM), filed 2000 and entitled System and Method for Optimizing the Performance of Email and Other Message Campaigns; and U.S. Patent Application Serial No. 60/173,689 filed 12/29/99 and entitled Optimizing the Performance of Emails, are related applications each of which is hereby incorporated by reference. Field of the Invention

This invention relates generally to systems, methods, and operating models for optimizing the performance of advertisements and other messages or content, and more particularly to systems, methods, and operating models for optimizing the performance of advertisements, messages, and other content in an interactive measurable medium such as a global network of interconnected information devices and appliances. An example of such a global network is the Internet.

BACKGROUND

This invention relates to executing and optimizing the performance of marketing campaigns in which advertisements or other messages are distributed over an interactive measurable medium such as the Internet. The marketing campaign involves a list of ad alternatives and a target customer population. The goal of the marketer is to allocate the ad alternatives to the customer population to optimize business objectives such as maximizing the number of responses received.

Various systems currently exist for the delivery and tracking of advertisements on the internet, for instance, ad servers for serving and tracking

"banner ads" on a web page. The users of these ad delivery or ad server systems have access to data on the performance of all the ads on all the locations. This data is updated by the delivery and tracking system on a periodic basis. The users are also provided with an array of parameters to configure the delivery and tracking system. In a typical conventional situation, an advertiser buys advertising space (ad space) on a number of web sites. The advertising space buy on each web site consists of a number of categories. Such categories may correspond to different sections within that web site, where a section is a specific web page or a set of related web pages within the site. A category may also correspond to keywords searched by a customer on a search engine. The term "zone" will be used to represent a unique site and category combination. There may typically be a number of banners that an advertiser wishes to deploy across these zones. A banner is either a graphic image that announces the name or identity of a site, a text link, or a rich media file. An impression occurs when an Internet visitor sees a banner. A clickthrough occurs when a visitor to a zone clicks on a banner. This redirects the visitor to the page on the advertisers web site.

We will use the term "placement" to refer to a particular banner-zone combination. The fraction of impressions (relative to the total number of impressions served for the particular zone) that should be allocated to a particular banner alternative is an important placement parameter that the advertiser can select and modify, to boost the advertising campaign performance.

Impressions can occur at any time, for example, whenever someone visits the appropriate page of a web site. However, the reports are typically updated at discrete times. We will call the intermediate time between two reports as a stage. At the end of each stage, the results are available for that stage. In particular, the following information is available for each placement: (1) the number of impressions delivered during a stage, and (2) the number of clickthroughs generated during a stage.

Additionally this information (that is, the number of impressions delivered during a stage, and number of clickthroughs generated during a stage) may be available separately for: (a) different time slots within a stage (e.g. hour of day, if each stage is a day); (b) different frequency levels i.e. the number of times that an ad was seen by the customer; (c) different operating systems used by the machine on which the customer is logged on; (d) different internet browsers used by the customer; and(e) different domain addresses of the computer on which the customer is logged on. This list is exemplary and not intended to be exhaustive.

In the current state-of- the-art, these reports are provided in printed form or in the electronic equivalent of printed form (for example, web-pages, spreadsheets, charts, or the like), and are manually analyzed by trained analysis personnel to derive new, improved advertisement configurations. For example, they are analyzed in an attempt to improve or optimize the clickthroughs generated by a pool of banner alternatives for a given zone, a given frequency level, and the like configuration information. This manual process is tedious and error-prone and has an inherent delay between the period of data collection and the time new advertisements are to be placed because of the large amount of data to be analyzed and a large number of parameters to be modified. Even if errors are not made and the user is able to overcome the tedium of the process, it is unlikely to yield optimal or even near- optimal recommendations for advertisement configurations. This is especially true in light of the typical delay of from a day to a week that lapse between data collection, analysis, and a new or modified ad campaign based on the analysis.

Optimization to provide an effective advertising campaign is in essence a multi-dimensional optimization problem, that by-and-large cannot be timely solved using conventional tools, methods, or systems. Furthermore, there are optimizations on multiple dimensions that are impractical to do manually and that will exacerbate the time delay. It is noted that these problems exist substantially independent of the type of advertisement or message, and that such issues and problems exist relative to advertisements for products and services, political campaigns, ballot measures and initiatives, media programming, lobbying, surveys, polling, news headlines, sports scores, as well as other directed marketing, promotions, surveys, news, information, other content generally, and the like.

Therefore, there remains a need for an automated system for optimizing allocation parameters for advertisement alternatives or message alternatives. There also remains a need for an automated system and method for rapidly and efficiently executing the optimized allocation parameters to place the advertisement or message on the Internet or other local or global communication system.

There also remains a need for a system and method that can learn and optimize across the various other parameters that can be reconfigured in advertisement delivery systems also commonly referred to as ad servers. There also remains a need for an ad server system and method that permits and advertiser to display different banners (or other content or messages) based on a time-of-day user web browsing profile which may include geographic location information, demographic information, or the like, as well as other user targeting information. Each banner can be described in terms of a set of attributes. An "attribute" is an important- characteristic of an advertisement alternative. For instance, in a given situation, a banner ad may be conceptualized as consisting of three attributes: banner background color, banner message, and banner picture. There may be several choices available for each attribute. We will use the term "level" to represent different choices along the same attribute. For example, there may be sixteen different banner background colors, ten different banner messages, and twenty different banner pictures from which an advertiser might ideally choose or select. The value or level along each attribute may have a profound and significant impact on a users response to the ad. For example, in an advertisement for a pet store, selection of a banner picture of a cat rather than of a dog may significantly effect the success of the ad.

Conventional ad server systems and methods treat individual ads as atomic units wherein constituent elements or parts, such as background color, message, picture, and the like are not separable. Hence, such elemental ad content attribute information is not available in conventional ad server systems and methods, which deal with the ad as an atomic unit. Therefore, there remains a need for system and method that generate a plurality of ads each having different elemental ad content attributes and which collects information on the success of such ads so that the system and method can learn which attributes are more effective in increasing performance of the message or advertising campaign, and then attempt to optimize the advertising campaign performance by generating an optimally placing a more effective pool of ads for subsequent campaigns.

There also remains a need for a system and method that can learn so as to optimize the objective specified by the advertiser. For example, one simple but frequently used optimization objective is to maximize the number of user clickthroughs from a banner or other message to the advertisers home-page or other site. Another objective may be to maximize the clickthroughs such that the total sales generated by all the consumers who visit the store after clicking on one of the banner alternatives is greater than a predetermined amount. There remains a further need to identify better performing or outperforming zones as quickly as possible so that the use and cost associated with poorer performing zones may be terminated in favor of better performing zones.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a diagrammatic illustration showing a first embodiment of an overall system and operating model according to the invention.

FIG. 2 is a diagrammatic illustration showing a second embodiment of an overall system and operating model according to the invention.

FIG. 3 is a diagrammatic illustration showing another detailed embodiment of the inventive system. FIG. 4 is a diagrammatic illustration showing an exemplary embodiment of an Optimization Engine according to the invention.

FIG. 5 is a diagrammatic illustration showing a flow chart of an exemplary optimization procedure according to the invention.

FIG. 6 is a diagrammatic illustration showing another alternative embodiment of the inventive system. FIG. 7 is a diagrammatic illustration showing a different alternative embodiment of the inventive system.

SUMMARY

This invention relates to executing and improving and desirably optimizing the performance of marketing campaigns in which advertisements or other messages are distributed over an interactive measurable medium such as the Internet. A message (marketing) campaign involves a list of message alternatives (ad alternatives) and a target customer population. The marketer's goal is to allocate the message alternatives to the customer population to optimize business objectives such as maximizing the number of positive responses received. The inventive system and method provides an automates message targeting and allocation scheme that is dynamic and adaptive

In one aspect the invention provides a system for autonomously selecting attributes of a message to improve message performance in an interactive measurable medium. The system includes means for receiving message attribute configuration information and message performance information, an Optimization Engine receiving the message performance information and making decisions based on the received message performance information and predetermined rules to generate future message attribute configuration information, and a message server to deliver messages based on the future message attribute configuration information.

In another aspect, the invention provides a system for autonomously selecting attributes of a message to improve message performance in an interactive measurable medium. The medium may be the internet. The system includes: a database storing message attribute configuration information, including future message attribute configuration information, and past message performance report information; an Optimization Engine receiving the past message performance information and making decisions based on the received past message performance information and predetermined rules to generate future message attribute configuration information; and a message server proxy that interacts with an external message delivery system over a communication link to configure the external system to deliver the future message based on the future message attribute configuration information. In one embodiment, the predetermined rules include past message attribute configuration information associated with the past message performance information. These predetermined rules may optionally further include message campaign data and optimization preference data.

In another aspect the invention provides a method for improving the stage-to- stage performance of a message in a multi-stage message campaign in an interactive measurable medium, where the method includes the steps of: reading prior stage message state pertaining to a prior stage in a message campaign; the prior stage message state including a cumulative number of trials and a cumulative number of successes for a particular message at the end of the prior stage; reading message performance results representing message trials and message successes from the previous stage based on the prior stage state; computing a current message state on the basis of the prior stage message state and the message performance results; and generating a current message allocation based on the current message state. Additional procedures and subsystems are provided that may be used separately and/or with external processing entities, or combined and integrated in various ways to yield different system configurations and processing distributions yet encompass the inventive concept.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

This invention provides system, method, computer program, and operating model for optimizing the performance of advertisements and other messages or content, and optionally for placing, or serving the advertisement or other message or content to a target device and ultimately to the user or operator of that target device. Among other features, the present invention provides an automated system for optimizing allocation parameters for each message placement and executes such placements on the Internet.

The invention can also learn and optimize across the various other parameters that are available for configuration message or ad server systems. An optimization algorithm is implemented as a computer program in an Optimization

Engine to accomplish this. For instance, ad server systems permit display of different banner ads based on the time of the day the profile of the user browsing the web site (e.g. their geographic location and other demographic information), as well as other targeting criteria. The inventive optimizing system and method learn and then set the appropriate parameters across multiple content and placement dimensions so as to optimize the objective specified by the message provider, such as an advertiser.

In addition, the inventive system and method gather user input and optimize for higher-level attributes that are more relevant to the advertiser. For instance, in a given situation, a banner ad may be viewed as consisting of three attributes: background color, message and picture. There may be several levels available to the user along each attribute. For instance, there may be five different background colors, four different messages, and five different pictures to choose from. The total number of banner ad alternatives which can be generated is equal to one hundred. The information about these attributes has not heretofore been available in the traditional ad server systems, which deal with the ad as an atomic unit. By allowing the user to give attribute information for the different banners, the optimizing ad server system can learn which attributes and which levels within an attribute are more effective in increasing performance of the advertising campaign. This information is also extremely useful in generating a more effective pool of banners for the next campaign and determining optimal placements.

Better performing or out-performing ads and zones for ads may also be more readily identified so that the use and cost associated with placing ads in poorer performing zones may be terminated in favor of placing such ads in better performing zones. This can be done, for example, by first estimating the clickthrough rate on every zone (or clickthrough rate of the best performing banners on the zone) followed by choosing the zones which generate the highest clicks per unit cost. Once the poor performing zones have been found, the advertiser would like to stop showing impressions on them. However, many of these ad zone buys are long-term (for example, from a month to two or more months) in nature and contractual obligations may not allow the advertiser to stop using the zone and stop paying for the impressions already bought. Even in the situation where a few weeks lead time is required to stop using a buy zone, the advertiser may swap the impressions bought with another advertiser such that the overall performance may improve for both the advertisers. The provision of the inventive method may therefore provide a method for interchanging ad space among advertisers or other businesses. These exchanges may be facilitated by a system and method that is automated and provides rapid analysis of performance data and rapid allocation of ads among the full pool of available ad allocation options.

Those workers having ordinary skill in the art will appreciate, in light of the description provided above and in greater detail below, that the system as described above can be easily extended to be useful for optimizing message types other than internet banner ads, including, for example: (1) other forms of advertisements such as emails, (2) message alternatives for marketing channels like pop-up windows, (3) web content alternatives for a web page, and (4) web layout alternatives for web page layout.

In one aspect, the invention is a system for the optimizing ads in an interactive medium and automating the configuration of an ad server system in a larger interactive medium, such as the internet. The system includes an optimizing engine and an optimization algorithm or procedure that takes as input an ad performance report and generates recommended message or ad configuration parameters for the optimal placement and targeting of ads. It loads the reports from the database and stores the parameter values into a database. The database stores user-configured information, Optimization Engine generated configuration information, and the performance report information from the ad server system. An advertising server interaction service is operative to loads performance reports from the ad server system and store them in the database as well as taking user or Optimization Engine generated settings and setting them into the ad server system. Finally, a user interface allows the user to enter campaign data and specifications about the optimization.

In another aspect, the invention provides a business or operating model for providing advertisement, message, or other content optimization and placement. The operating model is applicable for business-to-business electronic commerce as well as business-to-consumer and consumer-to-consumer electronic commerce (e- commerce).

Embodiments of each of the major elements of the system that contribute to the operation of the system, as well as certain optional but desirable elements are now described.

With reference to FIG. 1, there is illustrated a simple embodiment of the inventive system in the context of an internet advertising optimization environment, but the system and method readily applies to other types of messages on other measurable communication medium. Ad Optimization Service 102 receives data or information over the Internet 104 concerning ad alternatives and the performance results of these alternatives in trials of campaign stages so far completed. Procedures that utilize ad states and ad performance as well as any constraints on ad placement are used by optimization procedures within the Optimization Engine and service.

The advertiser 106 or other message manager is a point of control and may receive reports as to any unusual conditions or performance results. The advertiser 106 is also the most likely entity pay fees for the ad optimization service and to reap the rewards of the improved ad performance. Typically, however, the advertiser is not involved in stage-to-stage allocation decisions as the process is completely automated and does not require human analysis of decision making.

The ad target device 108 and the user of that device, typically a consumer with a personal computer and a web browser such as Microsoft Internet Explorer, receives one or more targeted ads (or other messages) and either ignores them (unsuccessful trial or impression) or positively responds to it (successful trial or impression) such as by clicking through a banner ad to the linked web site corresponding to the banner ad.

Ad content publisher 110 provides the content ad alternatives for which optimization of allocation is desired. A single ad content publisher may provide all of the alternatives or a plurality of different ad content publishers may provide the different alternatives.

The primary interaction of the ad optimization service 102 is with the ad server 112. The ad server 112 is responsible for actually placing the ads and for collecting performance results, such as success rate or clickthrough rate, for the ads so placed. It is from the ad server service 112 that ad optimization service receives performance information and to which ad optimization service sends ad placement allocation directives or recommendations.

In some embodiments, the ad optimization service 102 and the ad server 112 of the FIG. 1 embodiment may be combined into a single ad server and optimization service 114 as illustrated in FIG. 2. The ad content publisher may also be combined with the ad server and or with the ad server and the ad optimization service. Other combinations are also possible and not enumerated here.

This description applies, with changes that will be clear to workers having ordinary skill in the art, to messages other than advertisements. For example, the ad optimization service becomes the message optimization service, the ad server becomes the message server, and so forth.

The invention therefore also provides a business model and method or operating model. In one embodiment of the inventive business model and method of doing business, referred to as the web hosted model, the User-Interface and Message

Optimization are on a common web server site. The Optimization system and service interact with the underlying ad server system and a user of the system is levied a fee for each message trial or in the context of banner ads, for each impression. Volume discounts for different numbers of impressions or levels of use may advantageously be implemented. Revenues may also be collected on a per campaign basis for setting up the campaign and for analyzing the performance results and recommending allocations. Additional fixed or variable fees may be levied on a per-week or per-month basis.

In an alternative business model, the optimization system and method are provided to message agencies, such as for example, to ad agencies so that the Optimization Engine and service are integrated into a larger message or ad service system. Revenues are generated on the basis of the number of licensed installations and may be fixed, volume dependent, or charged as a percentage of revenues generated. Additional costs may be charged based on the size of system in terms of the total number of impressions delivered, size of the campaigns, size of the database, and the like factors.

Other aspects of the inventive business model may readily be appreciated from the description of the relationships between the Optimization Engine Service, the Database, the Message Server Interaction Service (MSIS), the Message Server, and the User or Message Control Agent. These relationships among these entities and the information and/or services interchanged between and among them may readily be accomplished on the Internet or other interconnected network of computers or information appliances. The high-speed interchange and measurable characteristics of the Internet as a communication medium provide technical advantages and effects that could not have been earlier achieved. One particularly useful benefit of the inventive system, and business operating model lie in its ability to rapidly measure the success of multiple messaging (e.g. advertising) schemes and select those scheme alternatives that outperform other schemes. Advantageously, the system and method permit poorer-performing schemes to be controllably retained (or discarded) so that in the event performance changes, they are automatically given a greater proportion of the message trials or impressions.

Reallocation of message alternatives may be made hourly, daily, weekly, or at any other time as the system is automated and at least in some embodiments directly collects performance information and allocates messages among message alternatives directly over the internet.

Another embodiment of the invention is now described relative to FIG. 3, which shows additional detail of the User Interface, database, Optimization Engine, message server interaction service, internet, and message server.

The User Interface (UI) 302 system gathers configuration information from the message sponsor, such as from the advertiser 106. The User Interface 302 may typically be a personal computer or other information device or information appliance that permits a user to enter data into the system and obtain reports to see what happened. In one embodiment of the system the user interface is a web site.

Generically, this configuration information includes where the message (for example the ad or email) should be displayed or otherwise presented, the different creative media that is to be displayed, information about the different attributes of the creative media, parameters about different options for the placement and instructions to the Optimization Engine 306 and optimization algorithm executing within the optimizing engine about which objectives to optimize for and other parameters, like "grace period", the period of time that the optimizing engine does not take action on the new creatives after they have been introduced. The grace period is an optional mechanism for the marketer to fine tune the optimization of his/her campaign. In an embodiment of the invention particular directed toward banner advertising on the internet, this configuration information includes a list of a banners, a list of sites and zones that the banners can be placed on, the different attributes of the banners such as background color, picture, message and animation, any specific placement constraints specified by the advertiser or ad agency attribute descriptions, and other optimization specifications and/or parameters. As with Message Servers (e.g. Ad Servers) 304, it is assumed that the message sponsor (e.g. advertiser or marketer) has already bought or otherwise contracted for impressions on these zones.

When the User Interface 302 system has gathered the above information, it schedules an Optimization Engine 306 targeted job in job scheduler 316 job table 317 of database 308 directing the engine service 306 to determine the initial set of allocations. This set of allocations is referred to as initial set of allocations because when new messages are placed into the system it needs to be determined where these are placed, and what fractions of the impressions for a zone that are going to be assigned to that message. The Optimization Engine service executing within Optimization Engine 306, on completion of the OE job writes an initial set of allocations for different placements into the database and schedules a Message Server Interaction Service 320 (e.g. an Ad Server Interaction Service) targeted job (MSIS Job) to modify the appropriate settings on the Message Server 304.

Periodically or according to other rules or procedures, the Message Server Interaction Service (MSIS) 320 pulls new performance report information from the Message Server 304. These performance reports contain data on performance metrics for the messages. In the context where the message is a banner ad on an internet web site, the performance metric may be, for example, the total number of clickthroughs for a given banner ad at a given zone and the total number of impressions delivered for that ad. The Message Server Interaction Service 320 stores this performance information into the Database 308 and schedules an Optimization Engine targeted job tasking the Optimization Engine service to generate a new set of message allocations.

The Optimization Engine 306 and the optimization algorithm 307 implemented by that engine provides the core piece of computer program code that implements the mathematical models to implement the optimization decisions. The engine service 309 wraps this Optimization Engine 306 to add the interaction with the database 308 and other parts of the system 300. One might conveniently use the two terms (Optimization Engine and optimization service) interchangeably but sometimes it is convenient to identify them separately.

All the historical message allocations that have been made and the performance results that have been collected to date for a particular message campaign are stored in the database. Based on the performance report information available in the database, the Optimization Engine 306 models the performance of different messages and message attribute alternatives. The Optimization Engine 306 uses this historical information to generate new message allocations from among the set of possible current message allocations that bias message impressions away from poor performing messages or message formats and/or attribute alternatives and toward what are or appear to be better performing messages or message formats. In the internet banner ad context, alternatives may include modifying the allocations for banner content (e.g. banner background color, banner message, and banner picture as applicable), zone (e.g. site-category combination), placement (e.g. banner- placement combination), different targeting attributes (such as for example, different banners for time of day, different geographical locations, browser types, frequency that ad has been shown, and any other demographic information that may be available about the end-user, their employer, or their operating environment), and the like message configuration information.

One embodiment of the invention will also generate these banner ads (and other messages) automatically based on these various attribute-alternatives and deploy the banner ads (or other messages) that it learns to be most effective. Thus, the inventive system provides for automated autonomous message optimization, generation, and placement.

The system and method of the present invention can also optionally provide reporting and alert functions to inform the message sponsor such as the advertiser of significant events that might require their attention. This is desirable so that there is some human review of anomalous situations. Examples of some of the events or situations for which a special report or alert might be issued include, for example: (i) no impressions were delivered even though purchased by the advertiser; (ii) a large change in the type or number of impressions delivered; (iii) a large change in click- through in a given zone; (iv) a large change in a specific banner's performance; and (v) the click-through performance achieved is below a specified contract minimum.

Other situations for which a report or alert may be generated may be specified for the system. Advantageously, the situations for which special reports or alerts are generated may readily be altered in real or near-real time, such as by interactive programming. The system and method also desirably provide the ability to monitor ones messages using the User Interface and to create custom reports.

The inventive system and method are inherently fault tolerant and can deal with a variety of potential failures including unavailability of the Message or Ad Server 304, and the unavailability of the most recent performance reports. In a situation where the message server 304 is unavailable, the system user (e.g. the message-sponsor) may still access past performance report information and current message allocations from the database 308 using the User Interface 302 instead of depending on the reliability of the underlying execution platform (e.g. the ad server). When the most recent or current reports are not available for what ever reason, the Optimization Engine 306 service can produce the best message allocations based on the information available in the historical data, and the time elapsed since the last report was obtained, even if all the reports, such as current performance reports which were due from the Message Server 304, but have not been received, are not available. A scheduling infrastructure is provided for scheduling tasks or jobs in the system, particularly for scheduling jobs to be performed by the Optimization Engine 306 and the Message Server Information Service 320.

Part of this scheduling infrastructure is provided by a Job Table data structure 316 defined in the database 308 that is used for scheduling the various jobs in the system. Jobs are scheduled for both the Optimization Engine and for the Messaging Server Interaction Service. In one embodiment, the data structure of the Job Table includes the following columns: Service Type, Category, Scheduled Time, Status, Start Time, and End Time. More, fewer, or different data structure columns or organization may be provided to accomplish the functionality provided by these scheduling information items.

"Service Type" specifies the target system module which is expected to execute this job, for example, the Optimization Engine Service (OE), or the Message Server Interaction Service (MSIS).

"Category" specifies the type of job which the service needs to execute. For example, in one embodiment, five job type categories are supported: (i) Optimization (OP), for the optimization engine; (ii) Performance Analysis (PA), for the Optimization Engine; (iii) Download Campaign (DC), for the Ad Server Interaction Service; (iv) Get Report (GR), for the Ad Server Interaction Service; and (v) Set Allocation (SA), for the Ad Server Interaction Service.

"Status" specifies the current status of the job within a service. In one embodiment, there are four possible values of this field: (i) New (N), indicating that the job has not yet been opened by the service; (ii) Work-in-Progress (W), indicating that the job is currently in progress; (iii) Success (S), indicating that the job was completed successfully; and (iv) Failure (F), indicating that the job was not completed successfully, for example, due to an exception or a system error.

The Job Table 317 also provides time information. Those workers having ordinary skill in the art will appreciate that data and time information may be provided in a variety of ways and formats. For example, absolute or relative time may be used, and that some time parameters may be computed from other time parameters. Therefore, the particular time parameters describe relative to one embodiment are exemplary.

The Job Table "Scheduled Time" field indicates the time at which the job is scheduled to be performed. This scheduled time may be manually entered, but is desirably determined by the system based on scheduling factors, such as for example the loading of the job target, the frequency of the jobs for a particular campaign, the expected time of availability of input data such as the availability of performance report data prior to an optimization, and the like factors. In one embodiment, Scheduled Time includes date information and time of day information and is encoded in conventional format.

The Job Table "Start Time" field indicates the actual starting time of the job. It may differ from the scheduled time. For example, if the job target has not completed a previous job or jobs or the loading on the job target is too great, the start of the job may be delayed somewhat. The "End Time" field indicates the time when the job was completed (successfully or unsuccessfully). Both Start Time and End Time may be considered to be optional parameters, however, in embodiments in which a service charge is levied based at least in part on the job computation time, knowledge of the actual start and end times permits computation of the elapsed computation time and provides an objective bases for service fees.

While each of these scheduling information items are present in the data structure of a preferred embodiment of the Job Table, it should be understood that not all of these information items is required in an operational system. For example, knowledge of the actual Start Time and End Time may be useful for monitoring the performance of the inventive system; however, the gathering or availability of such information does not affect the message optimization provided by the inventive system and method. In analogous manner, though knowledge that a job failed is advantageous so that the job may be rescheduled or other compensatory or corrective action taken, indications that the job is new or in-progress, ultimately have no effect on message optimization. Finally, though scheduling at predetermined times is desirable, the inventive system and method may be established in an environment where the jobs are merely scheduled sequentially or on a first-in-first-out basis. In this later instance, appropriate consideration and control should be exercised such that a first job generating results needed as inputs to a second job is scheduled and completed before the second job.

Table ! Exerr iplary Job Table for Database

The Engine Service of the Optimization Engine 306 polls the Job table 317 of the Database 308 at regular intervals (or according to some predetermined rules or schedule) for the list of pending jobs to process. The optimization engine 306 and the optimization algorithm implemented by that engine is the core piece of algorithmic and computer program code that implements the mathematical models to provide the optimization recommendations and decisions. The engine service wraps this Optimization Engine to add the interaction with the database and other parts of the system. For each job in the list of pending jobs, the engine service invokes the appropriate process in a new thread. More particularly, the engine service polls the database to identify any jobs that are targeted to the Optimization Engine (OE) and the Engine Service provided thereby. Those jobs in the Job Table having Service_Type=OE are targeted to the Optimization Engine and the engine service. (Those jobs in the Job Table having Service_Type=MSIS are targeted to the Message Server Interaction Service.)

Note that in some embodiments of the invention, the Optimization Engine 306 does not poll the database 308 and alternative mechanisms are provided to inform the Optimization Engine (or the message interactive service interface) that a job is ready for processing or execution. Therefore, either or both polling and non- polling implementations may be provided.

In one embodiment, the Optimization Engine 306 processes two categories of jobs: (i) Performance Analysis jobs and (ii) Optimization jobs. Normally, at least one performance analysis report is generated by a performance analysis job before an optimization job for the same message campaign is executed; however, an initialization set of performance data may be loaded in the database at the start of a new campaign so that optimization may begin. The initialization set of performance data may be established manually, generated by other analysis, be adopted from related campaigns, or by any other means. In one embodiment, such initialization set of performance data is a neutral or unbiased data that allows all candidate message attributes an opportunity to perform. In another embodiment, this data set is biased in favor of certain message attributes.

A performance analysis job uses the data on the current sets of allocations and placements, the historical performance of each of these and the most recent performance report of the current allocations and placements in play as inputs and generates a new set of message allocations and placements (or in particular embodiments for an advertising message, what banner ads should be turned on or off or allocated in which ratio and which placements should be added or deleted) as outputs. The message allocations and placements generated by the Optimization Engine 306 are stored in the Database 308. (In some embodiments message allocations and placements may be cached for later use in a memory storage device of the Optimization Engine, or directly affect the message server, without intermediate storage in the database.) An optimization job retrieves the message performance report information (or in particular embodiments directed to an advertising message, the ad performance report information) stored by the Optimization Engine in the Database and uses it as an input to generates message configuration parameters for the optimal placement and targeting of messages (or ads) as outputs, and stores them in the Database.

Recall that the term campaign refers to a set of media purchases over a period of time by a particular message sponsor, usually linked to a specific business objective, over which the creative messages need to be optimized It is not necessary for a campaign to have a time limit. For instance, the problem of optimizing the daily new as items at a news site is a continuous process every day, without a set termination time. For an advertisement (or ad), an ad campaign is a set of sites and sections within the site that a number of impressions have been purchased to be served over a period of time. Where the message is email, the email campaign is a list of email addresses and a time over which mail is to be sent to this set of addresses.

In the context of the invention, state refers to the encapsulation of relevant cumulative knowledge/information gathered over time since a given start time of the Campaign. Two types of state may be used by the Optimization Engine: global campaign state and individual cell state. Global campaign state refers to the encapsulation of those properties/aspects of the Campaign that are relevant at the Campaign, but not at the cell level. A cell is a representation of a specific placement of the Campaign. A placement in turn is the mapping of the message (or any web content) at a given site. A single message (or web content) can be mapped to multiple sites, and so can result in multiple cells. The Campaign is thus comprised of all the cells for the placements in the Campaign. Individual cell state refers to the encapsulation of the cumulative knowledge/information for that individual cell.

Upon invocation of the Engine Service, the Optimization Engine process executes one or more threads to accomplish processing for a given message campaign; and can kick off the processing of multiple campaigns at the same time. Processing is campaign based because each Campaign is completely independent from the other. Note that typically, the optimization and performance reporting are iterative in nature and that the longer the campaign runs, the greater the learning and optimization that can be accomplished. It is also interesting, that messaging trends and performance will frequently change over time and the iterative nature does not imply that the message criteria will become static once the campaign has run for a long period of time. Quite the contrary, as message saturation or staleness may occur after even a short period of time; such staleness being identified in the performance reports, and resulting in recommended changes by the Optimization Engine.

The thread-based nature of the Optimization Engine 306 process is now described. The Database 308 is the persistent store for the Campaign states as well as all information gathered on a regular basis in the form of performance reports. On arrival of a new report or reports, a job is scheduled for the optimization engine to process the new report or reports on a per Campaign basis. New reports can arrive for multiple Campaigns at the same time. The Optimization Engine 306 process polls the database 308 on a regular basis for jobs that are ready to be run. Once the

Optimization Engine gets a list of jobs that are ready to be run, it spawns a new thread 326 for processing each job independently of the other jobs.

From a top-level perspective and in simplified terms, processing involves reading the previous state of the Campaign (global state as well as cell state), processing the new results and updating the state of the Campaign, and finally generating the new recommendations. The new states and recommendations are also stored in the database.

The Optimization Engine and the process executed within the Optimization Engine are synonymous. The Optimization Engine 306 process executes the following steps: It reads in the Campaign state from the database and stores it into main memory data structures. Campaign state in one of the embodiments consists of a state for each placement, and a timestamp of the last state update. The state for each placement may, for example, comprise the cumulative number of impressions and the cumulative number of clickthroughs. The most current campaign state is stored in the database.

The Engine Optimization 306 then queries the database 308 for all new performance reports for that campaign since the last time the Campaign State was updated. Campaign state includes message state or data, performance results state or data, and allocation state or data. For each such report it: (a) Reads the report from the database 308 into Optimization Engine main memory 307 data structures; and (b) Processes the report and updates the campaign state data structures. Using the new state, and applying suitable allocation algorithms, it generates new allocations for the Campaign and stores the allocations into new data structures in memory.

One embodiment for performing the . allocation and performance optimization is described in co-pending United States Patent Application Serial No.

/ , (Attorney Docket No. A-69258 filed _ May 2000, and entitled Method,

Algorithm, and Computer Program for Optimizing the Performance of Messages Including Advertisements in an Interactive Measurable Medium, and is hereby incorporated by reference in its entirety. This optimization algorithm and procedure is also described below. It should however be understood that although embodiments of these particular algorithms and procedures may advantageously be used, the inventive system and method may alternatively utilize other processing routines, procedures, or algorithms in attempts to improve or optimize advertisement or message performance.

The process now begins a database transaction. The Engine Optimization 306 process stores the new Campaign states (including message state) from the data structures in memory 307 to the database 308. It also stores the new Campaign allocations (including message allocations) from the memory data structures to the database. Once these operations are successful, the process commits the database transaction.

This database commitment process prevents corruption of data in the database 308 (which is the only reliable data in the system) by the intelligent use of database transactions. A database transaction is a database feature that provides the application the ability to make multiple changes to the database 308 in an atomic fashion, that is, with the guarantee that either all changes to the database happen or that no changes happen. This prevents the possibility that the data in the database is updated only partially. In this particular embodiment, no other process in the system updates the Campaign states or generates allocations. This ensures the consistency and reliability of the Campaign states and allocations in the system.

Clearly, the above embodiment of the optimization procedure is just one possible embodiment of the algorithm that can be used for optimizing the total number of responses. Algorithms based on other techniques such as genetic algorithms, neural networks, machine learning, stochastic optimization or a combination of these techniques can be used instead of the algorithm described. The overall system architecture is independent of the technique used.

It can therefore be seen from the above description that the inventive optimization algorithm and procedure as well as indeed overall inventive system and method provide particularly advantageous benefits in the banner ad messaging context. The methodology applied to banner ads can readily be extended to other messaging situations, for example to email, interactive television, and the like.

In one embodiment, the core engine optimization procedure is implemented as computer program software procedures written in the Java language for maximum flexibility and also exposes a COM Application Program Interface (API). However, those workers having ordinary skill in the art in light of the description provided here will appreciate that various other language may alternatively or additionally be used.

The Message Server Interaction Service (MSIS) manages the interaction between the message product (such as for example an banner ad product, an email product, a splash page product, a news page product, a portal home page product, or other message product) and a message server. In one embodiment, the message server is an external internet based message server. In other embodiments the message server is integrated with the inventive system including the database, MSIS, and Optimization Engine. In either of these embodiments, the message served includes ads or advertisements and the message Server Interaction Service (MSIS) is referred to as the Ad Server Interaction Service (ASIS). The MSIS (ADIS) is responsible for retrieving performance reports from the Ad Server (or Ad Delivery Service) setting banner ad placements and allocations in the Ad Server, and downloading campaign settings from the Ad Server for use by the Optimization Engine and the User Interface.

The ADS advantageously utilizes a plug-in architecture, where the interface to the message server (e.g. to the Internet Ad Server) is a replaceable module which exposes a common application program interface (COM API). A "plug-in" is a module that implements this interface for a particular provider (e.g. a particular Ad Server). In the case of an external internet based Message Server, the plug-in is called an Agent (but this distinction between plug-in and agent is primarily semantic), accesses the Internet Message Server (or Ad Server) through the HTTP protocol. (Where communication links other than the internet are utilized, alternative supported protocols may be used.) The Agent within the MSIS simulates the interaction of a Internet Message Server customer, requesting pages of data via HTTP and parsing the needed information out of the returned HTML data. Other Agents would get the data in XML, or other proprietary format, based on what the underlying message server supports using HTTP:, FTP:, HTTPS: or another public or proprietary protocol. The information extracted from the returned data includes ad properties and names, zone properties and names, buy properties and names, campaign properties and names, performance data (such as clicks and impressions).

The parsing procedure includes the steps of deconstructing the HTML into its constituent tag structure and locating the desired data element at a known place within the tag structure. Standard parsing techniques of tokenizing the text stream and applying a simple grammar to parse out the HTML is used. Those workers having ordinary skill in the art in light of the description provided here will appreciate that any standard parsing technique could be used to accomplish the result. The MSIS (ADIS) interfaces with the Optimization Engine and the User

Interface via the database. The MSIS (ADIS) retrieves jobs targeted for the MSIS

(recall Service_Type=MSIS in the Job Table identifies MSIS targeted jobs) from the database and acts on the job, placing the results (performance report information) back in the database

The MSIS (ADIS) performs three types of jobs: Get Report, Set Allocations, and Download Campaign. For a Get Report (GetReport) job, the ADIS queries the Internet Ad Server for a detailed listing of impressions and clicks for each advertisement in the campaign. Recall that an impression is an instance of possible exposure of the message to the end-user, and clicks represent one possible performance indicia as to the successfulness of the impression. Other measures of success may alternatively be used - for example, whether the user was actually converted into a customer for the goods or services offered by the advertiser, what was the lifetime value of the customer to the advertiser or whether the brand of the advertiser was recalled more successfully after the exposure. The downloaded information is stored in a staging table in the databases, using for example, SQL server's BCP fast copy operations, and a stored procedure performs transformations on the data to put it in the correct databases tables from where it would be used by the other system components tables. The staging table allows the batch upload a set of data containing potentially invalid elements, and then the execute of a verification process against the existing data in the database. The other difference between the staging table and the report table is that the report table contains key constraints between it and other tables in the system where the staging table does not. This means that the staging table has the requirement that each row of data in it have a corresponding entry in some other table elsewhere in the database. The staging table cannot have this requirement, as we do not make the effort to insure that the related elements exists in the related tables before bulk-uploading the data. This is an optimization that allows fast upload of data into the staging tables.

For a Set Allocations (SetAllocations) job, the ADIS reads the current allocation settings for a campaign from the database and for each advertisement with a current allocation, the ADIS issues a request to the Internet Ad Server to modify the advertisement's properties in conformance with the allocation. Recall that an allocation specifies how the impressions are to be allocated amongst the different placements in the system. Typically, a Set Allocations job consists of a batch of allocations to be set for the entire campaign at the same time, rather than an allocation specific to one ad placement. Setting Allocations for more than one ad placement at the same time reduces the number of connections required with the external Message Server (Ad Server). The inventive system and method may alternatively be implemented without batched allocation setting.

For a Download Campaign (DownloadCampaign) job, the ADIS queries the Internet Ad Server for a data listing buys, sites, zones, and advertisements for a given customer. Recall that a zone is a unique site and category combination, and a "buy" is a combination of the zones on a site at which the ads have been purchased. The resulting data from the Ad Server is stored in a local database within the system and is visible to the message product (e.g. banner ad product) customer through the User Interface pages.

Having described the system and method generally, we now focus the description on several exemplary applications and a particular exemplary optimization algorithm and procedure that may be used with the inventive system. These particular applications and embodiments are described for purposes of illustration and not by way of limitation.

Exemplary Embodiment of an Optimization Engine and Procedure

As the inventive method is advantageously implemented as a computer program on a general purpose computer, such as for example a computer operating in an Internet or other network server environment, attention is now directed to an exemplary computer system which may be used in conjunction with the inventive method, procedure, computer program, and computer program product.

With respect to FIG. 4, computer 102 includes a processor or CPU 104 coupled to a fast random access memory 106 storing data 108 and procedures 110. Processor 104 is also conveniently coupled to one or more mass storage device 112 such as one or more hard disk drives of which many types are known, The mass storage devices may typically store information in one or more databases. In the context of the present invention, such databases may store performance data, allocation data, and other data and/or procedures or algorithms that are not currently being processed by the processor 104 or memory 106. Input/output devices 114 may also be conveniently coupled to the processor or other computer system 102 components for inputting data to the computer, retrieving results from the computer, monitoring activity on the computer or on other network devices coupled to the computer, or other operations as are known in the art. A network interface 116, such as a network interface card (NIC), modem, or the like is also conveniently provided.

Procedures 110 may for example, include an operating system 120, an optimization procedure 122, a state vector update procedure 124, a message allocation procedure 126, a pair-wise or other message alternative comparison procedure 128, as well as other procedures and algorithms 130.

Data 108 may for example include one or more state vector 142, results vector 144, allocations vector 146, constraints list or constraints items 148, stage discount factors or parameters 150, threshold values 152, proportional of impressions or trials factors 154, or other data or parameters 156.

Those workers having ordinary skill in the art in light of the description provided here will appreciate that the computer system 102 described here is merely exemplary and that various modifications may be made, or that other computer system architectures may be used in conjunction with the inventive system, method, and computer program.

Optimization Procedure and Algorithm Having described certain high level features of the inventive optimization procedure as well as providing an exemplary environment in which the inventive procedure has particular applicability, attention is now directed to details of a particular embodiment of the inventive algorithm or procedure. This embodiment is described in the internet banner ad context. In this application context, three principal types of data are used by the inventive procedure: (i) State Vector, (ii) Results Vector, and (iii) Allocations Vector which includes a Constraints List. (In one embodiment, the Constraints List is separate from the Allocation Vector, but used to generate the Allocation Vector.) These data types are described in turn. The term vector is used to refer to one form in which the data is stored and/or manipulated.

Zone/banner States and State Vector. For each combination of a zone and a banner, the inventive procedure retains a "state vector" that records all of the information that has been collected on the given banner at the given zone

(zone/banner combination). This state vector is updated at the end of each stage of the campaign to reflect the results of that stage. In particular, the state vector contains the history of impressions and clickthroughs for the given zone and banner.

Results Vector. The results vector is a vector of the results of the most recent stage of the campaign. It contains the unprocessed impression and clickthrough data for each zone/banner combination. If a banner was not displayed at some zone during the most recent stage, the results vector indicates this fact.

Allocations Vector. The allocations vector contains the recommended allocation of banners to impressions for the next stage of the campaign. Intuitively, these allocations are expressed in percentage terms. However, in practice the recommended allocations may be re-expressed in a form acceptable to the ad server. Hence, some translation or transformation may be provided. The Allocations Vector may also include a list of constraints, though the constraints list may be separately provided.

A list of any constraints on the allocation of banners to impressions is stored in the constraints list. At least two types of constraints can occur. First, the user can specify the minimum number of banners that must be retained and displayed at each stage of the ad campaign. Second, the inventive procedure implemented as a computer program on a general-purpose computer retains a table of "banner exclusions." The table of exclusions, includes for example, a table defined as a data structure in memory that indicates when a given banner should not be shown on a given zone. This table may include one or more entries or may be empty.

The inventive procedure is advantageously run at the end of each stage of an ad campaign. It allocates the available banner impressions or banner ad alternatives at each zone for the next stage among the available banners. That is, it generates a strategy or recommendation for displaying banners during the next stage of the campaign. In developing this recommendation, the algorithm uses the results from all prior stages. (Though in at least one embodiment, a selected range of data from older stages may optionally be discarded, and in a different embodiment, not all data from all stages has the same weight or value in the recommendation.) One objective is to allocate the banners at all stages so that the total number of positive results, such as user clickthroughs on a web site, recorded at all zones by all banners during the campaign is maximized. Other operations may optionally be performed, for example, during the ad campaign, the client advertiser or marketing manager may wish to delete or add a zone or banner.

An embodiment of the optimization procedure 202 is now described relative to the flow-chart diagram of FIG. 5. First, the prior stage message state is read (Step 204). Next, message performance results are read (Step 206). These performance data or results are typically received from another source, such as an ad service and stored (Step 208). The current (or new) message state is then computed (Step 210) using, for example, the state vector update procedure described in greater detail elsewhere herein. Message allocations, possibly with some format transformation to adapt them to the particular format needs of a receiver, are generated (Step 212) and sent to the receiver such as a message service (Step 214). As a primary application of the inventive method is for Internet advertising, the message service will frequently be an ad service. As the message campaign (ad campaign) is generally a multi-stage campaign, the procedure is repeated until all stages in the campaign have been completed. This requires that the prior message state used in step 204 be replaced with the current message stage (Step 216) generated in step 210 before the next iteration (Step 218). New message performance results received in step 206 are also used for the next iteration.

While it is generally intended that performance data or results are received for each prior stage before allocations are generated for the current or next stage in the campaign, and the inventive procedure benefits from such timely receipt, the invention does not require such timely receipt. In the event that expected prior stage results are not for some reason received as expected, the inventive procedures are sufficiently robust that the new allocations are merely based on the previous performance data or results. Therefore, when new performance results are not available, the old performance results are read or otherwise used in the computations. In some instances this may result in the same or substantially the same allocations as the previous stage, however, where discounting is applied, the allocations may not be the same. The extent of the difference may typically depend upon the discounting function and the historical prior performance results to which the discounting is applied.

Each of these operations is now described in greater detail. For purposes of clarity of description, it is assumed that the procedure is applied relative to a single zone. If multiple zones are available, the procedure is applied to each of the multiple zones one-at-a-time to derive the allocations for that zone. The banner ad performance may improve or deteriorate over time so that generally the performance of an ad is non-stationary in a statistical sense. There are a number of potential ^' reasons for a banner to have non-stationary behavior in the underlying performance.

For example, a banner that promises overnight delivery may be quite effective shortly before Christmas, but much less effective on the day after Christmas. Even absent a particular identifiable event, the performance of ads may change over time. For example, Internet visitors may see a given banner too many times and beyond some point the banner loses all attraction. As the fraction of visitors who have seen the ad too many times increases the performance of the banner deteriorates. State Vector Update Procedure

One embodiment of the inventive procedure optionally but advantageously uses a "discounting" of the performance data in computing or updating the State Vector. Such discounting may be in the form of a binary step function where data older than a certain stage is simply ignored, in the form of a linear or non-linear weighting function, or according to any other weighting or discounting scheme. In one embodiment, the discount is in the form of a geometrical discounting, that is, at the end of each stage all data (such as for example, the number of impressions and number of clickthroughs) is multiplied by a one-stage discount factor beta, β, where beta is a number less than one (β <1). Thus, data that is n stages old at the time when the procedure is executed will end up being multiplied by beta raised to the n power (β ⁿ). In this manner newer performance data is weighted more highly than older performance data. While not true in all circumstances, it is generally true that newer performance data provides better guidance as to what allocation scheme will be successful in the next stage than will older performance data. In any particular situation where this generalization does not hold true, different discounting schemes, including for example a discounting scheme that weights some segment of performance data more heavily than newer performance data, may be applied. If no discounting is desired, beta may be set to 1 (β=l). (Empirical and simulation studies have shown that a value of the one-stage discount factor (beta) of about 0.9 works well for a range of synthetic and real data sets, though values between about 0 and about 1 may be used, more typically between about 0.5 and 0.99, more usually between about 0.8 and about 1.0, and even more usually between about 0.85 and 0.95 may be used.)

Let Sj(t) denote the total number of discounted cumulative clicks and n;(t) denote the discounted cumulative number of impressions for banner i at the end of stage t. Let imp;(t) denote the total number of impressions and click;(t) represent the total number of clicks and impressions for a particular banner i in stage t. Then the discounted cumulative clicks S;(t) and the discounted cumulative impressions n,(t) are given by the following expressions:

Sj(t) = β Si(t-1) + clicki(t-l), for t > 2 ; and n_;(t) = β n_;(t-l) + impi(t-l), for t > 2;

where s;(l) = 0 and nι(l) = 0. Thus, s;(t) and nι(t) represent two components of the State Vector. Discounting reduces the impact of old data and allows the inventive algorithm to be more responsive to new data that may reflect changes in the clickthroughs rates or other performance indicators.

The above expressions are specialized for an embodiment in which the performance metrics are based on numbers of "impressions" and "clickthroughs," where for a particular impression the viewer has the binary choice of either clicking- though or not clicking through for each impression.

In the set of equations above, the discounting is provided by the factor β.

The above expressions can be easily generalized to an arbitrary or generalized discounting function G{...} which uses the clickthrough impressions from the previous stages to derive the discounted cumulative clicks s;(t) and the discounted cumulative impressions n,(t) using the following expressions:

s;(t) = G{ clicki(l), ..., clicki(t-2), clicki(t-l)}, for t ≥ 2 ; and

n_;(t) = G{ impi(l), ..., im_Pi(t-2), im_Pi(t-l)}, for t ≥ 2 ;

where s;(l) = 0 and n;(l) = 0. In this generalized discounting function G{...}, it is noted that none, some, or all, of the clickj(t) and or impj(t) may actually weighted or discounted, though in the preferred embodiment each is discounted.

In particular functions which give more weight to more recent values of S; and n; are of interest. When a different binary response performance indicator than clickthroughs such as positive responses or "successes" are used, and a more general term applied to impressions such as "trials" is used, along with a generalized discounting function H {...}, the general expressions for the total number of discounted successes s;(t) and the cumulative number of impressions n;(t) are given by the following expressions:

s;(t) = H{ successj(l), ..., success;(t-2), success;(t-l)}, for t ≥ 2 ; and

nj(t) = H{ triali(l), ..., triali(t-l), triali(t-l)}, for t > 2.

where Sj(l) = 0 and n;(l) = 0.

Thus, sj(t) and n;(t) represent the State Vector for banner i. As noted above for G {...}, none, some, or all, of the success;(t) and or triali(t) may actually weighted or discounted by the discounting function H {...}, though in the preferred embodiment each is discounted.

In some cases, before the inventive system or method are used for optimization, a campaign may already be executing. In this case, information may be available regarding the performance of the banners. This performance information can be used to initialize s;(l) and n;(l) as follows. A cumulative or discounting procedure as described in the previous section can be use to estimate s;(l) and n;(l) in the same fashion as s;(t) and n,(t) are calculated, for example, by either using the cumulative sum or some form of discounting. Those workers having ordinary skill in the art in light of the description provided here will appreciate that there are several similar variations of the above basic strategy.

Even when the campaign is started fresh, a campaign manager may have used the same banner in an earlier campaign or may have a prior judgement about the performance of the banner. In this case the value of s;(l) and nj(l) for banner i can be set as follows. The manager can provide an estimate of the average performance of mean; for the banner i over the previous campaigns. He/she may decide that the information from the previous campaign is worth Nj impressions. In this case s;(l) can be set to (mean; x Nj) and n;(l) can be set to N. The setting of s;(l) and n;(l) as described above can be used to incorporate managers prior experience or judgement of the banners. Larger values of N; imply that the manager is more confident about extrapolating the future performance of the banner based on his prior experience or judgement.

Those workers having ordinary skill in the art in light of the description provided here will appreciate that there are several similar variations of the above basic strategy-

Allocation Computation Procedure

Having now described several embodiments of the state vector update procedure, attention is now directed to the allocation computation procedure. At the initial stage (when no information has been collected on the banner ads), the inventive procedure may allocate the impressions in any one of several possible manners. Such initial allocation may be based on objective or subjective criteria. In one embodiment, an even or uniform initial allocation of impressions are made from among the available banners. That is, if there are a number b banners, then each banner will receive 1/b of the total number of allocated impressions.

At each of the remaining stages, the inventive procedure uses the available performance data (for example, the record of impressions and clickthroughs from the earlier stages), allocation information, and other information that may be available on the banners to divide the available banners into two classes: (i) a first group referred to as the contenders, and (ii) a second group referred to as the non- contenders. The "contenders" are the banners that are still in contention according to some criteria to be declared the best banner. The "non-contenders" are the banners that appear out of the running for best banner based on the available information. There may also be a (iii) third group of "unavailable banners" if the constraint list indicates that some banners cannot be used on the given zone during a particular stage. Banners that cannot be used for the next stage during which the new allocations are to be made would be placed in the third group, as they cannot be displayed during the next stage independent of how well they might perform. Hence, those ads that are available for the next stage are divided into either the contender or the non-contender group. Any one of these groups may have none, one, or a plurality of ads.

Given this division of the banners into contenders and non-contenders, one embodiment of the inventive procedure allocates the impressions among the banners as follows. Let γ<l be a constant that represents the proportion of impressions that will be allocated to the non-contenders at each stage. This γ is a constant selected at the beginning of the campaign. The value selected is not critical as the inventive procedure will reallocate between contenders and non-contenders at subsequent stages. The value may also be selected so that impressions are not allocated to non- contenders. Let b represent the total number of banners, bi denote the number of contenders and b₂=b-b₁ the number of non-contenders. Then each contender receives the fraction (l-γ)/b₁ of the available impressions, while each non-contender receives the fraction γ/b₂ of the available impressions. Using this approach, most of the impressions are spread evenly among the contenders and a few impressions are used to continue monitoring the non-contenders. As the campaign progresses the number of contenders will generally decrease over time since the inventive system and method will have more information upon which to judge which banners are more successful and which banners are less successful.

In an allocation task where there are five possible banners {Xj, X , X₃, X , X₅}, three of which are contenders {Xi, X , X₃} and two of which are non- contenders {X₄, X₅}, the Allocation Vector may therefore take the form of (0.3, 0.3, 0.3, 0.05, 0.05) where the contenders each have a 30% proportion and the non- contenders each have a 5% proportion of the total. In a situation where there are six possible banners but one in constrained by the constraint list for a particular stage, that banner would have a zero proportion, for example, allocation vector of the form (0.3, 0.3, 0.3, 0.05, 0.05, 0.0).

Exemplary Pair- Wise Test Procedure to Identify Contenders and Non-Contenders

In one embodiment, the division of banner ads into contenders and non- contenders is based on the following pair-wise test procedure for comparing two banners. When more than two banners are involved in a campaign a pair-wise test is used for each possible pair or combination of banners.

For each pair-wise test, suppose that two banners are given and let p; (for i=l, 2) denote the unknown clickthrough probability of the i^th banner. Let n; denote the total number of impressions for the i banner, s; the total number of successes and fi the total number of failures. (So, s; + f; = n;). Finally, define:

V = (njn₂ (si + s₂)(fι + f₂)/(nι + n₂)³

It can be shown that Z is approximately normally distributed with variance V and mean ΘV, where: θ

The pair-wise test attempts to determine or test whether the two unknown clickthrough rates for the two banners under consideration are equal. If they are equal, then θ= 0 and Z is normally distributed with mean 0 and variance V. The distance between the observed value of Z and 0 (the mean of its distribution) measured in standard deviations is given by Z divided by the square root of V. If

this statistic (ZΛ V) is greater (in absolute value) than some cutoff threshold value , then the procedure concludes that the click-through (or other performance indicia) rates are significantly different from each other. In one embodiment, the user (such as for example the marketing manager) specifies the cutoff threshold value prior to the campaign; however, the value may be modified during the campaign. In one embodiment, the value of the cutoff threshold value is selected to be between about Vi and about 1, based on empirical simulation results, but other values may be used.

The pair wise test is applied to all pairs of banners. If a particular banner loses one of these comparisons (that is, if a banner is determined to have a significantly lower click-through rate than another banner), then that banner is designated as a non-contender. This pair-wise procedure guarantees that at least one contender will remain.

Exemplary Procedure For More Than Two Sets of Alternatives

The algorithm described above divides the alternatives into two sets: contenders and non-contenders. The basic approach can be extended to decompose the alternatives into more than two sets. Consider the pair-wise test between alternative i and alternative j. Let Compare(ij) represent the Z(ij)/sqrt(V(ij)) statistic where:

Z(i j) = (n_jSj - njS_j)/(ni + n_j) , and

V(i j> (mnj (_Si + Sj)(fi + fj)/(ni + _nj)³

The value of Compare(ij) is positive if the performance of alternative i is better than alternative j and is negative if the performance of banner j is better than banner alternative i. Define Perf(k) = max_1≤i<n Compare(i,k). The algorithm deems an alternative k to be a non-contender if Perf(k) is above the cutoff threshold, otherwise k is a contender.

The above basic approach can be extended to decompose the alternatives into more than two sets. Under these circumstances, the alternatives are divided into several sets based on the corresponding Perf values. The impressions are allocated to different alternatives such that the alternatives for sets corresponding to higher values of Perf get smaller numbers of impressions. However, the number of impressions allocated to all alternatives within a set is equal. In the extreme case, the number of sets is equal to the number of alternatives. In this case, each alternative gets a different fraction of the total number of impressions (or trials).

This approach is formalized in the next section. Exemplary Procedure for Dividing Alternatives into Multiple Sets

The value of Compare(ij) is positive if the performance of alternative i is better than alternative j, and the value of Compare(i j) is negative if the performance of banner j is better than banner alternative i. Define a performance function (Perf(k)) to be the maximum of Compare(i,k) over the range of i from i=l, ..., n; that is, Perf(k) = maxi_≤i_≤n Compare(i,k). The algorithm deems an alternative k to be a non-contender if Perf(k) is above the cutoff threshold α, otherwise k is deemed to be a contender.

One embodiment uses the value of Perf(k) to segment the banners into two or more sets (r > 2 sets). In this context, larger values of the performance function Perf(k) as defined here will generally imply poor relative performance of banner alternative k. Let there be user defined thresholds αi, α , ... α,(_r-i) such that a first threshold α₀ =0, and the remaining thresholds are each greater than α₀, that is 0=α₀ < oti < α₂ <...< ot_(r-1) < α_r=∞. The algorithm assigns a banner k to set j, where l≤j≤r, if αo-i) < Perf(k) < α,-.

The impressions are allocated to different alternatives such that the alternatives for sets corresponding to higher values of Perf get smaller fraction of impressions. However, the number of impressions allocated to all alternatives within a set is equal. In the extreme case, the number of sets is equal to the number of alternatives. In this case, each alternative gets a different fraction of the total number of impressions (or more generally, total number of trials).

Those workers having ordinary skill in the art in light of the description provided here will appreciate that there are several similar variations of the above basic strategy that can use the Compare function to segment the banner into multiple sets.

One embodiment of the allocation algorithm is as follows. Define constant propu prop₂, prop₃, ..., prop_r such that propi < prop₂ < prop₃ < ... < prop_r. Prop_j represents the proportion of total impressions assigned to all the banners belonging to set j, where l≤j≤r. Let the number of impressions belonging to set j be given by numj, where l ≤j ≤ r. (Note that the Sum of numj = total number of banners). Then banner belonging to set i is assigned the fraction of total banners = Prop; /{∑i_≤j_≤r (numj x propj)}.

Unlike the allocation strategy for two sets, the above allocation does not guarantee a fixed fraction of impressions to be allocated to relatively under- performing sets. However, it guarantees that banners belonging to relatively poor performing sets get smaller number of impressions as compared to relatively higher performing sets.

Those workers having ordinary skill in the art in light of the description provided here will appreciate that there are several similar variations of the above basic strategy using the Z and V statistics for the pair-wise comparisons which can be used for decomposing the alternatives into two or more sets.

Procedure For Adding or Deleting Zones or Banners As mentioned previously, additional operations may be required when zones and/or banners are added or deleted. If a zone is deleted, then the number of available impressions for this zone is set to 0, and no further banners are allocated at this zone (unless the zone is re-introduced later). However, the data collected during earlier stages at this zone can be retained. Recall that in some embodiments, the earlier collected performance data is aged or discounted so that more recent performance data is given greater weight in the computations. If a banner is deleted, then this information is added to the constraint list so that the banner is no longer placed or not placed for a particular zone. Note that a banner may be deleted at one or more zones but remain for placement at other zones. After zones or banners have been deleted, all inputs to the inventive allocation computation procedure (for the remaining zones and banners) are available and allocation for the next stage is determined.

If a zone is added, any constraints that will affect the new zone are specified. If a banner is added, the new state vector is initialized (for example, with an empty history) and any constraints on the banner are specified. In one embodiment, at the next stage, this added banner is treated as though it were as good as the best existing banner, and is allocated the same number of impressions as the best banner (for each zone). However, this can be easily generalized to allocating more (or less) impressions than the best banner. Following this initial stage of data collection for the new banner, the allocation algorithm is applied as usual.

Additional Embodiments

The above described embodiment (FIG. 3) provides for an external Message Server coupled to the database via the Internet through the Message Server Interaction Service. This approach is advantageous because it allows a loose coupling between the optimization system and the Message Server where they may be operated by separate business and operational entities, for instance. This is also necessary when for various business, security or operational reasons the Optimization Service does not have direct access to the database that the Message Server uses. Another advantage of this embodiment is that it abstracts out the functions of each type of message servers so that the optimization system can work over different implementations of these servers possibly done by different business entities. It also allows the user experience with the overall interface presented by the optimization system to be independent of the Message Server, and can continue to operate even when the Message Serve configuration is unavailable. However, several other alternative configurations of the Message Server, Message Server Interaction Service, and Database may be used.

In one of these alternative configurations 200 illustrated in FIG. 6, the Message Server 204 is part of an integrated system 202 including the Database 206, Optimization engine 208 and the User Interface 210. In this system 200 the Optimization Engine 208is integrated right into the Message Server 204 and they share the same database. The User Interface 210 is integrated directly into the user interface of the Message Server - additional user interface elements are added on to the Message Server 204 user configuration interface to configure the optimization and get the optimization reports. This second alternative configuration is advantageously used when the Message Server 204 and the rest of the optimization system can closely be integrated without business or operational issues. It has the benefit that the database parameters do not need to be copied into another database (other than optimization that may be done on the message server side for purpose of run-time performance).

The detriment to integrating the Message Server 204 with the Optimization Engine 208 is that this may be infeasible because of business and operational necessities. Further more, this system will typically lose the flexibility of the being able to integrate with multiple message servers that my have different database schemas.

In this embodiment, the Message Server 204 directly enters the message performance reports into the database 206, and takes over all of the database activity that the MSIS of the embodiment of FIG. 3 would perform. The part of the MSIS that talks to the Message Server 204 is no longer needed since the Message Server can directly perform the functions internally. The need to get reports, download campaign settings, and communicated user and optimization engine settings now being performed within the integrated system or being eliminated. The functions of the optimization system would still exist in a similar fashion. The User Interface 210 component for interacting with an external user 218 would be altered to add just the additional parameters and reports that the optimization system needs or presents. Campaign Settings and Optimization Engine Generated Settings are established by the OE Engine Service 208 and communicated in a similar manner to the integrated database component 206. A user input table 216 may be provide within the Message server 204.

In this embodiment, the Job Table data structure is combined with the job structure that exists in the Message Server to generate periodic reports. In another embodiment, the Job Table remains unchanged, but all the jobs intended for MSIS in other embodiments are performed directly by the Message Server 204 itself.

A simpler alternative embodiment of the inventive system 230 illustrated in FIG. 7, integrates the optimizing engine (OE) 232 and message server 236 performing message allocation and having an internal local data storage 238 or database, to an even greater extent. It also places responsibilities for other parts of a larger system to other entities while retaining a core that provides optimization and allocation. These elements may be combined on or within the same machine or on collocated machines and operate with direct communication. The optimizing engine 232 receives optimizing specifications 240 from an external source and generates message allocations which it uses to server messages. As in other embodiments described here, the messages may be advertisements, emails, and the like, as described herein.

Target Message Destination Types Although the above described embodiment provides a message server that has been described as generally serving a message to a receiver, it is noted that the receiver may any one or more of a variety of receiver types. For example, the message will frequently be served to a personal computer connected to the world wide web via the internet. In other instances, the message will be served via interactive television, electronic mail, palm-top information appliances and/or Personal Data Assistants (PDAs), hand-held wireless devices, or other interactive media.

Advertising

In this embodiment, the message includes an advertisement for goods or services. Typically, the advertisement will be for a commercial product, good, or service; however, non-commercial advertisements are also anticipated. One type of non-commercial advertisement is a so called "public-service" announcement.

In this embodiment, the Message Server Interaction Service is an Advertisement Server Interaction Service, the Message Server is an Ad Server, the Message Performance Reports are Ad Performance Reports, the Message Control Agent is the advertiser, and the Message Campaign is an Ad Campaign. Email

Email is an important medium for advertising and marketing. Many businesses now readily use email to acquire new customers, build brands, advertise and promote products, measure consumer satisfaction and manage customer relationships. A typical email campaign involves sending emails to each address on a list of recipients. The list may typically be bought or otherwise acquired from an outside firm or collected internally over a period of time. The procedures and algorithms developed above can be easily extended for optimizing the responses generated by emails. A system and method for improving the performance of

Emails is described in United States Patent Application Serial No. __/___,

(Attorney Docket No. A-68760/RMA LXM), filed _ May 2000 and entitled System and Method for Optimizing The Performance of Email and Other Message

Campaigns, which is hereby incorporated by reference.

The inventive procedures and algorithms may, for example, be extending to the email environment by decomposing the email process into multiple stages and dividing the measurement process into multiple stages. In each stage a fraction of total emails to be sent out to each alternative is given by the allocation algorithm described above. The major difference in an email application as compared to the afore described banner ad application is that there is a non-uniform delay between the time an email is sent out and the time a response is received. (For banner ads this delay is either non-existent for practical purposes or fairly uniform, for example the delay for a banner ad clickthough may be between about 1 second and 60 seconds, well within one stage.) However, one can measure the total number of responses received for a given email alternatives by the end of a given stage and the total number of emails sent for that alternative as a way of measuring the performance of a given alternative. This information can then be used by the algorithm to determine and allocate the better alternatives and optimize the total number of responses over the entire campaign. The duration of the stage should advantageously be chosen to ensure that a reasonable fraction of overall responses are received by the end of each stage and the number of stages are large enough that the algorithm can learn the better performing alternatives in the earlier stages and exploit this information in later stages. News

In yet another embodiment, the message that is being served is a generic information item, such as for example, a news headline, a news story, a photograph or other graphic, a stock quotation or set of stock quotations, sports news or feature stories, marketing promotions or coupons, text or images on a web-site, text or images on a personal digital assistant, using wireless or other media to connect, is a message shown on interactive television.

Additional Embodiments and Description

The foregoing description, for purposes of explanation, used specific nomenclature to provide a thorough understanding of the invention. However, it will be apparent to one skilled in the art in light of the description provided that the specific details are not required in order to practice the invention. Thus, the foregoing descriptions of specific embodiments of the present invention are presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the invention to the precise forms disclosed, obviously many modifications and variations are possible in view of the above teachings.

The embodiments were chosen and described in order to best explain the principles of the invention and its practical applications, to thereby enable others skilled in the art to best utilize the invention and various embodiments with various modifications as are suited to the particular use contemplated. It is intended that the scope of the invention be defined by the following claims and their equivalents.

Claims

We claim:

1. A system for autonomously selecting attributes of a message to improve message performance in an interactive measurable medium, said system comprising: a database storing message attribute configuration information, including future message attribute configuration information, and past message performance report information; an optimization engine receiving said past message performance information and making decisions based on said received past message performance information and predetermined rules to generate future message attribute configuration information; and a message server proxy that interacts with an external message delivery system over a communication link to configure said external system to deliver said future message based on said future message attribute configuration information.

2. The system in claim 1, wherein said predetermined rules include past message attribute configuration information associated with said past message performance information.

3. The system in claim 2, wherein said predetermined rules further include message campaign data and optimization preference data.

4. The system in claim 1, further comprising a user interface that allows a message control agent to communicate message campaign data and message optimization preference data.

5. The system in claim 1, wherein said message attribute configuration information comprises message control agent configured information and optimization engine generated configuration information.

6. The system in Claim 5, wherein said message control agent comprises an external user.

7. The system in claim 1, wherein said communication link comprises a communication link coupling a network of interconnecting information devices.

8. The system in claim 1, wherein said communication link comprises a portion of the internet.

9. The system in claim 1, wherein said message server proxy comprises an advertising server proxy and said external message delivery system comprises an advertising server.

10. The system in claim 1, wherein said optimization engine includes a procedure for improving the stage-to-stage performance of a message in a multistage message campaign in an interactive measurable medium; said procedure including steps of:

(a) reading prior stage message state pertaining to a prior stage in a message campaign; said prior stage message state including a cumulative number of trials and th a cumulative number of successes for a particular (i ) message at the end of said prior stage;

(b) reading message performance results representing message trials and message successes from said previous stage based on said prior stage state;

(c) computing a current message state on the basis of said prior stage message state and said message performance results; and

(d) generating a current message allocation based on said current message state.

11. A method for improving the stage-to-stage performance of a message in a multi-stage message campaign in an interactive measurable medium; said method including steps of:

(a) reading prior stage message state pertaining to a prior stage in a message campaign; said prior stage message state including a cumulative number of trials and th a cumulative number of successes for a particular (i ) message at the end of said prior stage; (b) reading message performance results representing message trials and message successes from said previous stage based on said prior stage state;

(c) computing a current message state on the basis of said prior stage message state and said message performance results; and

(d) generating a current message allocation based on said current message state.

12. The method in claim 11, wherein said method further comprising step of:

(e) storing said current message state as prior stage state for a next iteration of said method.

13. The method in claim 11, wherein said cumulative number of trials and said cumulative number of successes comprise discounted cumulative number of trials and discounted cumulative number of successes.

14. The method in claim 11, wherein said prior stage message state comprises reading a state vector for the previous stage in a message campaign.

15. The method in claim 11, further comprising the step of storing said initial state vector in a database.

16. The method in claim 11, wherein said step of generating a current message allocation further includes applying a message allocation constraint.

17. The method in claim 10, further including the step of storing said current message allocation in a database.

18. The method in claim 12, further including repeating steps (a)-(e) for each stage in said message campaign.

19. The method in claim 10, wherein said message comprises a web banner ad.

20. The method in claim 10, wherein said message comprises an email.

21. The method in claim 10, wherein said method further comprising steps of:

(e) storing said current message state as prior stage state for a next iteration of said method; and

(f) repeating steps (a)-(e) for each stage in said message campaign; said cumulative number of trials and said cumulative number of successes comprise discounted cumulative number of trials and discounted cumulative number of successes; said prior stage message state comprises reading a state vector for the previous stage in a message campaign; said step of generating a current message allocation further includes applying a message allocation constraint; and said message selected from the group of messages consisting of an advertisement, an email, and combinations thereof.

22. A system for autonomously selecting attributes of a message to improve message performance in an interactive measurable medium, said system comprising: a database storing message attribute configuration information, including future message attribute configuration information, and past message performance report information; an optimization engine receiving said past message performance information and making decisions based on said received past message performance information and predetermined rules to generate future message attribute configuration information; a message delivery system for delivering said future message based on said future message attribute configuration information; and a message server proxy that interacts with said message delivery system over a communication link to configure said message delivery system to deliver said future message.

23. The system in claim 22, wherein said measurable interactive medium is selected from the group consisting of the internet, interactive television, email, handheld wireless devices, wireless telephones, personal data assistants, and combinations thereof.

24. The method in claim 22, wherein said message is selected from the group of messages consisting or an advertisement, an internet web site banner ad, an email, an email advertisement, an email having ad advertisement attachment, a solicitation, an interactive television message, and combinations thereof.

25. The system in claim 22, wherein said measurable interactive medium is selected from the group consisting of the internet, interactive television, email, handheld wireless devices, wireless telephones, personal data assistants, and combinations thereof.

26. The method in claim 22, wherein said message is selected from the group of messages consisting or an advertisement, an internet web site banner ad, an email, an email advertisement, an email having ad advertisement attachment, a solicitation, an interactive television message, and combinations thereof.

27. The system in claim 22, wherein said optimization engine includes a procedure for improving the stage-to-stage performance of a message in a multistage message campaign in an interactive measurable medium; said procedure including steps of:

(a) reading prior stage message state pertaining to a prior stage in a message campaign; said prior stage message state including a cumulative number of trials and a cumulative number of successes for a particular (i^th) message at the end of said prior stage;

(b) reading message performance results representing message trials and message successes from said previous stage based on said prior stage state;

(c) computing a current message state on the basis of said prior stage message state and said message performance results; and (d) generating a current message allocation based on said current message state; and said state comprises a first state vector component sj(t) for the total number of discounted cumulative clicks for message i at the end of stage t, and a second state vector component ni(t) for the cumulative number of impressions for banner i at the end of stage t; wherein:

Si(t) = G{ clicki(l), ..., click;(t-2), clicki(t-l) }, for t > 2 ; nι(t) = G { impi( 1 ), ... , impi(t-2), impι(t- 1 ) } , for t > 2 ; where Si(l) = 0 and nι(l) = 0; impi(t) is the total number of impressions for banner i in stage t; clickj(t) represent the total number of clicks for banner i in stage t; and

G{...} is a geometric discounting function.

28. A method for improving the • stage-to-stage performance of a message in a multi-stage message campaign in an interactive measurable medium; said method including steps of:

(a) reading prior stage message state pertaining to a prior stage in a message campaign; said prior stage message state including a cumulative number of trials and a cumulative number of successes for a particular (i^th) message at the end of said prior stage; (b) reading message performance results representing message trials and message successes from said previous stage based on said prior stage state;

(c) computing a current message state on the basis of said prior stage message state and said message performance results; and

(d) generating a current message allocation based on said current message state.

29. The method in claim 28, wherein said state comprises a first state vector component s;(t) for the total number of discounted cumulative clicks for message i at the end of stage t, and a second state vector component m(t) for the cumulative number of impressions for banner i at the end of stage t.

30. The method in claim 29, wherein:

Si(t) = G{ clicki(l), ..., click;(t-2), click_;(t-l) }, for t > 2 ; and m(t) = G{ impi(l), ..., imp;(t-2), impi(t-l) }, for t ≥ 2 ; where s;(l) = 0 and nι(l) = 0; and where: impi(t) is the total number of impressions for banner i in stage t; click;(t) represent the total number of clicks for banner i in stage t; and G{...} is a discounting function.

31. The method in claim 30, wherein G { ... } is a geometric discounting function.

32. A system for autonomously selecting attributes of a message to improve message performance in an interactive measurable medium, said system comprising: means for receiving message attribute configuration information and message performance information; an optimization engine receiving said message performance information and making decisions based on said received message performance information and predetermined rules to generate future message attribute configuration information; and a message server to deliver messages based on said future message attribute configuration information.

33. The system in claim 1, wherein said optimization engine includes a procedure for improving the stage-to-stage performance of a message in a multistage message campaign in an interactive measurable medium; said procedure selected from the group of procedures consisting of genetic algorithm based procedures, neural network based procedures, machine learning based procedures, stochastic optimization based procedures, and combinations thereof.

34. The method in claim 33, further including the step of computing a current message allocation and storing said current message allocation in a database.

35. The method in claim 33, wherein said message comprises a web banner ad.

36. The method in claim 33, wherein said message comprises an email.

37. The system in claim 27, wherein said optimization engine includes a procedure for improving the stage-to-stage performance of a message in a multistage message campaign in an interactive measurable medium; said procedure selected from the group of procedures consisting of genetic algorithm based procedures, neural network based procedures, machine learning based procedures, stochastic optimization based procedures, and combinations thereof.

38. The method in claim 37, further including the step of computing a current message allocation and storing said current message allocation in a database.

39. The method in claim 37, wherein said message comprises a web banner ad.

40. The method in claim 37, wherein said message comprises an email.