DIGITAL WATERMARKS AS DATA PROXIES
Field of the Invention The present invention relates to digital watermarks, and more particularly relates to use of watermarks as proxies for data in various applications.
Summary of the Invention As detailed in the assignee's prior applications, including 60/134,782, 60/141,538, and 09/343,104 (variously merged and supplemented in laid-open publications WO 00/70585 and WO 00/70523), digital watermark technology has numerous applications beyond its traditional role of simply communicating copyright information. One futuristic view foresees that all "content" should be watermarked, thereby enabling a great variety of operations and transactions whenever watermarked content is processed by digital devices equipped with watermark recognition and reading technology. All physical media objects can thereby be inherently and persistently digitally-enabled, permitting greatly simplified access to networks and execution of local and remote applications. The continuing growth of the Internet and beginnings of trends toward pervasive computing signal an opportunity to radically change the relationships between traditional media content and digital processing environments.
In this specification, content refers not just to electronic audio, image, and video files, but also includes the content aspects of physical objects and media, e.g., artwork, patterns, and labels on product packaging, concert tickets, etc.
In accordance with one aspect of the present invention, the processing of watermark data as pointer to shared resources is sometimes used in lieu of transmitting from point to point the object with which it is associated, thereby gaining efficiencies in speed and bandwidth.
In accordance with another aspect of the invention, watermarks are employed in OLE-like applications. Object linking and embedding ("OLE," sometimes also known as dynamic data exchange, or "DDE") is a well-known data processing construct by which a first digital object (e.g., a graph) can be embedded within a second digital object (e.g., a word
processing document). In some embodiments, the embedding is static. That is, once the embedding takes place, subsequent changes to the first digital object (e.g., the graph) are not reflected in the second, composite digital object (e.g., the document). In other embodiments, the embedding is dynamic (and thus more commonly termed linking rather than embedding). In such arrangements, if the graph is changed, the document is automatically updated to incorporate the latest version of the graph.
The technology underlying OLE is sophisticated, but is well understood by artisans in the field. Reference may be made to the many patents (e.g., 5,581 ,760 and 5,581,686) and reference books (e.g., Brockschmidt, Inside OLE 2, Microsoft Press, Redmond, WA, 1994) on the subject for further details.
In accordance with this aspect of the invention, OLE-like principles are implemented using watermark data in digital objects in order to effect object linking or embedding.
In one such embodiment, a photocopier scans an original paper document to produce image data. This image data is analyzed for the presence of watermark data that identifies the graphic(s) on the document. With this watermark identifier, the photocopier can query a remote image database for pristine image data corresponding to the graphic(s) on the document. This pristine data can be relayed from the remote database to the photocopier and substituted into the scanned image data. Output printed from the photocopier is thus based, at least in part, on pristine image data, rather than on image data that has been subjected to various corruption mechanisms (e.g., degradation of the original paper document, artifacts due to scanning, etc.).
This and other features and advantages of the present invention will be more readily apparent from the following detailed description, which proceeds with reference to the accompanying drawings.
Brief Description of the Drawings Fig. 1 is a diagram illustrating a network environment in which principles of the present invention may be employed. Fig. 2 is a flow chart illustrating aspects of one embodiment of the present invention.
Fig. 3 is a flow chart illustrating aspects of another embodiment of the present invention.
Fig. 4 is a flow chart illustrating aspects of yet another embodiment of the present invention. Fig. 5 shows an apparatus according to still another embodiment of the invention.
Detailed Description Referring to Fig. 1, consider an exemplary network 10 linking two devices - a first device 12 associated with an originating user, and a second device 14 associated with a recipient user. The first device 12 is coupled to the network through a relatively low bandwidth channel, whereas the second device 14 is coupled to the network through a relatively high bandwidth channel. (For example, the first device may be an internet-capable cell phone having low resolution-, still image only- capture capabilities, providing a 9600 baud data channel, or it may be a home PC, with an associated PC or digital single shot camera, coupled to the internet with a 28.8 kbps modem. The second device may be a computer coupled to the internet through a 1.45 megabit per second T-l line, a cable modem, etc.) The network 10 connecting the two devices includes various links - narrow bandwidth at some parts (e.g., 16), very broadband at other (e.g., internet backbone 18), etc.
Assume the user of device 12 encounters a printed image, e.g., an advertisement in a magazine, that may be of interest to the user of device 12. Using an imaging device (e.g., a CMOS- or CCD-camera built into a cell phone, a flatbed scanner connected to a PC, etc.), device 12 captures an image of the advertisement.
In prior art techniques, the image captured by device 12 would have been sent to device 14 over the network; the image received by the second device would be exactly the image sent by the first device.
In accordance with one embodiment of the invention, device 14 receives a better image than that sent from device 12. In one such embodiment, device 14 receives the image data captured by device 12. Device 14 recognizes that the image includes a watermark hidden within the image data, and decodes same. The watermark
payload includes an index by which a copy of the image can be accessed from a server 20 on the internet or other storage medium. With this index, the second device 14 queries the server 20, which returns the image corresponding to this watermark index (in this case, the advertisement) back to the second device 14. The image provided by the server can be higher resolution or pristine, i.e., it has no artifacts left from scanning at device 12, etc. Such a procedure is shown by the flowchart of Fig. 2.
The watermark payload identifying the sensed image can as long or as short as the application requires. Typically, payloads of between 16 and 64 bits are used, although this is not essential. Shorter payloads have the advantage that they can be more robustly encoded while maintaining a fixed degree of image quality; longer payloads offer a greater universe of identifiers with which the image can be labeled. Illustrative watermarking technology is detailed in the assignee's patent 5,862,260, and in copending application 09/503,881, filed February 14, 2000. A great variety of other watermarking arrangements may be used, including those proposed in patents 5,930,369, 5,933,798, 5,664,018, 5,825,892, 5,940,429 and 5,889,868.
In accordance with another embodiment of the invention (Fig. 3), the bandwidth bottleneck imposed by narrowband channel 16 (through which device 12 is coupled) is obviated by employing a watermark as a proxy for an image. In such an arrangement, the image data captured by device 12 is decoded, and a watermark payload hidden in the image is extracted. (This can be performed by hardware or software available in devicel2, e.g., a cell phone microprocessor, a desktop computer, dedicated decoder circuitry, etc. Alternatively, this decoding can be done remotely from device 12, but before device 14, e.g., by a smart router in the intervening network. In the following discussion, decoding in the device 12 is assumed.) Instead of transmitting the image data over the network, the watermark decoding device (e.g., device 12) simply transmits the watermark payload (or a part thereof). On receipt of the payload, device 14 again queries the server 20, and obtains the image (and/or additional content or functionality, as detailed below), corresponding to that watermark. The image is obtained over the high-speed channel(s) between the server and the second device; the low bandwidth channel linking the first device conveys just the low bandwidth watermark payload information.
By building filters into the low bandwidth devices, upon recognition of a class of watermarks indicating availability of the image as a shared resource, or upon user selection of "transmit only watermark data", the image [or content associated with it via the watermark] can be made available to the message recipient via more capable transmission means.
A variant of the foregoing does not transmit the watermark payload to the second device 14. Instead, the payload is dispatched by the first device 12 (or the smart router) directly to the server 20, with instructions that the corresponding desired image be sent to the second device 14. Such an arrangement is shown in Fig. 4. In some applications, the media delivered by the server may be richer than the simple image captured by device 12. For example, the watermark payload in the image captured by device 12 may index one or more files on server 20 that includes video, animation, sound, executable applications, aplets (e.g., JAVA, ActiveX) etc ("enhanced content"). Thus, scanning of a magazine ad at one device can prompt delivery of a video, a Macromedia Shock Wave presentation, etc., to the second device.
In some embodiments, the second device 14 identifies to the server 20 its media-playback capabilities. The server 20 can then respond to a watermark-based query with media appropriate to that particular media consumer.
One way the media capabilities of device 14 can be indicated to server 20 is by a data word comprising flag bits, with each set "1 " bit indicating a capability. A simplified 8-bit capability word may be as follows:
The data comprising this word may be automatically compiled on device 14, e.g., from the operating system database with which programs are registered on installation (the Registry database in Windows).
If device 14 sends the capability word 10101100 to server 20, the server knows the device 14 supports GIF and JPEG imagery (but not TLFF), and RealAudio and WAV sound (but not MP3 or WindowsMedia).
If server 20 has media content corresponding to the queried watermark in several supported formats, it can deliver certain ones according to a priority order (e.g., send JPEG if supported; else send GIF if supported; else send TIFF if supported). If the server 20 only has media in a format not supported by the second device
14 (e.g., TIFF in the foregoing example), the server may invoke a conversion routine to perform an on- the- fly conversion to a supported media type (e.g., JPEG) prior to sending to the second device 14.
If the watermark index is provided by the second device 14 (rather than directly from the first device 12), the capability data word can accompany the index.
If the watermark index is provided directly from the first device 12, the server can solicit from the second device 14 a data capability word before responding to the query. Alternatively, the server can keep, on-file, a database detailing the media capabilities of all known media consumers, and can tailor its query response according to such profile. (The second device 14 can be arranged to automatically inform server 20 of updates to its capability, e.g., each time a new media playback application is registered in the registry database.)
If the server 20 does not know, and cannot discern, the media capabilities of the second device 14, it can provide media in a default form that is most likely to be acceptable (e.g., JPEG, if the content captured by the first device 12 is imagery).
Fig. 5 shows a photocopier 50 according to another aspect of the invention. The photocopier includes a platen 52, a scanner assembly 54, a raw data memory 56, a watermark decoder 58, a processor 60, a network connection 62, a pristine image buffer 64, a compositing memory 66, and a reproduction engine 68. A paper document, such as document 70, is placed on platen 52, and scanner assembly 54 is activated to generate scan data corresponding to the document. The scanner assembly is conventional and may include a linear array of CCD or CMOS
sensor elements that optically scans along an axis of the platen to generate 2D image data. Alternatively, the scanner can comprise a 2D array of sensor elements onto which an image of the document is projected through one or more lenses. In the illustrated embodiment, the document 70 includes a picture 71 that is encoded with a plural-bit digital watermark. Document 70 may be referred to as a compound document since it incorporates plural components (e.g., text and picture).
The scan data from the scanner assembly 54 is stored in the raw data memory 56, where it is analyzed for the presence of watermark data by the watermark decoder 58. There are many different techniques by which imagery can be digitally watermarked and decoded. One is the noted Digimarc watermark system detailed, e.g., in patent 5,862,260, and in pending application 09/452,023, filed November 30, 1999, the disclosures of which are incorporated herein by reference. A great variety of other systems are known. All that is required is that the watermark permit the conveyance of plural-bit auxiliary data without objectionable image degradation.
Upon detection of the watermark in picture 71, the processor 60 is programmed to initiate communication with a remote server 72 (e.g., over the internet) through the network connection 62. The programmed processor sends to the server a query message identifying the detected watermark (which may be, e.g., an identifier of 16 - 64 bits). A database 74 at the server 72 searches its records 77 for a digital object indexed by that watermark ID 79 and, if located, causes a pristine version of the object 78 (in this case a pristine version of the picture 71) to be sent to the photocopier.
In the embodiment illustrated, the database has the pristine version of the object stored within the database record for that watermark ID, and relays same directly back to the photocopier. In other embodiments, the object itself is not stored in the database. Instead, the database stores (in a record associated with the watermark ID) the address of a remote data repository at which the pristine object is stored. In this case the object server 72 can transmit an instruction to the remote repository (e.g., again over the internet), requesting the remote repository to provide the pristine object. The object can be sent directly from the remote data repository to the photocopier, or may be relayed through the object server 72. In any case, the pristine object may be provided in TIFF, JPEG, GIF, or other format. (In some embodiment, the request signal from the
photocopier specifies the format desired, or may specify plural formats that the photocopier can accept, and the pristine object is then output by the server 72 or remote repository in such a format. In other embodiments, the request signal from the photocopier does not include any format data.) In some embodiments, the object server 32 can be of the sort more particularly detailed in copending applications 60/164,619 (filed 11/10/99), and 09/343,104 (filed 6/29/99), published in WO 00/70585, the disclosures of which are incorporated herein by reference.
In addition to detecting the ID of any watermark in the scanned image data, the photocopier's watermark detector also discerns the placement of the watermarked picture within the document image, and its state (e.g., size, rotation, etc.), and produces corresponding state information. In some embodiments, this state information is passed to the object server 72, permitting the pristine object 78 to be sizeαVrotated/etc. (e.g., by the object server) to match the object detected in the document image. In other embodiments, a generic version of the pristine object is passed back to the photocopier, and the processor 60 attends to sizing, rotating, etc., of the pristine picture 78 as necessary to match that of the original picture 71.
In some embodiments the picture 71 in the paper document has been cropped. (The watermark can nonetheless be detected from the cropped image.) When the pristine picture 78 is received from the remote location, it can be pattern-matched to the picture 71 detected in the original document to determine the cropping boundaries (if any), and corresponding cropping of the pristine picture can be effected.
Once the foregoing scaling/rotation/cropping, etc., adjustments (if any) have been made on the pristine picture 78 stored in buffer 64, the processed pristine picture is combined with the original document scan data in compositing memory 66, yielding a composite document image that includes the pristine picture data 78 in lieu of the scanned picture 71. (The substitution of the pristine picture for the original picture data can be accomplished by various known image processing techniques, including masking, overwriting, etc.) The composite document image is then passed to the reproduction engine 68 to produce a hard-copy output (i.e., an enhanced compound document 70') in the conventional manner. (The reprographic engine 68 can take many different forms including, e.g., xerography, ink-jet printing, etc.)
The pristine picture 78 received from the server 72 can, itself, be watermarked or not. If watermarked, the watermark will usually convey the same payload information as the watermark in the original picture 71, although this need not always be the case. In other embodiments, the pristine picture 78 received from the remote server 72 has no watermark. In such case the pristine picture can be substituted into the compound document 70 in its unwatermarked state. Alternatively, the apparatus 50 can embed a watermark into the picture prior to (or as part of) the substitution operation. If the substituted picture is watermarked, this permits later watermark-based enhancement or updating. For example, if the enhanced compound document 70' including the pristine picture 78 is printed by the photocopier, and the resulting photocopy is thereafter photocopied, the latter photocopying operation can again substitute pristine picture data for the scanned picture data produced by the second photocopier's scanner. Moreover, in applications where it is appropriate for a picture to be updated with the latest version whenever printed, the watermarking of the picture 78 permits substitution of a latest version whenever the document is scanned for printing.
In other situations, it is desirable for the picture 78 included in the enhanced compound document 70' to be unwatermarked. This is the case, for example, in certain archival applications where it is important that the document 70' not be changed after archiving. By assuring that the picture 78 is not watermarked, inadvertent changing of the picture in subsequent photocopying can be avoided. (In cases where the pristine image 78 is provided from server 72 in a watermarked state, the photocopier may remove or disable the watermark in response to corresponding instructions from a user through a user interface or the like.) From the foregoing description, it will be apparent that embodiments of the present invention provide various advantages over the prior art. One is the dispatch of high bandwidth enhanced content using a low bandwidth channel. Another is the receipt of higher-quality data than that originally captured. Another is delivering applications via low bandwidth channels to recipients by capturing images or watermark data from media content that serve as proxies for the applications. Another is the production of "photocopies" that are better than the "originals."
Having described and illustrated the principles of our invention with reference to specific embodiments, it will be recognized that the principles thereof can be implemented in other, different, forms.
For example, while the invention has been described primarily with reference to images, the same principles are equally applicable to video and audio.
Similarly, while the foregoing description has made reference to transmitting the watermark, in many implementations only a part of the watermark need be transmitted. (The watermark may include error correcting information, or other data, not necessary to identify the corresponding data on the server 20.) Still further, while the detailed embodiment contemplated a still or video camera system for first device 12, much of the functionality of such an image capture system isn't essential to the present invention. Instead, an input device that serves a simpler "watermark capture" function may be used instead. Such a device can omit, e.g., hardware or software components associated with pixel interpolation (commonly used to achieve a desired virtual resolution), formatting (e.g., to provide output in JPEG form), etc. Such components serve useful functions when the resulting imagery is to be displayed or printed, but are superfluous - or detrimental - when the image data is simply to be decoded to extract watermark data.
While the invention is illustrated with reference to steganographic watermark technology for identifying the initial content (i.e., that sensed by device 12), other technologies can alternatively be used. These include data glyphs, 1- and 2-D barcodes, magnetic ink, RF ID tags, UN or IR markings, etc.
While the detailed embodiment contemplated a single server 20 to serve as the repository of content corresponding to watermarks, in other embodiments such a server is implemented in distributed fashion. In some embodiments, one server may act as a default repository, and can dispatch queries to other servers if the first server cannot provide the requested data. Caching of frequently-requested content can be provided at various locations through the network. Additional details on such network configurations can be found in application 09/343,104 (WO 00/70585). In the photocopier embodiment, it will be recognized that the same principles are equally applicable in other systems, including personal computers (e.g., in conjunction with image editing software, such as Adobe Photoshop). In such case the
input image data needn't come from a scanner but may come, e.g., from a digital file, from a network location, etc. The reference to "documents" in this embodiment is illustrative only; the technology can similarly be employed with any compound object that includes a watermarked component - whether in digital or analog form. Moreover, while the Fig. 5 embodiment is described as using separate raw data memory 16, pristine image buffer 24, and compositing memory 26, more typically some or all of these functions are served by a single memory, which may be a computer system's main RAM memory.
While the Fig. 5 embodiment employs a processor 60 programmed in accordance with software instructions (e.g., stored in a memory or on a storage medium), in other embodiments some or all of the described functionality can be achieved using dedicated hardware (e.g., ASICs), or programmable hardware (e.g., PLAs).
Although the Fig. 5 embodiment is illustrated with reference to an arrangement in which a document includes a single watermarked photograph, it will be recognized that plural such watermarked components may be present in a compound document, and the system may be arranged to obtain pristine versions of each, and edit/composite same as necessary as to recreate an enhanced version of the original document. Likewise, while the illustrative embodiment contemplates that a watermarked photograph may be a component of the original document, in other embodiments the watermarked object may comprise the entirety of the original document.
While reference has been made to substitution of pristine image components, in some embodiments it may be desirable to substitute components that are not "pristine." Indeed, in some embodiments an object may be substituted that is visually dissimilar to the original object. Consider artwork for a Christmas card. The artwork may include a watermarked "generic" corporate logo. When encountered by a computer according to this aspect of the present invention, the generic logo may be replaced with a logo corresponding to the corporate owner of the computer. In such case, the substitute imagery may be stored within the computer itself, obviating the need for any network connection. The registry database maintained by the computer's operating system may include keys defined by watermark IDs. When a watermark ID is encountered, the registry database can be consulted to identify a corresponding graphic that can be
substituted into the object being processed. If none is found, the watermark ID can be passed to the remote server 72.
While, for expository convenience, the Fig. 5 embodiment was described as always substituting pristine data when available, more typically this is a function that would be enabled or disabled by an operator of the device, e.g., by an appropriate switch, button, or user interface control. In some embodiments, the device may be arranged to query the user when substitution of a pristine component is possible, in some cases presenting the user with a depiction of the image component proposed to be substituted. The Fig. 5 embodiment may be said to employ watermark-based object embedding, since the hard-copy output is static (i.e., cannot change) after printing. In other embodiments, the enhanced compound document 70' is not printed, but stored.
Each time the compound document is utilized (e.g., opened for editing, or printed), any watermarked component(s) therein can be updated to include the latest-available version(s) of the watermarked component(s). In such case, the document may be said to employ watermark-based object linking.
As is familiar to those skilled in the arts, all of the foregoing methods may be performed using dedicated hardware components/systems for the various devices, and/or through use of processors programmed in accordance with firmware or software, etc. In the latter case the processors may each include a CPU and associated memory, together with appropriate input and output devices/facilities. The software can be resident on a physical storage media such as disks, and can be loaded into the processors' memory for execution. The software includes instructions causing the CPU to perform the various processes detailed above. To provide a comprehensive disclosure without unduly lengthening this specification, applicants incorporate by reference the patents and applications cited above.
In view of the wide variety of embodiments to which the principles of our invention can be applied, it should be recognized that the detailed embodiments are illustrative only and should not be taken as limiting the scope of the invention. Rather, we claim as our invention all such embodiments as may come within the scope and spirit of the following claims, and equivalents thereto.