US20140375664A1 - System and method for video frame sequence control - Google Patents
System and method for video frame sequence control Download PDFInfo
- Publication number
- US20140375664A1 US20140375664A1 US14/373,512 US201214373512A US2014375664A1 US 20140375664 A1 US20140375664 A1 US 20140375664A1 US 201214373512 A US201214373512 A US 201214373512A US 2014375664 A1 US2014375664 A1 US 2014375664A1
- Authority
- US
- United States
- Prior art keywords
- output
- frame
- video
- frames
- input video
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims description 16
- 230000001419 dependent effect Effects 0.000 claims abstract description 8
- 230000009467 reduction Effects 0.000 claims description 55
- 238000003780 insertion Methods 0.000 claims description 2
- 230000037431 insertion Effects 0.000 claims description 2
- 238000012986 modification Methods 0.000 description 10
- 230000004048 modification Effects 0.000 description 10
- 230000000873 masking effect Effects 0.000 description 7
- 230000000007 visual effect Effects 0.000 description 6
- 238000004891 communication Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000012216 screening Methods 0.000 description 3
- 239000003086 colorant Substances 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 210000003254 palate Anatomy 0.000 description 2
- 239000002245 particle Substances 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000001413 cellular effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000009827 uniform distribution Methods 0.000 description 1
Images
Classifications
-
- H04N5/2357—
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/70—Circuitry for compensating brightness variation in the scene
- H04N23/745—Detection of flicker frequency or suppression of flicker wherein the flicker is caused by illumination, e.g. due to fluorescent tube illumination or pulsed LED illumination
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
- G09G5/36—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators characterised by the display of a graphic pattern, e.g. using an all-points-addressable [APA] memory
- G09G5/39—Control of the bit-mapped memory
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/04—Synchronising
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/74—Projection arrangements for image reproduction, e.g. using eidophor
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N9/00—Details of colour television systems
- H04N9/44—Colour synchronisation
- H04N9/475—Colour synchronisation for mutually locking different synchronisation sources
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2360/00—Aspects of the architecture of display systems
- G09G2360/18—Use of a frame buffer in a display terminal, inclusive of the display panel
Definitions
- the display When displaying a projected image and simultaneously capturing images of the projected scene and the display, there can be crosstalk between the projected images and the captured content.
- the crosstalk can reduce the image quality (i.e. brightness, color) of the captured image frames and additionally cause distracting flicker on the display.
- Various attempts have been made to reduce crosstalk in the projection/image capture systems.
- the display may be changed from a passive screen display screen an active switchable diffuser to reduce crosstalk.
- an active diffuser display screen can limit the useful duty cycle of both the projector and the image capture device used in the system.
- FIG. 1 shows a schematic drawing of a video sequence control system with binary modulation according to an example of the invention
- FIG. 2 shows the video sequence control system shown in FIG. 1 where the video frame sequence frames produced and output by the video sequence control system is displayed on a see-through display screen according to an example of the invention
- FIG. 3A shows a schematic drawing of a video sequence control system with spatial masking and modulation having a first modulation pattern according to an example of the invention
- FIG. 3B shows a schematic drawing of a video sequence control system with spatial masking and modulation having a second modulation pattern according to an example of the invention
- FIG. 3C shows a schematic drawing of a video sequence control system with spatial masking and modulation having a third modulation pattern according to an example of the invention
- FIG. 3D shows a schematic drawing of a video sequence control system with spatial masking and modulation having a fourth modulation pattern according to an example of the invention
- FIG. 4 shows a schematic drawing of a video sequence control system according to an example of the invention
- FIG. 5 shows a flow diagram for a method for controlling the output of a video sequence according to an example of the invention
- FIG. 6 shows a computer system for implementing the method shown in FIG. 5 described in accordance with examples of the present invention.
- the application describes a video sequence control system 100 comprising: an input video frame buffer 102 for receiving a video stream 104 of video frames 106 from a video source 110 ; and an output video frame selection component 112 for determining a video frame to be output according to a scheduler 116 that provides timing information 118 and modulation information 119 , wherein the timing information 118 includes information regarding when each video frame will be output, wherein modulation information varies dependent upon frame type to be output, wherein frame type includes at least the image frames from the input video frame buffer and functional frames, wherein the at least the image frames and functional frames are output according to a pattern defined by the scheduler, wherein based on the timing information 118 a synchronization output signal 124 is output corresponding to the output of the functional frame.
- the video sequence controller takes a video stream input from a video source and outputs a (1) modified video stream and (2) a synchronization output signal.
- the video sequence control system 100 receives a video stream 104 from the video source at a first rate.
- the video sequence controller 100 receives the input video stream, modifies the video stream and outputs the modified video stream 136 at a second frame rate.
- the modification of the video stream occurs based on a schedule 117 that details the insertion a functional video frame into the output video stream 136 .
- an input frame or other frame is modified to create the video output stream and how frames slotted for modification are changed is dependent upon frame type.
- At least two frame types are available for output, including input video frames and functional frames.
- Input video frames refer to the input video that are not modified and directly output in the output video frame sequence.
- the functional frames are frames other than the input video frames.
- the functional video frames often have a purpose other than the output of video images.
- the functional video frames are crosstalk reduction frames.
- the crosstalk reduction frames are added for the purpose of reducing crosstalk in the system in order to improve image quality.
- the functional frames are color channel varying frames.
- the purpose of varying the color channel is to add color channel varying frames to the output video stream to facilitate green-screening.
- the term functional frame and the alternative functional description of the frame i.e., crosstalk reduction frame, color channel varying frame
- FIG. 1 shows a schematic drawing of a video sequence control system 100 according to an example of the invention.
- the video sequence control system includes an input video frame buffer 102 for receiving a video stream 104 of video frames 106 from a video source 110 .
- Various sources may be used for the generating the input video stream 104 .
- video could originate from video streamed from the network.
- the video source could be a graphics adapter in a computing device.
- the video source could be a video camera that inputs the captured images to the graphics adapter in a computing device.
- the video sequence control system 100 includes an output video frame selection component 112 for determining the video frame to be output according to a scheduler 116 .
- the output video frame selection component 112 includes a modulation component 114 that modifies the input video stream that is coupled to a scheduler 116 .
- Timing information 118 is used by the output selection component 112 to determine the timing of when video frames are output.
- Modulation information 119 in the scheduler 116 can be used by the modulation component 114 to determine whether the input video frame needs to be modified and if the video frame needs to be modified which input video frames are modified and how the video input frames selected for modification are modified before being output.
- the output video frame selection component 112 can be described as a binary modulator.
- the number 1 can be representative of the image frame and the number 0 can be representative of the black crosstalk reduction frame.
- the number 1 can be representative of a black crosstalk reduction frame and the number 0 can be representative of the input image frame.
- the arrow 126 represents a schedule pointer that points to a frame that is under consideration (to be modified/output) according to the output video frame schedule 117 .
- the number 0 in the schedule is representative of an image frame and that the number 1 is representative of the black crosstalk reduction frame.
- the output video frame selection component 112 selects an image frame for output.
- the modulator changes the image in the input file according to the modification information in the schedule. Referring to FIG. 1 where the modulation schedule specifies a black crosstalk video frame—the modulator modifies the image component frame so that the pixels in the image frame are black. The selection component then outputs the modified image frame.
- a software or hardware implemented representation of the crosstalk reduction frame is used. For example, say for a black (crosstalk reduction) frame is required for every pixel in the video frame. The pixel value of zero could be stored and the value repeated for each pixel location in the video frame when a black frame is present on the schedule. For this case, the entire image frame may not need to be stored. In one case a single pixel value is stored. The timing information in the schedule can be used to determine when the crosstalk reduction frame is to be output. When the crosstalk reduction frame is output, the pixel value of zero is repeated for every pixel in the frame.
- the scheduler 116 of the output video frame selection component 112 includes timing information 118 and modulation information 119 .
- the timing information 118 includes information regarding when each video frame will be output.
- timing information 119 includes information about the number of frames to be output within a certain time period and when each frame in the scheduled sequence of video frames will be output.
- the scheduler determines the output sequence of the video frames at least in part based on the of the frame rate of the video input and the required video output. Thus for example, if the frame rate of the video source is 30 Hz and the required video output is 120 Hz, then for each video input image frame—four video frames are output. In the example shown in FIG.
- the schedule 117 alternates between image frames (frames 130 and 134 ) and crosstalk reduction frames (frames 128 and 132 ).
- a crosstalk reduction frame is output for each image frame in the video input buffer .
- This pattern (alternating image and crosstalk frames) is illustrated in the virtual representation 115 of the schedule and the corresponding video output sequence 136 .
- the video frames are output at equal time periods. Thus, for a 120 Hz output, a frame is output every 0.25 seconds.
- the video sequence control system 100 receives the video source input at a first frame rate and outputs a modified video stream at a second frame rate.
- the video output video frame selection component 112 repeats unmodified input image frames 106 and inserts the unmodified image frames into the output video stream 136 to accomplish the required frame rate.
- frames are dropped (lower frame rate) or alternatively some of the existing frames (i.e. same frame rate) are dimmed.
- the functional i.e. crosstalk reduction frame
- the dimmed image can be a reduced intensity version of the black frame (i.e. a gray frame) or alternatively, a dimmed version of the image frame.
- the frame rate and the video sequence pattern effects whether the input video frames are dropped, inserted or remain the same in the output video frames 136 .
- the schedule pattern includes an alternating sequence of image video frames from the input video stream 104 and crosstalk reduction frames, where the crosstalk are dimmed (black) video frames. If the input video streams and output video streams are both 60 Hz, then in one example, half of the video input frames could be modified (dimmed) to provide the video sequence pattern (alternating image and crosstalk reduction frames) shown on the visual representation 115 of the schedule.
- the scheduler 116 includes frame type information 120 , that provides information regarding the type of video frame that is currently being processed (the video frame in the visual representation of the schedule that the pointer 126 is pointing to.)
- the scheduled frame types include at least crosstalk reduction frame types and image frame types.
- modification information is associated with the frame type information to provide information on whether the frame type is selected for output with no modification or whether the frame type is modified. Further, if the frame type is to be modified—how each frame type is modified. For example, if the frame type is a crosstalk reduction frame—the image frame might be modified to be a uniform black frame, where the black frame is output in the output video stream for display.
- the scheduler 117 of the output video frame selection component 112 includes modification information 119 , timing information 118 , frame type information 120 , and sequence order information 122 which the scheduler 117 uses to determine whether input image frames are modified, how the input frames are modified, when the modified or unmodified frames are output, and the sequence order that the frame types are output.
- the sequence order information 112 provides information to the modification component 114 of the order and frame type that the video frames are output.
- the output video sequence can be described generally as at least one crosstalk reduction from followed by a sequence of n image frames.
- the repetitive pattern 138 is a pattern of alternating crosstalk reduction frames and image frames.
- the output video sequence could be different dependent upon the desired output and results.
- the video sequence could be a crosstalk reduction frame followed by three image frames. This video sequence may be preferred where the video sequence is displayed as the increased number of image frames in the sequence compared to the first example (single crosstalk reduction frame followed by a single image frame), improves the quality of the displayed image.
- the timing information 118 provides information regarding when and how often the selected (where schedule pointer 126 is currently pointing to) video frame is output. In one example, the timing information 118 is used to determine when a synchronization output signal 124 is output. The synchronization output signal 124 should correspond to the timing of the output of the crosstalk reduction frame.
- FIG. 2 shows the video sequence control system shown in FIG. 1 where the output video frame sequence frames are displayed on a see-through display screen 150 according to an example of the invention.
- the video sequence control system 100 outputs a video frame sequence 136 to the projector 148 that drives the display 150 .
- the video sequence control system 110 outputs a synchronization output signal 124 to the image capture device 146 to signal to the image capture device 146 when image capture of the scene should occur.
- the video sequence control system 100 acts as a specialized video processor. It can modify frame rate by inserting or removing frames and can modify image frames.
- the video sequence control system is a standalone device positioned between the video source 110 and the devices that it is outputting video frames and/or synchronization signals to.
- the video sequence control system 100 is positioned between the video source 110 and the projector 148 .
- the video sequence control system 100 is positioned between the video source 110 and the video camera 146 .
- the crosstalk reduction frame 112 is a dimmed or reduced intensity frame, by dimmed or reduced intensity we mean the crosstalk reduction frame has an intensity lower than the corresponding image frames in the same video sequence.
- the crosstalk reduction frames are black frames.
- the purpose of the lower intensity video frame is to provide a time interval when the image capture device 120 is capturing a video frame image of the scene behind the see-through display screen which minimizes the interference or crosstalk caused by the projected image.
- the video frame output 136 is projected onto a display screen 150 by a projector 148 .
- the display is a see-through display screen 150 comprised of a relatively low concentration of diffusing particles embedded within a transparent screen medium. The low concentration of diffusing particles allows a camera 146 or other image capture device to capture an image through the screen 150 (providing that the subject is well lit), while diffusing enough of the light from the projector 148 to form an image on the display screen.
- the display screen 150 can be a holographic film that has been configured to accept light from the projector 148 within a first range of angles and to transmit light that is visible to the remote participant within a different range of viewing angles. The holographic film is otherwise transparent.
- light projected onto the first surface 156 within the first range of angles can be observed by viewing the second surface 158 , but light striking the second surface 158 is transmitted through the screen to the camera.
- the camera also captures light from the projector diffused or scattered off of the first surface.
- the output crosstalk reduction frames are black frames (no light output) which simulates the projector 108 being in an off state.
- the image capture device 146 is a video camera. The video camera is synchronized to capture an image during the time period when the crosstalk reduction frame is being displayed. For the case where there is a see-through display screen, the video camera 146 captures the scene in front of the display screen, the display and the scene behind the see-through display screen. Thus, for a video conferencing example where there is a participant 152 on the other side of the display screen—the video camera 146 captures the image of the participant on the other side of the display screen. Because the projector is off when the image is captured, the camera does not capture the projected image.
- the video sequence control system 100 produces a synchronization output signal 124 .
- the synchronization output signal 124 is communicatively coupled to an image capture device 146 and is used to provide a signal to the image capture device 146 related to the occurrence of the at least one crosstalk reduction frame (e.g. 142 a , 142 b ).
- the synchronization output signal 124 is used to trigger image acquisition by the image capture device in order to capture an image when a crosstalk reduction frame 142 is being projected.
- the synchronization output signal 124 is directly coupled to the image capture device 146 .
- the image capture device is communicatively coupled to the synchronization output signal via an optional signal converter 154 .
- the signal converter could reduce the frequency of the synchronization trigger frequency.
- a image capture device 146 might require a specific signal pattern that follows an industry standard protocol. For this case, the signal converter 154 could translate a simple triggering synchronization output signal 124 to the pattern according to industry standard protocol.
- the synchronization output signal 124 provides both a synchronization signal and a command signal to the image capture device 146 .
- the synchronization output signal 124 provides information to the image capture device of when the crosstalk reduction frame is being projected.
- the command signal line can provide information to the image capture device of what camera function is to perform.
- the command signal to the image capture device provides a command to the image capture device to open or close its shutter. In the example shown in FIG. 2 , the timing of the image acquisition is done in synchronization with the projection of the crosstalk reduction frame.
- a synchronization output signal 124 is sent to the image capture device 146 to instruct it to open its shutter (take a picture) when the crosstalk rejection frames are projected onto the display screen.
- the black crosstalk reduction frame 142 a is projected at a time t porj1 .
- the camera opens its shutter at this time, captures an image and outputs a corresponding photograph 160 a .
- the corresponding photograph is of the display, the scene in front of the display and for the case of a see-through screen, the viewer behind the display screen.
- a second black crosstalk rejection frame 142 b is projected at time tpro, then the camera opens its shutter at this time, captures an image and outputs the corresponding photograph 160 b.
- FIG. 3A shows a schematic drawing of a video sequence control system shown in FIG. 1 with an alternative modulation pattern according to the schedule.
- the modulation component 114 of the video sequence control system 100 takes an input video and generates output video by dropping video frames, repeating video frames, or inserting dimmed, masked or otherwise modulated images. Similar to the example shown in FIG. 1 , the modulation pattern shown in the visual representation of the schedule in FIG. 3A alternates between the image frame and a crosstalk reduction frame. However, instead of a black crosstalk reduction as shown in FIG. 1 —the crosstalk reduction frame in FIG. 3A is modulated to output a gray frame.
- a dimmed image may be implemented instead of the crosstalk reduction frame having the highest possible darkness or intensity value (100%)—a black frame, a dimmed image may be implemented.
- a gray image having some percentage of the possible intensity range.
- the gray crosstalk reduction frame could have an intensity of 50%.
- the dimming can be applied at continuous varying levels.
- the modified output video could consist of modulated versions of the input video frames where the modulated versions of the input video frames are at 75%, 50% and 25% of the intensity of the input image.
- the darkness of the crosstalk reduction frame might be reduced from its highest possible intensity (black) to a lower intensity level.
- black the projected image frames are likely to be very bright.
- the crosstalk reduction frame color could be changed from black to a gray color (lower intensity value.)
- the display screen is not bright (in some cases where a see-through display is used)
- the crosstalk reduction frame 112 is a black frame (uniform distribution of black pixels across the video frame.)
- the color or intensity value of pixels can be applied in a spatially varying fashion across the modulated frame.
- FIG. 3B shows a schematic drawing of the video sequence control system 100 shown in FIG. 3A having a second alternative modulation (or masking) pattern.
- Both the modulation pattern shown in FIGS. 3A and 3B apply dimming to the video frame according to the schedule 117 , however, the image frames are modulated according to a schedule that applies dimming to each color channel of the multichannel input video frames.
- dimming can be applied uniformly to each color channel.
- dimming can be applied at a different level to each color channel.
- dimming can also be applied differently to each channel of multi-channel video frames. For example, in one implementation dimming could be applied only to the red channel of the video frames.
- One potential application of the video output sequence 100 shown in FIG. 3B is to use the video output to effectively perform “green-screening” techniques using the video output frames.
- the video sequence control system 100 shown in FIG. 2 where the video sequence control system outputs a video output sequence 136 on a display screen 150 .
- the display screen 150 is opaque and that a weatherman is standing in front of the display screen while a video camera is capturing the weatherman standing in front of a green display screen.
- Previous use of the green-screening techniques may have been to cut out the image of the weatherman and recomposite the weatherman in front of a weather map afterwards. However, it may be desirable for the weatherman to be able to see the display as a TV viewer would see it—instead of seeing a green display screen.
- the weatherman is photographed standing in front of a weather map.
- the modulation pattern shown in the schedule in FIG. 3B could be implemented by scheduling a green channel in one time slot (frame 128 ).
- the image of the weatherman could be captured against the green channel and in the next time slot the image of the weatherman could be captured against the combined blue and red color channel (frame 130 ).
- the video camera captures an image of the weatherman in front of the green backdrop.
- the camera captures an image of the weatherman in front of a combined blue and red backdrop.
- frame rate of the video output is sufficiently high, the viewer's eyes average the green, blue and red channels so that the viewer sees a full color weather map.
- the image is modulated to present different color video frames that can be used as different backdrops when performing image processing. For example, if an image is captured of a person against a black background and the person is wearing a black jacket—the black jacket will not show up well in the captured image. However, if an image of the same person is against a green background—the black jacket will show up. Alternating image capture against at least two substantially different color backgrounds—means no matter what color the person is wearing, for example, it will be easy to determine all of the colors in the image.
- FIG. 3C shows a schematic drawing of the video sequence control system 100 shown in FIG. 3A having a third alternative modulation pattern.
- the video frame output 136 is comprised of modulated image frames where the modulation occurs according to a schedule 117 that has a pattern that varies in a spatially varying manner.
- the third modulation pattern shown in the visual representation 115 of the schedule 117 is a crescent shaped pattern that is used as a mask to create a crescent shaped pattern on the input image frame.
- the mask is used to perform both spatial masking and color channel masking. For example, for the pattern shown in FIG. 3C , the crescent shaped area could be green while the areas outside of the crescent shape could be blue.
- FIG. 3C shows a crescent shape
- other shapes and patterns could be used.
- the schedule 117 shows a modulation pattern that instead of dividing the frame into crescent shaped regions—divides each frame into four rectangular regions.
- FIG. 3D shows a schematic drawing of a video sequence control system shown in FIG. 3A having a fourth alternative modulation pattern. In the example shown in FIG.
- each of the four rectangular regions 128 a - d , 130 a - d , 132 a - d and 134 a - d have two different color channels—two rectangular regions having a red color channel (represented by the letter “R”) and two rectangular regions having a combined green+blue color channel (represented by the letters G+B).
- R red color channel
- G+B green+blue color channel
- two sequential frames alternate color patterns so that similar to the example described with respect to FIG. 3B , the color patterns alternate to provide all of the colors (RGB) within two sequential frames—so that the viewer sees the full color palate.
- Each of the video frames in the schedules has four rectangular regions 128 a - 128 d and 130 a - 130 d .
- Each of sequential video frames have two rectangular regions having a red color channel ( 128 b , 128 c and 130 a , 130 d ) and two rectangular regions having a combined green+blue color channel ( 128 a , 128 d and 130 a , 130 d ).
- the color patterns of the two sequential video frames alternate—for example, if the color channel is red in video frame 128 , it is combined green+blue in video frame 130 and vice versa.
- video frame region 128 b is red and in its next (sequential) video frame the video frame region 130 b is combined+blue.
- video frame region 128 a is green+blue, while the corresponding sequential video frame 130 a is red. Similar to the alternating color channel video frames in FIG. 3B , the result of the pattern of alternating regions in the video frames video is that the viewer of the video output sequence 136 sees the full color palate of the input video frame images 104 .
- FIG. 4 shows a schematic drawing of a time multiplex control system with binary modulation according to an example of the invention.
- the configuration shown in FIG. 3 includes an input video frame buffer 102 for receiving a video stream 104 of input video frames 106 from a video source 110 and an output selection component 112 for determining video frames to be output according to a schedule.
- the output selection component 112 is implemented using a modulator for modifying the input video stream
- the output selection component is implemented using a multiplexer that selects the appropriate input from either the input video frame buffer 102 or the crosstalk reduction buffer 108 .
- the implementation shown in FIG. 4 stores the desired crosstalk reduction video frames in a crosstalk buffer.
- a crosstalk reduction frame buffer is added in FIG. 4 .
- the output selection component 112 includes a multiplexer 114 which selects a particular frame type for output.
- the multiplexer has as possible inputs: video frames from the input frame buffer 102 or crosstalk reduction frames from the crosstalk reduction frame buffer 108 . Based on the schedule 115 , the multiplexer 114 chooses which video frame from which input buffer (input frame buffer or crosstalk reduction buffer) will be selected for output.
- the multiplexer 114 is implemented in hardware, software or a combination of the two.
- FIG. 5 shows a flow diagram for a method for controlling the output of a video sequence according to an example of the invention.
- the method includes the steps of: receiving a sequence of input video frames from a video source (step 510 ); determining when to output the input video frames from timing information provided to a scheduler 116 of the output selection component (step 520 ); determining whether the input video frames will be modified based on scheduler modulation information, wherein modulation information varies dependent upon the frame types available to be output, wherein the available frame types include at least image frames from the input video frame buffer 102 and crosstalk reduction frames (step 530 ); outputting the at least input video frames and modified crosstalk reduction frames according to the scheduler (step 540 ); and outputting an synchronization output signal 124 , wherein the synchronization output signal corresponds to the crosstalk reduction frame (step 550 ).
- the secondary memory 608 includes, for example, one or more hard drives 610 and/or a removable storage drive 612 , representing a removable flash memory card, etc., where a copy of the program code for the method 400 may be stored.
- the removable storage drive 612 reads from and/or writes to a removable storage unit 614 in a well-known manner.
- Non-transitory computer readable storage devices that may be used to implement the present invention include but are not limited to conventional computer system RAM, ROM, EPROM, EEPROM and magnetic or optical disks or tapes. Concrete examples of the foregoing include distribution of the programs on a CD ROM or via Internet download. In a sense, the Internet itself is a computer readable medium. The same is true of computer networks in general. It is therefore to be understood that any interfacing device and/or system capable of executing the functions of the above-described examples are encompassed by the present invention.
- any of the memory components described 606 , 608 , 614 may also store an operating system 630 , such as Mac OS, MS Windows, Unix, or Linux; network applications 632 ; and a display controller component 630 .
- the operating system 630 may be multi-participant, multiprocessing, multitasking, multithreading, real-time and the like.
- the operating system 630 may also perform basic tasks such as recognizing input from input devices, such as a keyboard or a keypad; sending output to the display 620 ; controlling peripheral devices, such as disk drives, printers, image capture device; and managing traffic on the one or more buses 604 .
- the network applications 632 includes various components for establishing and maintaining network connections, such as software for implementing communication protocols including TCP/IP, HTTP, Ethernet. USB, and FireWire.
- the computing apparatus 600 may also include an input devices 616 , such as a keyboard, a keypad, functional keys, etc., a pointing device, such as a tracking ball, cursors, mouse 618 , etc., and a display(s) 620 .
- a display adaptor 622 may interface with the communication bus 604 and the display 620 and may receive display data from the processor 602 and convert the display data into display commands for the display 620 .
- the processor(s) 602 may communicate over a network, for instance, a cellular network, the Internet, LAN, etc., through one or more network interfaces 624 such as a Local Area Network LAN, a wireless 402.11 ⁇ LAN, a 3G mobile WAN or a WiMax WAN.
- a network interface 624 such as a Local Area Network LAN, a wireless 402.11 ⁇ LAN, a 3G mobile WAN or a WiMax WAN.
- an interface 626 may be used to receive an image or sequence of images from imaging components 628 , such as the image capture device.
Abstract
Description
- When displaying a projected image and simultaneously capturing images of the projected scene and the display, there can be crosstalk between the projected images and the captured content. The crosstalk can reduce the image quality (i.e. brightness, color) of the captured image frames and additionally cause distracting flicker on the display. Various attempts have been made to reduce crosstalk in the projection/image capture systems. For example, the display may be changed from a passive screen display screen an active switchable diffuser to reduce crosstalk. However, although crosstalk may be reduced when using the active switchable diffuser for the display, an active diffuser display screen can limit the useful duty cycle of both the projector and the image capture device used in the system.
- The figures depict implementations/embodiments of the invention and not the invention itself. Some embodiments are described, by way of example, with respect to the following Figures.
-
FIG. 1 shows a schematic drawing of a video sequence control system with binary modulation according to an example of the invention; -
FIG. 2 shows the video sequence control system shown inFIG. 1 where the video frame sequence frames produced and output by the video sequence control system is displayed on a see-through display screen according to an example of the invention; -
FIG. 3A shows a schematic drawing of a video sequence control system with spatial masking and modulation having a first modulation pattern according to an example of the invention; -
FIG. 3B shows a schematic drawing of a video sequence control system with spatial masking and modulation having a second modulation pattern according to an example of the invention; -
FIG. 3C shows a schematic drawing of a video sequence control system with spatial masking and modulation having a third modulation pattern according to an example of the invention; -
FIG. 3D shows a schematic drawing of a video sequence control system with spatial masking and modulation having a fourth modulation pattern according to an example of the invention; -
FIG. 4 shows a schematic drawing of a video sequence control system according to an example of the invention; -
FIG. 5 shows a flow diagram for a method for controlling the output of a video sequence according to an example of the invention; -
FIG. 6 shows a computer system for implementing the method shown inFIG. 5 described in accordance with examples of the present invention. - The drawings referred to in this Brief Description should not be understood as being drawn to scale unless specifically noted.
- For simplicity and illustrative purposes, the principles of the embodiments are described by referring mainly to examples thereof. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the embodiments. It will be apparent, however, to one of ordinary skill in the art, that the embodiments may be practiced without limitation to these specific details. Also, different embodiments may be used together. In some instances, well known methods and structures have not been described in detail so as not to unnecessarily obscure the description of the embodiments.
- The application describes a video
sequence control system 100 comprising: an inputvideo frame buffer 102 for receiving avideo stream 104 ofvideo frames 106 from avideo source 110; and an output videoframe selection component 112 for determining a video frame to be output according to ascheduler 116 that providestiming information 118 andmodulation information 119, wherein thetiming information 118 includes information regarding when each video frame will be output, wherein modulation information varies dependent upon frame type to be output, wherein frame type includes at least the image frames from the input video frame buffer and functional frames, wherein the at least the image frames and functional frames are output according to a pattern defined by the scheduler, wherein based on the timing information 118 asynchronization output signal 124 is output corresponding to the output of the functional frame. - The video sequence controller takes a video stream input from a video source and outputs a (1) modified video stream and (2) a synchronization output signal. Referring to
FIG. 2 , the videosequence control system 100 receives avideo stream 104 from the video source at a first rate. Thevideo sequence controller 100 receives the input video stream, modifies the video stream and outputs the modifiedvideo stream 136 at a second frame rate. In one example, the modification of the video stream occurs based on aschedule 117 that details the insertion a functional video frame into theoutput video stream 136. - Whether an input frame or other frame is modified to create the video output stream and how frames slotted for modification are changed is dependent upon frame type. At least two frame types are available for output, including input video frames and functional frames. Input video frames refer to the input video that are not modified and directly output in the output video frame sequence. The functional frames are frames other than the input video frames. The functional video frames often have a purpose other than the output of video images. For example, in the implementation described in
FIGS. 1 , 2 and 3A the functional video frames are crosstalk reduction frames. The crosstalk reduction frames are added for the purpose of reducing crosstalk in the system in order to improve image quality. In the example shown inFIGS. 3B and 3D , the functional frames are color channel varying frames. In one example, the purpose of varying the color channel is to add color channel varying frames to the output video stream to facilitate green-screening. Throughout the application, the term functional frame and the alternative functional description of the frame (i.e., crosstalk reduction frame, color channel varying frame) may be used interchangeably according to the implementation. -
FIG. 1 shows a schematic drawing of a videosequence control system 100 according to an example of the invention. The video sequence control system includes an inputvideo frame buffer 102 for receiving avideo stream 104 ofvideo frames 106 from avideo source 110. Various sources may be used for the generating theinput video stream 104. For example, video could originate from video streamed from the network. In another example, the video source could be a graphics adapter in a computing device. In another example, the video source could be a video camera that inputs the captured images to the graphics adapter in a computing device. - Referring to
FIG. 1 , the videosequence control system 100 includes an output videoframe selection component 112 for determining the video frame to be output according to ascheduler 116. In one example, the output videoframe selection component 112 includes amodulation component 114 that modifies the input video stream that is coupled to ascheduler 116.Timing information 118 is used by theoutput selection component 112 to determine the timing of when video frames are output.Modulation information 119 in thescheduler 116 can be used by themodulation component 114 to determine whether the input video frame needs to be modified and if the video frame needs to be modified which input video frames are modified and how the video input frames selected for modification are modified before being output. - For the example shown in
FIG. 1 , how the input video stream is modified and the resultant video stream output is dependent upon the output videoframe selector schedule 117. Referring toFIG. 1 shows avisual representation 115 of the videoframe output schedule 117. In the example shown inFIG. 1 , the output videoframe selection component 112 can be described as a binary modulator. In one implementation of a binary modulator, thenumber 1 can be representative of the image frame and thenumber 0 can be representative of the black crosstalk reduction frame. Alternatively, thenumber 1 can be representative of a black crosstalk reduction frame and thenumber 0 can be representative of the input image frame. In one example, thearrow 126 represents a schedule pointer that points to a frame that is under consideration (to be modified/output) according to the outputvideo frame schedule 117. - Assume for purposes of example, that the
number 0 in the schedule is representative of an image frame and that thenumber 1 is representative of the black crosstalk reduction frame. In the example shown inFIG. 1 , when the pointer of the selection component points to an image frame (represented by a 0 in the schedule), the output videoframe selection component 112 selects an image frame for output. However, for the case where the pointer is pointing to the functional frame (the crosstalk reduction frame) in the schedule, the modulator changes the image in the input file according to the modification information in the schedule. Referring toFIG. 1 where the modulation schedule specifies a black crosstalk video frame—the modulator modifies the image component frame so that the pixels in the image frame are black. The selection component then outputs the modified image frame. - In an alternative example, instead of the modulator modifying each pixel of the corresponding image frame, a software or hardware implemented representation of the crosstalk reduction frame is used. For example, say for a black (crosstalk reduction) frame is required for every pixel in the video frame. The pixel value of zero could be stored and the value repeated for each pixel location in the video frame when a black frame is present on the schedule. For this case, the entire image frame may not need to be stored. In one case a single pixel value is stored. The timing information in the schedule can be used to determine when the crosstalk reduction frame is to be output. When the crosstalk reduction frame is output, the pixel value of zero is repeated for every pixel in the frame.
- The
scheduler 116 of the output videoframe selection component 112 includestiming information 118 andmodulation information 119. In one example, thetiming information 118 includes information regarding when each video frame will be output. For example, timinginformation 119 includes information about the number of frames to be output within a certain time period and when each frame in the scheduled sequence of video frames will be output. In one example, the scheduler determines the output sequence of the video frames at least in part based on the of the frame rate of the video input and the required video output. Thus for example, if the frame rate of the video source is 30 Hz and the required video output is 120 Hz, then for each video input image frame—four video frames are output. In the example shown inFIG. 1 , theschedule 117 alternates between image frames (frames 130 and 134) and crosstalk reduction frames (frames 128 and 132). Thus for each image frame in the video input buffer a crosstalk reduction frame is output. This pattern (alternating image and crosstalk frames) is illustrated in thevirtual representation 115 of the schedule and the correspondingvideo output sequence 136. The video frames are output at equal time periods. Thus, for a 120 Hz output, a frame is output every 0.25 seconds. - Referring to
FIG. 1 , the videosequence control system 100 receives the video source input at a first frame rate and outputs a modified video stream at a second frame rate. In one example, where the videosequence control system 100 is outputting video at a higher frame rate than the video input, the video output videoframe selection component 112 repeats unmodified input image frames 106 and inserts the unmodified image frames into theoutput video stream 136 to accomplish the required frame rate. Where the videosequence control system 100 is outputting video at the same frame rate than the input video or lower, in one example frames are dropped (lower frame rate) or alternatively some of the existing frames (i.e. same frame rate) are dimmed. In one example, by dimmed we mean that the functional (i.e. crosstalk reduction frame) has an intensity lower than the corresponding input image frames in the same video sequence. The dimmed image can be a reduced intensity version of the black frame (i.e. a gray frame) or alternatively, a dimmed version of the image frame. - The frame rate and the video sequence pattern (sequence of frame types) effects whether the input video frames are dropped, inserted or remain the same in the output video frames 136. For the video sequence pattern shown in the visual representation of the
schedule 117 inFIG. 1 for example, the schedule pattern includes an alternating sequence of image video frames from theinput video stream 104 and crosstalk reduction frames, where the crosstalk are dimmed (black) video frames. If the input video streams and output video streams are both 60 Hz, then in one example, half of the video input frames could be modified (dimmed) to provide the video sequence pattern (alternating image and crosstalk reduction frames) shown on thevisual representation 115 of the schedule. - In one example, the
scheduler 116 includesframe type information 120, that provides information regarding the type of video frame that is currently being processed (the video frame in the visual representation of the schedule that thepointer 126 is pointing to.) In one example, the scheduled frame types include at least crosstalk reduction frame types and image frame types. In one example, modification information is associated with the frame type information to provide information on whether the frame type is selected for output with no modification or whether the frame type is modified. Further, if the frame type is to be modified—how each frame type is modified. For example, if the frame type is a crosstalk reduction frame—the image frame might be modified to be a uniform black frame, where the black frame is output in the output video stream for display. - In one example, the
scheduler 117 of the output videoframe selection component 112 includesmodification information 119, timinginformation 118,frame type information 120, and sequenceorder information 122 which thescheduler 117 uses to determine whether input image frames are modified, how the input frames are modified, when the modified or unmodified frames are output, and the sequence order that the frame types are output. In one example, thesequence order information 112 provides information to themodification component 114 of the order and frame type that the video frames are output. In the example shown inFIG. 1 , the output video sequence can be described generally as at least one crosstalk reduction from followed by a sequence of n image frames. In the example shown inFIG. 1 , therepetitive pattern 138 is a pattern of alternating crosstalk reduction frames and image frames. However, the output video sequence could be different dependent upon the desired output and results. For example, the video sequence could be a crosstalk reduction frame followed by three image frames. This video sequence may be preferred where the video sequence is displayed as the increased number of image frames in the sequence compared to the first example (single crosstalk reduction frame followed by a single image frame), improves the quality of the displayed image. - In one example, the
timing information 118 provides information regarding when and how often the selected (whereschedule pointer 126 is currently pointing to) video frame is output. In one example, thetiming information 118 is used to determine when asynchronization output signal 124 is output. Thesynchronization output signal 124 should correspond to the timing of the output of the crosstalk reduction frame. -
FIG. 2 shows the video sequence control system shown inFIG. 1 where the output video frame sequence frames are displayed on a see-throughdisplay screen 150 according to an example of the invention. In the example shown inFIG. 2 the videosequence control system 100 outputs avideo frame sequence 136 to theprojector 148 that drives thedisplay 150. The videosequence control system 110 outputs asynchronization output signal 124 to theimage capture device 146 to signal to theimage capture device 146 when image capture of the scene should occur. - The video
sequence control system 100 acts as a specialized video processor. It can modify frame rate by inserting or removing frames and can modify image frames. In one example, the video sequence control system is a standalone device positioned between thevideo source 110 and the devices that it is outputting video frames and/or synchronization signals to. For example, inFIG. 2 the videosequence control system 100 is positioned between thevideo source 110 and theprojector 148. Also, the videosequence control system 100 is positioned between thevideo source 110 and thevideo camera 146. - In one example, the
crosstalk reduction frame 112 is a dimmed or reduced intensity frame, by dimmed or reduced intensity we mean the crosstalk reduction frame has an intensity lower than the corresponding image frames in the same video sequence. In the example shown inFIG. 2 , the crosstalk reduction frames are black frames. The purpose of the lower intensity video frame is to provide a time interval when theimage capture device 120 is capturing a video frame image of the scene behind the see-through display screen which minimizes the interference or crosstalk caused by the projected image. - In the example shown in
FIG. 2 , thevideo frame output 136 is projected onto adisplay screen 150 by aprojector 148. In one example, the display is a see-throughdisplay screen 150 comprised of a relatively low concentration of diffusing particles embedded within a transparent screen medium. The low concentration of diffusing particles allows acamera 146 or other image capture device to capture an image through the screen 150 (providing that the subject is well lit), while diffusing enough of the light from theprojector 148 to form an image on the display screen. In another example, thedisplay screen 150 can be a holographic film that has been configured to accept light from theprojector 148 within a first range of angles and to transmit light that is visible to the remote participant within a different range of viewing angles. The holographic film is otherwise transparent. In both examples, light projected onto thefirst surface 156 within the first range of angles can be observed by viewing thesecond surface 158, but light striking thesecond surface 158 is transmitted through the screen to the camera. In both examples the camera also captures light from the projector diffused or scattered off of the first surface. - In the example shown in
FIG. 2 , the output crosstalk reduction frames (e.g. 142 a, 142 b) are black frames (no light output) which simulates theprojector 108 being in an off state. In one example, theimage capture device 146 is a video camera. The video camera is synchronized to capture an image during the time period when the crosstalk reduction frame is being displayed. For the case where there is a see-through display screen, thevideo camera 146 captures the scene in front of the display screen, the display and the scene behind the see-through display screen. Thus, for a video conferencing example where there is aparticipant 152 on the other side of the display screen—thevideo camera 146 captures the image of the participant on the other side of the display screen. Because the projector is off when the image is captured, the camera does not capture the projected image. - Referring to
FIG. 2 , the videosequence control system 100 produces asynchronization output signal 124. Thesynchronization output signal 124 is communicatively coupled to animage capture device 146 and is used to provide a signal to theimage capture device 146 related to the occurrence of the at least one crosstalk reduction frame (e.g. 142 a, 142 b). Thesynchronization output signal 124 is used to trigger image acquisition by the image capture device in order to capture an image when a crosstalk reduction frame 142 is being projected. - In one example, the
synchronization output signal 124 is directly coupled to theimage capture device 146. In the example shown inFIG. 2 , the image capture device is communicatively coupled to the synchronization output signal via anoptional signal converter 154. In one example, the signal converter could reduce the frequency of the synchronization trigger frequency. In another example, aimage capture device 146 might require a specific signal pattern that follows an industry standard protocol. For this case, thesignal converter 154 could translate a simple triggeringsynchronization output signal 124 to the pattern according to industry standard protocol. - In one example, the
synchronization output signal 124 provides both a synchronization signal and a command signal to theimage capture device 146. In one example, thesynchronization output signal 124 provides information to the image capture device of when the crosstalk reduction frame is being projected. The command signal line can provide information to the image capture device of what camera function is to perform. In one example, the command signal to the image capture device provides a command to the image capture device to open or close its shutter. In the example shown inFIG. 2 , the timing of the image acquisition is done in synchronization with the projection of the crosstalk reduction frame. - In one example, a
synchronization output signal 124 is sent to theimage capture device 146 to instruct it to open its shutter (take a picture) when the crosstalk rejection frames are projected onto the display screen. For example, assuming the blackcrosstalk reduction frame 142 a is projected at a time tporj1. Then the camera opens its shutter at this time, captures an image and outputs acorresponding photograph 160 a. The corresponding photograph is of the display, the scene in front of the display and for the case of a see-through screen, the viewer behind the display screen. Similarly assuming a second blackcrosstalk rejection frame 142 b is projected at time tpro, then the camera opens its shutter at this time, captures an image and outputs thecorresponding photograph 160 b. -
FIG. 3A shows a schematic drawing of a video sequence control system shown inFIG. 1 with an alternative modulation pattern according to the schedule. Themodulation component 114 of the videosequence control system 100 takes an input video and generates output video by dropping video frames, repeating video frames, or inserting dimmed, masked or otherwise modulated images. Similar to the example shown inFIG. 1 , the modulation pattern shown in the visual representation of the schedule inFIG. 3A alternates between the image frame and a crosstalk reduction frame. However, instead of a black crosstalk reduction as shown in FIG. 1—the crosstalk reduction frame inFIG. 3A is modulated to output a gray frame. - In one example, instead of the crosstalk reduction frame having the highest possible darkness or intensity value (100%)—a black frame, a dimmed image may be implemented. For example, a gray image having some percentage of the possible intensity range. For example, the gray crosstalk reduction frame could have an intensity of 50%. In another example, the dimming can be applied at continuous varying levels. For example, the modified output video could consist of modulated versions of the input video frames where the modulated versions of the input video frames are at 75%, 50% and 25% of the intensity of the input image.
- In another example, the darkness of the crosstalk reduction frame might be reduced from its highest possible intensity (black) to a lower intensity level. One case where this might be a reasonable option would be the case where the projected image frames are likely to be very bright. For the case where the projected images go between a very bright projected image to a black video frame (the crosstalk reduction frame), the high amount of change between the two projected images that the human eye perceives—increases the perception of flashing and flicker. To reduce this effect, in one example, the crosstalk reduction frame color could be changed from black to a gray color (lower intensity value.) In another example, where the display screen is not bright (in some cases where a see-through display is used), it is not necessary to have the crosstalk reduction frame be completely black in order to substantially reduce crosstalk. In one example, fully software-controllable dimming allows crosstalk reduction to be turned on/off or to vary continuously between zero and full. In the example shown in
FIG. 1 , thecrosstalk reduction frame 112 is a black frame (uniform distribution of black pixels across the video frame.) In another example, the color or intensity value of pixels can be applied in a spatially varying fashion across the modulated frame. -
FIG. 3B shows a schematic drawing of the videosequence control system 100 shown inFIG. 3A having a second alternative modulation (or masking) pattern. Both the modulation pattern shown inFIGS. 3A and 3B apply dimming to the video frame according to theschedule 117, however, the image frames are modulated according to a schedule that applies dimming to each color channel of the multichannel input video frames. In one example, dimming can be applied uniformly to each color channel. In another example, dimming can be applied at a different level to each color channel. In another example, dimming can also be applied differently to each channel of multi-channel video frames. For example, in one implementation dimming could be applied only to the red channel of the video frames. - One potential application of the
video output sequence 100 shown inFIG. 3B is to use the video output to effectively perform “green-screening” techniques using the video output frames. Assume for purposes of example, the videosequence control system 100 shown inFIG. 2 where the video sequence control system outputs avideo output sequence 136 on adisplay screen 150. In one example, assume that thedisplay screen 150 is opaque and that a weatherman is standing in front of the display screen while a video camera is capturing the weatherman standing in front of a green display screen. Previous use of the green-screening techniques may have been to cut out the image of the weatherman and recomposite the weatherman in front of a weather map afterwards. However, it may be desirable for the weatherman to be able to see the display as a TV viewer would see it—instead of seeing a green display screen. - In one example, instead of taking video of the weatherman standing in front of a green screen display, the weatherman is photographed standing in front of a weather map. The modulation pattern shown in the schedule in
FIG. 3B , for example, could be implemented by scheduling a green channel in one time slot (frame 128). The image of the weatherman could be captured against the green channel and in the next time slot the image of the weatherman could be captured against the combined blue and red color channel (frame 130). In the first time slot, the video camera captures an image of the weatherman in front of the green backdrop. In the second time slot, the camera captures an image of the weatherman in front of a combined blue and red backdrop. As long as frame rate of the video output is sufficiently high, the viewer's eyes average the green, blue and red channels so that the viewer sees a full color weather map. - In image processing, there are advantages to using more than one color channel background—for example when you are trying to subtract out a person standing in front of a background. In one example, the image is modulated to present different color video frames that can be used as different backdrops when performing image processing. For example, if an image is captured of a person against a black background and the person is wearing a black jacket—the black jacket will not show up well in the captured image. However, if an image of the same person is against a green background—the black jacket will show up. Alternating image capture against at least two substantially different color backgrounds—means no matter what color the person is wearing, for example, it will be easy to determine all of the colors in the image.
-
FIG. 3C shows a schematic drawing of the videosequence control system 100 shown inFIG. 3A having a third alternative modulation pattern. In the example shown inFIG. 3C , thevideo frame output 136 is comprised of modulated image frames where the modulation occurs according to aschedule 117 that has a pattern that varies in a spatially varying manner. In the example shown inFIG. 3C , the third modulation pattern shown in thevisual representation 115 of theschedule 117 is a crescent shaped pattern that is used as a mask to create a crescent shaped pattern on the input image frame. In one example, the mask is used to perform both spatial masking and color channel masking. For example, for the pattern shown inFIG. 3C , the crescent shaped area could be green while the areas outside of the crescent shape could be blue. - Although the shape in
FIG. 3C is a crescent shape, other shapes and patterns could be used. For example, referring toFIG. 3D theschedule 117 shows a modulation pattern that instead of dividing the frame into crescent shaped regions—divides each frame into four rectangular regions.FIG. 3D shows a schematic drawing of a video sequence control system shown inFIG. 3A having a fourth alternative modulation pattern. In the example shown inFIG. 3D , each of the fourrectangular regions 128 a-d, 130 a-d, 132 a-d and 134 a-d have two different color channels—two rectangular regions having a red color channel (represented by the letter “R”) and two rectangular regions having a combined green+blue color channel (represented by the letters G+B). In the example shown inFIG. 3D , two sequential frames alternate color patterns so that similar to the example described with respect toFIG. 3B , the color patterns alternate to provide all of the colors (RGB) within two sequential frames—so that the viewer sees the full color palate. - For purposes of example, consider the two sequential video frames in the schedule video frames 128 and 130. Each of the video frames in the schedules has four
rectangular regions 128 a-128 d and 130 a-130 d. Each of sequential video frames have two rectangular regions having a red color channel (128 b, 128 c and 130 a, 130 d) and two rectangular regions having a combined green+blue color channel (128 a, 128 d and 130 a, 130 d). The color patterns of the two sequential video frames alternate—for example, if the color channel is red invideo frame 128, it is combined green+blue invideo frame 130 and vice versa. Thus,video frame region 128 b is red and in its next (sequential) video frame the video frame region 130 b is combined+blue. Similarly,video frame region 128 a is green+blue, while the corresponding sequential video frame 130 a is red. Similar to the alternating color channel video frames inFIG. 3B , the result of the pattern of alternating regions in the video frames video is that the viewer of thevideo output sequence 136 sees the full color palate of the inputvideo frame images 104. -
FIG. 4 shows a schematic drawing of a time multiplex control system with binary modulation according to an example of the invention. Similar to the configuration shown inFIG. 1 , the configuration shown inFIG. 3 includes an inputvideo frame buffer 102 for receiving avideo stream 104 of input video frames 106 from avideo source 110 and anoutput selection component 112 for determining video frames to be output according to a schedule. However, instead of theoutput selection component 112 being implemented using a modulator for modifying the input video stream, the output selection component is implemented using a multiplexer that selects the appropriate input from either the inputvideo frame buffer 102 or thecrosstalk reduction buffer 108. In contrast to the implementation shown inFIG. 1 which modifies the input video stream to create a desired crosstalk reduction video frame, the implementation shown inFIG. 4 stores the desired crosstalk reduction video frames in a crosstalk buffer. Thus, compared to the configuration shown inFIG. 1 , a crosstalk reduction frame buffer is added inFIG. 4 . - Referring to
FIG. 4 , in one example theoutput selection component 112 includes amultiplexer 114 which selects a particular frame type for output. The multiplexer has as possible inputs: video frames from theinput frame buffer 102 or crosstalk reduction frames from the crosstalkreduction frame buffer 108. Based on theschedule 115, themultiplexer 114 chooses which video frame from which input buffer (input frame buffer or crosstalk reduction buffer) will be selected for output. In one example, themultiplexer 114 is implemented in hardware, software or a combination of the two. -
FIG. 5 shows a flow diagram for a method for controlling the output of a video sequence according to an example of the invention. The method includes the steps of: receiving a sequence of input video frames from a video source (step 510); determining when to output the input video frames from timing information provided to ascheduler 116 of the output selection component (step 520); determining whether the input video frames will be modified based on scheduler modulation information, wherein modulation information varies dependent upon the frame types available to be output, wherein the available frame types include at least image frames from the inputvideo frame buffer 102 and crosstalk reduction frames (step 530); outputting the at least input video frames and modified crosstalk reduction frames according to the scheduler (step 540); and outputting ansynchronization output signal 124, wherein the synchronization output signal corresponds to the crosstalk reduction frame (step 550). -
FIG. 6 shows a computer system for implementing the methods shown inFIGS. 5A and 5B and described in accordance with the examples herein. Thecomputing apparatus 600 includes one or more processor(s) 602 that may implement or execute some or all of the steps described in the method 400. Commands and data from theprocessor 602 are communicated over acommunication bus 604. Thecomputing apparatus 600 also includes amain memory 606, such as a random access memory (RAM), where the program code for theprocessor 602, may be executed during runtime, and asecondary memory 608. Thesecondary memory 608 includes, for example, one or morehard drives 610 and/or aremovable storage drive 612, representing a removable flash memory card, etc., where a copy of the program code for the method 400 may be stored. Theremovable storage drive 612 reads from and/or writes to aremovable storage unit 614 in a well-known manner. - These methods, functions and other steps described may be embodied as machine readable instructions stored on one or more computer readable mediums, which may be non-transitory. Exemplary non-transitory computer readable storage devices that may be used to implement the present invention include but are not limited to conventional computer system RAM, ROM, EPROM, EEPROM and magnetic or optical disks or tapes. Concrete examples of the foregoing include distribution of the programs on a CD ROM or via Internet download. In a sense, the Internet itself is a computer readable medium. The same is true of computer networks in general. It is therefore to be understood that any interfacing device and/or system capable of executing the functions of the above-described examples are encompassed by the present invention.
- Although shown stored on
main memory 606, any of the memory components described 606, 608, 614 may also store anoperating system 630, such as Mac OS, MS Windows, Unix, or Linux;network applications 632; and adisplay controller component 630. Theoperating system 630 may be multi-participant, multiprocessing, multitasking, multithreading, real-time and the like. Theoperating system 630 may also perform basic tasks such as recognizing input from input devices, such as a keyboard or a keypad; sending output to thedisplay 620; controlling peripheral devices, such as disk drives, printers, image capture device; and managing traffic on the one ormore buses 604. Thenetwork applications 632 includes various components for establishing and maintaining network connections, such as software for implementing communication protocols including TCP/IP, HTTP, Ethernet. USB, and FireWire. - The
computing apparatus 600 may also include aninput devices 616, such as a keyboard, a keypad, functional keys, etc., a pointing device, such as a tracking ball, cursors,mouse 618, etc., and a display(s) 620. Adisplay adaptor 622 may interface with thecommunication bus 604 and thedisplay 620 and may receive display data from theprocessor 602 and convert the display data into display commands for thedisplay 620. - The processor(s) 602 may communicate over a network, for instance, a cellular network, the Internet, LAN, etc., through one or
more network interfaces 624 such as a Local Area Network LAN, a wireless 402.11×LAN, a 3G mobile WAN or a WiMax WAN. In addition, aninterface 626 may be used to receive an image or sequence of images from imagingcomponents 628, such as the image capture device. - The foregoing description, for purposes of explanation, used specific nomenclature to provide a thorough understanding of the invention. However, it will be apparent to one skilled in the art that the specific details are not required in order to practice the invention. The foregoing descriptions of specific embodiments of the present invention are presented for purposes of illustration and description. They are not intended to be exhaustive of or to limit the invention to the precise forms disclosed. Obviously, many modifications and variations are possible in view of the above teachings. The embodiments are shown and described in order to best explain the principles of the invention and its practical applications, to thereby enable others skilled in the art to best utilize the invention and various embodiments with various modifications as are suited to the particular use contemplated. It is intended that the scope of the invention be defined by the following claims and their equivalents:
Claims (14)
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/US2012/027004 WO2013130051A1 (en) | 2012-02-28 | 2012-02-28 | System and method for video frame sequence control |
Publications (2)
Publication Number | Publication Date |
---|---|
US20140375664A1 true US20140375664A1 (en) | 2014-12-25 |
US9413982B2 US9413982B2 (en) | 2016-08-09 |
Family
ID=49083093
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/373,512 Active 2032-07-03 US9413982B2 (en) | 2012-02-28 | 2012-02-28 | System and method for video frame sequence control |
Country Status (2)
Country | Link |
---|---|
US (1) | US9413982B2 (en) |
WO (1) | WO2013130051A1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112235554A (en) * | 2020-10-15 | 2021-01-15 | 上海交通大学医学院附属第九人民医院 | Method for realizing software synchronization of projector and camera |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP6700731B2 (en) * | 2015-11-13 | 2020-05-27 | キヤノン株式会社 | Projection device and projection system |
US11582422B2 (en) * | 2021-02-24 | 2023-02-14 | Gn Audio A/S | Conference device with multi-videostream capability |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20050017939A1 (en) * | 2003-07-25 | 2005-01-27 | Nec Electronics Corporation | Display driver and image display method |
US20080001881A1 (en) * | 2006-06-30 | 2008-01-03 | Kabushiki Kaisha Toshiba | Liquid crystal display and image display method |
US20110025726A1 (en) * | 2009-07-28 | 2011-02-03 | Canon Kabushiki Kaisha | Hold-type image display apparatus and display method using the hold-type image display apparatus |
US20110187840A1 (en) * | 2010-02-04 | 2011-08-04 | Chunghwa Picture Tubes, Ltd. | Three dimensional display and method thereof |
US20120098825A1 (en) * | 2010-10-22 | 2012-04-26 | Hannstar Display Corp. | Driving method for display |
US20120189212A1 (en) * | 2011-01-24 | 2012-07-26 | Alcatel-Lucent Usa Inc. | Method and apparatus for comparing videos |
US20130077057A1 (en) * | 2010-09-07 | 2013-03-28 | Dai Nippon Printing Co., Ltd. | Projection-type footage display device |
Family Cites Families (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2001292408A (en) | 2000-04-07 | 2001-10-19 | Sony Corp | Video editing device, video editing method, vide editing system, and computer-readable recording medium recorded with video editing program |
JP2003087601A (en) | 2001-09-10 | 2003-03-20 | Nec Corp | Video frame sequence holding method |
JP4887750B2 (en) | 2005-11-16 | 2012-02-29 | カシオ計算機株式会社 | Image processing apparatus, control method, and program |
JP5344846B2 (en) | 2008-03-31 | 2013-11-20 | ゴールドチャームリミテッド | Display panel control device, liquid crystal display device, electronic device, and display panel drive control method |
US8300087B2 (en) | 2009-10-23 | 2012-10-30 | Broadcom Corporation | Method and system for response time compensation for 3D video processing |
US20110234777A1 (en) | 2009-11-02 | 2011-09-29 | Panasonic Corporation | Three-demensional display apparatus and three-dimensional display system |
WO2011071465A1 (en) | 2009-12-09 | 2011-06-16 | Thomson Licensing | Progressive video reformatting for film-based content |
TWI413804B (en) | 2010-03-22 | 2013-11-01 | Amtran Technology Co Ltd | 3d video display method and system for enhancing black frame insertion effect |
JP2012029235A (en) | 2010-07-27 | 2012-02-09 | Toshiba Corp | Electronic device and image processing method |
CN102196290A (en) | 2011-03-23 | 2011-09-21 | 深圳创维-Rgb电子有限公司 | Three-dimensional (3D) image display control method and system |
-
2012
- 2012-02-28 US US14/373,512 patent/US9413982B2/en active Active
- 2012-02-28 WO PCT/US2012/027004 patent/WO2013130051A1/en active Application Filing
Patent Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20050017939A1 (en) * | 2003-07-25 | 2005-01-27 | Nec Electronics Corporation | Display driver and image display method |
US20080001881A1 (en) * | 2006-06-30 | 2008-01-03 | Kabushiki Kaisha Toshiba | Liquid crystal display and image display method |
US20110025726A1 (en) * | 2009-07-28 | 2011-02-03 | Canon Kabushiki Kaisha | Hold-type image display apparatus and display method using the hold-type image display apparatus |
US20110187840A1 (en) * | 2010-02-04 | 2011-08-04 | Chunghwa Picture Tubes, Ltd. | Three dimensional display and method thereof |
US20130077057A1 (en) * | 2010-09-07 | 2013-03-28 | Dai Nippon Printing Co., Ltd. | Projection-type footage display device |
US20120098825A1 (en) * | 2010-10-22 | 2012-04-26 | Hannstar Display Corp. | Driving method for display |
US20120189212A1 (en) * | 2011-01-24 | 2012-07-26 | Alcatel-Lucent Usa Inc. | Method and apparatus for comparing videos |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112235554A (en) * | 2020-10-15 | 2021-01-15 | 上海交通大学医学院附属第九人民医院 | Method for realizing software synchronization of projector and camera |
Also Published As
Publication number | Publication date |
---|---|
WO2013130051A1 (en) | 2013-09-06 |
US9413982B2 (en) | 2016-08-09 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US8780161B2 (en) | System and method for modifying images | |
EP3054424B1 (en) | Image rendering method and apparatus | |
CN101242476B (en) | Automatic correction method of image color and digital camera system | |
US9280034B2 (en) | Dynamic lighting | |
EA018293B1 (en) | Video and telepresence processing method | |
US20180025521A1 (en) | Dynamic modulation for near eye display | |
Daly et al. | Preference limits of the visual dynamic range for ultra high quality and aesthetic conveyance | |
KR101765428B1 (en) | Live scene recognition allowing scene dependent image modification before image recording or display | |
CN106782385B (en) | A kind of light-dimming method and its device for wearing display equipment | |
JP2011053373A (en) | Stereoscopic video display device and stereoscopic video display method | |
US8570357B2 (en) | Systems and methods for reducing video crosstalk | |
US20170280090A1 (en) | Binocular display and method for displaying images | |
US9413982B2 (en) | System and method for video frame sequence control | |
CN116569248A (en) | System and method for PWM control of video display screens | |
US9489102B2 (en) | System and method of modifying lighting in a display system | |
JP2020532821A (en) | Lighting methods and systems that improve the perspective color perception of images observed by the user | |
US20120320144A1 (en) | Video modification system and method | |
US9332162B2 (en) | System and method for video frame sequence control | |
US8947563B2 (en) | Reducing crosstalk | |
Farrell et al. | A cinema luminance range by the people, for the people: viewer preferences on luminance limits for a large-screen environment | |
CN116324827A (en) | Glare reduction in images | |
CN108628570A (en) | Display control unit, display control method and storage medium | |
Iwaida et al. | P‐216L: Late‐News Poster: Reproduction of Perceptual Reality in Standard‐Dynamic‐Range (SDR) Environments Using High‐Dynamic‐Range (HDR) Images Compressed by Global Tone Mapping | |
EP4300945A1 (en) | Image processing device, image processing method, and projector device | |
WO2023026543A1 (en) | Information processing device, information processing method, and program |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: HEWLETT-PACKARD DEVELOPMENT COMPANY, L.P., TEXAS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:TAN, KAR-HAN;SOBEL, IRWIN E;JAM, MEHRBAN;REEL/FRAME:033509/0694 Effective date: 20120229 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
CC | Certificate of correction | ||
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |
|
FEPP | Fee payment procedure |
Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |