CA2065668A1

CA2065668A1 - Spectroscopically correlated light scanning microscopy

Info

Publication number: CA2065668A1
Application number: CA002065668A
Authority: CA
Inventors: Wolfgang Jacobsen
Original assignee: Individual
Current assignee: Bayer AG
Priority date: 1991-04-12
Filing date: 1992-04-09
Publication date: 1992-10-13
Also published as: DE4111903A1; EP0508257A3; DE59206974D1; JPH05142144A; DE4111903C2; EP0508257B1; US5329352A; EP0508257A2

Abstract

Spectroscopically correlated light scanning microscopy Abstract The present invention relates to a process for producing and correlating light microscope images and spectroscopic data resolved according to wavelength of a sample by scanning individual elements of the sample surface to be imaged once or twice with a confocal scanning light microscope, launching a portion of the light from the imaging beam path into a spectrometer and correlating the image data with the spectroscopic data by storing the spectroscopic data in a two-dimensional area, with one dimension being used to store the measured spectrum of the individual elements and the second dimension being activated by means of the light intensity diffusely reflected by the scanned elements or by means of a criterion obtained from the total data of the sample image by image processing.

The advantage of said process is that the potential of a confocal scanning light microscope and of the various spectroscopic processes may be fully exploited.

Le A 28 365-foreign countries

Description

2 ~

Spectroscopically correlated light scanning microscopy The present invention relates to a process for producing and correlating light microscope images and spectroscopic data resolved according to wavelength of a sample by means of confocal scanning light-microscopy and spectrometers.

It is known practice (Applied Physics 22 (1980), p.119) to produce light microscope images of transparent or semitransparent test objects using scanning light microscopy. If the beam path corresponds to the principle of the confocal light microscope (CLSM), optical sectional images are obtained, i.e. images of a narrow zone around the focal plane of the microscope lens. If the focal plane lies within the tested object, then owing to the confocal principle the intensities from the regions of the sample lying above and below are to a large extent eliminated.
For some years now, appliances operating on said principle have been commercially available.

It is further known practice to acquire data about the chemical structure of a test object by examining it using spectroscopic techniques, in particular by using light from the visible region of the spectrum or from regions close to the visible spectrum.

The combination of light-microscopy and spectroscopic techniques is also known from literature:
A concept already realized by most manufacturers of commercial confocal light microscopes is that of producing diffuse reflection images in the fluorescence contrast.
For this purpose, the sample is illuminated by a monochromatic light source. There is inserted into the imaging beam path an optical filter which as completely as possible retains the light having the wavelength of the Le A 28 365 - foreign count~ies 2~5~

illuminating light source, so that only the fluorescent light having a longer wavelength reaches the photoelectric detector.

It is further known, from Microscopia Acta 79 (1977), 3, p.267-276, to add to the imaging beam path of a conventional light microscope a spectrograph which has been adjusted to a specific Raman line of a selected substance present in the test object. In said manner, a microscopic Raman dark field image of the object is produced. Said image reproduces the local distribution of the selected substance in the test object.

From Nature 347, (20.09.1990) p.301-303, it is known practice to use the concept of the confocal optical beam path in order to be able to record the complete Raman spectrum of a small pre-selectable measuring volume within the test object. For said purpose, the test object is illuminated by a laser by way of a stationary arrangement according to the concept of the confocal beam path, and the light passing throuqh the aperture of the imaging beam path is analyzed in a Raman spectrometer. In said manner, the Raman spectrum of a selected measuring volume in the order of magnitude of 1 ~m**3 is obtained~
The drawback of all the~e processes ~nown from literature which combine light-microscopy and spectroscopic techniques is that it is impossible to exploit the full potential of both techniques simultaneously:
The standard process of confocal fluorescence microscopy used up till now utilizes only the mean intensity of fluorescence transmitted by the measuring filt~r to build up the image. The fine details of the fluorescence spectrum are on the other hand not utilized. The process 3S according to Microscopia ~cta 79 ~1977), 3, p.267-276, does not supply three-dimensional microscopic data, like Le A 28 365 3 2~65~68 confocal light microscopy, but only two-dimensional images and, moreover, of the data available in the Raman spectra it uses only those of a previously selected line when producing an image.

The process according to Nature 247, p.301-303, because of the absence of scanning microscope image production, likewise does not offer any three-dimensional microscopic data but does provide the entire Raman spectrum of the examined sample volume.

The aim therefore was to discover a process which allows the full potential of confocal scanning light microscopes and spectroscopic techniques to be exploited.

In particular, the aim was to discover a data-reducing type of correlation between the image of a sample obtained by the confocal scanning light microscope and its spectrum.
Said aim was achieved by carrying out the following steps:

a) scanning the individual elements of the sample surface to be imaged once or twice with a confocal scanning light microscope;

b) launching a portion of the light from the imaging beam path into a spectrometer;

c) correlating the image data with the spectroscopic data by storing the spectroscopic data in a two-dimensional area, with one dimension being used to store the measured spectrum of the individual elements and the second dimension being activited by means of the light intensity diffusely reflected by the scanned elements or by means of a criterion obtained from the total Le A 28 365 4 2 a ~ s data of the sample image by image processing.

Further preferred forms of implementation are indicated in the sub-claims.

As a confocal scanning light microscope, all arrangements are suitable in which the sample piece is scanned by a focusing light beam and in which a portion of the transmitted or scattered light is imaged by an imaging optical beam path onto an aperture or onto a system of apertures and in which the portion of said light passing through the aperture is measured by a photoelectric detector and, with the aid of said measuring signal, an image of the entire sample piece is produced. In particular, arrangements based on the concept of the confocal laser scanning microscope (cf. e.g. T. Wilson Ed., Confocal Microscopy, Academic Press London etc. 1990) and arrangements based on the concept of the rotating aperture disk may be cited. The image of the sample piece produced in said manner may be both a sectional image perpendicular to the optical axis of the microscope as well as the image of a cutting plane of any desired orientation, in particular of a cutting plane which extends in the direction of the optical axis of the microscope.
The spectroscopic measuring processes may be any methods which are based on recording the wavelength-dependent intensity of visible light or of light having a wavelength in the vicinity of visible light, e.g. of lO0 nm to 20 ~m.
Methods to be cited in particular are absorption spectroscopy in the ultraviolet, visible, near-infrared and infrared region as well as fluorescence and Raman spectroscopyO The excitation wavelength for fluorescence spectroscopy again lies in the region of lO0 nm to 20 ~m, preferably in the region of 300 nm to 700 nm. The excitation wavelength for Raman spectroscopy also lies in Le A 28 365 5 ~ 0 ~ 8 the region of 100 nm to 20 ~m, preferably in the region of 250 nm to 1.5 ~m.

As a simultaneously recording spectrometer, all arrangements may be cited which allow, for the entire wavelength range applicable to the spectroscopic process in question or for parts thereof, time resolution measurement of the radiant intensity contained in the measuring light per wavelength interval. A preferred arrangement is a single or multiple spectrograph, with a particularly preferred arrangement being a triple spectrograph arrangement with a line or monoplane photoelectric detector. A detector which is particularly preferred is a two-dimensional CCD array (charge coupled device).
It is astonishing that, despite the confocal light microscope having been in commercial use for many years, no full combination with spectroscopic techniques has as yet been achieved. However, an immediate and direct combination, without special data reducing measures, leads to enormous data rates and data volumes, as the following numerical example demonstrates: spectroscopic intensity measurements with 8 bit resolution and at 1,024 wavelength interpolation points lead, in conjunction with the production of an image of 512*512 pixels, to a data volume of 268 MB per image and hence, given an image frequency of 1 Hz, to a data rate of ~68 MB/sec.

The process according to the invention allows the simultaneous or consecutive production of a confocal, light-microscope, optical sectional image on the one hand and a set of spectroscopic intensity distributions on the other hand, and does so in such a way that, on the basis of said data, the associated mean spectrum of the spectroscopic process used may then be indicated for each suitably selected portion of this optical sectional image.
Le A 28 365 6 20~3~i~8 In other words, separate data may be acquired relating to the chemical structure of the morphological structural elements which become evident as a result of the differing contrast in the light microscope image. Examples are material identification of the individual phases in multiphase polymers, location and identification of additives and impurities in polymers or location of active substances in biological preparations.

The process according to the invention is explained in greater detail using the following examples and the drawings.

Fig.l shows the basic set-up for effecting the process, Fig.2 illustrates how the process is effected with mechanical allocation of the spectra to the pixels, Fig.3 illustrates how the process is effected with electronic allocation of the spectra to the pixels, Fig.4 shows the division of a sample image into five objects and the background by an image-analyzing system.
The basic manner of effecting the process is described using the example of a confocal laser scanning microscope:

The light emitted by the laser 1 is focused by a focusing lens 2 onto the entrance aperture 3. Said aperture is imaged by the microscope lens 4 onto or into the sample 5 to be examined. The deflection unit 6 lying between aperture and lens leads to linear scanning of the selected sample piece by said laser beam.
The light diffusely reflected by the sample is collected Le A 28 365 7 2~5~&~8 again by the microscope lens 4 and focused by the beam splitter 7 onto a second aperture 8. A portion of the light passing through said aperture is guided through the beam splitter 9 by way of a further lens 10 to a photoelectric detector 11. The electric signal produced by the photoelectric detector is amplified and converted from analog to digital in the signal processing unit 12. The digitized CLSM image may be further processed and stored in a computer system 13 and may be displayed at an output device 14.

The portion of the backscatter intensity deflected by the beam splitter 9 is imaged by the lens 15 onto the entrance slit of the spectrograph 16. From said portion, the spectrograph 16 simultaneously produces a spectrum analysis which is detected by the line or monoplane photoelectric detector 17. The electric signal of said photoelectric detector is amplified in the spectroscopic evaluation unit 18.
In both examples, the combination of a confocal laser scanning microscope with a Raman spectrograph is described.
The examples differ from one another in their manner of recording the spectroscopic measuring data.
Example l describes deflection of the Raman scattered light by means of a rotating mirror.

The technical implementation details for this example are illustrated in Fig.2.

A commercial CLSM 21 is combined with a commercial triple spectrograph 16 in that a portion of the measuring light passing through the measuring aperture is separated from the measuring channel of the CLSM by a beam splitter 9 and introduced by way of the optical fibres 24 into the Le A 28 365 8 2~6~8 spectrograph. The first two stages of the spectrograph retain the elastically (i.e. with no wavelength change) scattered laser light. What remains is the light scattered by fluorescence or the Ra~.an effect. Said light is subjected to spectrum analysis in the third stage of the spectrograph and imaged onto the cooled two-dimensional CCD
array 17. The local coordinates on the CCD array, alo-g which spectrum analysis of the measuring light is effected, are referred to hereinafter as x.
Situated between the exit slit of the spectrograph and the CCD array is a relay lens 26 with an electromotively controlled rotating mirror 27. The rotating mirror allows the measuring light to be deflected in the y direction of the array, perpendicular to the x direction mentioned above.

Said rotating mirror is then controlled by means of the intensity of the light scattered elastically in the sample, said intensity being measured in the second stage of the spectrograph 16 by the photoelectric detector 28. On the other hand, this is also the variable which produces the contrast in the conventional CLSM diffuse reflection image.
The signal of the photoelectric detector is amplified in the amplifier 29.

The Raman (and fluorescence) intensity measured during a complete scanning process for producing a CLSM image in the CCD array is then read out and transmitted to the computer 18 for further processing. To improve the signal-to-noise ratio, an average may be taken from a plurality of CLSM
scans.

A critical factor for the reliability of the evaluations 3~ just described is the tuning of the setting time of the rotating mirror to the scanning frequency of the CLSM. For Le A 28 365 9 5 ~

it is necessary to ensure that the rotating mirror can react quickly enough to the variations with time of the diffuse reflection intensity which correspond to the structure of the examined sample. If a local resolution of 5 pixels is required for the spectroscopic allocation and the CLSM image has a resolution of 512*512 pixels and an image frequency of 1 Hz, a mirror setting time of about 20 ~sec is required. If the image frequency is reduced, e.g.
to 0.05 Hz, a mirror setting time of about 0.4 msec is adequate.

Example 2 describes electronic sorting of the Raman scatter intensities.

The technical implementation details of this example are illustrated in Fig.3.

The optical and optoelectronic design of the confocal laser scanning microscope 31 and of the spectrograph 32 is identical to the arrangement described in Example 1. In this example, the Raman (and fluorescence) intensity emerging from the spectrograph directly strikes a one-dimensional CCD array ~7 with 1,024 elements. Spectrum analysis of the spectrograph is so selected that the relevant spectral region is just picked up by said CCD
array.

Transfer of the Raman intensity from the primary photoelectric detectors to the output line is effected synchronously to detection of the actual CLSM image in that the synchronizing signal 34 of the CLSM electronics is transmitted to the CCD array. Said Raman intensities are amplified in the readout electronics 35 and result, after analog-to-digital conversion, in a data record of 1024*8bit=1 kB per pixel. Said data records are, synchronously to the building of the conventional CLSM
Le A 28 365 10 2 ~

image, added up in a memory array 36 of N locations each for 1 Kb. The number N usually lies in the region of l<=N<=10. Allocation is effected by means of a look-up table (LUT) in which one of the addresses 1 to N is input for each pixel. If allocation is effected by determining the intensity, resolved into N regions, of the light elastically scattered in the sample and measured in the second stage of the spectrometer, said allocation may be easily achieved by a single scan of the sample by, for example, using the intensity signal, which in the first example controlled th~ mirror, to address the memory locations of the Raman spectra in the memory array 36.
Otherwise, the LUT is calculated in a step precedlng pick-up of the Raman intensities:
For this purpose, a conventional CLSM backscatter image is taken of the same sample piece and transmitted to the image-analyzing system 37. Said device carries out image cleaning and image processing operations on said image, using methods to be defined in a problem-specific manner, with the aim of object identification. For example, all singly or doubly connected objects may be detected, which in themselves present an approximately equal backscatter intensity and are sufficiently distinct in this intensity from their environment and which moreover have a specific minimum size. Thus, the imaged sample piece is divided into a number of objects and - residually - their environment. The latter may in turn comprise a plurality of sub-regions. In other words, thP overall result is N
objects, of which the image of the sample piece is composed. Fig.4 reproduces a typical example of such a division (a, b, c, d, e and background f). If the number N is still too great for the subsequent Raman detection, the objects are classified on the basis of additional criteria. Finally, the objects or classes of object are allocated consecutive numbers. All that has to be done to Le A 28 365 11 20~5~8 set up the above-mentioned LUT is to determine for each pixel, to which of these N objects it belongs. Said LUT is then transmitted from the image-analyzing compu~er to the control electronics of the Raman signal detector 38.

On completion of the two steps:

a) production of the conventional CLSM image and image-analytical calculation of the LUT and b) pick-up of the Raman intensities separately for each of the objects defined under a), the Raman spectrum of each of the objects identified in the CLSM image may be analyzed. If the chemical composition of the sample as a whole is known, e.g. as a result of other analytical processes, the measured Raman spectrum of each of said objects may be compared with the known Raman spectra of the chemical constituents contained in the sample. In particular, using known mathematical methods (e.g. known from Appl. Spectrosc.33 ~1979), p.351-357), the proportion of said chemical constituents in the individual objects of the CLSM backscatter image may be quantitatively determined.

Le A 28 365 12

Claims

1. Process for producing and correlating light microscope images and spectroscopic data resolved according to wavelength of a sample comprising steps of a) scanning of individual elements of the sample surface to be imaged once or twice with a confocal scanning light microscope;

b) launching of a portion of the light from the imaging beam path into a spectrometer;

c) correlation of the image data with the spectroscopic data by storing the spectroscopic data in a two-dimensional region, with one dimension being used to store the measured spectrum of the individual elements and the second dimension being activated by means of the light intensity diffusely reflected by the scanned elements or by means of a criterion obtained from the total data of the sample image by image processing.

2. Process according to claim 1, whereby the spectrum is measured and digitized using a photoelectric detector array and is stored in a memory selected from a finite number of data memories by means of the digitized value of the diffusely reflected light intensity.

Le A 28 365-US 13

3. Process according to claim 1, whereby the spectrum of a scanned element is focused as a line onto a two-dimensional photoelectric detector array and said spectrum is displaced perpendicular on said array using a deflection unit within the beam path of the spectrometer with said unit being controllable by the diffusely reflected light intensity of the scanned element.

4. Process according to claim 1, whereby one- or two-dimensional arrangements of light sensitive semi-conductors or photoelectric multipliers are used as photoelectric detectors.

5. Process according to claim 1, whereby the spectrum of a scanned element is focused as a line onto a two-dimensional photoelectric detector array and said spectrum is displaced perpendicular on said array using a electrically driven rotating mirror within the beam path of the spectrometer with said unit being controllable by the diffusely reflected light intensity of the scanned element.

6. Process according to claim 1, whereby the sample image obtained in a first scanning operation is divided by the image processing system into a plurality of regions, to which the spectra obtained from a second scanning of the sample are allocated.

Le A 28 365-US 14

7. Process according to claim 1, whereby a microscope with a monochromatic light source or a microscope with a light source emitting a continuous spectrum is used as a confocal scanning light microscope.

8. Process according to claim 1, whereby one or a combination of the following spectrometers is used as a spectrometer:

a) Raman spectrometer;
b) fluorescence spectrometer;
c) ultraviolet spectrometer;
d) visual spectrometer;
e) infrared spectrometer.
Le A 28 365-US 15