Deep-ROCS: from speckle patterns to superior-resolved images by deep learning in rotating coherent scattering microscopy

Rotating coherent scattering (ROCS) microscopy is a label-free imaging technique that overcomes the optical diffraction limit by adding up the scattered laser light from a sample obliquely illuminated from different angles. Although ROCS imaging achieves 150 nm spatial and 10 ms temporal resolution, simply summing different speckle patterns may cause loss of sample information. In this paper we present Deep-ROCS, a neural network-based technique that generates a superior-resolved image by efficient numerical combination of a set of differently illuminated images. We show that Deep-ROCS can reconstruct super-resolved images more accurately than conventional ROCS microscopy, retrieving high-frequency information from a small number (6) of speckle images. We demonstrate the performance of Deep-ROCS experimentally on 200 nm beads and by computer simulations, where we show its potential for even more complex structures such as a filament network.

For more details:

A. Saguy, F. Jünger, A. Peleg, B. Ferdman, E. Nehme, A. Rohrbach, and Y. Shechtman, "Deep-ROCS: from speckle patterns to superior-resolved images by deep learning in rotating coherent scattering microscopy", Optics Express, 29 (15), 23877-23887 (2021)

Multicolor Single-Particle-Tracking by Multiplexed PSF Engineering

PSF engineering enables 3D tracking of single sub-diffraction emitters by encoding depth into the measured image. The variability in measured PSF shapes can be further exploited to obtain spectral information of the emitters. Such benefit is obtained by separating the emission into separate spectral channels, and allotting different PSFs to each of the channels. This enables the use of single-channel phase masks, which are fairly easy to fabricate. By multiplexing the channels, spectral information is provided by the PSF while maintaining high-resolution 3D localizations. The use of multiple PSFs enables full use of the camera sensor, facilitating a large field of view.

To the right, a schematic drawing of the optical system is added, and the analysis of a large number of diffusing fluorescent beads is shown. The beads whose trajectories are shown, marked A,B,C, are visible through different spectral channels, colored by blue, orange and red respectively. The channel is determined by the orientation of the Tetrapod PSF.

For more details:

N. Opatovski, Y. S. Ezra, L. E. Weiss, B. Ferdman, R. Orange and Y. Shechtman, "Multiplexed PSF Engineering for Three-Dimensional Multicolor Particle Tracking", Nano Letters (2021)


Learning Optimal Wavefront Shaping for Multi-channel Imaging


Fast acquisition of depth information is crucial for accurate 3D tracking of moving objects. Snapshot depth sensing can be achieved by wavefront coding, in which the point-spread function (PSF) is engineered to vary distinctively with scene depth by altering the detection optics. In low-light applications, such as 3D localization microscopy, the prevailing approach is to condense signal photons into a single imaging channel with phase-only wavefront modulation to achieve a high pixel-wise signal to noise ratio. Here we show that this paradigm is generally suboptimal and can be significantly improved upon by employing multi-channel wavefront coding, even in low-light applications. We demonstrate our multi-channel optimization scheme on 3D localization microscopy in densely labelled live cells where detectability is limited by overlap of modulated PSFs. At extreme densities, we show that a split-signal system, with end-to-end learned phase masks, doubles the detection rate and reaches improved precision compared to the current state-of-the-art, single-channel design. We implement our method using a bifurcated optical system, experimentally validating our approach by snapshot volumetric imaging and 3D tracking of fluorescently labelled subcellular elements in dense environments.

For more details:

E. Nehme*, B. Ferdman*, L. E. Weiss; T. Naor; D. Freedman; T. Michaeli; Y. Shechtman. "Learning optimal wavefront shaping for multi-channel imaging", in IEEE Transactions on Pattern Analysis and Machine Intelligence, doi: 10.1109/TPAMI.2021.3076873 (2021). (*equal contribution)

3D Printable Diffractive Optical Elements by Liquid Immersion

Diffractive optical elements (DOEs) are used to shape the wavefront of incident light. This can be used to generate practically any pattern of interest, albeit with varying efficiency. A fundamental challenge associated with DOEs comes from the nanoscale-precision requirements for their fabrication. Here we demonstrate a method to controllably scale up the relevant feature dimensions of a device from tens-of-nanometers to tens-of-microns by immersing the DOEs in a near-index-matched solution. This makes it possible to utilize modern 3D-printing technologies for fabrication, thereby significantly simplifying the production of DOEs and decreasing costs by orders of magnitude, without hindering performance. We demonstrate the tunability of our design for varying experimental conditions, and the suitability of this approach to ultrasensitive applications by localizing the 3D positions of single molecules in cells using our microscale fabricated optical element to modify the point-spread-function (PSF) of a microscope.

For more details:

R. Orange-Kedem, E. Nehme, L. E. Weiss, B. Ferdman, O. Alalouf, N. Opatovski and Y. Shechtman, "3D printable diffractive optical elements by liquid immersion", Nature Communications 12, 3067 (2021)


Automated Analysis of Fluorescence Kinetics in Single-Molecule Localization Microscopy Data Reveals Protein Stoichiometry

Understanding the function of protein complexes requires information on their molecular organization, specifically, their oligomerization level. Optical super-resolution microscopy can localize single protein complexes in cells with high precision, however, the quantification of their oligomerization level, remains a challenge. Here, we present a Quantitative Algorithm for Fluorescent Kinetics Analysis (QAFKA), that serves as a fully automated workflow for quantitative analysis of single-molecule localization microscopy (SMLM) data by extracting fluorophore “blinking” events. QAFKA includes an automated localization algorithm, the extraction of emission features per localization cluster, and a deep neural network-based estimator that reports the ratios of cluster types within the population. We demonstrate molecular quantification of protein monomers and dimers on simulated and experimental SMLM data. We further demonstrate that QAFKA accurately reports quantitative information on the monomer/dimer equilibrium of membrane receptors in single immobilized cells, opening the door to single-cell single-protein analysis. 

Figure caption: Oligomerization state of HGF-stimulated and unstimulated MET-mEos4b receptors in a stable HEK293T cell line. (a) Temporal sum over time of a PALM experiment of a single HEK293T cell (marked with white dashed line) that expresses MET-mEos4b (scale bar = 2.5 um). The oligomerization state of MET-mEos4b was analyzed with QAFKA in resting and HGF-stimulated cells. (b) Cumulative density function of the number of blinking events without ligand stimulation and in presence of 1 nM HGF. (c) Estimation of monomer and dimer fractions with QAFKA in absence and presence of 1 nM HGF.

For more details:

A. Saguy, T. N. Baldering, L. E. Weiss, E. Nehme, C. Karathanasis, M. S. Dietz, M. Heilemann, and Y. Shechtman, "Automated Analysis of Fluorescence Kinetics in Single-Molecule Localization Microscopy Data Reveals Protein Stoichiometry", The Journal of Physical Chemistry B Article ASAP  DOI: 10.1021/acs.jpcb.1c01130 (2021)


Microscopic scan-free surface profiling over extended axial ranges by point-spread-function engineering

The shape of a surface, i.e. its topography, influences many functional properties of a material; hence, characterization is critical in a wide variety of applications. Two notable challenges are profiling temporally changing structures, which requires high-speed acquisition, and capturing geometries with large axial steps. Here we leverage point-spread-function (PSF) engineering for scan-free, dynamic, micro-surface profiling. The presented method is robust to axial steps and acquires full fields of view at camera-limited framerates. We present two approaches for surface profiling implementation:

  1. Fluorescence-based: using fluorescent emitters scattered on a surface of a 3D dynamic object (inflatable membrane in this example).

  2. Label-free: using pattern of illumination spots projected onto a reflective sample (tilting mirror in this example).

Both implementations demonstrate the applicability to a variety of sample geometries and surface types.

For more details:

R. Gordon-Soffer, L. E. Weiss, R. Eshel, B. Ferdman, E. Nehme, M. Bercovici, and Y. Shechtman, "Microscopic scan-free surface profiling over extended axial ranges by point-spread-function engineering", Science Advances 6, 44 (2020).


DeepSTORM3D: dense 3D localization microscopy and PSF design by deep learning

Localization microscopy is an imaging technique in which the positions of individual point emitters (e.g. fluorescent molecules) are precisely determined from their images. Localization in 3D can be performed by modifying the image that a point-source creates on the camera, namely, the point-spread function (PSF). The PSF is engineered to vary distinctively with emitter depth, using additional optical elements. However, localizing multiple adjacent emitters in 3D poses a significant algorithmic challenge, due to the lateral overlap of their PSFs. In our lastest work, DeepSTORM3D, we presented two different applications of CNNs in dense 3D localization microscopy:

  1. Learning an efficient 3D localization CNN for a given PSF entirely in silico (Tetrapod in this example)

  2. Learning an optimized PSF for high density localization via end-to-end optimization

For more details:

Nehme, E., Freedman, D., Gordon, R. et al. "DeepSTORM3D: dense 3D localization microscopy and PSF design by deep learning". Nature Methods (2020).

Three-dimensional localization microscopy in live flowing cells


Here, we have demonstrated that by merging two technologies: point-spread-function engineering and imaging flow cytometry, we can attain excellent spatial detail with extremely high sample throughput! Essential to our approach is calibrating the imaging system. This is accomplished with a novel method that analyzes the statistical distributions of tiny fluorescent beads that are imaged alongside cells suspended in media. This is represented in the attached graphic, where the images of randomly positioned objects on the left are analyzed collectively to produce the shape-to-depth calibration on the right. This calibration is then applied to the images of fluorescently labeled positions within cells.

For more details:

L. E. Weiss, Y. Shalev Ezra, S. Goldberg, B. Ferdman, O. Adir, A. Schroeder, O. Alalouf, Y. Shechtman. "Three-dimensional localization microscopy in live flowing cells"  Nature nanotechnology (2020).

VIPR: Vectorial Implementation of Phase Retrieval

With work previously done by a former lab member, we have developed a powerful tool for use in the identification and characterization of the processes in our model system. A major advantage of this development is its improved sensitivity, which allows it to detect subtle dynamic property changes in response to our experimentation.

For more details:

B. Ferdman, E. Nehme, L. E. Weiss, R. Orange, O. Alalouf, Y. Shechtman "VIPR: Vectorial Implementation of Phase Retrieval for fast and accurate microscopic pixel-wise pupil estimation" bioRxiv (2020)


Deep learning for diffusion characterization

Diffusion characterization.jpg

We implement a neural network to classify single-particle trajectories by diffusion type: Brownian motion, fractional Brownian motion (FBM) and Continuous Time Random Walk (CTRW). Furthermore, we demonstrate the applicability of our network architecture for estimating the Hurst exponent for FBM and the diffusion coefficient for Brownian motion on both simulated and experimental data. The networks achieve greater accuracy than MSD analysis on simulated trajectories while requiring as few as 25 steps. On experimental data, both net and MSD analysis converge to similar values, with the net requiring only half the number of trajectories required for MSD to achieve the same confidence interval.

For more details:
N. Granik, L. E. Weiss, E. Nehme, M. Levin, M. Chein, E. Perlson, Y. Roichman, Y. Shechtman "Single particle diffusion characterization by deep learning",  Biophysical Journal 117, 2,185-192 (2019).

Ultrasensitive refractometry


We present a refractometry approach in which the fluorophores are preattached to the bottom surface of a microfluidic channel, enabling highly-sensitive determination of the Refractive Index using tiny amounts of liquid by detecting the Supercritical Angle Fluorescence (SAF) effect at the conjugate back focal plane of a high NA -obejective.


The SAF effect (presented above) is the propagation of evanescent waves in the higher refractive index immersion medium, which captures the change in the transfer coefficients, observed as a strong transition ring.

We demonstrate the relevance of our system for monitoring changes in biological systems. As a model system, we show that we can detect single bacteria (Escherichia coli) and measure population growth.

For more details:
B. Ferdman*, L.E. Weiss*, O. Alalouf, Y. Haimovich, Y. Shechtman "Ultrasensitive refractometry via supercritical angle fluorescence", ACS Nano, 12, 11892-11898 (2018). (*Equal Contribution)
*** See also ACS Nano perspective (12/2018)

Deep-learning for super-resolution localization microscopy


Deep-learning for multicolor localization microscopy

Deep learning has been shown to be an effective tool for image classification. Here we demonstrate that capability extends to distinguishing the colors of single emitters from grayscale images as well. This was done by training a convolutional neural network (CNN) on a library of images containing up to four types of quantum dots with different emission wavelengths embedded in a polymer matrix, then evaluating the net with new images. Surprisingly, we found that the same approach was applicable to the much more challenging problem of classifying moving emitters as well, where the chromatic-dependent subtleties in the point-spread function (PSF) are distorted by motion blur. The performance of the neural net in these two applications show that such an approach be used to simplify the design of multicolor microscopes by replacing hardware components with downstream software analysis.
In a second application of neural nets, we have shown how a phase-modulating element, which can be used to control the shape of the PSF, can be designed in parallel with net training in order to optimize the ability of the net to distinguish the position and color of the object. This approach produces novel phase masks that increase the net’s ability to categorize emitters while maintaining other desirable properties, namely, the localizability of emitters. This approach for mask optimization solves a longstanding problem in PSF-engineering: how a phase mask can be optimally designed to encode for any parameter of interest.

For more details:
E. Hershko, L.E. Weiss, T. Michaeli, Y. Shechtman, “Multicolor localization microscopy and point-spread-function engineering by deep learning“, Optics Express 27, 5, 6158-6183 (2019)


In localization microscopy, regions with a high density of overlapping emitters pose an algorithmic challenge. Various algorithms have been developed to handle overlapping PSFs, all of which suffer from two fundamental drawbacks: data-processing time and sample-dependent paramter tuning.
Recently we demonstrated a precise, fast, parameter-free, super-resolution image reconstruction by harnessing Deep-Learning. By exploiting the inherent additional information in blinking molecules, our method, dubbed Deep-STORM,  creates a super-resolved image from the raw data directly. Deep-STORM is general and does not rely on any prior knowledge of the structure in the sample.

For more details:
E. Nehme, L.E. Weiss, T. Michaeli, and Y. Shechtman, "Deep-STORM: Super Resolution Single Molecule Microscopy by Deep Learning", Optica 4,  458-464 (2018).
*** See also Nature Methods highlight ( May 2018)

And also piece in Technion-Magazine (in Hebrew) here


How, and to what precision, can one determine the 3D position of a sub-wavelength particle by observing it through a microscope? This is the problem at the heart of methods such as single-particle-tracking and localization based super-resolution microscopy (e.g. PALM, STORM). One useful way of achieving such 3D localization at nanoscale precision is to modify the point-spread-function (PSF) of the microscope so that it encodes the 3D position in its shape.We have recently asked the basic question – what is the optimal way to modify a microscope’s PSF in order to encode the 3D position (x,y,z) of a point emitter in the most efficient way? We approach this challenge by solving an optimization problem: Find a pupil-plane phase pattern that yields a PSF which is maximally sensitive to small changes in the particle’s position. Formulated mathematically, this sensitivity corresponds to the Fisher-Information of the system. The result is the saddle-point PSF (bottom right panel in figure above)

For more details:
Y. Shechtman, S.J. Sahl, A.S. Backer and W.E. Moerner, "Optimal point spread function design for 3D imaging", Physical Review Letters 113, 133902 (2014).

Optimal-3D and multicolor PSF engineering

Background: Optimal-3D and multicolor PSF engineering

Extremely-large-range PSFs for 3D localization microscopy


Our PSF optimization method can be used to generate PSFs with unprecedented capabilities, like an extremely large, modular axial (z) range of up to 20 um. The resulting optimal large-range PSFs belong to a family of PSFs we call the Tetrapod PSFs (see image above)

Simulation of different Tetrapod PSFs as a particle is scanned over a 20 micron axial (z) range (from -10 um to +10 um and back)

We demonstrate experimentally the applicability of these Tetrapod PSFs in micro-fluidic flow profiling over a 20um z range, and in tracking under noisy biological conditions.

For more details:
Y. Shechtman, L.E. Weiss,  A.S. Backer, S.J. Sahl and W.E. Moerner, "Precise 3D scan-free multiple-particle tracking over large axial ranges with Tetrapod point spread functions", Nano Letters,DOI:10.1021/acs.nanolett.5b01396 (2015).

Multicolor 3D PSFs 

Often in fluorescent microscopy one is interested in observing several different types of fluorescently labeled objects. Commonly, this is done by labeling different objects with different colors. How can you distinguish between different colors using a highly sensitive grayscale detector (e.g. EMCCD)?

One approach is to separate the emission light into different color channels with dichroic elements. Alternatively, is possible to  switch between emission filters and image sequentially. But how do  you simultaneously image multiple colors on a single multiple channel with no additional elements (other than a 4F system with a phase mask)? The answer is multicolor PSF engineering.

By exploiting the spectral response of the phase modulating element, it is possible to design masks that create different phase delays for different colors, and therefore enable simultaneous multicolor 3D tracking or multicolor super-resolution imaging.

For more details:
Y. Shechtman, L.E. Weiss,  A.S. Backer, M.Y. Lee and W.E. Moerner, "Multicolor localization microscopy by point-spread-function engineering“,Nature Photonics 10, (2016).
*** See also Nature Methods highlight ( September 2016)