Inference in artificial intelligence with deep optics and photonics – Nature.com

Inference in artificial intelligence with deep optics and photonics – Nature.com

  • 1.

    LeCun, Y. et al. Handwritten digit recognition with a back-propagation network. In Advances in Neural Information Processing Systems 2 (NIPS 1989) (ed. Touretzky, D. S.) 396–404 (1990).

  • 2.

    Krizhevsky, A., Sutskever, I. & Hinton, G. E. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25 (NIPS 2012) (eds Pereira, F. et al.) 1097–1105 (2012).

  • 3.

    LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

    ADS  CAS  PubMed  Google Scholar 

  • 4.

    Miller, D. A. B. Waves, modes, communications, and optics: a tutorial. Adv. Opt. Photonics 11, 679–825 (2019).

    ADS  Google Scholar 

  • 5.

    Brunner, D., Soriano, M. C., Mirasso, C. R. & Fischer, I. Parallel photonic information processing at gigabyte per second data rates using transient states. Nat. Commun. 4, 1364 (2013).

    ADS  PubMed  PubMed Central  Google Scholar 

  • 6.

    Goodman, J. W., Leonberger, F. J., Kung, S.-Y. & Athale, R. A. Optical interconnections for VLSI systems. Proc. IEEE 72, 850–866 (1984). The first paper to provide a substantial analysis and reasons for the use of optics in interconnection (rather than for logic) in digital systems.

    ADS  Google Scholar 

  • 7.

    Miller, D. A. B. Rationale and challenges for optical interconnects to electronic chips. Proc. IEEE 88, 728–749 (2000).

    Google Scholar 

  • 8.

    Miller, D. A. B. Attojoule optoelectronics for low-energy information processing and communications. J. Lightwave Technol. 35, 346–396 (2017).

    ADS  CAS  Google Scholar 

  • 9.

    Miller, D. A. B. Are optical transistors the logical next step? Nat. Photon. 4, 3–5 (2010).

    ADS  CAS  Google Scholar 

  • 10.

    Athale, R. & Psaltis, D. Optical computing: past and future. Opt. Photon. News 27, 32–39 (2016).

    Google Scholar 

  • 11.

    Goodman, J. W. Introduction to Fourier Optics (Roberts and Co, 2005).

  • 12.

    Liutkus, A. et al. Imaging with nature: compressive imaging using a multiply scattering medium. Sci. Rep. 4, 5552 (2014).

    CAS  PubMed  PubMed Central  Google Scholar 

  • 13.

    Saade, A. et al. Random projections through multiple optical scattering: approximating kernels at the speed of light. In 2016 IEEE Intl Conf. Acoustics, Speech and Signal Processing (ICASSP) 6215–6219 (IEEE, 2016).

  • 14.

    Lin, X. et al. All-optical machine learning using diffractive deep neural networks. Science 361, 1004–1008 (2018). An optical implementation using multiple optimized layers for all-optical image classification.

    ADS  MathSciNet  CAS  MATH  PubMed  Google Scholar 

  • 15.

    Chang, J., Sitzmann, V., Dun, X., Heidrich, W. & Wetzstein, G. Hybrid optical–electronic convolutional neural networks with optimized diffractive optics for image classification. Sci. Rep. 8, 12324 (2018). An optical implementation of a single CNN layer demonstrated for hybrid optical–electronic image classification.

    ADS  PubMed  PubMed Central  Google Scholar 

  • 16.

    Rosenblatt, F. The Perceptron, A Perceiving and Recognizing Automaton Report no. 85-460-1 (Project Para, Cornell Aeronautical Laboratory, 1957).

  • 17.

    Hebb, D. O. The Organization of Behavior (Wiley, 1949).

  • 18.

    Widrow, B. & Hoff, M. E. Adaptive switching circuits. In 1960 IRE WESCON Convention Record 96–104 (Institute of Radio Engineers, 1960).

  • 19.

    Hopfield, J. J. Neural networks and physical systems with emergent collective computational abilities. Proc. Natl Acad. Sci. USA 79, 2554–2558 (1982).

    ADS  MathSciNet  CAS  MATH  PubMed  Google Scholar 

  • 20.

    Carpenter, G. A. & Grossberg, S. A massively parallel architecture for a self-organizing neural pattern recognition machine. Comput. Vis. Graph. Image Process. 37, 54–115 (1987).

    MATH  Google Scholar 

  • 21.

    Kohonen, T. Self-organized formation of topologically correct feature maps. Biol. Cybern. 43, 59–69 (1982).

    MathSciNet  MATH  Google Scholar 

  • 22.

    Rumelhart, D. E., Hinton, G. E. & Williams, R. J. Learning representations by back-propagating errors. Nature 323, 533–536 (1986).

    ADS  MATH  Google Scholar 

  • 23.

    Mead, C. Neuromorphic electronic systems. Proc. IEEE 78, 1629–1636 (1990).

    Google Scholar 

  • 24.

    Farhat, N. H., Psaltis, D., Prata, A. & Paek, E. Optical implementation of the Hopfield model. Appl. Opt. 24, 1469–1475 (1985). Optical implementation of content-addressable associative memory based on the Hopfield model for neural networks and on the addition of nonlinear iterative feedback to a vector–matrix multiplier.

    ADS  CAS  PubMed  Google Scholar 

  • 25.

    Denz, C. Optical Neural Networks (Springer Science & Business Media, 2013).

  • 26.

    Psaltis, D., Brady, D., Gu, X.-G. & Lin, S. Holography in artificial neural networks. Nature 343, 325–330 (1990). Introduction of nonlinear photorefractive crystals for optical computing.

    ADS  CAS  PubMed  Google Scholar 

  • 27.

    Li, H.-Y. S., Qiao, Y. & Psaltis, D. Optical network for real-time face recognition. Appl. Opt. 32, 5026–5035 (1993).

    ADS  CAS  PubMed  Google Scholar 

  • 28.

    Miller, D. A. B. Self-configuring universal linear optical component. Photon. Res. 1, 1–15 (2013). Proof that arbitrary linear operations such as singular value decompositions can be performed in optics—not just Fourier transforms and convolutions as in early optical computing.

    ADS  Google Scholar 

  • 29.

    Shen, Y. et al. Deep learning with coherent nanophotonic circuits. Nat. Photon. 11, 441 (2017). A silicon photonic neural network using meshes of MZIs for vowel recognition.

    ADS  CAS  Google Scholar 

  • 30.

    Fang, M. Y.-S., Manipatruni, S., Wierzynski, C., Khosrowshahi, A. & DeWeese, M. R. Design of optical neural networks with component imprecisions. Opt. Express 27, 14009–14029 (2019).

    ADS  CAS  PubMed  Google Scholar 

  • 31.

    Wilkes, C. M. et al. 60 dB high-extinction auto-configured Mach–Zehnder interferometer. Opt. Lett. 41, 5318–5321 (2016).

    ADS  CAS  PubMed  Google Scholar 

  • 32.

    Hughes, T. W., Minkov, M., Shi, Y. & Fan, S. Training of photonic neural networks through in situ backpropagation and gradient measurement. Optica 5, 864–871 (2018).

    ADS  Google Scholar 

  • 33.

    Feldmann, J., Youngblood, N., Wright, C. D., Bhaskaran, H. & Pernice, W. H. P. All-optical spiking neurosynaptic networks with self-learning capabilities. Nature 569, 208–214 (2019). A photonic circuit that exploits wavelength division multiplexing techniques for pattern recognition directly in the optical domain.

    ADS  CAS  PubMed  PubMed Central  Google Scholar 

  • 34.

    Tait, A. N. et al. Neuromorphic photonic networks using silicon photonic weight banks. Sci. Rep. 7, 7430 (2017).

    ADS  PubMed  PubMed Central  Google Scholar 

  • 35.

    Huang, C. et al. Giant enhancement in signal contrast using integrated all-optical nonlinear thresholder. In 2019 Optical Fiber Communications Conference and Exhibition (OFC) 415–417 (IEEE, 2019).

  • 36.

    Nahmias, M. A., Shastri, B. J., Tait, A. N. & Prucnal, P. R. A leaky integrate-and-fire laser neuron for ultrafast cognitive computing. IEEE J. Sel. Top. Quantum Electron. 19, 1800212 (2013).

    Google Scholar 

  • 37.

    Amin, R. et al. ITO-based electro-absorption modulator for photonic neural activation function. APL Mater. 7, 081112 (2019).

    ADS  Google Scholar 

  • 38.

    Williamson, I. A. D. et al. Reprogrammable electro-optic nonlinear activation functions for optical neural networks. IEEE J. Sel. Top. Quantum Electron. 26, 7700412 (2020).

    CAS  Google Scholar 

  • 39.

    Miller, D. A. B. Novel analog self-electrooptic-effect devices. IEEE J. Quantum Electron. 29, 678–698 (1993).

    ADS  Google Scholar 

  • 40.

    Srinivasan, S. A. et al. High absorption contrast quantum confined stark effect in ultra-thin Ge/SiGe quantum well stacks grown on Si. IEEE J. Quantum Electron. 56, 5200207 (2020).

    Google Scholar 

  • 41.

    Ferreira de Lima, T., Shastri, B. J., Tait, A. N., Nahmias, M. A. & Prucnal, P. R. Progress in neuromorphic photonics. Nanophotonics 6, 577–599 (2017).

    Google Scholar 

  • 42.

    Nahmias, M. A. et al. Photonic multiply–accumulate operations for neural networks. IEEE J. Sel. Top. Quantum Electron. 26, 7701518 (2020). A review article on the state-of-the-art of photonic MACs along with detailed characterizations and comparisons of the performance of photonic and comparable electronic hardware.

    CAS  Google Scholar 

  • 43.

    Gupta, S., Agrawal, A., Gopalakrishnan, K. & Narayanan, P. Deep learning with limited numerical precision. In Proc. 32nd Intl Conf. Machine Learning (eds Bach, F. & Blei, D.) 1737–1746 (PMLR, 2015).

  • 44.

    Hamerly, R., Bernstein, L., Sludds, A., Soljačić, M. & Englund, D. Large-scale optical neural networks based on photoelectric multiplication. Phys. Rev. X 9, 021032 (2019).

    CAS  Google Scholar 

  • 45.

    Lugt, A. V. Signal detection by complex spatial filtering. IEEE Trans. Inf. Theory 10, 139–145 (1964). The introduction of optical correlators.

    MATH  Google Scholar 

  • 46.

    Gregory, D. A. Real-time pattern recognition using a modified liquid crystal television in a coherent optical correlator. Appl. Opt. 25, 467–469 (1986).

    ADS  CAS  PubMed  Google Scholar 

  • 47.

    Manzur, T., Zeller, J. & Serati, S. Optical-correlator-based target detection, recognition, classification, and tracking. Appl. Opt. 51, 4976–4983 (2012).

    ADS  PubMed  Google Scholar 

  • 48.

    Javidi, B., Li, J. & Tang, Q. Optical implementation of neural networks for face recognition by the use of nonlinear joint transform correlators. Appl. Opt. 34, 3950–3962 (1995).

    ADS  CAS  PubMed  Google Scholar 

  • 49.

    Koppal, S. J., Gkioulekas, I., Zickler, T. & Barrows, G. L. Wide-angle micro sensors for vision on a tight budget. In 2011 IEEE Conf. Computer Vision and Pattern Recognition (CVPR 2011) 361–368 (IEEE, 2011).

  • 50.

    Hughes, T. W., Williamson, I. A. D., Minkov, M. & Fan, S. Wave physics as an analog recurrent neural network. Sci. Adv. 5, eaay6946 (2019).

    ADS  PubMed  PubMed Central  Google Scholar 

  • 51.

    Duarte, M. F. et al. Single-pixel imaging via compressive sampling. IEEE Signal Process. Mag. 25, 83–91 (2008).

    ADS  Google Scholar 

  • 52.

    Moretti, C. & Gigan, S. Readout of fluorescence functional signals through highly scattering tissue. Nat. Photonics 14, 361–364 (2020).

    ADS  CAS  Google Scholar 

  • 53.

    Rahmani, B., Loterie, D., Konstantinou, G., Psaltis, D. & Moser, C. Multimode optical fiber transmission with a deep learning network. Light Sci. Appl. 7, 69 (2018).

    ADS  PubMed  PubMed Central  Google Scholar 

  • 54.

    Caramazza, P., Moran, O., Murray-Smith, R. & Faccio, D. Transmission of natural scene images through a multimode fibre. Nat. Commun. 10, 2029 (2019).

    ADS  PubMed  PubMed Central  Google Scholar 

  • 55.

    Li, Y., Xue, Y. & Tian, L. Deep speckle correlation: a deep learning approach toward scalable imaging through scattering media. Optica 5, 1181–1190 (2018).

    ADS  Google Scholar 

  • 56.

    Horisaki, R., Takagi, R. & Tanida, J. Learning-based imaging through scattering media. Opt. Express 24, 13738–13743 (2016).

    ADS  PubMed  Google Scholar 

  • 57.

    Ando, T., Horisaki, R. & Tanida, J. Speckle-learning-based object recognition through scattering media. Opt. Express 23, 33902–33910 (2015).

    ADS  CAS  PubMed  Google Scholar 

  • 58.

    Mahoney, M. W. Randomized Algorithms for Matrices and Data (Now Publishers, 2011).

  • 59.

    Dong, J., Rafayelyan, M., Krzakala, F. & Gigan, S. Optical reservoir computing using multiple light scattering for chaotic systems prediction. IEEE J. Sel. Top. Quantum Electron. 26, 7701012 (2019).

    Google Scholar 

  • 60.

    Gupta, S., Gribonval, R., Daudet, L. & Dokmanić, I. Don’t take it lightly: phasing optical random projections with unknown operators. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019) (eds Wallach, H. et al.) 14855–14865 (2019).

  • 61.

    Marshall, J. & Oberwinkler, J. The colourful world of the mantis shrimp. Nature 401, 873–874 (1999).

    ADS  CAS  PubMed  Google Scholar 

  • 62.

    Thoen, H. T., How, M. J., Chiou, T.-H. & Marshall, J. A different form of color vision in mantis shrimp. Science 343, 411–413 (2014).

    ADS  CAS  PubMed  Google Scholar 

  • 63.

    Wetzstein, G., Ihrke, I., Lanman, D. & Heidrich, W. Computational plenoptic imaging. Comput. Graph. Forum 30, 2397–2426 (2011).

    Google Scholar 

  • 64.

    Hinton, G. E. & Salakhutdinov, R. R. Reducing the dimensionality of data with neural networks. Science 313, 504–507 (2006).

    ADS  MathSciNet  CAS  MATH  PubMed  PubMed Central  Google Scholar 

  • 65.

    Sitzmann, V. et al. End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging. ACM Trans. Graph. 37, 114 (2018). The first demonstration of end-to-end optimization of optics and image processing for a computational camera design with computer vision applications.

    Google Scholar 

  • 66.

    Chakrabarti, A. Learning sensor multiplexing design through back-propagation. In Advances in Neural Information Processing Systems 29 (NIPS 2016) (eds Lee, D. D. et al.) 3081–3089 (2016).

  • 67.

    Martel, J. N. P., Muller, L. K., Carey, S., Dudek, P. & Wetzstein, G. Neural sensors: learning pixel exposures for HDR imaging and video compressive sensing with programmable sensors. IEEE Trans. Pattern Anal. Mach. Intell. 42, 1642–1653 (2020).

    PubMed  Google Scholar 

  • 68.

    Horstmeyer, R., Chen, R. Y., Kappes, B. & Judkewitz, B. Convolutional neural networks that teach microscopes how to image. Preprint at https://arxiv.org/abs/1709.07223 (2017).

  • 69.

    Marco, J. et al. DeepToF: off-the-shelf real-time correction of multipath interference in time-of-flight imaging. ACM Trans. Graph. 36, 219 (2017).

    Google Scholar 

  • 70.

    Su, S., Heide, F., Wetzstein, G. & Heidrich, W. Deep end-to-end time-of-flight imaging. In 2018 IEEE Conf. Computer Vision and Pattern Recognition (CVPR) 6383–6392 (IEEE, 2018).

  • 71.

    Kellman, M., Bostan, E., Repina, N. & Waller, L. Physics-based learned design: optimized coded-illumination for quantitative phase imaging. IEEE Trans. Comput. Imaging 5, 344–353 (2019).

    Google Scholar 

  • 72.

    Sinha, A., Lee, J., Li, S. & Barbastathis, G. Lensless computational imaging through deep learning. Optica 4, 1117–1125 (2017).

    ADS  Google Scholar 

  • 73.

    Metzler, C. A., Ikoma, H., Peng, Y. & Wetzstein, G. Deep optics for single-shot high-dynamic-range imaging. In 2020 IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR) 1372–1382 (IEEE, 2020).

  • 74.

    Luo, Y. et al. Design of task-specific optical systems using broadband diffractive neural networks. Light Sci. Appl. 8, 112 (2019).

    ADS  PubMed  PubMed Central  Google Scholar 

  • 75.

    Haim, H., Elmalem, S., Giryes, R., Bronstein, A. M. & Marom, E. Depth estimation from a single image using deep learned phase coded mask. IEEE Trans. Comput. Imaging 4, 298–310 (2018).

    Google Scholar 

  • 76.

    Chang, J. & Wetzstein, G. Deep optics for monocular depth estimation and 3D object detection. In 2019 IEEE/CVF Intl Conf. Computer Vision (ICCV) 10192–10211 (IEEE, 2019).

  • 77.

    Wu, Y., Boominathan, V., Chen, H., Sankaranarayanan, A. & Veeraraghavan, A. Phasecam3D—learning phase masks for passive single view depth estimation. In 2019 IEEE Intl Conf. Computational Photography (ICCP) 19–30 (IEEE, 2019).

  • 78.

    Bertero, M. & Boccacci, P. Introduction to Inverse Problems in Imaging (CRC Press, 1998).

  • 79.

    Barbastathis, G., Ozcan, A. & Situ, G. On the use of deep learning for computational imaging. Optica 6, 921–943 (2019).

    ADS  Google Scholar 

  • 80.

    Rivenson, Y. et al. Deep learning microscopy. Optica 4, 1437–1443 (2017).

    ADS  Google Scholar 

  • 81.

    Wu, Y. et al. Extended depth-of-field in holographic imaging using deep-learning-based autofocusing and phase recovery. Optica 5, 704–710 (2018).

    ADS  Google Scholar 

  • 82.

    Nehme, E. & Weiss, L. E., Michaeli, T. & Shechtman, Y. Deep-storm: super-resolution single-molecule microscopy by deep learning. Optica 5, 458–464 (2018).

    ADS  CAS  Google Scholar 

  • 83.

    Ouyang, W., Aristov, A., Lelek, M., Hao, X. & Zimmer, C. Deep learning massively accelerates super-resolution localization microscopy. Nat. Biotechnol. 36, 460–468 (2018).

    CAS  PubMed  Google Scholar 

  • 84.

    Christiansen, E. M. et al. In silico labeling: predicting fluorescent labels in unlabeled images. Cell 173, 792–803 (2018).

    CAS  PubMed  PubMed Central  Google Scholar 

  • 85.

    Wu, Y. et al. Three-dimensional virtual refocusing of fluorescence microscopy images using deep learning. Nat. Methods 16, 1323–1331 (2019).

    CAS  PubMed  Google Scholar 

  • 86.

    Wang, H. et al. Deep learning enables cross-modality super-resolution in fluorescence microscopy. Nat. Methods 16, 103–110 (2019).

    CAS  PubMed  Google Scholar 

  • 87.

    Rivenson, Y., Zhang, Y., Günaydın, H., Teng, D. & Ozcan, A. Phase recovery and holographic image reconstruction using deep learning in neural networks. Light Sci. Appl. 7, 17141 (2018).

    CAS  PubMed  PubMed Central  Google Scholar 

  • 88.

    Boyd, N., Jonas, E., Babcock, H. & Recht, B. DeepLoco: Fast 3D localization microscopy using neural networks. Preprint at https://doi.org/10.1101/267096 (2018).

  • 89.

    Weigert, M. et al. Content-aware image restoration: pushing the limits of fluorescence microscopy. Nat. Methods 15, 1090 (2018).

    CAS  PubMed  Google Scholar 

  • 90.

    Nehme, E. et al. DeepSTORM3D: dense 3D localization microscopy and PSF design by deep learning. Nat. Methods 17, 734–740 (2020). An end-to-end optimization approach for point spread function engineering and neural-network-based locations for 3D fluorescence superresolution microscopy.

    CAS  PubMed  Google Scholar 

  • 91.

    Liu, T. et al. Deep learning-based super-resolution in coherent imaging systems. Sci. Rep. 9, 3926 (2019).

    ADS  PubMed  PubMed Central  Google Scholar 

  • 92.

    Zhang, H. et al. High-throughput, high-resolution deep learning microscopy based on registration-free generative adversarial network. Biomed. Opt. Express 10, 1044–1063 (2019).

    PubMed  PubMed Central  Google Scholar 

  • 93.

    Escudero, M. C. et al. Digitally stained confocal microscopy through deep learning. In Proc. 2nd Intl Conf. Medical Imaging with Deep Learning (eds Cardoso, M. J. et al.) 121–129 (PMLR, 2019).

  • 94.

    Rivenson, Y. et al. Deep learning enhanced mobile-phone microscopy. ACS Photonics 5, 2354–2364 (2018).

    CAS  Google Scholar 

  • 95.

    Goy, A., Arthur, K., Li, S. & Barbastathis, G. Low photon count phase retrieval using deep learning. Phys. Rev. Lett. 121, 243902 (2018).

    ADS  CAS  PubMed  Google Scholar 

  • 96.

    Rivenson, Y. et al. Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning. Nat. Biomed. Eng. 3, 466–477 (2019).

    CAS  PubMed  Google Scholar 

  • 97.

    Wu, Y. et al. Bright-field holography: cross-modality deep learning enables snapshot 3D imaging with bright-field contrast using a single hologram. Light Sci. Appl. 8, 25 (2019).

    ADS  PubMed  PubMed Central  Google Scholar 

  • 98.

    Rivenson, Y. et al. PhaseStain: the digital staining of label-free quantitative phase microscopy images using deep learning. Light Sci. Appl. 8, 23 (2019).

    ADS  PubMed  PubMed Central  Google Scholar 

  • 99.

    Mengu, D., Luo, Y., Rivenson, Y. & Ozcan, A. Analysis of diffractive optical neural networks and their integration with electronic neural networks. IEEE J. Sel. Top. Quantum Electron. 26, 3700114 (2019).

    PubMed  Google Scholar 

  • 100.

    Dagenais, M., Sharfin, W. F. & Seymour, R. J. Optical digital matrix multiplication apparatus. EU patent EP0330710A1 (1988).

  • Source

    Leave a Reply

    %d bloggers like this: