Perspective | Open Access
Xiaofeng Li, Yuan Zhou, Fan Wang, "Advanced Information Mining from Ocean Remote Sensing Imagery with Deep Learning", Journal of Remote Sensing, vol. 2022, Article ID 9849645, 4 pages, 2022. https://doi.org/10.34133/2022/9849645
Advanced Information Mining from Ocean Remote Sensing Imagery with Deep Learning
In the past decades, the increasing ocean-research-oriented satellites, sensors, acquisition, and distribution channels have brought new tasks and challenges to mine information from such big data with complex and sparse information. The information mining requirements from big data and the advance in deep learning (DL) technology showed mutual promotive benefits in practical ocean information extraction and DL-based framework development. In 2020, scientists showed that most information retrievals from ocean remote sensing images could be accomplished using existing DL network frameworks, i.e., U-net for semantic segmentation and SSD (Single-Shot Multi-box Detection) for object detection . The U-Net’s almost symmetric encoder-decoder structure and the skip connection between encoder-decoders have an excellent performance in retrieving fundamental semantic segmentation information in the ocean remote sensing imagery, such as coastal inundation area extractions . SSD extracts feature maps of different data scales and takes a priori frames of different scales. Therefore, it has an excellent performance in detecting fundamental object detection problems in the ocean field, such as ship detection .
Although the off-the-shelf DL-based models are helpful, new developments in this field lead to a new era of DL-based technology for ocean remote sensing information mining. Specifically, two developments should be incorporated into the specific task-driven DL model: network architecture advance and domain-knowledge-based (expert knowledge) guidance in model parameter selection.
2. Deep Network Architecture with Attention Mechanism
Ocean remote sensing images and time series resemble the image/video data stream in computer vision. This similarity provides a strong argument that various emerging DL architectures used in computer vision can be adopted to solve critical oceanography problems. A significant emerging trend in the computer vision field has been the attention-based neural networks  that have made exciting progress in classification, regression, anomaly detection, and dynamic modeling . The core of the attention-based neural networks is the attention function. Typically, an attention function represents a mapping between a query, a set of key-value pairs, and an output, where the input, output, and query are all vectors. The output is calculated as a weighted sum of the values, where the weight allocated to each value is determined by a compatibility function between the query and the corresponding key.
The need to add the attention mechanism is two folds. First, according to the U-Net architecture, most existing DL-based ocean models use convolution operations, such as inundation area detection . Convolutional operations process a local neighborhood, either in space or time. However, the convolution operator has a limited receptive field, thus preventing it from modeling long-range pixel dependencies. Secondly, the convolution filters have static weights at inference and cannot flexibly adapt to the input content. This disadvantage is not conducive to capturing signals of some fast-changing variables. The attention mechanism calculates response at a given pixel by a weighted sum of all other positions, thus capturing long-range dependencies with deep neural networks and overcoming the abovementioned issue.
In recent years, the attention-based transformer architecture has outperformed the previous convolutional-based architectures in various DL tasks. Furthermore, several studies have also applied the attention mechanism to ocean remote sensing image processing, such as sea ice detection , sea ice prediction , and cyclone intensity estimation .
Ren et al.  integrated the position and channel attention modules into an original U-Net model to form a dual-attention U-Net model (DAU-Net) for sea ice detection. The DAU-Net integrates the SAR image’s long-range and local-range dependencies, which helps extract more discriminating feature representations for classifying sea ice and open water. Experiments showed that the dual-attention mechanism helps DAU-Net extract more discriminating features than the original U-Net. Ren et al.  proposed an attention-based data-driven model for predicting daily sea ice concentration (SIC) of the Pan-Arctic, termed SICNet. For the Pan-Arctic, spatiotemporal dependencies exist on both global and local ranges. Thus, Ren et al.  designed a temporal-spatial attention module (TSAM) to help the SICNet capture accurate spatiotemporal dependencies. The TSAM employed a temporal convolutional network (TCN) as a temporal attention module to capture the long-range temporal dependencies and a spatial attention module to capture the long-range spatial dependencies. Wang et al.  added the spatial and channel attention mechanisms in the DL model of tropical cyclone (TC) intensity estimation using satellite images. The channel attention layer weights indicate that satellite images at 10.4 and 12.3 μm channels play a significant role in the estimation model. The spatial attention layer weights demonstrate that the DL model focuses on areas with low brightness temperature and TC eye.
The attention mechanism emphasizes the combination of global and local information, which is also in line with the oceanographic problems that require multiscale combined analysis. Figure 1 middle panel shows the newly proposed DL framework with the attention mechanism as a purple box module. As shown in the expanded view of the attention mechanism module, an image is divided into multiple patches. The attention network is modeled between every two patches to capture global information.
3. Incorporating Domain Science Knowledge into the DL Architecture Design
Knowledge-driven and data-driven approaches are complementary to each other. The knowledge-driven approach is based on physical rules to establish governing equations that are directly interpretable. The data-driven approach is based on statistical knowledge, is highly flexible in adapting to the data, and facilitates detecting signals that the governing equations ignore. However, the “black box” nature of the DL structure lacks interpretation. The fusion of domain and data sciences is an increasing trend in solving particular problems in science by DL . Oceanography research is no exception requiring the combination of ocean theory with DL methods. Using domain knowledge can deal with complex ocean problems and alleviate the demand for data in the modeling process. Specifically, ocean theoretical domain knowledge can reduce the degree of freedom of input data dimensionality and thus the training difficulty of DL models.
Ocean domain knowledge can be divided into physical constraints and spatiotemporal data processing methods. Integrating domain knowledge into DL models can be achieved using a multibranch network structure.
Physical knowledge can help to constrain the model construction. For example, using satellite sea surface height (SSH) and sea surface temperature (SST), Liu et al.  designed a dual-branch convolutional neural network with dense connections to simultaneously obtain ocean eddies’ mesoscale dynamic and thermal characteristics. Zhang et al.  solved the small training dataset problem in retrieving internal solitary wave amplitude by combining satellite observations and lab experiments with transfer learning techniques. The lab experiment was specially designed following the similarity law and basic fluid mechanics principles. Furthermore, during the model establishment, ocean background information and internal solitary wave characteristics, which affect the internal solitary wave amplitude, were considered following the domain knowledge guidance. The results show that with domain knowledge informed, the input parameters and model structures can be carefully designed, and better model performance can be achieved.
The knowledge of spatiotemporal data processing has contributed to sea fog detection and mesoscale eddies detection. A dual-branch sea fog detection network is proposed comprising a statistical extraction module and a dual-branch optional module. Specifically, Chen et al.  designed a DL model for efficient detection of sea fog. Sea fog detection is more complicated than other segmentation tasks because of the difficulty of separating clouds and fogs. Aiming at the indistinguishable problem,  analyzed the difference from the reflection principle and designed a knowledge extraction module to extract statistical information in the visual space using prior knowledge. By introducing domain knowledge, the proposed method outperforms advanced semantic segmentation algorithms in sea fog detection; especially, it can effectively detect sea fog in an image with mixed cloud and fog. Mu et al.  developed a hurricane winds retrieval model that fully exerted DL’s powerful data fusion advantage and deeply mined the hurricane information in synthetic aperture radar images. It showed that the model significantly improved the wind speed retrieval accuracy by simultaneously utilizing SAR measured physical parameters in backscattering energy, the texture feature represented by the grey level co-occurrence matrix, and the unique morphological hurricane feature. All that domain science knowledge considered by the DL-based framework is helpful for model training and fitting.
The above studies use domain knowledge and perform well in sea fog detection, mesoscale eddies detection, internal solitary wave retrieval, and wind field retrieval. Furthermore, these methods jointly extract discriminative features from both visual and knowledge domains. Thus, we believe that in other fields of oceanographic research, multibranch networks can also be used to combine domain knowledge to improve the model’s performance further. The newly proposed DL framework (Figure 1 middle panel) is a typical dual-branch network, an example of a simple multibranch network structure. First, the visual branch extracts features from ocean remote sensing images. Then, the features extracted through expert knowledge are input into the network through the knowledge branch. Compared with extracting features from images in the existing AI framework, the dual-branch network provides richer modellable features, reducing modeling difficulty.
This article points out that network architecture advance and domain-knowledge-based (expert knowledge) guidance should be incorporated into the specific task-driven ocean remote sensing imagery processing. The attention mechanism emphasizes the combination of global and local information. Ocean theoretical domain knowledge provides compelling input features for DL models and reduces the degree of freedom of input data dimensionality.
There is no data associated with this article.
Conflicts of Interest
The authors declare that there is no conflict of interest regarding the publication of this article.
X.L and Y.Z conceived the ideas. All authors contributed to the writing and revision of the article.
This work was supported in part by the Strategic Priority Research Program of the Chinese Academy of Sciences (CAS) (XDA19060101 and XDB42040401), the Key Research and Development Project of Shandong Province under (2019JZZY010102); the CAS programs COMS2019R02 Y9KY04101L; and the National Natural Science Foundation of China (U2006211).
- X. Li, B. Liu, G. Zheng et al., “Deep-learning-based information mining from ocean remote-sensing imagery,” National Science Review, vol. 7, no. 10, pp. 1584–1605, 2020.
- B. Liu, X. Li, and G. Zheng, “Coastal inundation mapping from bitemporal and dual-polarization SAR imagery based on deep convolutional neural networks,” Journal of Geophysical Research: Oceans, vol. 124, no. 12, pp. 9101–9113, 2019.
- Y. Ren, X. Li, and H. Xu, “A deep learning model to extract ship size from Sentinel-1 SAR images,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–14, 2022.
- A. Vaswani, N. Shazeer, N. Parmar et al., “Attention is all you need,” in In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 6000–6010, Red Hook, NY, USA, 2017.
- K. Han, Y. Wang, H. Chen et al., “A survey on vision transformer,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. PP, pp. 1–1, 2022.
- Y. Ren, X. Li, X. Yang, and H. Xu, “Development of a dual-attention U-net model for sea ice and open water classification on SAR images,” IEEE Geoscience and Remote Sensing Letters, vol. 19, pp. 1–5, 2022.
- Y. Ren, X. Li, and W. Zhang, “A data-driven deep learning model for weekly sea ice concentration prediction of the Pan-Arctic during the melting season,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–19, 2022.
- C. Wang, G. Zheng, X. Li, Q. Xu, B. Liu, and J. Zhang, “Tropical cyclone intensity estimation from geostationary satellite imagery using deep convolutional neural networks,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–16, 2022.
- D. M. Blei and P. Smyth, “Science and data science,” Proceedings of the National Academy of Sciences, vol. 114, no. 33, pp. 8689–8692, 2017.
- Y. Liu, L. Yu, and G. Chen, “Characterization of sea surface temperature and air-sea heat flux anomalies associated with mesoscale eddies in the South China Sea,” Journal of Geophysical Research: Oceans, vol. 125, no. 4, 2020.
- X. Zhang, H. Wang, S. Wang et al., “Oceanic internal wave amplitude retrieval from satellite images based on a data-driven transfer learning model,” Remote Sensing of Environment, vol. 272, p. 112940, 2022.
- Y. Zhou, K. Chen, and X. Li, “Dual Branch Neural Network for Sea Fog Detection in Geostationary Ocean Color Imager,” 2022, http://arxiv.org/abs/2205.02069.
- S. Mu, X. Li, and H. Wang, “The fusion of physical, textural, and morphological information in SAR imagery for hurricane wind speed retrieval based on deep learning,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–13, 2022.
Copyright © 2022 Xiaofeng Li et al. Exclusive Licensee Aerospace Information Research Institute, Chinese Academy of Sciences. Distributed under a Creative Commons Attribution License (CC BY 4.0).