计算机视觉/深度学习/医学影像 (Computer Vision/ Deep Learning/ Medical Imaging)

Histopathology is the study of diseases of tissues that involves the examination of microscopic slides consisting of tissues, cells, etc, that has been extensively used for the diagnosis of various forms of cancer. Histopathologists are medical experts who analyze cells or tissues under the microscope to make a diagnosis in order to come to a consensus regarding nature, the severity of diseases, and plan of action with regards to patient care. With the advent of advanced equipment such as specialized scanning machines, strides in storage/cloud capabilities, it has now become quite easy to store Microscopic glass slides in the form of Digital slides on a computer for processing them. It has enabled remote diagnosis, faster analysis, and systematic and safe storage of pathology information. Recent events such as the global pandemic have shown us the importance of automating medical activities such as diagnostics. Not only will it help improve the accuracy and capacity, removing a lot of redundancy, but it will ensure less exposure for the Doctors and diagnosticians on the frontline.

组织病理学是对组织疾病的研究，涉及检查由组织，细胞等组成的微观载玻片，该载玻片已广泛用于诊断各种形式的癌症。 组织病理学家是医学专家，他们在显微镜下分析细胞或组织以做出诊断，以便就自然，疾病的严重程度以及有关患者护理的行动计划达成共识。 随着诸如专用扫描机之类的先进设备的出现，存储/云功能的飞跃发展，现在已经非常容易以数字载玻片的形式将显微载玻片存储在计算机上进行处理。 它实现了远程诊断，更快的分析以及病理信息的系统和安全存储。 最近的事件(例如全球大流行)向我们表明了自动化医疗活动(如诊断程序)的重要性。 它不仅有助于提高准确性和容量，消除大量冗余，而且可以确保减少对一线医生和诊断人员的影响。

Due to the rising applicability, scalability, and success of Artificial Intelligence and Machine Learning and its multidisciplinary nature, it is increasingly being applied to various fields. Medical Science is no different. A large number of procedures from Automated diagnosis to surgical procedures to drug discovery are leveraging Machine learning with promising results. Medical imaging for diagnosis using Machine Learning and Classical computer vision is a very fastly growing area of research.Deep Learning, especially CNNs has become the methodology of choice when it comes to digital histopathology. This blog is a summarised review of the current state of research, the best practices, and the challenges in the field of Deep Learning for Histopathology. The review is structured based on the different deep learning approaches and discusses some of the popular, state of the art papers.

由于人工智能和机器学习的适用性，可伸缩性以及成功的不断发展以及其多学科的性质，它正越来越多地应用于各个领域。 医学也是如此。 从自动化诊断到外科手术再到药物发现的大量程序正在利用机器学习取得可喜的结果。 使用机器学习和经典计算机视觉进行诊断的医学成像是一个快速发展的研究领域。深度学习，尤其是CNN，已成为数字组织病理学的首选方法。 该博客是对当前研究状态，最佳实践以及组织病理学深度学习领域挑战的总结。 审查是根据不同的深度学习方法构建的，并讨论了一些流行的最新技术论文。

挑战性 (Challenges)

Unlike normal image data used for General computer vision tasks, dealing with Whole slide Digital pathology images poses a unique set of challenges.

与用于一般计算机视觉任务的普通图像数据不同，处理整张幻灯片数字病理图像带来了一系列独特的挑战。

The most significant challenge is the size of Whole slide images. These slides are usually gigabytes in size and thus pose a computation bottleneck both in terms of storage, processing, and compatibility with deep learning algorithms. A large percentage of literature dealing with deep learning deals with this problem using a Patch extraction/tiling method. They mostly use a sliding window-based approach to extract smaller patches which are then sent for further processing. Another common approach is to downsample the whole slide images to a tractable size but this leads to information loss, which in turn affects the performance of deep learning algorithms.

最大的挑战是整张幻灯片图像的大小。 这些幻灯片的大小通常为千兆字节，因此在存储，处理以及与深度学习算法的兼容性方面都构成了计算瓶颈。 有关深度学习的大量文献都使用补丁提取/平铺方法来解决此问题。 他们大多使用基于滑动窗口的方法来提取较小的补丁，然后将其发送以进行进一步处理。 另一种常用的方法是将整个幻灯片图像降采样为易于处理的大小，但这会导致信息丢失，进而影响深度学习算法的性能。

Researchers have begun to notice limiting flaws in the patch extraction approach too.

研究人员也开始注意到补丁提取方法中存在的缺陷。

• To make sure important information is not lost, significant attention needs to be given to ensure maximum overlap among patches by carefully adjusting the strides
  为确保重要信息不会丢失，需要特别注意以通过仔细调整步幅来确保补丁之间的最大重叠
• In order to perform any classification/prediction at the whole slide image level, we need to go through the entire slide, and hence a large number of patches need to be processed which makes the training and prediction process tedious and computationally expensive.
  为了在整个幻灯片图像级别执行任何分类/预测，我们需要遍历整个幻灯片，因此需要处理大量的补丁，这使得训练和预测过程繁琐且计算量大。
• When we deal with patches we are dealing with a very small region at a time, hence a lot of global contextual information is lost which limits the ceiling performance.
  当我们处理补丁时，我们一次只处理一个很小的区域，因此会丢失很多全局上下文信息，从而限制了最高性能。

Overcoming these limitations of the patch extraction method on whole slide digital pathology image analysis using deep learning has become a hot topic of research on its own. A number of solutions have been proposed and experimented with.

克服使用深度学习在整张幻灯片数字病理图像分析上补丁提取方法的这些限制，已成为其自身研究的热门话题。 已经提出并尝试了许多解决方案。

One of the significant viable solutions proposed is to use Sparse Coding of pathology slides for learning features and inferring representations of Cancer histology slides.

提出的重要可行解决方案之一是使用病理幻灯片的稀疏编码来学习特征并推断癌症组织学幻灯片的表示形式。

LSTMs and Conditional Random Fields are increasingly being used in combination with CNNs to model the correlation between neighboring patches, in order to capture larger contextual information.

为了捕获更大的上下文信息，越来越多的LSTM和条件随机场与CNN结合使用，以对相邻补丁之间的相关性进行建模。

Cascaded CNNs and Visual attention-based methods are being utilized to enforce a region selection method so that only the most relevant, diagnostically useful areas of the whole slide image is used for prediction and training.

级联的CNN和基于视觉注意的方法被用于实施区域选择方法，以便仅将整个幻灯片图像中最相关的，诊断上有用的区域用于预测和训练。

A highly ingenious solution called slide graph¹ proposed in a work presented very recently in CVPR’20 addresses the issue of size by using a Graph Convolutional neural network-based model.

最近在CVPR'20上发表的一项工作中提出了一种非常新颖的解决方案，称为幻灯片图 ¹ ，它通过使用基于图卷积神经网络的模型解决了尺寸问题。

The central idea of the work is to model the whole slide image into a graph and then apply Graph neural networks.

这项工作的中心思想是将整个幻灯片图像建模为图形，然后应用图形神经网络 。

This works by localizing and classifying the individual nuclei using a regression model i.e HoVer-Net² before using spatial clustering to group together nuclei that share common properties.Each of the tissue clusters is then modeled as a node in a graph, with edges of the graph capturing the possible tissue signaling mechanisms. Graphs are highly memory-efficient data structures, hence converting WSI to graphs is a computationally effective method that also allows us to preserve large scale contextual information and relationship between different regions of the slide. The graphs can then be used to train the various Graph learning algorithms such as GCN to learn prediction/classification.

这个作品通过定位和使用回归模型即个体细胞核分级悬停净使用空间聚类分组在一起细胞核使得组织簇的份额共同properties.Each然后建模为在图中的节点之前²，与边缘图捕获可能的组织信号传导机制。 图形是具有高度存储效率的数据结构，因此将WSI转换为图形是一种计算有效的方法，该方法还允许我们保留大规模的上下文信息以及幻灯片不同区域之间的关系。 然后可以将这些图用于训练各种图学习算法(例如GCN)以学习预测/分类。

Apart from size, another significant challenge is the availability of data. Due to the sensitive nature of the pathology data, and hence the stringent data privacy policies to ensure appropriate confidentiality and protection, the open-source availability of such data is limited. Even when the data is available, data annotation is a big issue, since it requires a high level of domain expertise and hence is mostly done by pathologists and medical practitioners and is a time taking process to ensure maximum precision. Poor annotation might lead to spurious results of our learning algorithms which is a dangerous prospect when it comes to medical diagnosis. Hence we are more often than not left with a limited amount of data and/or weakly annotated data with only slide level annotations. Data augmentation techniques such as introducing random affine transformations such as rotation, scaling, etc. , introducing noise/blur, color value variations in order to multiply the dataset are widely used to address the lack of enough data problem. Data augmentation also helps in introducing the necessary variability to the learning paradigm so that the learning algorithms can be generalized.

除了规模外，另一个重大挑战是数据的可用性。 由于病理数据的敏感性质，以及因此确保适当机密性和保护的严格数据隐私政策，此类数据的开源可用性受到限制。 即使数据可用，数据注释也仍然是一个大问题，因为它需要高水平的领域专业知识，因此大多由病理学家和医学从业人员完成，并且要花费大量时间来确保最大的准确性。 注释不当可能导致我们学习算法的虚假结果，这对医学诊断而言是一个危险的前景。 因此，我们常常只剩下带有幻灯片级别注释的有限数量的数据和/或弱注释的数据。 诸如引入随机仿射变换(例如旋转，缩放等)，引入噪声/模糊，颜色值变化以使数据集相乘之类的数据增强技术被广泛用于解决数据不足的问题。 数据扩充还有助于将必要的可变性引入学习范式，以便可以对学习算法进行概括。

Methodologies such as unsupervised learning help us learn useful patterns and representation from the data when we don’t have labels for the data. These learned representations can be used for further downstream tasks or for self-supervised learning. There is another class of approaches viz Weakly supervised learning, that helps us when we have weak coarse-grained(image-level) annotations but want to perform fine-grained(pixel level) prediction.

当我们没有数据标签时，无监督学习等方法可以帮助我们从数据中学习有用的模式和表示形式。 这些学习的表示可以用于进一步的下游任务或自我监督学习 。 还有另一类方法，即弱监督学习，当我们具有较弱的粗粒度(图像级)注释但想要执行细粒度(像素级)预测时，该方法可以为我们提供帮助。

In the next part, we shall be discussing the different deep learning approaches that have been applied to histopathology slides for cancer detection, their state of research, various tasks they help perform, and some prominent relevant work.

在下一部分中 ，我们将讨论已应用于癌症检测的组织病理学幻灯片的不同深度学习方法，它们的研究现状，它们帮助执行的各种任务以及一些重要的相关工作。

PS: I have linked technically important terms to respective resources explaining them.

PS：我已将技术上重要的术语链接到解释它们的相应资源。