Machine learning is data-driven. Most artificial intelligence (AI) practitioners would agree that dataset ingestion, data processing, data cleansing, and data management take more than 90% of the development effort of a total machine learning project. Even though many tested data collection tools (such as Spark, Kafka, and Hadoop) are very well discussed, the topic of dataset management, surprisingly, doesn’t come up very often within the machine learning community.

机器学习是数据驱动的。 大多数人工智能(AI)从业人员都同意，数据集提取，数据处理，数据清理和数据管理将花费整个机器学习项目90％以上的开发工作。 尽管对许多经过测试的数据收集工具(例如Spark，Kafka和Hadoop)进行了很好的讨论，但令人惊讶的是，在机器学习社区中，数据集管理的主题并不经常出现。

At Salesforce, we knew this was a crucial part of our internal AI development process, so we designed a unique system for organizing datasets that aims to protect data integrity for updates, improve data retrieval efficiency, and facilitate lifelong learning.

在Salesforce，我们知道这是内部AI开发流程的关键部分，因此我们设计了一个用于组织数据集的独特系统，旨在保护数据完整性以进行更新，提高数据检索效率并促进终身学习。

数据集管理挑战 (Dataset Management Challenges)

Before talking about our system design, let’s first take a look at the major challenges of a typical dataset management system.

在讨论我们的系统设计之前，让我们首先看一下典型数据集管理系统的主要挑战。

终身学习 (Lifelong Learning)

As we said from the beginning of this post, machine learning training is data-driven. A model is typically trained with stationary batches of data, but our world is changing everyday; users of the model demand that it perform well in a dynamic environment, meaning that the dataset management system needs to absorb incrementally available data from non-stationary data sources and keep the training data updated.

正如我们从本文开头所说的那样，机器学习培训是由数据驱动的。 通常使用固定的数据批次训练模型，但是我们的世界每天都在变化。 模型的用户要求它在动态环境中表现良好，这意味着数据集管理系统需要吸收来自非平稳数据源的增量可用数据并保持训练数据更新。

数据集版本控制 (Dataset Versioning)

Machine learning is an iterative process. When we observe model performance regression after a retrain, comparing the datasets in the two training runs could reveal a lot of details that are critical for troubleshooting. Unlike a single file, a training dataset is usually a large group of binary files. How to version different groups of files and efficiently retrieve these files with the version requires deliberate thinking.

机器学习是一个反复的过程。 当我们在重新训练后观察模型性能回归时，比较两次训练运行中的数据集可能会发现许多对故障排除至关重要的细节。 与单个文件不同，训练数据集通常是一大组二进制文件。 如何对不同组的文件进行版本控制以及如何通过版本有效地检索这些文件需要认真思考。

使用严格的访问控制策略训练数据 (Training data with strict access control policy)

There are some highly sensitive datasets that require strict access, such as customer sales records and payment records. But training systems usually operate with normal security standards, making it impossible to train with these valuable but highly sensitive datasets.

有些高度敏感的数据集需要严格的访问权限，例如客户销售记录和付款记录。 但是培训系统通常以正常的安全标准运行，因此无法使用这些有价值但高度敏感的数据集进行培训。

多租户数据集管理 (Multi Tenant Dataset Management)

There could be different users and organizations sharing one training system. This requires the data management system to restrict data access only to the authenticated data owner and not to share data or grant access outside of its tenant.

可能会有不同的用户和组织共享一个培训系统。 这要求数据管理系统仅将数据访问限制为仅通过身份验证的数据所有者，而不能共享数据或在其租户外部授予访问权限。

数据可访问性 (Data Accessibility)

The Internet is full of data. The list of data collectors and tools seems to keep growing forever. We are building new data ingestion pipelines almost everyday, bringing in fresh and valuable data, but this also introduces lots of complexity to downstream model training components. A well-thought-out dataset management system should encapsulate all the details (such as data source, tools etc) and only expose some simple APIs for retrieving training data.

互联网上充满了数据。 数据收集器和工具的列表似乎永远都在增长。 我们几乎每天都在建立新的数据摄取管道，以获取新鲜且有价值的数据，但这也为下游模型训练组件带来了很多复杂性。 一个经过深思熟虑的数据集管理系统应该封装所有细节(例如数据源，工具等)，并且只公开一些简单的API来检索训练数据。

可扩展性 (Scalability)

Scalability is a basic requirement for any data system nowadays. From a machine learning training perspective, dataset fetching performance shouldn’t degrade as data volume increases. Distributed training and hyper parameter training will put lots of load on performance and needs to be accounted for in the system’s design.

可伸缩性是当今任何数据系统的基本要求。 从机器学习训练的角度来看，数据集的获取性能不应随数据量的增加而降低。 分布式训练和超参数训练将给性能带来很多负担，并且需要在系统设计中加以考虑。

我们的方法 (Our Approach)

In a nutshell, our dataset management system has three components: dataset ingestion API, dataset fetching API and dataset storage API. The generic data ingestion API allows any data collectors or clients to upload and update a dataset. The Dataset storage API organizes datasets in a special format (Dataset Storage Layout section) which is designed for protecting data integrity and fast data retrieval. The unified dataset fetching API offers users a simple and unified experience to retrieve data regardless of data type and source.

简而言之，我们的数据集管理系统包含三个组件：数据集提取API，数据集提取API和数据集存储API。 通用数据提取API允许任何数据收集器或客户端上载和更新数据集。 数据集存储API以特殊格式组织数据集(“ 数据集存储布局”部分 )，旨在保护数据完整性和快速数据检索。 统一的数据集获取API为用户提供了一种简单而统一的体验，无论数据类型和来源如何，都可以检索数据。

Let’s dive into the details of the major components pictured in the above system architecture diagram.

让我们深入研究以上系统架构图中所示的主要组件的细节。

数据集存储布局 (Dataset Storage Layout)

As we can see from the above architecture diagram, data is organized into several boxes — different types of datasets. So how is the data stored in a given dataset?

从上面的架构图中可以看出，数据被组织到几个盒子中-不同类型的数据集。 那么数据如何存储在给定的数据集中呢？

Above is the concept graph for a dataset storage layout. The layout is designed to support continuous data updates. Every update operation to the dataset will result in creating a new changeset, and each changeset contains an index file representing the current training data view (include examples to be trained upon and labels for these examples). No actual file is stored in the changeset folder. The newly received example file will be placed with all previous examples together. The index file in each changeset only keeps a reference to these actual example files, avoiding data duplication and saving storage space.

上面是数据集存储布局的概念图。 该布局旨在支持连续数据更新。 对数据集的每次更新操作都将导致创建一个新的变更集，并且每个变更集都包含一个表示当前训练数据视图的索引文件(包括要训练的示例和这些示例的标签)。 Changeset文件夹中没有存储任何实际文件。 新接收到的示例文件将与所有先前的示例放在一起。 每个变更集中的索引文件仅保留对这些实际示例文件的引用，从而避免了数据重复并节省了存储空间。

For consuming training data from a restricted access datastore (such as billing details or user payment records), our researcher could choose to upload a query instead of real files to the dataset. During training, the query will be executed at the defined remote secure datastore, and the returned data will be sent to the training program in real time.

为了使用受限访问数据存储区中的培训数据(例如账单明细或用户付款记录)，我们的研究人员可以选择将查询而不是实际文件上传到数据集中。 在训练期间，将在定义的远程安全数据存储区执行查询，并将返回的数据实时发送到训练程序。

Our storage system acts like a broker in this case. The changeset only contains the document for the data pulling query and pulling instructions. With this design, the actual sensitive data won’t persist in our dataset store. Our system consumer, which could be researchers or other model developers, could have data analysis reports and build models from the data it doesn’t have access to. This is a huge win for production model training since data security is always a blocker for data scientists to be able to experiment on such valuable data.

在这种情况下，我们的存储系统就像一个经纪人。 变更集仅包含用于数据提取查询和提取指令的文档。 通过这种设计，实际的敏感数据将不会保留在我们的数据集存储中。 我们的系统使用者(可能是研究人员或其他模型开发人员)可以拥有数据分析报告，并从无法访问的数据中构建模型。 这对于生产模型培训来说是一个巨大的胜利，因为数据安全性始终是数据科学家能够对如此有价值的数据进行实验的障碍。

统一数据集提取API (Unified Dataset Fetching API)

In order to make it easier to consume data from our dataset management system, we decided to provide only two APIs: prepareDatasetTarball and getDatasetTarball. These two APIs work for all types of training scenarios: image, text, and audio.

为了使从数据集管理系统中消费数据更容易，我们决定仅提供两个API：prepareDatasetTarball和getDatasetTarball。 这两种API适用于所有类型的训练场景：图像，文本和音频。

The schema of returned training data (image, text, and audio) is also in the same format, so the downstream trainer just needs to follow one standard pattern to read. Here is the sample code for how to use the API.

返回的训练数据(图像，文本和音频)的格式也相同，因此下游训练者只需遵循一种标准模式即可读取。 这是有关如何使用API​​的示例代码。

// Build training data selectors, all filters are optional.DatasetTarballParams params = new DatasetTarballParams()
 .setLabelFilters(labelFilters) // Filter labels.
 .setTimeRange(timeRange) // Filter data by modification time.
 // Long number, 0.8 means return 80% dataset from latest, 1 means all..setDataPercentage(percentage)
 .setChangesets(changesetList); // List of changesets.// Send dataset retrieval request to datastore.
DatasetPrepareResponse datasetPrepareResponse = dataFetcherApi.prepareDatasetTarball(orgIdStr, datasetId, params);// Get training data version.String datasetVersion = datasetPrepareResponse.getVersion();// Retrieve training data by providing its version.
DatasetTarballStatus datasetTarballStatus = dataFetcherApi.getDatasetTarball(orgIdStr, datasetId, datasetVersion);// Get actual training data url.String datasetUrl = datasetTarballStatus.getUrl();

You may have noticed that a string named dataset version is employed here for representing a training dataset from a given query. We will discuss it further in the dataset version section.

您可能已经注意到，此处使用了一个名为数据集版本的字符串来表示来自给定查询的训练数据集。 我们将在数据集版本部分中进一步讨论它。

“虚拟”数据集 (“Virtual” Dataset)

Now we want to walk you through a sample scenario to elaborate what exactly “virtual” dataset means.

现在，我们将引导您完成一个示例场景，以详细说明“虚拟”数据集的确切含义。

Normally, when we talk about a dataset in machine learning training, it means a compressed group of training examples. It works perfectly fine if we just want to explore an idea or make some ad-hoc experimentations, but it will struggle once we move to a serious development cycle. To demonstrate that, let’s look at an example.

通常，当我们在机器学习训练中谈论数据集时，它意味着一组压缩的训练示例。 如果我们只是想探索一个想法或进行一些临时实验，它就可以很好地工作，但是一旦进入一个严肃的开发周期，它就会很困难。 为了证明这一点，让我们看一个例子。

Bob is assigned an object detection training task. The task is to train a model to recognize the trademark of an insurance company “LifeA.” At the beginning, Bob collected some trademark (tm1) images from LifeA to build a dataset named “lifeA_trademark_tm1.zip.” Bob then trained the model from it; it works perfectly and everyone is happy. After a while, “LifeA” launched two new trademarks — tm2 and tm3. To adapt to that change, Bob built a new dataset, “lifeA_trademark_tm1_tm2_tm3.zip,” which contains all three kinds of trademark images. Then suddenly company LifeA lost a lawsuit, so it gave its trademark tm2 away to its opponent. In order to reflect this in the model, Bob had to exclude tm2 images and built another dataset “lifeA_trademark_tm1_tm3.zip.” To make things worse, when LifeA is about to release a new trademark tm4, its trademark design department refused Bob’s access request to their datastore due to security concerns. How could Bob update his trademark recognition model to recognize trademark tm4 without having access to tm4 images?

Bob被分配了对象检测训练任务。 任务是训练一个模型来识别保险公司“ LifeA”的商标。 最初，Bob从LifeA收集了一些商标(tm1)图像，以建立一个名为“ lifeA_trademark_tm1.zip”的数据集。 然后，鲍勃从中训练了模型。 它运行完美，每个人都很高兴。 不久之后，“ LifeA”推出了两个新商标-tm2和tm3。 为了适应这种变化，Bob建立了一个新的数据集“ lifeA_trademark_tm1_tm2_tm3.zip”，其中包含所有三种商标图像。 然后突然间，LifeA公司败诉，因此将商标tm2转让给了对手。 为了在模型中反映这一点，Bob必须排除tm2图像并构建另一个数据集“ lifeA_trademark_tm1_tm3.zip”。 更糟的是，当LifeA即将发布新商标tm4时，由于安全方面的考虑，其商标设计部门拒绝了Bob对其数据存储区的访问请求。 鲍勃(Bob)如何在不访问tm4图片的情况下更新他的商标识别模型以识别商标tm4？

From the above example, we can see where the problems are. In order to make the trademark recognition model meet the business’s needs, Bob has to produce three different datasets: “lifeA_trademark_tm1.zip,” “lifeA_trademark_tm1_tm2_tm3.zip,” and “lifeA_trademark_tm1_tm3.zip.” It’s clear that these training files are correlated and have many duplicate contents. As time goes on, the data preparation work would become super cumbersome; for example, think about what Bob would have to do to make a dataset zip that only contains trademark tm2 and the new trademark tm4. He would have to do file to file comparison among all the zip files he has to create a “tm2.zip.”

从上面的示例中，我们可以看到问题出在哪里。 为了使商标识别模型满足业务需求，Bob必须产生三个不同的数据集：“ lifeA_trademark_tm1.zip”，“ lifeA_trademark_tm1_tm2_tm3.zip”和“ lifeA_trademark_tm1_tm3.zip”。 显然，这些训练文件是相关的，并且有很多重复的内容。 随着时间的流逝，数据准备工作将变得非常繁琐。 例如，考虑鲍勃(Bob)要制作仅包含商标tm2和新商标tm4的数据集zip所要做的事情。 他将不得不在所有创建“ tm2.zip”的zip文件之间进行文件到文件的比较。

We also noticed that the correlation between these different datasets is based on file naming convention. The association among dataset files could be easily broken when the project is shared among multiple people or teams that follow different conventions.

我们还注意到，这些不同数据集之间的相关性基于文件命名约定。 当项目在遵循不同约定的多个人或团队之间共享时，很容易破坏数据集文件之间的关联。

Our thinking is to consider a dataset to be a virtual (logic) group. We separate the data collected and the data provided for training in the dataset and build a virtual dataset on top of them. One virtual dataset will be created for one machine learning task. Using the LifeA trademark recognition model training task as an example, one dataset (LifeA_trademark) would accompany the training task for its entire development cycle.

我们的想法是将数据集视为虚拟(逻辑)组。 我们将收集的数据与提供用于训练的数据分离在数据集中，并在它们之上构建虚拟数据集。 将为一项机器学习任务创建一个虚拟数据集。 以LifeA商标识别模型培训任务为例，一个数据集(LifeA_trademark)将伴随培训任务的整个开发周期。

From Bob’s example, we can see that machine learning task development is a continuous process. Even though the training data we collected is persisted statically in the datastore, the actual training files used in the model training are dynamic. Instead of creating duplicate data snapshots like Bob did, we persist data once by only storing the delta to avoid data duplication but to produce different training data to satisfy business needs.

从Bob的例子中，我们可以看到机器学习任务开发是一个连续的过程。 即使我们收集的训练数据静态地保存在数据存储中，模型训练中使用的实际训练文件也是动态的。 我们没有像Bob那样创建重复的数据快照，而是仅通过存储增量来避免数据重复，而是生成不同的培训数据来满足业务需求，从而对数据进行一次持久化。

The above diagram visualizes Bob’s example. We can see that data within a dataset could keep changing and training data retrieved from it could be different based on query parameters, but the dataset as a logic group remains stationary, which gives us a handle in this dynamic environment.

上图显示了Bob的示例。 我们可以看到，数据集内的数据可能会不断变化，并且根据查询参数，从数据集中检索到的训练数据可能会有所不同，但是作为逻辑组的数据集保持不变，这为我们提供了在动态环境中的处理能力。

One last thing to mention is that the data group (changeset) within the dataset could be just a query. The actual data will be pulled and compressed into training data during the model training at the runtime. This fits the need of training sensitive data.

最后要提到的是，数据集中的数据组(变更集)可能只是一个查询。 在运行时的模型训练期间，实际数据将被提取并压缩为训练数据。 这符合训练敏感数据的需要。

数据集版本控制 (Dataset Versioning)

Some readers may be wondering why we need versioning in dataset management.

一些读者可能想知道为什么我们需要在数据集管理中进行版本控制。

First, we need versions to track training data used in each training trial. If we were to find the new produced model’s performance to be degraded, then we would need to analyze and compare the training data which was used for producing the current and previous models to figure out the root cause. Without a version, it would be hard to find all the actual files used; using a filename to do so is not a reliable way.

首先，我们需要版本来跟踪每个培训试验中使用的培训数据。 如果要发现新产生的模型的性能下降，则需要分析和比较用于生成当前模型和先前模型的训练数据，以找出根本原因。 没有版本，将很难找到所有实际使用的文件。 使用文件名这样做不是一种可靠的方法。

Second, we need versions to track data updates inside a datastore. For example, we need to be able to revert certain data updates on a dataset if they are made by mistake. Versioning every data operation will make things a lot easier when it comes to data failure recovery and data deletions.

其次，我们需要版本来跟踪数据存储中的数据更新。 例如，如果错误地进行了数据集上的某些数据更新，我们需要能够还原它们。 对每个数据操作进行版本控制将使数据故障恢复和数据删除变得容易得多。

Many teams use existing version control tools like Git, which works well with code, but, due to a combination of file size constraints and the fact that git is not ideal for binaries, doesn’t work as well for datasets and trained models.

许多团队使用像Git这样的现有版本控制工具，该工具可以很好地与代码配合使用，但是由于文件大小的限制以及git对于二进制文件不理想的事实的结合，因此不适用于数据集和训练有素的模型。

In our proposal, we assign versions based on user requests, not on actual files. For data change versioning (dataset updates), each data update request is mapped with one changeset, incrementing the next changeset version up by 1: changeset 1, 2, … N. For training data versioning, we generate a version string by hashing the data query in the submitted dataset preparation request (DatasetTarballParams — see detail in “Unified Dataset Fetching API” section). This runs much faster and requires much less computation than hashing on large groups of files.

在我们的提案中，我们根据用户请求而不是实际文件分配版本。 对于数据更改版本控制(数据集更新)，每个数据更新请求都映射有一个变更集，将下一个变更集版本向上递增1：变更集1、2，…N。对于训练数据版本控制，我们通过对数据进行哈希处理来生成版本字符串。在提交的数据集准备请求中查询(DatasetTarballParams-请参见“统一数据集获取API”部分中的详细信息)。 与对大型文件组进行哈希处理相比，此方法运行速度更快，所需的计算量也更少。

综上所述 (In Summary)

In short, what we’ve done with our dataset management system is to break the convention that a dataset necessarily equals a static group of files. We’ve separated the concepts of data in a datastore training data used by a given training experiment, the former being stationary but the latter being dynamic. Then, we defined the dataset as a virtual group to bring the two types of data together.

简而言之，我们对数据集管理系统所做的工作是打破惯例，即数据集必须等于一组静态文件。 我们在给定训练实验中使用的数据存储区训练数据中分离了数据的概念，前者是固定的，而后者是动态的。 然后，我们将数据集定义为虚拟组，以将两种类型的数据结合在一起。

Another advantage of using this virtual dataset is it makes training with highly restricted data possible by storing only queries instead of real files, which allows us to satisfy customer requirements while also respecting their data.

使用此虚拟数据集的另一个优势是，通过仅存储查询而不是真实文件，可以对严格受限的数据进行训练，这使我们能够在满足客户要求的同时尊重其数据。

To optimize for performance and to allow for versioning, we make a dataset version number from the user’s request context instead of using file hash, and we approach dataset fetching with an API-led approach.

为了优化性能并允许版本控制，我们从用户的请求上下文中创建数据集版本号，而不是使用文件哈希，并且我们采用API主导的方法来获取数据集。

In designing the system in this manner, we satisfied the major challenges of dataset management at scale.

在以这种方式设计系统时，我们满足了大规模数据集管理的主要挑战。