Month: April 2021

What to look for in a MLOps tool –…

In an earlier blog, we discuss the various non-technical features to be considered when buying/building/subscribing to an MLOps tool. In this article, we review the main technical considerations to be kept in mind when selecting such a tool.

It is useful to list these considerations in the context of the development process of an AI/ML project. This typically consists of steps, such as those shown below:

1. Data Management

AI/ML projects typically start with a business problem statement and some sample data, which data scientists use to start figuring out modeling strategies. At this point, the following questions usually arise:

• Does the tool have connectors to different data sources (local/remote file systems, RDBMS, NOSQL databases, cloud data sources, etc.)?
• Does the tool support data versioning? (extremely critical since training data is the most critical input to the model training process)
• Does the tool support automated data exploration and visualization?
• If yes, can developers perform their own custom exploration and visualization, or is this restricted to analysis supported by the tool?
• Can exploration and visualization output be exported and distributed?
• Does the tool support easy mechanisms for automated Feature Engineering?
• Does the tool support the reuse and share of features through Feature Stores?

2. Auto ML

AutoML enables users to create sophisticated models with a click of a button.

• Does the tool support AutoML?
• Does the tool support “white box AutoML”, wherein the code for the automatically generated model is available to developers to customize and enhance?

3. Experiment Management

Data Scientists spend a lot of time running experiments with different models. It is critical to keep track of these experiments, as well as model lineage, for experiment repeatability, as well as to reuse experiments across projects.

• Does the tool track data scientists’ experiments?
• Does the tool support model versioning?
• Does the tool support model lineage (ability to trace a model version back to its inputs)?
• Can the tool compare models across various parameters?
• Does the tool support the most popular libraries for Machine Learning and Deep Learning?
• Does the tool keep track of various metrics associated with the training process?
• Can the developer report custom metrics during the experiment run?
• Does the tool support best practices of software architecture, e.g., Object-Oriented Design, modularity, and reuse?
• Can developers compare models to determine which one is better?

4. Model Explainability

Once a model has been trained and selected, it is critical that it is explainable, i.e., the behavior is well understood and the output free from bias. A better-performing model may often be replaced by a slightly less accurate model, which is easier to explain. While Explainable AI (XAI) is a burgeoning field of research, some basic features need to be supported by MLOps tools:

• Does the tool provide means to understand model behavior?
• Does the tool enable bias detection in a model?
• Does the tool enable “what-if” analysis?

5. Model Deployment – containerization, Inference pipelines, and A/B Testing

Once a model is ready, it has to be deployed to a production environment, where it has to respond to requests, often with extremely low latency. For example, a credit card fraud detection model has to respond to thousands of requests a second. 

• Does the tool support industry-standard deployment techniques, such as containerization?
• Specifically for AI/ML models, does the tool deploy these models using industry-standard techniques, e.g., REST APIs?
• Can the deployment environment perform well with heavy request load? Does the environment scale horizontally if required?

6. Model Monitoring

After a model has been deployed to production, it has to be monitored continually to check for decay in performance and/or accuracy. 

• Does the tool provide means to monitor deployed models?
• What kind of metrics are provided for monitoring models?
• Can the tool monitor models create elsewhere?
• Does the tool enable the creation of alerts if metrics cross pre-specified values?
• Does the tool enable automated retraining of models in case of model decay?

What to look for in an MLOps tool –…

In an earlier blog, we explain how MLOps helps to reduce inherent risks associated with the development and deployment of AI/ML models in enterprises.

Implementing MLOps workflows, as with any other workflows, involves the age-old choice of “Build vs. Buy” – whether to implement a tool from scratch or buy (or subscribe to) an existing off-the-shelf tool. Each organization may choose to respond to this dichotomy in its own manner, keeping its unique requirements and capabilities in mind. Either way, it would help to list these requirements to ensure that these are kept in mind during the decision process.

An MLOps tool should enable all stakeholders of the AI/ML development and deployment process to perform their tasks with improved efficiency and productivity. From this point of view, the requirements of such a tool can be classified into two categories:

• Technical – features that address specific technical requirements of data scientists, business analysts, project managers, and other users
• Non-technical – features that straddle user roles and are typically required by any enterprise-wide software.

In this blog, we enumerate the non-technical feature requirements, focusing on technical features in the next article in this series.

Non-technical features required from an MLOps tool are the same as those offered by any enterprise-grade software, with some tweaks to cater to their unique positioning. These include:

1. Installation Flexibility

• Does the tool tie the organization into a specific cloud? 
• Or is it cloud-agnostic? 
• Does it have an on-premise offering? 
• Does it provide a PaaS (Platform as a Service) version? 

2. Development Best Practices

The primary users of the tool will be the Data Scientist team. 

• Does it support architectural best practices – e.g., modularity, reusability? 
• Does it integrate with code versioning systems? 
• What languages does it support (a minimum set should include Python, R, Java, Scala, and GoLang)? 
• Is it flexible – does it impose custom ML libraries on developers, or does it support all popular ML libraries?

3. Collaboration

Many tools look at data scientists as “lone rangers”, working on their own to develop, test and deploy models. However, the entire AI/ML development and deployment process involves a team of data scientists, business analysts, Production Engineers, and Project Managers to complete the entire workflow. 

• Does the tool support collaboration among all the stakeholders through a mix of appropriate workflows and collaboration tools?

4. Security

• Does it integrate with the enterprise security systems? 
• Does it provide support for industry-standard security best practices?
  Is the data within the tool secure, both at rest and in transit? 
• Does it provide VPN-based access as a standard? 

5. Access Control

• Does the tool provide Role-Based Access Control (RBAC) features? 
• Can enterprises define custom roles and assign functions to these roles? Ideally, administrators should be able to assign individual actions permitted in the tool to a specific role(s).

6. Governance

• Does the tool provide governance mechanisms through workflow approvals? 
• Can you set multiple levels of approval authorities for any action? 
• Are the logs adequate for failure analysis? 
• Does it provide audit trails to ensure traceability? 
• Is there model provenance defined by entity (which data, algorithm, and hyperparameter versions), activity (when deployed and the changes since the last deployment), and agent (who all signed off on the deployment)?

7. Performance

• Does the tool support containerization? 
• Are the models deployed to a scalable, fault-tolerant environment? 
• Can you increase/decrease the deployment cluster size as required?

8. Ease of Use

• Does the tool have an intuitive GUI to enable even non-technical business users to use it effectively? 
• Is there sufficient documentation available? 
• Does it include starter projects to reduce the learning curve? 

9. Support

• Does the tool provider provide adequate support? One possible way to reduce the learning curve is to provide a managed service offering, wherein an engineer from the provider will support your AI/ML operations for a specific period of time before the final handover

10. Provider Reputation

• What’re the provider credentials like? 
• Have they implemented any AI/ML projects themselves? (In other words, do they eat their own dog food?) 
• Does the leadership team have a Data Science background?

These are some of the questions every enterprise must ask before embarking on this choice. There are several MLOps tools available today, including xpresso.ai, AWS Sagemaker, Microsoft Azure ML, Allegro, Iguazio, DataRObot, Dataiku, etc. Make sure you evaluate all of them using these criteria and select the one best suited for your needs. Best of Luck!