MLOps: pitfalls and success factors

1 Opportunistic design of the MLOps pipeline

One of the most costly pitfalls in applying data science actually manifests itself the moment you start it. Not surprisingly, because data science generally starts with small experiments to gain experience. Once the potential is experienced and results are picked up by stakeholders, the temptation is great to continue on the experimental basis. As a result, the organization is not properly prepared for what MLOps is intended to do. Those who neglect the success factors below will put the success of data science at risk in no time.

1a. Clear objectives and strategy

A well-defined strategy and clear objectives define the path that involved departments follow and how those departments contribute to business goals. Start by identifying the specific goals of the MLOps pipeline. What does the company want to achieve with MLOps? Then develop a strategic plan to achieve these goals, which specifically defines how the MLOps pipeline integrates seamlessly with existing business processes. Include the departments that MLOps will first target: the analytics buyer!

1b. Collaboration between teams

In addition to close collaboration with business stakeholders, MLOps requires collaboration between different teams under the hood. This includes the data scientists themselves, as well as the software engineering teams and operational teams, such as IT-administrators, DBAs and application managers. Establish open communication channels between these teams. Without those "short lines," there is no effective collaboration and MLOps will lose performance and quality.

1c. Data management and governance

Ensure that a well-defined data governance strategy is in place to ensure data quality and regulate data access. Implement data catalogs and data line processes to ensure traceability, transparency and quality of source data. In addition to data, proper control of algorithms is critical to the accuracy and reliability of Machine Learning models. A solid data management and governance approach for both data and algorithms, ensures consistency in high-quality insights.

1d. Invest in mature technology in the value chain

Like everywhere else, optimized and scalable infrastructure helps improve performance and reduce costs. In the experimental phase of data science, none of this is important yet, but the moment business operations become dependent on it, it is a different story. Therefore, incrementally, in accordance with the strategic plan, provide a robust and scalable infrastructure to develop, implement and maintain Machine Learning models. Choose tools and technologies that fully support the MLOps pipeline, such as automated data preparation and model deployment, version control, API management and monitoring.

1e. Automated performance monitoring

Monitoring from within the MLOps team itself is crucial for early detection of performance setbacks. Monitoring also allows you to enable teams to intervene proactively, that is, before there are signals from the business that a drop in performance has occurred. However, manual monitoring is costly and mind-numbing work, which can put it at the bottom of the to-do list. So automate robust automated monitoring mechanisms to track model performance in production and respond to model degradation in a timely manner, without sacrificing development work.

2 Tunnel vision of technology

When data scientists or ML engineers get started building models, they often focus primarily on the technical quality of the model. This is certainly not a bad thing and is a crucial step in the development process. However, this narrow focus can lead to problems in implementing and operationalizing the model. For example, a model that works perfectly in an isolated test environment may run into problems when placed in the real world, with other systems, data sources and user interactions. In addition, end-user expectations and needs may be overlooked, leading to a model that is technically solid but does not meet the needs or usability requirements of those who will use it. Ignoring these factors can lead to a mismatch between the capabilities of the model and actual business needs or technical requirements. This makes the model less effective or even unusable in practice.

2a. Holistic planning

Begin by understanding the broader ecosystem in which the model will operate. This includes understanding the business objectives, the technical environment, the intended users and the overall impact the model will have. This approach ensures that the model is not only technically sound, but also aligned with the real world.

2b. User engagement

Involve end users and stakeholders early and regularly in the development process. Through their input and feedback, developers can ensure the model meets actual needs and expectations. This can also help identify potential problems early on.

2c. Flexible architecture and design

Take a flexible and modular approach to building the model. Maintain a clear and singular responsibility for each module within the system and consider splitting modules if they become too complex or begin to perform multiple functions. By doing so, you optimize maintainability and adaptability. Adaptations and integration with other systems and data sources also run more smoothly, allowing the model to work effectively in its intended operational environment.

2d. Thorough documentation

Carefully documenting the development process, the methodology used and the decision-making within a model, is essential for the understanding and applicability of the model by different team members. Good documentation facilitates maintenance, improves collaboration and ensures better alignment with business goals.

2e. Robust testing and implementation strategy

Implement a robust testing framework that assesses not only technical quality but also performance in the intended operational environment. After all, you want the impact of the model on operations demonstrated. This should also include a plan for gradual implementation and monitoring in the production environment so that any problems can be quickly identified and addressed.

3 The great temptation of new use cases

The iterative cycle of MLOps suggests it: continuous development. But there is a temptation to focus the Agile approach primarily on new projects. After all, the previously developed models have amply proven their value. But nothing could be further from the truth! One of the main pitfalls is the gradual decline in the quality of the models, or degeneration. In fact, in the world of generative ML models (GANs and UAEs), people are already talking about the risk of model collapse.

3a. Regular retraining and updates

This is necessary to keep your models current, as the world around us is constantly changing. Regularly retraining the models with current data is the minimum you can do. Recalibration of the previously delineated data set may require adjustment as additional data features become available. This allows models to adapt to changes in data distribution and incorporate relevant patterns and trends.

3b. Detection of data drift and concept drift

Use techniques for detecting data drift and concept drift to determine whether the relationship between input and output changes over time. For example, monitoring the performance of a model can indicate concept drift if performance declines, while using statistical testing can help identify data drift by detecting changes in the distribution of input data. By detecting drift, you can make timely adjustments to the models to accommodate the new data or changed data distribution (data drift) and new conditions in the previously optimized business process (concept drift). Moreover, when applying generative models, such as ChatGPT, the question is how to prevent model collapse. Through this service, more and more text is generated that is somehow later used again as training data to train the same model. The last word has not yet been said on this.

3c. Version management

A liability of the young data scientist, but still... Provide up-to-date documentation of changing models and maintain a version control system to maintain a clear audit trail. This facilitates understanding the models and any changes over time. In addition, it is a requirement at the time when accountability for the use of the model needs to be demonstrated again. Also describe what to do with previous analyses from the old model. These can by no means always be contrasted with the new results.

3d. Constant attention to data quality

Invest in data quality to ensure that the data used to train and validate models remains of high quality.

3e. Quick to respond to scaling up and down

In the MLOps pipeline setup, we already briefly talked about performance monitoring of the models. But there is another aspect at play. In addition to qualitative performance of the analytic models, the performance of the underlying cloud infrastructure remains of great importance. As the number of models, data or users increases, the infrastructure must be able to handle this growth without sacrificing performance. So the performance of the platform as a whole and the numerous data connection must be closely monitored and preferably adjusted to the intensity of use using standard scripts. In addition to the technical maintenance aspect, the cost aspect also comes into play here. With the scaling up of the MLOps pipeline as a whole, costs will increase and you want to avoid unpleasant surprises. So forecasting and reporting platform costs is also part of it!

This article marks the end of our article series on MLOps. We hope we have been able to contribute to your journey to a smooth Machine Learning operation. Do you have any questions or comments on this topic? If so, you can always contact us via Roman Nekrasov (for setting up an MLOps pipeline) or Frank de Nijs (for business alignment).

Also read our first two articles on MLOps:

• MLOps: for a perfect Machine Learning Pipeline
• MLOps: How do you quickly extract value from data?