What is a good data project made of – a data scientist's point of view

Support from data

The project's goal was to make visible the data flows and key figures related to the supply chain in a company manufacturing consumer products. Material sourcing, production, inventory management, logistics, sales, and maintenance are the same continuum in which changes in one function – for the better or worse – directly affect other links in the supply chain.

Therefore, reliable data, and harmonised indicators and forecasts refined from it, are prerequisites for data-driven decision-making in daily operations. In this way, up-to-date data-based information is equally visible to all, and we can avoid situations where data creates divergent truths at different stages of the supply chain.

From user stories to goals

Before my part in the project began, the team had already made careful preparations. A group of experts in production, sales, maintenance, and sourcing had described their routine work as user stories, outlining the kind of information and key figures that would support them. Instead of manually prepared Excel files stored in shared directories, the hopes were to update reporting views automatically.

During these preparations, business experts and end-users themselves had been able to paint a better future, and the seed for how to put their working hours to better use had been sown. Thanks to this, end-users were committed to the project because the goals were their own.

Extensive expertise

The project team had both technical and business expertise. My role was to act as a developer who implements the data reading from different sources, looks at the quality and suitability of data, codes, aggregates, and visualises it from different angles with various filters. In order to get results out quickly and accumulate development expertise internally, two of the customer's own developers were also involved in the implementation with me. 

A product owner acted as an interpreter for how the process works in practice at different stages of the supply chain and how this was reflected in the source data. Because a developer is not an operational business expert, understanding the connection between source data and business was crucial.

A scrum master took care of everyday running matters: access rights, organising sprints, dailies, and expert meetings. An analyst mapped and studied the quality of different data sources, and an IT team provided views to the data. In addition, the end-users tested and commented on the reports from the beginning and gave immediate feedback on the solutions developed.  

Information sharing

Every day started with a 15-minute daily, where we quickly recapped what we had achieved and what is on the table today. Not only we immediately spotted possible problems, but this was also an excellent opportunity to sense daily moods.

We organised the work into two-week sprints and broke down our goals derived from user stories into actionable tasks. These were further prioritised to backlog items together with the end-users. At the end of each sprint, we reviewed our achievements. These sessions always had a good participation rate, and people conducted lively discussions after the demos. At this point, we also compared progress with the budget and outlined how to use the remaining amount best.

There were plenty of problems to solve, and things didn't always go as we hoped. Instead of making excuses for delays and sticking bullishly to the plan, we immediately solved any upcoming issues together as a team. If a problem appeared to be unsolvable within the limits of the budget or the schedule, we reviewed the backlog and continued on another front to make progress.

Promoting results

As soon as we had completed the new reporting views and gone through their information, we transferred them to the production environment. We also organised end-user training to cover the functionalities and the information included in the reports: how the metrics were constructed, where the data came from, how often the reports were updated, and what phase of the supply chain they portrayed. 

This way, the reports were made available to all users simultaneously, and confidence in the new reporting views began to grow.  With all this, a broader and more coherent picture of the entire supply chain opened up as data from different stages became visible throughout the chain. It became easier to understand the supply chain as a whole. 

As the project was closely linked to operational business, it also revealed possible differences in operating practices. For example, different data entry practices could distort the data, and as a result, the developed metrics showed anomalies. Also, further needs related to data quality, gathering, and maintenance needs became visible when the reported numbers were examined together. This is how the project also added value by improving operational practices.

Takeaways

As a result, I would summarise the cornerstones of success as follows:

• Building a clear goal is based on concretely describing the need for a change, which crystallises it for both the development team and end-users. The trick of concrete user stories is that they make the bottlenecks in everyday operational work understandable to the development team and motivate them to seek solutions. This way, the business is also engaged in the project from the very beginning. Support and immediate end-user feedback are essential for fast progress. They are also fuel for the team, which creates a flow of working together.

• Agile working methods (Scrum) keep the project work on track when the routines are clearly framed. You don't have to waste your capacity on agreeing on extra things when you have a knowledgeable scrum master and a clear methodology to follow.

• A capable team includes technical and business expertise, and the common goal is equally clear to everyone. The team is forward-focused and doesn’t seek causes for any issues by looking at the rear-view mirror. In addition, a good mood and enthusiasm are contagious: the first one who has the skills and is available does the job.

• Training ensures the results are taken into use widely across the organisation, and creates confidence in them. This way, we can gradually get rid of scattered reporting practices and competing truths.

• Business harmonisation can be a valuable by-product of an exercise if data validation and built metrics also reveal potential process deviations and differing practices across the organisation.

I believe these tips pave the path for a good data project, and success will follow when you prepare for it!

Are you interested in hearing more about the work of a data engineer and Nitor as a workplace? Read more about the open Senior Data Engineer -position!