Data Lake

Considerations of Moving to the Data Lake

So, what exactly are the limitations and frustrations with Data Lakes and switching to them?

Data Lake technologies are not a new phenomenon, traditionally they have been used with high volume data requirements and are something that most, if not all technology users within the workplace will have heard of and have at least a basic understanding of.

What is new, is using and utilising Data Lake technology within organisations for things such as operational purposes and operational analytics. The language and navigation of this technology can be a challenge within organisations as the configuration is very different to the SQL databases for which they may already have existing skilled individuals specifically trained in SQL processes. A mind shift in the way we approach data is required to take full advantage of the Data Lake technology, as well as training in new skillsets, data processes and language for all employees, users and data teams. A reasonable transition period will be necessary to onboard everyone within an organisation to the new setup.

Shifting to this new technology is not an insignificant task. The code which runs your current data and analytics platform will be of limited use when migrating to the Data Lake, at least if again you wish to take full advantage of the new technologies. Finding and maintaining a team with these skills can be a challenge for an SME. Salaries for individuals with these skills are high and competition to secure the very best talent in this space his hot. Companies need to think long and hard about their investments in this space and the ongoing costs associated.

Change management must form part of your approach to the shift to Data Lake and it will likely impact your DevOps process.  As previously mentioned, employees will need time to transition to a new skillset and learn new programming languages. Without investing in these changes and requirements, like any project, the transition will not be easy and may not deliver the benefits expected.

In the case of some applications, the change is inevitable, with deadlines for the transition to happen coming as soon as the end of 2022 (currently November deadline for Dynamics 365 F&O).

At 5Y, we have developed approaches and processes, and a sophisticated platform application, which enables us to support our clients in their move to the Data Lake in a quick and efficient transaction. We’re able to provide help and in-depth mentoring to our clients with their migration to a new data landscape. Please get in touch with us if you’d like to know more on how 5Y Technology can help you with any Data Lake considerations.

‍