Subscribe to the Teradata Blog

Get the latest industry news, technology trends, and data science insights each week.

I consent that Teradata Corporation, as provider of this website, may occasionally send me Teradata Marketing Communications emails with information regarding products, data analytics, and event and webinar invitations. I understand that I may unsubscribe at any time by following the unsubscribe link at the bottom of any email I receive.

Your privacy is important. Your personal information will be collected, stored, and processed in accordance with the Teradata Global Privacy Policy.

Stages of Grief for Data Scientists and It Alike: Making Open Source Work in Paranoid Corporations

Stages of Grief for Data Scientists and It Alike: Making Open Source Work in Paranoid Corporations
We’ve all seen it, haven’t we? The explosion of new analytic tools and techniques, like R and Tensorflow, that promise the world huge advances from using machine learning and artificial intelligence.

The tide is definitely flowing in the direction of these new tools. Leaders in big organisations get excited, set up big budget programmes and recruit expensive specialist staff and then the IT department stops them doing their job. That is at least the data scientist’s perspective.

This leads to frustration and grief for both the data scientists and IT teams, but it doesn’t have to become a problem for either – as we look at common problem scenarios and ways for data scientists and IT security to collaborate and deliver a long-term solution for the company.

But, before you can find a solution, you often have to go through the different stages of grief.

Stage one: anger and denial

After a data scientist joins a company and wants to start using open source tools that have not yet been approved, they often need to put a request in. A conversation might go like this:

Data scientist: “I need to be able to install R Studio and download libraries from CRAN (, so I can build some machine learning algorithms to generate real business value. How do I do it as my laptop is locked down?”

IT: “Is this on the Approved Supplier list?”

Data scientist: “No. But it is open source technology. You don’t need to buy it and you can easily download it.”

IT: “Well, you’ll have to get it approved first.”

Data scientist: “OK, well how do I do that?”

This strict process is understandable though. The core responsibility of IT teams is to ensure that data is secure and controlled, that there are no risks to core IT systems, that these services run without hindrance and all SLAs are met.

Allowing the use of just any piece of software, that is not fully enabled and supported by the platform vendor, would be negligent on their part. Ultimately, IT security would the first person/people blamed if something went wrong. So, they need to be cautious.

But, at this point the data scientist wishes they’d joined a more progressive business, and then often does just that – leaves for a ‘better’ company, meaning that the project fails to deliver on its promises. Or, maybe worse, out of frustration someone hacks the system to get their tool installed anyway and crashes the production system.

This has happened in the real world and this is where the story often ends: however, it doesn’t have to. The question should be whether the traditional enterprises can make use of new and open source tools at all?

Stage two: bargaining

As is the case with every business, employees must work within the boundaries (such as security) set by the company and often this means being creative in their thinking. The first thing to do is to take a step back and ask: “What is the problem to be solved?”

In order to serve the business with solutions they really need, the different departments must collaborate and agree on an analytics strategy; outlining what they want to achieve, how they want to achieve it and what tools they need to do so. 

There really is no substitute for sitting around the table and collaborating to come up with a solution that works. We really believe this should be actually sitting around a table: it is far superior than a faceless web meeting or an extended email chain.

The advantage of insiders collaborating is that compliance with internal processes can be woven into the actual solution more effectively – not circumventing the process but exploiting the experience people have of actually making things happen.

Building the business case for change – quantifying the benefit from a ‘proper’ fix and considering this against the effort to solve it in the context of a future architecture vision – becomes a critical activity that the team needs to undertake.

For data scientists looking to introduce new software and tools to their companies, they should try to answer some core questions:
  • What is the use case and why was an open source library identified to meet this?
  • Can this requirement be met in a different way using tools the organisation already has (e.g. using the traditional tools such as SQL or SAS)?
  • What is the wider scope for the output of this work – e.g. what systems/applications does it need to integrate with to drive business value going forward?
In our next article, we’ll look at how the teamwork between data scientists and IT can help to deliver a better long-term solution for the company, one that can keep both parties happy and deliver beneficial results for the business.
If you have found this interesting and want to explore further with Teradata, feel free to reach out to the authors – Stewart to talk about exciting use cases, or Greg for the boring technical stuff.
Portrait of Greg Loxton

Greg Loxton

Greg is an accomplished Solution Architect with over 25 years’ experience of Data and Analytics in a variety of industries. Specialising in Analytical and Big Data Architecture, Greg has extensive experience of building solutions in numerous enterprise environments. He compliments his technical expertise with a business focus, ensuring that customer requirements are understood so that solutions are developed that meet the business’ needs and satisfy their on-going requirements.
  View all posts by Greg Loxton
Portrait of Stewart Robbins

Stewart Robbins

Stewart Robbins is responsible for providing expertise within the Financial Services sector and in defining strategic data and analytics capabilities within banking, delivering business-focused support to all dimensions of projects. He brings subject matter expertise in the areas of Analytics Consulting, Data Science, Customer Insight, Predictive Modelling, and CRM in presales, solution development and delivery activities.

Stewart has extensive client-side and consulting experience in the design of analytic capabilities and delivering measurable value through data discovery and analysis. Whilst Stewart has past hands-on experience of data science, he is also an experienced leader of such teams and programmes of activity. He has helped senior business leaders develop their analytics visions with detailed supporting strategies and roadmaps.  

Prior to joining Teradata, Stewart was an analytics consultant at Hewlett Packard (now DXC Technology) and spent 25 years as an analytics practitioner and leading analytics teams at E.ON Energy, Barclays Bank and Nationwide Building Society.

Stewart is a graduate of University College London with a BSc in Economics and Geography and is also an Associate of the Chartered Institute of Banking.

View all posts by Stewart Robbins

Turn your complex data and analytics into answers with Teradata Vantage.

Contact us