I’ve been thinking a lot lately about the tension between science and commerce—particularly the tension between data science and commerce. For most of my career, I’ve been on the seller’s side of the table, but from reading Robert Klopp’s “Story of Hadoop Disillusionment” and mathbabe’s experiences with “I don’t know”, I think my experience is pretty valuable to customers, too.

So, a product meeting with upper management and/or marketing in a large software company will often go like this:

Them: “And it can do this!”

Me: “Yes. With enough developer time and a lot of testing, it could probably do that. Probably.”

Them: “So it can do this.”

Me: “It could do that. It doesn’t do that now.”

Them: “It can do this!”

Me: “Doesn’t ‘can’ imply that it does it now?”

To my colleagues who are still having that conversation, I would point out that there are two issues at work here.

Issue #1 is the difference between can and could. Could implies a certain amount of uncertainty. Customers hate uncertainty, and so the most customer-facing elements of the company, i.e. the marketing team, hate it too.

Issue #2 is the Dunning-Kruger effect. I knew the product very well. I knew it so well that I was intimately aware of its flaws and failings and how terribly a “We can do this!” project could  go wrong. Neither upper management nor the marketing team knew my product like that. It wasn’t their job to know the product like that. That was my job. It was why I was hired, and it was why I had a seat in meetings where “It can do this!” got tossed around the table like a conch shell in Lord of the Flies.

To my data science customers (and potential customers) I would point out that this conversation affects you greatly. It is the Marketing department’s job to sell you software. They may care about building a long-term relationship with you (or they may not), but their primary job is to sell software, and if they don’t do that, they will not have a job next quarter. They are not trying to screw you. They are trying to pay their kids’ college tuition and not have their house foreclosed upon. They are trying to keep their company solvent so it can make more software.

It’s the customer’s job, then, to ask questions. “Does the product do that now?” “Can you give me a demonstration?” “Can you discuss some successes you’ve had with other customers?”

If you don’t know what questions to ask or aren’t sure you can evaluate the answers, you should bring someone technical to the meeting. If you don’t have someone technical who can come to the meeting with you, call me. If your sales representative tells you you’re being nosy, you’re onto something. Keep asking.

In the meantime, you can have some sympathy for the people across the table from you. They’re trying to do their jobs, and those jobs are hard. From the C-level down to the programmers, Big Data/Data Science software vendors are in a weird position right now, and every person in the company has a different perspective on the problem.

The problem from marketing’s perspective: The market is changing quickly. It is changing so quickly the product development team has to be flexible in order to keep up with changing demand.

The problem from my perspective: We were never going to get a shippable product because the goal posts kept moving. Every feature added to the testing. Everything added to testing increased the test space exponentially. One feature at the end of a project takes 10X longer than a feature added at the beginning of the project.

The problem from management’s perspective: We had to sell software, and every ship date was precious. If the product was out of date before it shipped, if it didn’t sell because it lacked one feature everyone wanted, if we had unknown bugs or ineffective code, none of us were going to be at that table next quarter.

For my data science colleagues, then, I offer some solutions to keep you sane.

I had to realize that planning a software project meant planning for what you could do as well as what you’ve been asked to do. I was dragged into the Agile methodology with much kicking, screaming, and biting, but I got there and I think its value is much higher in data science and big data projects. Getting the product into the hands of the people who use it gave us information early on about what  our customers would want it to do—and reduced our “October surprises” by quite a bit.

Marketing had to recognize that trends are trends and you can’t follow all of them—any more than you can make all the customers happy all the time. We had to figure out which trends would be there long enough to pay off our development costs and be prepared to re-brand our work as the market changed—because we knew what we were doing was valuable no matter what we called it.

Upper management had to realize that development resources are not free. It is very tempting to fix process problems by lengthening the work week and yelling at everyone to “Be team players!” It doesn’t work that way. Years of productivity studies tell us our software teams can’t work more than 40 hours for long periods of time. You may have a marvelous development team who can produce under terrible conditions, but if you keep fixing all your problems by asking people to just work harder, you won’t have them for long.

Development decisions have to be made strategically, with both the company’s and the customer’s benefit in mind.  If someone in your project meeting is speaking without advocating for at least one of those groups, take the conch shell away and tell them the grown-ups are talking.

Customers: Be patient with us. We’re trying. In the meantime, think about what you’re buying and what you really need. Software is a tool, and it’s a tool you want to use effectively.  Thinking about it in that way makes you less susceptible to buzzwords and more likely to get what you need–no matter what you’re buying.