In this month’s issue:
We discover what evolutionary anthropology can teach us about handling the uncertainties of a data science brief.
I offer some advice for starting out by yourself in Semi-Supervised.
Academia lays into Market Mix Modelling (while I cheer from the sideline) in this month’s The Dunghill.
We begin, of course, in 300,000 BCE …
Prehistoric tools for data science
Casual reading is tricky in statistics and data science. The choice is usually between the trivial (the accessible but disappointing pop science book in the smart thinking section) and the terrifying (the serious but stultifying academic textbook with the low-effort cover). The secret is not to look in the obvious places; specifically, not to choose books about statistical analysis or algorithms (avoid anything with “data“ in the title). Instead, keep a look out for books that might use these techniques as a means to an end, particularly an end that is fascinating in its own right.
For me, there is no better example of this than two books by the biological anthropologist Robin Dunbar: Human Evolution and How Religion Evolved. Both are published as part of the new Pelican Book series. Dunbar is well known for Dunbar’s number - an apparent upper limit on the number of stable social relationships a human being can maintain. The number itself is controversial, with some recent attempts at validation resulting in credible intervals so wide as to be unusable (although Professor Dunbar has challenged these claims).1 But I think focusing on this one number misses the point: Dunbar’s field is evolutionary anthropology, a topic that involves an enormous amount of speculation about a tiny amount of data. It’s close to miraculous that anything at all can be said about how early humans and their ancestors lived. And Dunbar says a lot.
So what, you are now asking, has this got to do with my day job? After all, the modern-day business environment and the Late Pleistocene could not be more different: we have plenty of data, facts are in abundance, and if we don’t know something we can go and find it out.
But really?.. I confess, I do most of my work waiting for data sets that never arrive; facts may be in abundance but they change with each new conversation; and just try finding something out in an organisation of more than 150 (and I use that number for a reason!)
The fact is we need the tools that Dunbar and his peers created in order not to be overwhelmed by the sheer number of possibilities that present themselves at every hurried briefing - if only so we can get started on the work. These two books are rich in ideas for how to speculate intelligently and usefully with just scraps of information. Possibly my favourite is the use of a simple but obvious constraint to narrow down the space of possibilities. In Human Evolution, Dunbar manages to extract an astonishing amount from the simple assumption that there are only so many hours in the day for the hard-working hominid. Elsewhere he uses causal diagrams to frame distinct hypotheses about complex and dynamic social systems. The point is not that these are necessarily right (that much has to be argued for) it is that the conversation can now continue without endless circles of misunderstanding. Another thing to watch for is how he integrates research from many fields to revise or enrich his argument.
None of this is revolutionary from a methodological point of view - it’s all just part of science. The novelty, and the opportunity to learn, is in seeing it done so well.
Semi-supervised
This month’s question is refreshingly practical. Sara Gaspar has asked me for tips on how to go freelance. I’ve filtered out the obvious and included only those things I found surprising:
Make it clear to your clients that you are neither a part-time employee nor a contractor - otherwise you will be treated like one. You won’t be going to company meetings; you won’t be on-boarded; you won’t be in their offices (except for essential meetings); you won’t be using their hardware. There are various ways of doing this without causing offence: draw attention to concurrent clients, point them to your website, and quote fixed-cost projects rather than a day rate. Your choice of words is important: talk about services, deliverables, products, costings.
Get yourself a sugar daddy/mamma: a first client who will provide you with a regular source of income while you get on your feet. This is usually a current or former employer who likes what you have done in the past and is sympathetic towards what you are trying to do now. Offer them a reduced rate for a retainer, but take care not to fall into the employee trap described above.
Carefully scope out work. Scope creep is annoying when you work for someone else but potentially fatal when you work for yourself. First, cost up the full breakfast - everything you can think of delivering, and they could expect, for a given brief. Then, if the client wants a lower price, you can start taking things out. It is a subtle way of marking out the scope while emphasising your value.
Offer a wide range of services - don’t think you can survive on just one product or speciality. (You’d need a sales team and a marketing department for that).
As soon as possible, start to build up your network. Very little work comes in from advertising, and I’ve never even tried cold calling. It’s all about recommendation and therefore reputation.
Relationships can take a very long time to mature into work. Be patient. And for this reason, pay careful attention to the pipeline - even when you are busy delivering work. A coffee with a potential client today may be your dinner in six month’s time.
Don’t be frightened by your own day rate. It’s going to look enormous at first, but estimate how many days of the year are going to be spent in paid employment. Multiply that by your day rate and then divide this by the number of days you will be working. Not so big now.
I won’t eulogise the upsides. If you are considering the move, you already know what they are!
Please do send me your questions and work dilemmas. You can DM me on substack or email me at simon@coppelia.io.
The dunghill
At long last someone in academia has taken an axe to the gnarly old trunk of market mix modelling. I am referring to a recent paper, Your MMM is Broken: Identification of Nonlinear and Time-varying Effects in Marketing Mix Models by Ryan Dew et al. It should give an entire industry pause for thought, although I doubt it will.
Market mix modelling is the practice of building regression models to understand the drivers of sales, with a particular interest, of course, in the contribution of marketing activity. Now I’ve been out of this area for over a decade, so I don’t know how different it is now, but it was certainly bad back then. There were some very good people, with an honest interest in cracking a devilishly difficult problem. I worked with them then and still work with some of them. (Much of what is now in Google’s Meridian - time varying parameters, hierarchical modelling, Bayesian priors, the use of DAGs - we were working on at the time.) But it was also full of crooks and cynics (who knew the models were wrong but didn’t care) and true believers (who had no idea that there was a problem). Unsurprisingly, it was a hotbed of QRPs (see Glasseye July issue) - most prominently overfitting, p-hacking and harking, double dipping, and the gratuitous use of researcher degrees of freedom. As one old-timer recently told me, “At the time I left [a very well-known agency], their TV function had six parameters... We used to stay up until 2 am chasing t-stats to pin those down”.
But the authors of this paper have been generous - too generous - they have assumed good practice on all other fronts and concentrated on the one devastating flaw in the whole endeavour - what philosophers of science call underdetermination and what Leo Breiman called, more picturesquely, the Rashomon effect (again see Glasseye July issue.) Underdetermination occurs when there is insufficient information in the data to determine which out of many beliefs we should accept; lots of models fit the data equally well. The problem for marketing mix models arises when these equally compatible beliefs imply completely different marketing strategies and therefore budget allocations. The problem is compounded when the modeller, steered by an eagerness to please, has taken just one route to their chosen model: they are aware of only one model that fits the data well, and that’s the model they want.
Underdetermination is carefully ignored in a business environment where data is the answer to everything. The basic conviction is that with enough brains anything can be squeezed out of the data - the problem just hasn’t been cracked yet. But sometimes you really can go no further; the answer just isn’t in there, and attempts to reach it have the same pitiful futility as an alchemist’s experiments or a gambler’s “system”.
But back to the paper. The specific aspect of underdetermination that the authors have chosen to focus on is that between models using time-varying effects to model the impact of marketing and models using non-linear effects to do the same. In many cases the two approaches fit the data equally well, but the practical implications could not be more different - one suggests that there is an optimal spend for advertising and it is constant over time, the other suggests that the effect of advertising is constantly evolving (perhaps it is more effective in the summer, or is in gradual decline.)
This is an exceptionally subtle instance of the Rashomon effect. Other cases are more brazen. I once found an agency modelling an upward-sloping trend by selecting from the infinite pool of steadily upward moving phenomena the one that would most please the client!
Of course, no one does that anymore? If they did you’d tell me right?
If you have some particularly noxious bullshit that you would like to share then I’d love to hear from you. DM me on substack or email me at simon@coppelia.io.
From Coppelia
Been busy with lots of technology this month. In particular:
Tensorflow probability - gave a short training session to a client on its potential. Interesting to consider the extent to which it bridges the two cultures of machine learning and statistical modelling as described by Leo Brieman.
MLOps - diving deeper into this tangled world than perhaps I wanted to go. Conclusion: don’t use too many tools!
Looking at alternatives to VSCode for no better reason than self-sabotage. Don’t quite yet understand all the fuss about Cursor although it sure does look pretty.
Github Copilot paid for itself by summarising a mountain of code that was passed to me by a client. The secret, I discovered, is context: use and # to tell it where to direct its attention.
And thanks this month to
for pointing me in the direction of the paper: Your MMM is Broken.If you’ve enjoyed this newsletter or have any other feedback, please leave a comment.
See for example this paper and Dunbar’s response.
I very much agree on MMM. It's definitely a mixture of people who know it's all dodgy but need to keep the customer happy, and people who have never seen outside of this area, so don't understand what they're doing is wrong!
I'm interested what you've been using tensorflow probabilities for, having never looked at it? Is it a replacement for e.g. Pymc? Or something different?
as someone who has not tried MMM, but is skeptical having read up on it a bit, it's reassuring to see your take on them. Well done on this newsletter, it's consistently an interesting and thought provoking read, please keep going!