In previous posts, I've described what federated learning is and what it would actually require. This post is about what happened when I tried to understand the algorithm that the project was supposed to implement - and what that understanding revealed about the project itself.
The Algorithm
The Pathway Generator is a predictive tool originally developed at Janus Rehabilitation in Reykjavik, with academic involvement from the University of Stirling (Haraldsson et al., 2017). It takes historical data about individuals - their health, circumstances, skills, employment history - and uses this to predict which rehabilitation pathways are most likely to succeed. Thomson et al. (2022), a paper co-authored by several of the consortium's academic partners and much of the management documented 188 predictor variables used in the Icelandic system's factor analysis, giving some indication of the scale of data infrastructure involved. By the time I arrived, the algorithm had been adopted - symbolically at least as something that was going to be 'implemented' in our organisation, or organisational context. The algorithm was what made the project "data-driven"; it was what justified the language of machine learning and AI in the funding application.
When I arrived as a designer, nobody could describe how it worked in any detail. People could say what it was for - predicting rehabilitation outcomes or pathways - and they could point to the Icelandic implementation as 'proof' that it worked, but there wasn't a demo of it or despite repeated enquiries any chance to see or talk to people in Iceland using this tool. I was given access to what still seemed quite crude or experimental python code. The data pipeline for the underlying data for actually running the algorithm was opaque. But the questions I needed to answer in order to do useful service design work - what data it requires, in what format, from what sources, assessed and governed by what protocols, stored in what systems - had no clear answers. Not because people were withholding information, but because the algorithm had been adopted at a level of abstraction that didn't require those questions to be answered, or hadn't before now.
This wasn't unique to the Pathway Generator. "Data science" itself was functioning as a floating signifier across the project - a term that different stakeholders filled with entirely different content, enabling apparent agreement without actual alignment. For some partners it meant the Pathway Generator itself. For others it meant the idea of using data to inform decision-making in some way, without any specific algorithm in mind. For others it meant the promise of a future state where data would be used to support rehabilitation, without any specific tools or processes in mind.
None of this, obviously, bares any relation to Federated Learning, which is a specific technical approach to training machine learning models on distributed data.
Working Backwards from the Algorithm
The approach I adopted was to work backwards from the algorithm. Start from what the Pathway Generator claims to produce, and trace its dependencies down through every layer: what decision logic it applies, what types of assessment feed it, what variables it expects, in what format, from what institutional systems.
Burns and Hajdukiewicz (2017) describe something similar in Work Domain Analysis, which uses an abstraction hierarchy to decompose a system from its functional purposes down through its abstract and generalised functions to its physical forms and material constraints. This, and the wider Ecological Interface Design approach it is part of has been a core background influence on my own professional practice since my time at Brunel.
Opening the Black Box
Hollanek (2020) argues that opening the algorithmic black box is a design challenge rather than a purely engineering one: the task isn't just to understand the code, but to make the system's assumptions and dependencies visible to the people who would need to work with it. That captures what I was trying to do. The Pathway Generator wasn't a black box because its code was hidden - I had access to the python files and could run a version on synthetic data and could examine its structure - but because the relationship between its requirements and the Swedish institutional context had never been specified. Nobody had asked what it would take to actually run this thing here.
I produced three kinds of artefact, each operating at a different level of specificity: concept maps that synthesised how what - tangibly we had on the project so far - this python code, integrates with more normative or literature informed models of the machine learning process; typed interface definitions that specified the algorithm's data structures variable by variable; and architecture diagrams that showed what the system would look like as an implemented whole. What Sevaldson (2022) calls multi-scalar mapping - spanning from policy context down to individual data fields.
I also drew on Maass and Storey's (2021) model of how data progresses through a machine learning system - from raw data through structured data, information, and recommendations to the point of user encounter - as a way of making visible what the full pipeline would need to look like. My visualisation of their model maps the entire journey that the Pathway Generator's outputs would need to travel before reaching a caseworker, and therefore the infrastructure, interpretation layers, and communication design that would need to exist at each stage.
I also adapted Schoppe's (2020) model of ML development into a process diagram showing the steps from understanding context through to deploying a working system, adding design-led activities - identification of needs, frames and values, consequence scanning - that I thought should precede and accompany ML development when approached from a more service-design or user centred perspective.
What the Algorithm Requires
At the concept-map level, the project held together. Categories of data were visible - psychological health, life situation, employment status - without forcing questions about how each would be gathered in Sweden. Stakeholders could point at categories and confirm their relevance without needing to say whether the data existed.
At the interface level, every variable forced a material question:
// Physical Health - unclear what protocol for gathering
mean_physical : number
stddev_physical : number
range_physical : number
That comment - "unclear what protocol for gathering" - appears for nearly every domain the algorithm covers: physical health, psychological health, social health, financial situation, self-discipline, self-reflection, creativity, technical skills, time management, empowerment, resourcefulness. Each comment marks a place where a concrete assessment protocol should be and isn't or where some degree of alignment or standardisation would need to occur. Some variables raised questions beyond logistics:
// Hard to interpret this...
// a boolean on whether they consider they are a victim of being bullied??
// - what protocol is used to assess this?
bullying : boolean
Even understanding the existing Icelandic system - never mind implementing it in Sweden - requires answering questions about assessment validity, clinical protocols, and ethical implications that, it seemed, nobody in the project had addressed or given much thought to. The typed interface definitions were, as some representation of the "data contracts" or programmatic interfaces in effect specifying what the algorithm would need, to support the process of mapping the upstream service processes or protocols that either gather compatible data, or beginning to think through how we would do so.
What I was doing was the preliminary work of reverse engineering: identifying the algorithm's conceptual structure and data dependencies as a first step toward understanding what data pipelines or sources would need to exist before any integration with an existing service could even be discussed. But the reverse engineering kept running into the same problem - there was nothing on the Swedish side to reverse-engineer toward. The infrastructure the algorithm depends on didn't exist, and the institutional capacity to build it was unclear.
The systems map I produced of the service landscape around the algorithm attempted to visualise this at a higher level: the institutional actors, user journeys, and data flows that would or could underpin the data production lifecycle for the Pathway Generator to function. It represents the service context into which the algorithm was supposed to be embedded - the coordination between national, regional and local organisations, the caseworker interactions, the data journeys that would feed the model and carry its recommendations back into practice.
A Technoimaginary
What I was mapping turned out to be neither an as-is state nor a to-be state - the two standard modes of service design mapping that Kalbach (2016) describes as foundational. There was no AI-supported rehabilitation service to document as it currently exists. But I also wasn't designing a future state grounded in research about current needs and capabilities. Instead, I was trying to make specific something that had already been promised - in funding bids, job descriptions, consortium agreements - but had no material substrate.
I've started calling this a technoimaginary, borrowing from Jasanoff and Kim's (2015) work on sociotechnical imaginaries: a collectively performed vision of a technological future that has become organisationally real through its institutional expression, even though it has no technical reality whatsoever, or a very limited technical reality in another country with very different scale, data protection and infrastructure from the context that we were working in. The technoimaginary holds together at high abstraction precisely because "data science" or even "the pathway generator" functions as a sacred concept - a representation whose meaning is bound up with collective identity rather than technical content. Its power derives from the constitutive performance of invoking it, from what I called in an earlier post a rationalised myth: a practice adopted because it signals legitimacy, not because its instrumental effects match expectations.
What This Means
The algorithm archaeology didn't set out to expose the project's contradictions. I was trying to understand what we were building so I could contribute to building it. The exposure was a side effect of doing the work at the level of specificity that competent design work requires. Each successive layer peeled away more of the ambiguity that held the project together; the more specific I got, the more visible the absence became, or fragile the existing premises or artefacts became.
What happened next is a subject for later posts in this series.
References
Burns, C.M. and Hajdukiewicz, J. (2017). Ecological Interface Design. CRC Press.
D'Ignazio, C. and Klein, L.F. (2020). Data Feminism. MIT Press.
Falk, P. (in press). Assemble Care // Align Data: An Ethnographic Study of Datafication in Swedish Public Care.
Haraldsson, S.O., Brynjolfsdottir, R.D., Woodward, J.R., Siggeirsdottir, K. and Gudnason, V. (2017). The Use of Predictive Models in Dynamic Treatment Planning. Proceedings of the IEEE Congress on Evolutionary Computation.
Hollanek, T. (2020). AI Transparency: A Matter of Reconciling Design with Critique. AI & Society.
Jasanoff, S. and Kim, S-H. (2015). Dreamscapes of Modernity: Sociotechnical Imaginaries and the Fabrication of Power. University of Chicago Press.
Kalbach, J. (2016). Mapping Experiences: A Complete Guide to Creating Value through Journeys, Blueprints, and Diagrams. O'Reilly Media.
Maass, W. and Storey, V.C. (2021). Pairing Conceptual Modeling with Machine Learning. Data & Knowledge Engineering, 134, 101909.
Schoppe, S. (2020). The Role of Design in Machine Learning. Salesforce UX Blog. Available at: https://medium.com/salesforce-ux/the-role-of-design-in-machine-learning-ae968ea90aac
Sevaldson, B. (2022). Designing Complexity: The Methodology and Practice of Systems Oriented Design. Springer.
Thomson, S.L., Adair, J., Bergström, M., Brynjolfsdottir, R.D., Falk, P., Gudnason, V., Haraldsson, S.O., Hjaltason, O., Lund, R., Siggeirsdottir, K. and Tzang, S. (2022). The Use of Logistic Regression in Vocational Rehabilitation Factor Analysis.