In 2015, researchers at Google published the following diagram in Hidden Technical Debt in Machine Learning Systems. The almost unreadable, black box labeled “ML code” is where a lot of engineering of that era focused.
Just like in 2015, the same is true today with AI. From an engineering perspective, we have several to dozens of boxes across data, infra, observability, inference, and more. Similarly, we also have to consider process integration, change management, training and talent management, support, financial validation to fully capture value from the system.
The diagram is just as relevant today as it was then. Teams and leaders often focus on the model (i.e., hallucination / cost / vendor lock in, etc.). While this is incredibly important, results depend on the full supporting system.
The FULL system, the one going much beyond the model, is also why I’m writing this newsletter. The Rest of the System covers the work, judgment, ownership, controls, and long term maintenance around the model. These aspects drive whether AI creates long term value for people and organizations, or the pilot gets canned or relegated to the backlog.
I spent the last ~15 years implementing systems inside larger orgs, most of which are regulated. I walked manufacturing floors in pharma / consumer products, implemented automations and software, worked with LOB finance teams to qualify opportunities, and HR to carefully plan change management and long term talent management.
I work with leaders most days to plan, design, build, deploy, and retire solutions today, so I will share what I am building, seeing, and reading from that lens, and I promise to share real examples when I have them.
Three recurring lenses will organize the publication (likely more to follow):
Field Notes: translation from the physical or business process into software.
Shaped Like Success: systems that look green while quietly failing (particularly insidious with AI / nondeterministic systems).
After the Demo: adoption, ownership, maintenance, and the operating life of a build.
Expect one issue every week or two.
The full detailed essays, models, and figures live at tomsullivan.dev, but each email version will stand on its own.
The complete flagship essay behind the first issue is currently live here:
Request: Please drop me a line and tell me what part of the AI implementation / management process your organization is still figuring out how to build and / or own. I read every reply, and the answers steer what I investigate next.
Have a wonderful week,
Tom