<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd" xmlns:googleplay="http://www.google.com/schemas/play-podcasts/1.0"><channel><title><![CDATA[The Rest of the System]]></title><description><![CDATA[Field notes on the machinery around the model, work, judgment, ownership, and what happens after the demo, for the operators and leaders who have to own the results.]]></description><link>https://news.tomsullivan.dev</link><image><url>https://substackcdn.com/image/fetch/$s_!zoId!,w_256,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd2b1092f-0e83-4ba3-b33e-16853284c7ca_1024x1024.png</url><title>The Rest of the System</title><link>https://news.tomsullivan.dev</link></image><generator>Substack</generator><lastBuildDate>Wed, 14 Oct 2026 19:58:05 GMT</lastBuildDate><atom:link href="https://news.tomsullivan.dev/feed" rel="self" type="application/rss+xml"/><copyright><![CDATA[Tom Sullivan]]></copyright><language><![CDATA[en]]></language><webMaster><![CDATA[restofthesystem@substack.com]]></webMaster><itunes:owner><itunes:email><![CDATA[restofthesystem@substack.com]]></itunes:email><itunes:name><![CDATA[Tom]]></itunes:name></itunes:owner><itunes:author><![CDATA[Tom]]></itunes:author><googleplay:owner><![CDATA[restofthesystem@substack.com]]></googleplay:owner><googleplay:email><![CDATA[restofthesystem@substack.com]]></googleplay:email><googleplay:author><![CDATA[Tom]]></googleplay:author><itunes:block><![CDATA[Yes]]></itunes:block><item><title><![CDATA[The Maintenance Ceiling]]></title><description><![CDATA[Shipping code is cheaper, but owning what you shipped is harder than ever.]]></description><link>https://news.tomsullivan.dev/p/the-maintenance-ceiling</link><guid isPermaLink="false">https://news.tomsullivan.dev/p/the-maintenance-ceiling</guid><dc:creator><![CDATA[Tom]]></dc:creator><pubDate>Thu, 24 Sep 2026 01:56:39 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!88C2!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4215a6c3-9596-44e8-bd78-48bfc9e92ad5_2912x1638.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<h2><strong>Bottom Line Up Front (BLUF)</strong></h2><p>A small team of engineers can now build more software than it has the capacity to reliably maintain. The time it takes to hit the <em><strong>maintenance ceiling</strong></em> has dropped dramatically with how fast code can be generated. The cheapest response is to build less and / or eliminate bloat, but will ultimately require us to implement at least one intentional maintenance model to operate the system.</p><h2><strong>The Maintenance Ceiling</strong></h2><p>Every software release adds a small, permanent maintenance requirement to the system (e.g., models, prompts, integrations, permissions), but builder capacity doesn&#8217;t grow at the same pace. None of these maintenance obligations are inherently capacity sinks. Eventually though, maintaining previously built apps <em>can</em> consume most of the team&#8217;s current capacity, and I refer to this limit as the <em><strong>maintenance ceiling</strong></em>. </p><p>The following roles are pretty self explanatory, but best to make sure we are on the same page up front.</p><ul><li><p><em><strong>The builder</strong></em> - the team or engineer mainly accountable for creating / building the thing</p></li><li><p><em><strong>The operator</strong></em> - the team or engineer mainly accountable for ongoing maintenance work, enhancements, etc.  </p></li></ul><p>To avoid hitting the maintenance ceiling, builders and operators can&#8217;t add additional handoffs. Domain and / or product teams still need to own the outcomes their systems produce.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://news.tomsullivan.dev/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://news.tomsullivan.dev/subscribe?"><span>Subscribe now</span></a></p><h2><strong>What happens after deployment?</strong></h2><p><a href="https://news.tomsullivan.dev/p/the-work-underneath-forward-deployed">In the previous issue</a>, I proposed that FDE is a new title for an implementation consultant / specialist, etc. Implementation starts with a P&amp;L and business process. Then, the process follows the general flow of the below diagram to move something from idea into production. But, the process below doesn&#8217;t answer <em><strong>who</strong></em> keeps the system both running and <em>current</em> after go-live. <em><strong>Go-live does not transfer a capability by itself, someone must still explain, test, change, recover, and eventually retire the system, and it&#8217;s best to know that person / team up front.</strong></em> </p><blockquote><p>More importantly, what happens when the implementation team / partner / developer leaves?</p></blockquote><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!88C2!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4215a6c3-9596-44e8-bd78-48bfc9e92ad5_2912x1638.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!88C2!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4215a6c3-9596-44e8-bd78-48bfc9e92ad5_2912x1638.png 424w, https://substackcdn.com/image/fetch/$s_!88C2!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4215a6c3-9596-44e8-bd78-48bfc9e92ad5_2912x1638.png 848w, https://substackcdn.com/image/fetch/$s_!88C2!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4215a6c3-9596-44e8-bd78-48bfc9e92ad5_2912x1638.png 1272w, https://substackcdn.com/image/fetch/$s_!88C2!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4215a6c3-9596-44e8-bd78-48bfc9e92ad5_2912x1638.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!88C2!,w_2400,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4215a6c3-9596-44e8-bd78-48bfc9e92ad5_2912x1638.png" width="1200" height="675" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4215a6c3-9596-44e8-bd78-48bfc9e92ad5_2912x1638.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:false,&quot;imageSize&quot;:&quot;large&quot;,&quot;height&quot;:819,&quot;width&quot;:1456,&quot;resizeWidth&quot;:1200,&quot;bytes&quot;:118866,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://news.tomsullivan.dev/i/217003183?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4215a6c3-9596-44e8-bd78-48bfc9e92ad5_2912x1638.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:&quot;center&quot;,&quot;offset&quot;:false}" class="sizing-large" alt="" srcset="https://substackcdn.com/image/fetch/$s_!88C2!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4215a6c3-9596-44e8-bd78-48bfc9e92ad5_2912x1638.png 424w, https://substackcdn.com/image/fetch/$s_!88C2!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4215a6c3-9596-44e8-bd78-48bfc9e92ad5_2912x1638.png 848w, https://substackcdn.com/image/fetch/$s_!88C2!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4215a6c3-9596-44e8-bd78-48bfc9e92ad5_2912x1638.png 1272w, https://substackcdn.com/image/fetch/$s_!88C2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4215a6c3-9596-44e8-bd78-48bfc9e92ad5_2912x1638.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p></p><h2><strong>History Repeats: The Previous Automation Ceiling</strong></h2><p>Back in 2017/2018 I watched an automation program reach the maintenance ceiling inside a regulated environment. The automation teams were genuinely strong, staffed correctly, and the work was genuinely well done. Even more rare, our governance program was largely a value exporter. The COE team provided infra, office hours, peer collaboration, and some shared operational support.</p><p><em><strong>Note: </strong>While the rest of this article is mainly focused on automation, the following stories and logic can also apply to feature delivery and capability augmentation as well.</em></p><p>As time went on and the volume of automations grew, each additional solution created some additional operational need. None were individually huge, but collectively, it started to add up. Source applications changed, new exceptions surfaced, new edge cases were identified, and business processes were reworked as the industry changed. Each time something changed, the solutions needed rework.</p><p>For some automations like those that didn&#8217;t have API capabilities (I know it sounds crazy today, but it&#8217;s absolutely still relevant in banking), the technology was brittle. Changes to vendor or internal application user interfaces required routine maintenance, which was often unscheduled. Being at the mercy of upstream applications changing required agile pivots to ensure business continuity of the automations.</p><p>Eventually, we floated the idea of staffing and hiring a specific / dedicated operations role. We needed capacity for new work, and it had to come from somewhere. The idea was to hire or find a more junior engineer / resource who would be reskilled on the capabilities while learning the tools to execute full projects. Admittedly, that talent funnel and the training path were never easy to build. <br><br>At the same time, that role is also somewhat antithetical to DevOps in principle. I firmly believe you should own the solutions you build, because ownership is what incentivizes you to build them correctly. But the capacity had to come from somewhere, and a lot of the tooling for that automation wave was less mature than traditional software engineering, especially of today.</p><blockquote><p>Therein lies the dilemma; builders should keep and own the capability, but eventually they run out of capacity and can&#8217;t.</p></blockquote><p>At our org, we landed on a hub-and-spoke model. We worked really hard to enable folks with domain expertise to self organize, judge what work should be done, and execute it as they saw fit (obviously within certain guardrails).</p><p>Now, you may be wondering, <em><strong>&#8220;how do we solve the same issue when we have more citizen development&#8221;</strong></em>? Which, at present is especially urgent in middle-market organizations and orgs without large / dedicated IT departments.</p><h2><strong>Back to the Present: AI Accelerates the Problem</strong></h2><p>As mentioned above, code generation is becoming more expedient. At the cost of a few thousand dollars in tokens, someone can ship a working prototype of an application out the door to users for feedback in days to weeks. Even non-software engineers are building and managing AI applications (whether these be vibe-coded or built with purpose / agentic engineering).</p><p>In the past builders were trained alongside a support network or a team, typically with some apprenticeship and systems in place for getting safe reps. COEs and Engineering Guilds also provided office hours, peer collaboration, and shared operational support. Because code is quicker and easier to generate, a lot of code is written by non-engineers without those support systems in place.</p><p>Consequently, I have spoken to many leadership teams who experimented and built software internally in the past 2 years. Early on, custom built AI software was cheaper and more purpose built than off the shelf SaaS (particularly at the application layer). Now leaders are struggling with non engineers maintaining the applications they built:</p><ul><li><p>Sales folks adding features to a custom CRM they built instead of selling</p></li><li><p>Marketing resolving bugs overnight for a campaign in the morning</p></li><li><p>Finance manually reconciling data between manual processes and the newly built applications before month-end</p></li></ul><p>None of that work is on any of these folks&#8217; original job description; they built tools out of excitement (or pressure from leadership), but the long tail of maintenance comes after. Every hour these teams spend on maintenance typically can be better spent performing their core job function. When they hit that limit, they hit the maintenance ceiling. Now, business folks are experiencing something previously felt most acutely by engineering teams.</p><p>This is especially critical for organizations without large tech / IT departments. These orgs may have strong operating owners and lean technical teams. At their current scale and operational complexity, they do not need (nor can they afford) permanent specialists for every dependency. BUT, they still need a maintenance model, including the right discipline (people / process) and technology to support it. But, how do we decide which model and who ultimately should make that decision?</p><h2><strong>What Does the Research Say?</strong></h2><p><strong>Caveat Up Front:</strong> I have not yet found a study that measures the maintenance ceiling in a meaningful way (though if you have one to share, I would be grateful). The diagram below shows how I mentally think about it, and should be read directionally. However, I would argue that the following studies support the concept.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!XkB0!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8e0384c-0a03-4adc-ac49-cbd7c59c57aa_2912x1638.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!XkB0!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8e0384c-0a03-4adc-ac49-cbd7c59c57aa_2912x1638.png 424w, https://substackcdn.com/image/fetch/$s_!XkB0!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8e0384c-0a03-4adc-ac49-cbd7c59c57aa_2912x1638.png 848w, https://substackcdn.com/image/fetch/$s_!XkB0!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8e0384c-0a03-4adc-ac49-cbd7c59c57aa_2912x1638.png 1272w, https://substackcdn.com/image/fetch/$s_!XkB0!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8e0384c-0a03-4adc-ac49-cbd7c59c57aa_2912x1638.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!XkB0!,w_2400,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8e0384c-0a03-4adc-ac49-cbd7c59c57aa_2912x1638.png" width="1200" height="675" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b8e0384c-0a03-4adc-ac49-cbd7c59c57aa_2912x1638.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:false,&quot;imageSize&quot;:&quot;large&quot;,&quot;height&quot;:819,&quot;width&quot;:1456,&quot;resizeWidth&quot;:1200,&quot;bytes&quot;:155757,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://news.tomsullivan.dev/i/217003183?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8e0384c-0a03-4adc-ac49-cbd7c59c57aa_2912x1638.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:&quot;center&quot;,&quot;offset&quot;:false}" class="sizing-large" alt="" srcset="https://substackcdn.com/image/fetch/$s_!XkB0!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8e0384c-0a03-4adc-ac49-cbd7c59c57aa_2912x1638.png 424w, https://substackcdn.com/image/fetch/$s_!XkB0!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8e0384c-0a03-4adc-ac49-cbd7c59c57aa_2912x1638.png 848w, https://substackcdn.com/image/fetch/$s_!XkB0!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8e0384c-0a03-4adc-ac49-cbd7c59c57aa_2912x1638.png 1272w, https://substackcdn.com/image/fetch/$s_!XkB0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8e0384c-0a03-4adc-ac49-cbd7c59c57aa_2912x1638.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p></p><ol><li><p>Google SRE writes that toil is <em>&#8220;the kind of work tied to running a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows&#8221;</em> (<a href="https://sre.google/sre-book/eliminating-toil/">Google SRE</a>). From their documentation, they prescribe keeping the operational work below 50% of each SRE&#8217;s time. Furthermore, if left unchecked, they state that toil can quickly fill 100% of everyone&#8217;s time. As we will discuss in the <em><strong>Productize Shared Operations</strong></em><strong> </strong>section below, being intentional about working ON the platform can help mitigate the risk that any team will become solely focused on Ops.</p></li></ol><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Ktxu!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3417c20f-0bc5-4bbe-b683-947fd853b514_1072x461.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Ktxu!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3417c20f-0bc5-4bbe-b683-947fd853b514_1072x461.png 424w, https://substackcdn.com/image/fetch/$s_!Ktxu!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3417c20f-0bc5-4bbe-b683-947fd853b514_1072x461.png 848w, https://substackcdn.com/image/fetch/$s_!Ktxu!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3417c20f-0bc5-4bbe-b683-947fd853b514_1072x461.png 1272w, https://substackcdn.com/image/fetch/$s_!Ktxu!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3417c20f-0bc5-4bbe-b683-947fd853b514_1072x461.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!Ktxu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3417c20f-0bc5-4bbe-b683-947fd853b514_1072x461.png" width="728" height="313.06716417910445" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3417c20f-0bc5-4bbe-b683-947fd853b514_1072x461.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:false,&quot;imageSize&quot;:&quot;normal&quot;,&quot;height&quot;:461,&quot;width&quot;:1072,&quot;resizeWidth&quot;:728,&quot;bytes&quot;:123313,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://news.tomsullivan.dev/i/213767810?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3417c20f-0bc5-4bbe-b683-947fd853b514_1072x461.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:&quot;center&quot;,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!Ktxu!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3417c20f-0bc5-4bbe-b683-947fd853b514_1072x461.png 424w, https://substackcdn.com/image/fetch/$s_!Ktxu!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3417c20f-0bc5-4bbe-b683-947fd853b514_1072x461.png 848w, https://substackcdn.com/image/fetch/$s_!Ktxu!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3417c20f-0bc5-4bbe-b683-947fd853b514_1072x461.png 1272w, https://substackcdn.com/image/fetch/$s_!Ktxu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3417c20f-0bc5-4bbe-b683-947fd853b514_1072x461.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><ol><li><p>A 2019 study worked with 43 developers and estimated that tech debt consumed ~23% of the engineers&#8217; working hours (<a href="https://research.chalmers.se/en/publication/511450">Besker, Martini, and Bosch</a>). My SWE friends will be quick to point out that technical debt is NOT the same as total application / platform maintenance. Maintaining existing systems can steal meaningful development capacity for feature delivery or platform improvement.</p></li><li><p>The current AI evidence makes the who / what / how of the supporting system urgent. DORA&#8217;s 2025 research compiled survey responses from nearly 5,000 tech professionals, and <em><strong>90%</strong></em> reported using AI at work. I suspect this is higher for SWEs today. In that study, DORA observed a positive relationship between AI adoption and <em><strong>delivery throughput</strong></em> but negative relationship between AI adoption and <em><strong>delivery stability</strong></em> (<a href="https://cloud.google.com/blog/products/ai-machine-learning/announcing-the-2025-dora-report">DORA, 2025</a>). They label this broader pattern <strong>&#8220;downstream disorder&#8221;</strong> (i.e., the work downstream of the code). Velocity gains from writing code agentically can be quickly absorbed by additional required testing, security, and/or deployment bottlenecks. DORA guidance treats the platform as a required internal product, NOT an infra queue (more on that topic in the <em><strong>Productize Shared Operations</strong></em><strong> </strong>section below).</p></li></ol><blockquote><p><strong>Shipping code faster does not create the downstream capacity required to maintain the app(s).</strong></p></blockquote><h2><strong>Be Selective on What to Ship</strong></h2><p>A cheaper lever worth considering before choosing any maintenance and support model is elimination. As process improvement specialists will recognize immediately before building automation, the target process should be <strong>known</strong>, <strong>understood</strong>, <strong>optimized / eliminated</strong>, and only then <strong>automated</strong> (this has deep roots in Lean Process Improvement).</p><ul><li><p><em><strong>Known</strong></em> - the process is identified and documented / articulated as it actually works. This is actually more often NOT exactly as the spec / procedure describes it. Documentation has a nasty habit of going stale the moment it&#8217;s written.</p></li><li><p><em><strong>Understood</strong></em> - we can articulate and describe the outcome, benefit, or requirement for the process to exist.</p></li><li><p><em><strong>Optimization </strong></em>- improve the process. This is fairly broad, but could be as easy as facilitating a conversation between two employees, rearchitecting the process around new capabilities, or making existing capabilities run more smoothly. In some cases, the best process optimization would be to <em><strong>eliminate the process entirely. </strong></em>Note: typically, this isn&#8217;t a FULL enterprise workflow, but some subprocess in a decomposed workflow can be axed.</p></li></ul><blockquote><p><strong>Elimination &#187; automation. Elimination can be the most efficient and cheapest way to improve a process.</strong></p></blockquote><p>In process improvement (PI), skipping one or more steps in the PI sequence used to be painful down the road. Automating an unmapped or undocumented process took months of upfront process conversations, business interviews, and multiple build and test cycles. Then, requirements were firmed along the way but one or two inevitably always landed during testing. Now, AI makes navigating through the PI sequence faster, which is a double edged sword.</p><ul><li><p><strong>Pro: </strong>On one hand, process discovery <em>can</em> now move much faster. Source systems can be reviewed quickly (e.g., mining traces, logs, etc.), which can help alleviate the pain of manually mapping each system connection. But this still doesn&#8217;t completely eliminate discovery and trust-building in interviews.</p></li><li><p><strong>Con: </strong>On the other hand, because building and testing is now much faster, delivery teams are incentivized to accelerate the discovery and jump quickly to build and test cycles. Unfortunately, the skipped steps still exist, but they now pop up as missed edge cases, a change requiring maintenance, or less trust for the business team or customer benefiting from the work.</p></li></ul><p>I&#8217;m not saying we have to be fully dogmatic every time with the PI process, especially in the case of experimentation and prototyping. Prototyping and A/B testing is one important way to achieve shared consensus on what should be built and what the lift would look like. However, we do need to ensure that the experiment is cost controlled and time bound. We also should ensure that the experiment lifecycle is <em>roughly</em> determined before we hit go.</p><p>One risk that&#8217;s even more relevant today is that the experiment unintentionally becomes a permanent feature. This experiment &#8212;&gt; feature risk has always been a challenge of software development, but the speed at which we can create MVPs makes the risk more likely.</p><p><strong>Relevance to Feature Delivery</strong></p><p>We have been more focused on discussing software automation, but this also applies for product features delivery too. For feature development, core product management is almost more critical now than ever. Deciding what&#8217;s above the line in the build queue / backlog avoids customer feature fatigue, and potentially maintaining features that long term detract from your ability to ship more impactful ones. I&#8217;m not an expert product manager, but I would argue that the following questions are still extremely relevant:</p><ul><li><p><strong>Juice Worth the Squeeze: </strong>Does your market research suggest that your customers would actually pay for this feature?</p></li><li><p><strong>Buy vs. Build: </strong>Are you building something that you could buy for cheaper? Does the feature you are planning to ship align with your intentional product vision / competitive advantage, or are you building away from the core product?</p></li></ul><h2><strong>Potential Maintenance Choices</strong></h2><p>Eliminating process steps or killing non-essential features helps reduce the speed at which teams reach the maintenance ceiling, but the ceiling doesn&#8217;t go away. After removing these features / process steps, we likely have a better window to actively prepare for and manage the maintenance overhead.</p><p>Below are a few potential models for managing the maintenance / operations of these apps / services. Selecting one or several will depend heavily on an organization&#8217;s implementation model (e.g., internally developed or built by third party), business complexity, shape, maturity, etc. These aren&#8217;t mutually exclusive, especially for larger organizations, but there are advantages to adopting a consistent model.</p><p><strong>Maintenance Models:</strong></p><ol><li><p><strong>Delegated Build, Operations Ownership:</strong> After a building team completes the work, the operating team will manage and can independently change / update the application without depending on the initial delivery (builder) team. In this case, the builders could be internal engineering teams or FDEs.</p><ol><li><p>Ideally, this handoff should include much more than just a code repo and documentation. The building team must also have a transition plan for humans or agents: how to access the code, evaluate the model results, release changes to prod, manage incidents, etc.</p></li><li><p><strong>Note: </strong>I will dive deeper into the different models for operator owned capability shapes in a future issue.</p></li></ol></li><li><p><strong>Partner Manages the Maintenance:</strong> An external partner assumes the operator role on behalf of the organization with specific, documented Service Level Agreements (SLAs), escalation paths, etc.</p><ol><li><p>Mentally, similar to how MSPs operate for IT applications. For middle-market orgs, especially those with lean IT teams, this can be a practical answer, though selection of firms is limited.</p></li><li><p>As with any service, I would recommend explicit language around pricing, governance, and exit path(s) to ensure quality service management.</p></li></ol></li><li><p><strong>Build and Operation Shared:</strong> Both builders and the operating teams build and run the app together, with overlapping but not fully equivalent responsibility. This is most closely aligned to DevOps collaboration principles. While this model can be extremely effective, it can be tricky for less mature / lean engineering teams.</p></li></ol><p>We still need to identify an owner up front. None of these maintenance models remove the requirement that someone or some team needs to <em><strong>own</strong></em> the application or service throughout the build, handoff, maintenance, and retirement phases.</p><h2><strong>Productize Shared Operations</strong></h2><p>It&#8217;s rare (though not impossible) for teams to build and market internal tools that scale well beyond their team. This would be wonderful in theory, but is unlikely to be unanimously successful without strong internal community and leadership buyin. Internal products still need product management and product vision, or these products can even consume capacity from the team&#8217;s normal duties.</p><p>For example, I have seen a team build a feature flag service that was useful for their own delivery and safe feature rollouts before the equivalent SaaS was available internally. After other teams started using the service, up to ~40% of the team&#8217;s capacity was consumed to enhance and manage the feature flag application. But their code mandate hadn&#8217;t changed, so their velocity took a hit before an owner was named for the flagging service (which was eventually retired).</p><ul><li><p><em><strong>Central group manages software that was chucked over the wall:</strong></em> We have been working hard for years to further integrate engineering, product, and business teams; this undoes a lot of that transformation. This group will have to try their best to manage more than they can reasonably deliver. Hopefully others agree, but the gap between pilot / MVP and production is <em>large</em>. Productionizing and operationalizing code is still hard work (especially with non-SWE citizen development). The handoffs will lead to long lead times, and games of telephone (e.g., builder agent to builder to operator to operator agent).</p></li><li><p><em><strong>Centralized AI / Engineering support: </strong></em>A central group that only answers questions about what to do and how to do it. The engineers will likely disengage, and outcomes could suffer.</p></li></ul><p>Google SRE treats this problem as a true risk, which they work to actively manage. In their documentation, they promise new hires that SRE is not the typical Ops organization, intending to &#8220;keep that promise by not allowing the SRE organization or any subteam within it to devolve into an Ops team.&#8221;</p><p><strong>What to do instead:</strong></p><ul><li><p><em><strong>Productize Shared Operations:</strong></em> That product can / should include infrastructure, observability / traceability, AI evaluation harnesses, release management, incident management, and plans for software retirement.</p><ul><li><p>The product creates the paths to help reduce the operating work for each new application (e.g., something that lowers the cost of safe operation for <em><strong>every</strong></em> subsequent build). </p></li><li><p>Ask the question: &#8220;<em>does this reduce the work for the next app, or only this one?&#8221; </em>Anything that only helps the current build is a service. </p></li></ul></li><li><p><em><strong>Ensure that the governance function is a value exporter:</strong></em> Governance often gets a bad rap because it&#8217;s often a red tape machine that slows builds and re-litigates the same problems. The governance teams then retool their processes, frustrating everyone that has to relearn a new process again.</p><ul><li><p>Build the systems focusing on Developer Experience (DevEx) that make the fast path the easiest path (e.g., pre-approved architecture patterns, automated risk based approvals, consistent CI/CD reference patterns, etc.).</p></li></ul></li></ul><p>While the measurement is different (i.e., application count vs. service count in the DORA / SRE literature), I am trying to advocate addressing the same root problem. Without intentional work at both the team and organizational level, we approach a similar ceiling. Domain teams should still own both judgement on what to do and the outcome of the work. The shared function makes operating the system easier over time (without having to recreate learnings after each improvement). </p><p><em><strong>Productizing Shared Operations</strong></em><strong> </strong>helps address the global capacity ceiling, not the team level ceiling, but both need to be addressed. A single team with saturated overhead can be supported with one or more of the strategies mentioned above in the <em><strong>Potential Maintenance Choices </strong></em>section. But, an overall operating saturation cannot be addressed by the same strategies (each team solves similar problems independently, which prevents knowledge and solutions from compounding).</p><blockquote><p>Go-live doesn&#8217;t mean &#8220;done&#8221;; the building team has to collaborate with the operating team to enable the full application lifecycle (operating a live system, making changes, and eventually retiring the app).</p></blockquote><h2><strong>Key Takeaway: Choose the Operator Before Launch</strong></h2><p>I focused the previous issue on broadening the definition of who can translate operating work into software. In this piece, I walked through the concept of the maintenance ceiling, and how to operate the system when the builder leaves to avoid hitting the ceiling.</p><p>In that spirit, before your next AI application ships, I would ask several questions to make sure that you can set the operators up for success:</p><ol><li><p>Should this process or feature exist at all?</p></li><li><p>Who owns the outcome?</p></li><li><p>Who will operate it?</p></li><li><p>Has the intended operator been notified?</p></li><li><p>Who&#8217;s on the hook for bugs? Who&#8217;s on the hook for enhancements?</p></li><li><p>Which operating work should become a reusable part of the shared system?</p></li><li><p>Who decides when the automation / feature gets retired? And who performs that retirement?</p></li></ol><p>If an operator can&#8217;t be named before launch, it may default to who finds the bug overnight, whether or not they have the capability to fix it is a separate problem.</p><p>Have others seen any similar problems?</p><p>Have a wonderful week!</p><p>-T</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://news.tomsullivan.dev/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Rest of the System! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p></p>]]></content:encoded></item><item><title><![CDATA[The Work Underneath Forward Deployed Engineering (FDE) is Not New]]></title><description><![CDATA[Code is getting cheaper to teach. Twenty years of operating judgment is not.]]></description><link>https://news.tomsullivan.dev/p/the-work-underneath-forward-deployed</link><guid isPermaLink="false">https://news.tomsullivan.dev/p/the-work-underneath-forward-deployed</guid><dc:creator><![CDATA[Tom]]></dc:creator><pubDate>Thu, 27 Aug 2026 21:58:28 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!xg6m!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc2adf38e-4049-4564-bdda-b1d2261ffea9_2400x1254.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<h3>Bottom Line Up Front (BLUF)</h3><p>Forward deployed engineering is implementation consulting rebuilt for new probabilistic based software systems. AI agents will make building these software solutions cheaper, but durability sits in the translation between a Profit and Loss statement and software behavior (including the software controls, change management, and owned capabilities). AI is quickly changing the tools to implement and increasing the pace of change, but it does not remove the need for the implementation discipline, nor the work needed to bridge the talent supply shortage we are currently facing. </p><h3>Time Machine to 2014</h3><p>In 2014, implementing a Laboratory Information Management System (LIMS) meant wiping my clipboard with isopropyl alcohol, gowning up in my blue bunny suit, and walking into ISO 8 and ISO 7 manufacturing areas.</p><p>The site made injectable IV bags across multiple manufacturing lines: saline, Lactated Ringer&#8217;s, and other terminally sterilized products (i.e., non-aseptic filled products). I was helping implement some of the LIMS workflows for biological and chemistry quality (BQ and CQ) on what was then Lonza&#8217;s MODA platform.</p><p>That meant locating the environmental monitoring points across the actual manufacturing plant. For example, that could be identifying physical tank-sampling valves or viable and total-particulate monitoring locations. Then, determining how frequently each one needed to be sampled. The BQ and CQ teams needed to know what sample to collect, where to collect it, and the sample scheduling to make sure they were compliant.</p><p>Also, those aligned answers had to be consistent across the process, the standard operating procedure, and the LIMS system configuration. If one of those aspects was incorrect, the software was wrong, and we could be at risk of a potential investigation (if you haven&#8217;t written a CAPA before, you&#8217;re in for a treat). The implementation also had to go through pharmaceutical software validation and hold up during an eventual FDA audit.</p><p>This overall exercise taught me one of the most valuable lessons of my career:</p><blockquote><p><strong>None of the implementation exercise started with the code. The process started with walking the floor, physically speaking with the operators doing the work, and translating that work into the software. </strong></p></blockquote><h3>Implementation as a Discipline</h3><p>Today, many folks would describe much of that implementation process as forward deployed engineering. At the time, my badge said <em>engineer</em>, but we didn&#8217;t call that work &#8220;Forward Deployed Engineering,&#8221; we just called it software implementation.</p><p>I am not making the old-man-on-the-internet claim that we secretly invented FDE first (I hope I&#8217;m not there yet, but my kids may say otherwise). Admittedly, the FDE title is also still useful, and the current AI assisted tooling <em>genuinely</em> changes what one person can build. My point is that the discipline underneath this new motion already had a name, a methodology, and history several decades before the current label arrived.  </p><p>That methodology really narrowed down to business // technology translation; how can one translate a problem into a system that changes (and hopefully improves) the operation in a meaningful way. </p><p>See the below metaphorical translation &#8220;chain&#8221; that starts with the Profit &amp; Loss (P&amp;L) of the business and walks forward all the way to &#8220;reality&#8221; where things are executing and working within that business. </p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!xg6m!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc2adf38e-4049-4564-bdda-b1d2261ffea9_2400x1254.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!xg6m!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc2adf38e-4049-4564-bdda-b1d2261ffea9_2400x1254.png 424w, https://substackcdn.com/image/fetch/$s_!xg6m!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc2adf38e-4049-4564-bdda-b1d2261ffea9_2400x1254.png 848w, https://substackcdn.com/image/fetch/$s_!xg6m!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc2adf38e-4049-4564-bdda-b1d2261ffea9_2400x1254.png 1272w, https://substackcdn.com/image/fetch/$s_!xg6m!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc2adf38e-4049-4564-bdda-b1d2261ffea9_2400x1254.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!xg6m!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc2adf38e-4049-4564-bdda-b1d2261ffea9_2400x1254.png" width="1456" height="761" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c2adf38e-4049-4564-bdda-b1d2261ffea9_2400x1254.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:761,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:95508,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://news.tomsullivan.dev/i/212998148?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc2adf38e-4049-4564-bdda-b1d2261ffea9_2400x1254.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!xg6m!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc2adf38e-4049-4564-bdda-b1d2261ffea9_2400x1254.png 424w, https://substackcdn.com/image/fetch/$s_!xg6m!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc2adf38e-4049-4564-bdda-b1d2261ffea9_2400x1254.png 848w, https://substackcdn.com/image/fetch/$s_!xg6m!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc2adf38e-4049-4564-bdda-b1d2261ffea9_2400x1254.png 1272w, https://substackcdn.com/image/fetch/$s_!xg6m!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc2adf38e-4049-4564-bdda-b1d2261ffea9_2400x1254.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p></p><p>In the above translation chain, the software box is only one small part of driving improvements that translate back to the P&amp;L of a business. If you enter the improvement process at the &#8220;software behavior&#8221; box, you are gearing up to learn an expensive lesson. By starting downstream, you inherently agree to every upstream assumption, and treat them as settled.</p><p>Starting this late, it&#8217;s much harder to challenge the process. </p><blockquote><p>Outside of technical implementation chops, an FDE should be able to work through tough implementation questions about the process and push back where the best solution is NOT AI. </p></blockquote><p>Likewise, as a buyer, I would hope and expect that an FDE would ask questions like:</p><ul><li><p><strong>Process Hops: </strong>Is this process handoff necessary? </p></li><li><p><strong>Regulatory Requirements: </strong>Is it required by a regulation / law?</p></li><li><p><strong>Change Management: </strong>Does this process exist because it is just &#8220;the way we have always done it?&#8221;</p></li><li><p><strong>Exception Management: </strong>Is the exception actually exceptional or should it be a warning?</p></li><li><p><strong>Outcome Measurement / KPI / OKR: </strong>Does the measurement metric measure the desired outcome of the process?</p></li><li><p><strong>Optimization: </strong>Do we even need this process at all? (the most efficient process is an eliminated one)</p></li></ul><p>Often, one of those assumptions / questions actually highlights the breakdown in the value chain.</p><p>ERP vendors built an enormous talent market around folks who could cross the gap between SAP transaction codes and a business process. Over time seasoned consultants would also build a heuristic understanding of where these flows would typically fall apart within an organization AND how to help redesign process around their system (partly for the betterment of the organization, but also partly because it made the ERP more sticky).</p><p>Over time, the implementation process became repeatable, and a competitive advantage for consultants. There was much time spent honing implementation methods, building talent development pipelines (through materials and a lot of on-the-job training), and even rate differentials for on/nearshore delivery. As the market saturated, the generic delivery DID get cheaper, but the domain expertise, business judgment, and apprenticeship / capability building genuinely remained scarce.</p><h3>Back to the Present</h3><p>With new capabilities (i.e., AI), the time it will take us to build repeatable implementation processes, build new implementation methods, talent development pipelines, etc. will almost assuredly decrease. Agent / software factories, lowcode agent builders, managed code environments (i.e., runtimes) and model provider tooling are already compressing the build and execution. At the same time, the frontier labs like Anthropic &amp; OpenAI are investing billions in &#8220;last mile&#8221; implementation and delivery because access to foundation models does not guarantee a change in client operations.</p><p>I argued in <a href="https://news.tomsullivan.dev/p/the-model-is-the-small-box">my first article</a> that the model was the small piece in a much larger ecosystem required to create value. When it comes to the FDE / talent play, the model IS a real difference in this wave because it enlarges the whole diagram rather than just one box. When you have these probabilistic models, you must embrace and defensively design systems that handle receiving different answers to the same question (the model is a software slot machine after all).</p><p>To resolve this, there are several best practices we have to consider in designing and building these systems (i.e., building continuous evaluation, ensuring your failures are loud and not shaped like successes, facilitating meaningful organizational change management, and being ready to manage the complexity of ongoing maintenance). </p><p>Consequently, these factors make the implementation discipline MORE important. With this in mind, it makes sense that the current demand for these folks is outstripping supply, but that leaves us with a new question&#8230;</p><h3>Ok, how do we mint more of these people?</h3><p>My current working prediction is that many excellent FDEs will come from business analysis, operations, domain roles, and automation engineering, not only conventional software engineering. Code is getting cheaper to teach and produce, and access to world class training materials is being created for free on YouTube and made available by the foundation model providers. However, twenty years of expertise and judgment from a niche domain continues to be scarce.</p><blockquote><p><strong>Therefore, while we shouldn&#8217;t lower the talent standard, we must consider broadening the talent pool to meet the demand.</strong></p></blockquote><div><hr></div><p><a href="https://tomsullivan.dev/writing/fde-before-the-label/">Please check out the full essay including proposed hiring models here. </a></p><p>I&#8217;m curious where you draw the line between implementation and forward deployed engineering!</p><p>Have a wonderful week and upcoming weekend,</p><p>-T</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://news.tomsullivan.dev/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Rest of the System! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div>]]></content:encoded></item><item><title><![CDATA[The model is the small box]]></title><description><![CDATA[The AI model is likely the most reliable part of your entire system]]></description><link>https://news.tomsullivan.dev/p/the-model-is-the-small-box</link><guid isPermaLink="false">https://news.tomsullivan.dev/p/the-model-is-the-small-box</guid><dc:creator><![CDATA[Tom]]></dc:creator><pubDate>Mon, 24 Aug 2026 20:06:46 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!zoId!,w_256,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd2b1092f-0e83-4ba3-b33e-16853284c7ca_1024x1024.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>In 2015, researchers at Google published the following diagram in <a href="https://research.google/pubs/hidden-technical-debt-in-machine-learning-systems/">Hidden Technical Debt in Machine Learning Systems</a>. The almost unreadable, black box labeled &#8220;ML code&#8221; is where a lot of engineering of that era focused. </p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!68W8!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa38d373e-5748-4c44-b376-1307d55324dd_639x201.png 424w, https://substackcdn.com/image/fetch/$s_!68W8!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa38d373e-5748-4c44-b376-1307d55324dd_639x201.png 848w, https://substackcdn.com/image/fetch/$s_!68W8!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa38d373e-5748-4c44-b376-1307d55324dd_639x201.png 1272w, https://substackcdn.com/image/fetch/$s_!68W8!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa38d373e-5748-4c44-b376-1307d55324dd_639x201.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!68W8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa38d373e-5748-4c44-b376-1307d55324dd_639x201.png" width="639" height="201" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a38d373e-5748-4c44-b376-1307d55324dd_639x201.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:201,&quot;width&quot;:639,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:19364,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://news.tomsullivan.dev/i/212599260?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa38d373e-5748-4c44-b376-1307d55324dd_639x201.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!68W8!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa38d373e-5748-4c44-b376-1307d55324dd_639x201.png 424w, https://substackcdn.com/image/fetch/$s_!68W8!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa38d373e-5748-4c44-b376-1307d55324dd_639x201.png 848w, https://substackcdn.com/image/fetch/$s_!68W8!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa38d373e-5748-4c44-b376-1307d55324dd_639x201.png 1272w, https://substackcdn.com/image/fetch/$s_!68W8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa38d373e-5748-4c44-b376-1307d55324dd_639x201.png 1456w" sizes="100vw" fetchpriority="high"></picture><div></div></div></div></figure></div><p>Just like in 2015, the same is true today with AI. From an engineering perspective, we have several to dozens of boxes across data, infra, observability, inference, and more. Similarly, we also have to consider process integration, change management, training and talent management, support, financial validation to fully capture value from the system. </p><p>The diagram is just as relevant today as it was then. Teams and leaders often focus on the model (i.e., hallucination / cost / vendor lock in, etc.). While this is incredibly important, results depend on the full supporting system. </p><p>The FULL system, the one going much beyond the model, is also why I&#8217;m writing this newsletter. <em>The Rest of the System</em> covers the work, judgment, ownership, controls, and long term maintenance around the model. These aspects drive whether AI creates long term value for people and organizations, or the pilot gets canned or relegated to the backlog.</p><p>I spent the last ~15 years implementing systems inside larger orgs, most of which are regulated. I walked manufacturing floors in pharma / consumer products, implemented automations and software, worked with LOB finance teams to qualify opportunities, and HR to carefully plan change management and long term talent management. </p><p>I work with leaders most days to plan, design, build, deploy, and retire solutions today, so I will share what I am building, seeing, and reading from that lens, and I promise to share real examples when I have them.</p><p><strong>Three recurring lenses will organize the publication (likely more to follow):</strong></p><ul><li><p><em><strong>Field Notes</strong></em><strong>:</strong> translation from the physical or business process into software.</p></li><li><p><em><strong>Shaped Like Success</strong></em><strong>:</strong> systems that look green while quietly failing (particularly insidious with AI / nondeterministic systems).</p></li><li><p><em><strong>After the Demo</strong></em><strong>:</strong> adoption, ownership, maintenance, and the operating life of a build.</p></li></ul><p>Expect one issue every week or two.</p><p>The full detailed essays, models, and figures live at <a href="https://tomsullivan.dev/writing/">tomsullivan.dev</a>, but each email version will stand on its own.</p><p>The complete flagship essay behind the first issue is currently live here:</p><ul><li><p><a href="https://tomsullivan.dev/writing/fde-before-the-label/">Forward-Deployed Engineering Is New. The Work Underneath It Is Not.</a></p></li></ul><p><strong>Request</strong>: Please drop me a line and tell me what part of the AI implementation / management process your organization is still figuring out how to build and / or own. I read every reply, and the answers steer what I investigate next.</p><p>Have a wonderful week,</p><p>Tom</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://news.tomsullivan.dev/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading The Rest of the System! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p></p>]]></content:encoded></item></channel></rss>