Your rewritten codebase will presumably look much cleaner and more maintainable that what it's aiming to replace, but it's easy to miss the purpose/necessity of all of the old "cruft" which need not be just poor coding but rather later enhancements not anticipated by the original design whose implementation was therefore crufty.
Does anyone here work at Cisco, and have any insight as to why a rewrite is considered necessary and might be expected to succeed?
A rewrite has historically been a huge mistake in many situations, some of which I've personally had the agony of witnessing firsthand, powerless to stop it. But AI completely changes the calculus because it doesn't waste significant developer time or energy crawling back to the previous working state, it just does it while you go get coffee. And this I've found to be a huge benefit, that I can continue making my codebases nicer to work with and better documented/integrated with bugs stamped out for free. There are upsides to a rewrite, and they're now much more likely to be worth it as the downsides are automated away.
An LLM doesn't have enough context to know what you so and don't want. If you give it call logs for every function that was called and how, maybe, but a seasoned engineer will have more context around what the company needs.
Honestly though, it's pretty clear the AI rewrites work, we have a ton of high profile examples. It's just a holdover attitude about rewrites from before AI. In reality, AI is going to make the code better when it rewrites
Such as?
pgrust, high profile rewrites all over the place.
When you have an oracle to check behavior against, it is really easy to build a robust testing machine with property tests against the original.
You don't get to pick what parts of the API you're porting during something like that, it has to work the same or some of the ecosystem around it will break.
Obviously this fails if this is a large org with many teams and microservices, but for a leaner org this isn’t really a problem.
I realize this thread is about Cisco so my comment doesn’t apply here haha.
And AI is really good at all the stages here.
I'm not a lawyer and don't like writing sentences with "probability bordering on certainty", etc.
- Their output is hyper-customized/optimized for their personal use case, constraints and quirks
- They have zero interest in being an open source contributor and maintainer. Usually because it's a lot of additional work, sometimes for personal safety reasons (harassment targets etc.) In a workplace it can also mean dealing with an extremely tedious legal process with months of uncertainty.
- They figure anyone who wants something similar can just make it themselves instead of using theirs. If I spent between fifteen minutes and two days to make some software, in 3-6 months someone with fewer skills will probably make it in 5 minutes to a few hours using a more powerful model.
- They may not have full rights to redistribute their source code.
Basically most people - individuals and corporations alike - aren't actually interested in donating to open source and in many cases have anti-incentives to do so.
With the current sota models code is free.
But there is pretty big risk of false positives and negatives; some "bugs" are design decisions that were not documented well/at all, and inevitably some new ones will appear when it reasoned something wrong.
AI has no magic powers to figure out what the “API/etc contracts” are in a nasty old codebase. This problem is fundamentally hard and not always well defined.
AI is also (at least for now) not consistently good at writing tests that test the behavior that matters. Recent models seem to be getting slightly less bad, but again, this is a hard problem.
I’ve personally used AI to rewrite gnarly code where the goal is to have exactly identical results to the old code. GPT-5.6 Sol could do this correctly with some extra help guiding it to rest the right things, and I haven’t tried this particular task on a newer model.
But ancient code like in the article is probably an enormous buggy mess full of accidental behaviors and implementing protocols in ways that do not fully match the spec. Some, but not all, of the mismatches will be cases where the spec is wrong. Sure, an AI can probably generate a new implementation with a similar set of bugs, but is this what is actually wanted? Also, depending on the gnarly set of bugs, a clean implementation may be unable to reproduce the bugs, leading to a messy implementation instead.
AI makes these projects significantly more feasible, at least in my domain. We have some critical-path software I wouldn't touch with such an approach, but the vast majority of our code is internal and/or supporting tooling.
For those cases, burning it down and rewriting via AI tooling has proven far more productive than anything we have attempted in the past. This is for old technical debt laden stuff everyone has been afraid to touch for well over a decade.
It's not perfect, but it moves the needle where before a lot of such projects were stuck on the backburner for years.
It certainly introduces new bugs and such, but at least this way forward momentum is happening vs. just being stuck in decision/analysis paralysis forever.
This also assumes some very senior SME folks are driving the process. I would not expect similar results with junior or even mid-level folks in these specific domains.
One of the reasons to avoid rewriting has long been that the time spent rewriting code that works is time not spent writing code to meet new business needs. In other words, it's usually a huge opportunity cost, rarely balanced out by sufficient gains to be net positive.
I do think AI does challenge this prior, as the time needed to do a rewrite shrinks dramatically. It's not zero, of course, but it now becomes worth asking if it's worth doing, rather than defaulting to avoiding rewrites in nearly all cases.
Until one day you leave and the next guy comes and says “what is all this shit”
Ha, Ha. Tell that to Gnome or KDE teams.
Idiocracy (2006): https://en.wikipedia.org/wiki/Idiocracy
So in a way, he was a more effective president than Trump. He was smart enough to know he wasn’t the smartest person in the room.
What we have now is, in a way, worse than idiocracy: we have kakistocracy.
We had to turn off a bunch of on-by-default features in the system I'm deploying now because IOS matches too aggressively and decides that arbitrary UDP datagrams are DHCP requests or PTP packets that need rewriting (and then dropping when they turn out to not be DHCP/PTP). I have to think that an AI rewrite with a today-good model would recognize the lax checks and do them better.
A critical security firmware update once brought down redundant pair of firewalls we used because the bug didn’t manifest until a counter buffer overflowed months after application. Three hours into the outage the tech power cycles the thing and our services came back to life.
Cisco is only doing this because some other alternative is more expensive. They had more than a decade to get started.
Legacy code is typically a euphemism for old unmaintained code, no one knows how it works so it can be edited, and it probably doesn't align with current work practices.
Not to say a rewrite is necessary right, but avoiding rewrites of working code is how we ended up with more COBOL code than COBOL programmers in 2026.
I think people are assuming the worst but AIs most helpful contribution beyond churning out code is explaining the code base.
Who knows if they will do a good or bad job that's down to the engineers and not the tools imo.
That said, there's also research pointing to LLMs encoding existing bugs as test requirements: https://arxiv.org/abs/2607.05139
That CS PHD researcher, Konstantinou, in particular, has written a lot on this: https://arxiv.org/search/cs?searchtype=author&query=Konstant...
Once AI came along, I became like Captain Picard (RIP) and would just sit back and let the AI write all unit tests. "Try harder, make sure coverage is 100%". It was very exhilarating to sit back and watch the models churn away, creating 100s of lines of unit tests. Of course, I checked them all manually before submission ;-)
The issue with using "the code as the spec" is that humans can't read 100K lines of code in a day, can't hold the entire ball of unstructured mud in their brains all at once, and generally require hierarchical "structure".
Modern AIs just don't have these issues! Their context windows are big enough to encompass an entire codebase, so they can "see everything", all at once.
Each line of new code they write is compared against every relevant line of code in the original, which they treat as a spec.
My main issue in my experience is that that AIs tended to reproduce the legacy behaviour "bug for bug", which was not my intention.
Similarly, the AIs were hamstrung by external constraints: I typically prefer to leave the DB schema unchanged, which meant the torturous workarounds were needed for design errors in some stored procedures, etc.
The older the code, the bigger the issues with baked-in assumptions about transactions, stateful sessions, data types, string formats, etc.
This trips up AIs just as much as humans. You can't trivially maintain backwards compatibility and modernise the codebase! Those are conflicting goals, no matter what.
There’s clearly a range from small to large where this is not true. Plus it’s well known llm degrade with large context favoring content at the ends more than the middle.
Surely we have more code than programmers for every language in existence?
Except for the part where the concept isn't even well-defined, this sounds like a complaint that the world contains more bricks than bricklayers. Of course it does.
Not necessarily. It's the name for the last thing(s) before the new thing, but you can't shut it down because people are using it and/or the new thing actually isn't so great.
E.g. some critical security vulnerability that is not backported to the library/compiler/runtime you are using in production, or you can't get spare parts for your servers any more so need to upgrade to something newer but you can't run your old OS version on new hardware so you have to upgrade the OS too and now it's dropped support for foo etc etc. Lots of things pile up.
When they started making smallish dedicated routers I remember opening one to see a dodgy backplane and a number of big boards plugged in held up only by minimal slides. The minute it got hot (which it did because no air management) those boards sagged in the middle and it just looked slapdash. The 3Com equivalents were always better built but Cisco had such mindshare it didn’t matter the products were a bit crap
And they've overstretched their product line to too many things, trying to be the "be all" company. Fortinet is making the same mistake, but they're earlier in that process.
I mean, it's not really a mistake on their parts, their primary goal is to get rich and they are succeeding.
Citation: I've been working with Cisco technologies since the mid-1990s. At my current job, we are a few weeks away from phasing out the last of our Cisco junk. And since people will ask: to Juniper/Mist and Arista.
If history has taught us anything. It’s that the measure of a successful product is usually more to do with its sales people.
Cisco did provide updates via CCO and anonymous ftp server. They quite openly told that just download new image and test in your lab, and then if feature set did require licensing, please do it before deploying production. Trust was put on customers and they did very little to check and verify if customers did comply. Which is quite contrary to what for example Oracle practise is been.
>If history has taught us anything. It’s that the measure of a successful product is usually more to do with its sales people.
As someone who worked Cisco gear over thirty years I still remember why us networking did want cisco instead of many other brand devices and it hadn't anything to do with marketing and sales people did.
The Cisco IOS text based configuration was in early days of internet which made it so popular.
Many other manufacturers like HP, 3Com, Cabletron, Madge (IBM), Bridge, etc. did not get why we wanted and complained “give us textual configuration format like IOS has”.
With a plain text based configuration it was easy to adapt configuration from one model to have ready made templates for each model, upgrades and migration to new models were easier, you could use version controls (rcs,cvs, svn,... long before git) and then when RANCiD came it was easy to automate config backups, revert and rollback changes, pass audit with human readable configs and what was in production even long time after etc. In short text based configs made it possible to build large network scale configurations management tools.
IOS has compact, but quite easy to read config snippets made it much easier emailing others, posting examples newsgroups, opening TAC at Cisco CCO, books with examples, learning and teaching etc.
It took quite long before for example HP understood and switched text formats right after turn of century. Juniper seemed to understood right from beginning. But some like Fore, Nortel, Alcatel (ATM-switch makers) and many others did not understand before going out of business. Most of these others had either menus (3Com, HP) or a lousy CLI with obscure unique for just that device model and quite often not even option to print config or if they had not just with changes from default config but printing everything and providing no means to drop showing default. This made it manually while online very hard without external version control to even know what is current status and what had been changes.
BTW, this is still an issue with some manufacturers haven't understood and makes their devices much worse than they understand. Fortigate firewalls CLI is still like this. You can select a subsection of config to print, but it will no indication what differs from default configs. And the whole firewall config is hundreds of kilobytes. Good luck finding what's been changed unless you had saved default config when you started using that device!
When you have thousands to even tens of thousands of network devices in use as many were working you need to be rely that even when you encounter device failure and you don’t have same model any more available, you are able to adapt and find working fix configuring something else doing what is needed.
Binary and any non textual configurations may fit the purpose SOHO category devices, but when network grows even bit larger and configurations more complex there isn’t anything better than text base configs as far as us humans will need to configure and maintain those devices.
Cisco have had issues with software quality for sure, but it was Cisco IOS that was the reason it became so popular and we wanted to use it.
And we did put more weigh on manageability which that IOS text based configuration provided than for example speed some manufacturers got with their devices (Madge, etc.) and even when some Cisco switches dropped some Network World tests more packets we still wanted that manageability more and relied Cisco will fix those issues with software upgrades which it often did.
Juniper when it came end of 90's did understand this and made even better configuration system for Junos than IOS was and still is. Cisco have improved things with IOS-XR, but it's still not up to what Junos is currently.
e: fixed few typos and added that manageability chapter near end. Places where I did work none were always everything Cisco places. No, but either so called ABC (Anything But Cisco) places either. Cisco was and is still popular for those reasons mentioned above.
"even better" is an understatement ! I was working on a dual-vendor network at the time, we ran Cisco and Juniper in parallel.
Working on Cisco always just felt like going back to the 18th century, I mean the whole candidate and active config thing alone in Junos was insanely valuable:
show | compare # diff the config
commit check # lint it
commit confirmed 10 # apply it, but don't persist unless confirmed
commit # confirm
My “favourite” was “admin”, which was used as the login user, was also short form for “administer-all-port-shutdown” (or words to that effect; it was a long time ago). So if you were distracted when logging in, you could easily soft-lock the entire device meaning you’d have to then run into the DC, find the hardware, and physically power cycle it.
Absolute madness.
However there are also things Junos vs Cisco was that Cisco had made bit easier. Just to mention few
1) a simple and quick way to add aliases to config file, which made life easier with long commands that could be otherwise always in hurry hard to remember. Now later versions Junos they have made this better, before it involved twiddling with XML files OS-level etc.
2) IOS TCL scripting, that proved to be quite useful automation tool.
3) OS upgrades of course with a single flash blob made it easier than upgrading or downgrading FreeBSD based Junos.
The last large scale networking project I did was more than 10 years ago and we went with HP.
Personally I don’t want direct console access. I want a modern DevOps pipeline with version controlled config. I’ve got too many battle scares from accidental typos in IOS. I have no intention or reliving that error prone way of network management again.
Almost every new product of theirs I’ve touched in the last 20 years has been trash, and even the R&S golden goose is being dragged in that direction with customer-hostile functionality like Smart Licensing and a general decline in quality.
https://www.cisco.com/c/en/us/support/docs/field-notices/743...
"Cisco Access Points That Are Running, Or Have Run, Releases 17.12.4, 17.12.5, 17.12.6, or 17.12.6a May Exhaust Flash Space, Preventing Upgrades"
Recovery path basically involves either factory resetting every single AP or up to, if it's already bricked, Cisco TAC + you having to console into every single bricked AP you have to pass a secret challenge/response at the bootloader to flash a recovery image :)
Being a mess isn't a problem when you have more than 6 items context.
Otherwise he's just going to replicate all the legacy bugs and questionable design decisions.
Right? ...
[1] https://github.com/enterprisequalitycoding/fizzbuzzenterpris...
Anyone who has used the built in date picker knows why everyone rolls their own.
It's a bunch of text files in a repo with massive churn (all new version 5!), but somehow even the technical debt 'skill' (i.e. text file) has technical debt itself:
https://github.com/DietrichGebert/ponytail/blob/main/skills/...
> ...every ponytail shortcut is marked with a shortcut: comment ... (older code may say ponytail:).
It doesn’t matter if it’s satire. Developers seem to be so vulnerable to all these AI promises of ‘automating’ everything.
BTW it’s terrible. It gave two very hollow suggestions that did not solve the real problem. Hour wasted, angry colleagues, no code change.
https://www.reddit.com/r/programming/comments/eaqgk/java_is_...
"XML is like violence: if it doesn't solve your problem, you aren't using enough of it." -Chris Maden
https://www.reddit.com/r/ProgrammerHumor/comments/1mbgih/xml...
"XML is a giant step in no direction at all" -Erik Naggum
https://quotefancy.com/quote/1643080/Erik-Naggum-XML-is-a-gi...
What have been achieved using this port?
And how did you make sure the port is correct?
I'd ask "did you make sure the port is correct?"
Followup question: did anyone read the automated test suite?
Silence.
The software rewrite is shelved.
The goal and objective is to deliver value for your user/customers; to the speed, reliability, and assurance requirements your market segment expects.
I suspect that the average Cisco customer expects a bit more stability than $RANDOM_SAAS.
If you have good test coverage, and the tests are reliable, then what's the problem?
If your engineering culture is bad already and you don't have good test coverage or they're flaky or whatever then that is different. If that is the case, it is probably from smart-ass attitudes like yours asking these "clever" rhetorical questions to derail people trying to improve things that led it to be like that. Do everyone a favour and keep your mouth shut and let people fix things.
Reading the tests is meant to check that.
Why not just ask the coding assistant?
I’ll take the system written in a language that people can maintain and fix
The answer is always that the rewrite was bug-compatible on purpose, the bugs were to be fixed later on the more tractable system. Every time development was abandoned right away, because there was no gain in using the new system.
27 is still quite young though and very possibly over confident.
[0] https://www.cnet.com/tech/tech-industry/why-microsoft-murder...
A change in behavior maybe construed as a bug, a breaking change or a regression, and a lot of time it's not so obvious. Sometimes you want the original behavior, sometimes you don't. Sometimes changing one for the better means another behavior needs to go, and you don't know if the user already depended on that other behavior. That's the real staying power of legacy software despite universal agreement that they suck.
it's the testing in real world usage to shake out all the edge cases and bugs that's hard and that you can't just write unit tests for
Not clear to me what he's doing. Re-implement switching? Simplify features? Reduce LoC? Add test coverage? Either way, AI in that world will be helpful. I'm sure it'll spot all kinds of problems in opaque drivers that are hard to follow from MF specs.
rewrite can be done to quickly refactor and/or close the feature parity, improve the UI/UX for end customer, add missing features or integration needs
I’d love to know what internal services and tools that are included in this, as well as any deliverables that shipped as software products to customers.
If it’s legacy code for IOS firmware, however, I’d be very worried.
-Works according to a defined specification
-Within the bounds of that specification, has a lot of code written to optimize speed, cost, latency, etc - all measureable goals
-Written in c, with all it's weakness
-Given limitless engineering work, some could be ported to better hardware
-the new system could be automatically compared to the old system for functional correctness
All those factors fit well with agent based development.
Btw, many of them are true in general to embedded systems.
There are also tight time budgets (basically SLAs) that must be met along with memory considerations (both in SW and on the NPU/HBM), not to mention how the device needs to respond to load and various failure scenarios.
if any hardware does not confirm to specs, thats on them and its actually a reason to pull out and replace legacy hw
The whole design where you need to get a magical list of IDs to decode the metrics and the list is not even shipped with device is entirely asinine
Then consider that not all vendors implement the standards identically, because the standards can be a bit woolly and in some cases followed rather than lead live implementations.
There will be a bunch of code that looks useless, but is actually there because of specific interoperability scenarios.
I wonder whether AI is any good at looking at that code,running the system, and documenting those edge cases?
Lots of this stuff should work in theory, you only find out that it doesn't in practice when you try it on real hardware. And there are thousands of bits of hardware.
This whole thing sounds like a deluded vanity project birthed in an ivory tower.
But yes, fact it uses a bunch of very well documented standards with multiple implementations gonna help massively
I think it is a reasonable risk to encounter these bugs again in order to get a more clean and documented codebase. Especially when asking AI to plan and A/B release is also trivial.
Fortune's source was the CEO himself:
> “I’ve given him unlimited tokens because he’s going to refactor code,” Patel told Fortune at a small roundtable dinner. “So I’m like, you can go, you can go nuts.”
I think that Fortune story is better than the Tom's Hardware one. It ties the Cisco anecdote into a larger trend of some people deliberately hiring younger, less experienced developers for their AI projects - because they are more likely to be "AI native".
I frequently find myself in conversations with people who are resistant to AI and turn out to have a very skewed idea of what it can and cannot do.
Or they were reading your comments and articles and couldn't reproduce any of the fantastic results that you were claiming, even back then. Things have indeed changed since the first Opus release about a year ago and LLMs are good at programming now. But still, experienced professionals are not so keen to sign off 1000s of lines of code that they never read, take responsibility for it etc. Also, it is a different job and it will take some time to adjust. Can you blame them?
Moreover, other trades have their own guilds or are otherwise protected through regulation. Can you blame programmers for fighting to keep doing things as they were doing it until recently?
What did you have trouble replicating?
> Also, it is a different job and it will take some time to adjust. Can you blame them?
Not in the slightest. This stuff is really hard. The changes it makes to our production are disruptive and, in many cases, unwelcome.
The main takeaway though is we had one person, in a single month, do the equivalent of at least a couple years of work for a small team. We view bugs and performance issues as a currently inevitable part of the process and we are dealing with them with models just like we do with humans on every other project already. Overall though it is representing an absolutely massive speedup. If you work in an environment where the models make egregiously more mistakes than your human counterparts then I am quite jealous.
yeah, i think they'll survive whatever the outcome
I'm worried for us users of the internet rather than them.
https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2Fv...
https://slimemoldtimemold.com/2026/08/05/a-stupid-idea-for-a...
The ability to clean room Adobe products came at about the same time Claude's RE abilities suddenly improved.. Hmm lets think about this a little more.
Anthropic sucked the bones dry of the public domain, literature and all of opensource. It seems likely that Anthropic RL'd Claude into being a great reverse engineer because it needed more training data.
I believe that Anthropic turned Claude into a digital software shredder and started decomping proprietary code, like adobes.
So the reason Claude can suddenly clean room Photoshop isnt because it became a more capable programmer, but rather, the Adobe code is literally in its training data now.
Ciscos internal legacy code is not going to be in Claude's data (yet).
The decision to increase Claudes ability to RE is brilliant, because now everyone is doing the source code decomp for them, and sending them fresh data from previously closed source repositories of code.
Thanks for the very thought provoking reply tho!!!!!!!!!!!
Demonstrates fanatical devotion to capital’s golden idol - “yes, all of the code; yes, unlimited budget”. Doesn’t actually cost the company anything - only one engineer off task, any nuggets of value can be harvested as it progresses but any costs or big changes can be deferred until completion of the impossibly large open-ended mandate (ie never), one guy won’t have the personal bandwidth to spend so much that the “unlimited” part actually matters, and in case of emergency it only costs six months severance to make it all go away. It even rescues a fellow national from the ignominy of being employed so far below his previous station, there’s so much to like about it. Hmm? The code? What about it?
I see you haven’t met my coworkers and their fanatical use of parallel “workflows”
Tell me you don’t use agents effectively without telling me you don’t use agents effectively.
The feeling of coding like that is a bit like realizing that an elevator is 10cm of aluminum and a 100 meters of empty space under your feet.
After the first few bouts of existential dread you get uses to it.
Lightly manually edit for speech rec failures.
LLM takes that and converts it into a much much denser, much more structured technical spec in markdown. This is issue 1 in the repo.
I edit this directly extensively, adding detail, removing misconceptions, adding sections with more dictated text.
LLM takes this and further densifies it, closer to an actual implementor’s spec. I carefully note what got added, removed, edited in this step.
I edit this carefully and extensively again. Some removals get added back, some additions get removed. This is the first commit and closes issue 1.
I then have an LLM write an implementation plan and basic project scaffolding. Makefiles, smoke tests, logging, test framework, DI. I edit the plan, and the plan is the second commit. An LLM uses both plus scaffolding plus my repo policy documents (global) to write a README. We’re still single threaded.
An agent files the first n top level implementation tickets. These refer to and sometimes quote any interface contracts in the spec or implementation plan. These usually map 1-to-1 to a module/directory. These sometimes have stub commits to create the module directory and commit interface definitions. Some larger modules have a README in their module directory if their purpose or structure or inputs or outputs are complicated or non-obvious.
Parallel agents write commit level issues that are the task items for each top level implementation ticket. One agent per module.
Then, spawn parallel agents with detailed global instructions, the full spec+plan, and the full readme, and the full code from the module it is working in, plus the issue text. Its output is a single commit on the feature branch and a PR that closes out one feature ticket.
This spawns an adversarial reviewer which has detailed agent instructions as well as our code style guides and repo policies. The scaffolding includes tests and linting and checks. Any failures spawn rework agents until the feature branch is up to snuff.
I review the feature branch diff and merge it, having read the review and rework comments on the PR. I frequently add comments about the implementation (“don’t do x, do y”) and add WIP: back to loop it back for more rework. I don’t see anything that isn’t green, formatted, passes linting, passes tests, and a (dumb) bot thinks is ready to land. Most of the time, it is. Sometimes, it did it the wrong way and I correct it.
The more modular the code, the easier it is to parallelize (obviously). It is ridiculously easy to build large software quickly now, provided it is sufficiently modular so the implementation doesn’t have to cross modules very much. The detailed spec and implementation plan up front helps a lot with this. It’s a lot like writing multithreaded programs really.
Any bugs discovered are tackled with strict TDD, one commit with a failing test, one bugfix commit that must be green and not touch the test.
Individual modules must conform to good contracts, meet best OOP practices, and fit into context. Too big and they get broken down further. Context size is actually a useful constraint for code structure here.
This workflow, the code standards, the styleguide, the linter settings, the fmt settings (obviously only applicable to non-Go as the Go fmt is not configurable), everything is global and stored in another repo and is vendored into each AI managed project repo. It has a fair bit of “I shouldn’t have to say this, but…” stuff (in the style of “fire codes are written in blood” - each warning sign came from experience).
My results are quite good. It’s as the saying goes, much like managing a team of junior devs. It is easy to end up with a LOT of parallel implementors, reviewers, reworkers, and tons of CI, but it’s still so much vastly cheaper than humans that it doesn’t matter.
Someone will make a LOT of money wringing out the inefficiencies from this workflow, because I think the “agent as replacement for third-world contracted junior dev” model is the one almost everyone is using right now.
Just like you just need to ask the LLM to "make it bug free".
(At this point even I'm more sure if any of this is sarcasm.)
“If you don’t hire the early-in-career [workers], and you just let your workforce continue the way that they were, where do fresh ideas get injected? How do you challenge the status quo?” he said."
-------------------
This would be an interesting approach for a bleeding edge startup that's betting big on producing something novel or going bankrupt. If you're going to bet the company on one guy and some AI, that's bold. He'd better have some serious chops and not be merely a slick salesman.
However, Cisco is an infrastructure company. Status quo is what most people want from an infrastructure company. You don't want your Cisco stuff to suddenly start doing new, innovative things that require a complete redux of everything built on top of it. This, of course, assumes that having one guy do a refactor of your entire code-base produces something "innovative" rather than a cavalcade of snafu's.
It's probably not the best time to be holding CSCO.
For anyone who wants to run the game in your browser, here it is - https://bonkheads-port-by-kapil.pages.dev/
Good memories of playing this at a friends place, way back when
It is almost an exact copy of the Mario battle game, Im surprised Nintendo didnt nuke it.
Yes it will lead to new bugs, but the older code probably contains a ton of bugs already and is probably difficult to update (original writers long gone, docs out of date etc), so this is probably the best way out of that situation, as long as proper support is given afterwards so they don't end up in the same situation a year later (this is the hard bit).
The ra-ra commentary from the CEO is somewhat off-putting, but the idea is not all bad.
Testing is still the long pull, even if you have an engineer with good expertise.
good thing they have unlimited token, now they would have 5 different model reviewing the work
Also, extensive commenting inside codebases is useful documentation, incl. these kinds of incidents. Good code documents the how, but not why. Comments shall also include the "why" where applicable.
They have to modernize the code while keeping all weird edge case behaviour. I think this can be done. And I suspect that is probably the request.
The issue is deciding what is weird edge case behavior and what are bugs. It can be a fine line.
Thanks!
(Unless I'm misunderstanding your comment, and "horrifying" was not meant to describe the code itself.)
I was only a passive observer on most of this, but the feeling was “oh look, an unchecked copy, maybe this is exploitable, four layers out towards the attack surface, oh nope they have explicit checks here that lock everything down, rinse repeat N times.”
ETA: Maybe I’ll get GLM to write up its thoughts and post them here later.
ETA: Here's GLM 5.3's opinion, also posted above. https://gist.github.com/matthewdgreen/38430b5e91780b70fdb5e9...
I haven't done C or C++ in a decade, but the fact that it's always allocating a max sized structure makes it sound like they just have a struct Thing { struct Header hdr; char[Thing::max_size] buf}, and then they are just doing `new(struct Thing) `? Then elsewhere they can just check that hdr->len < sizeof(Thing::buf) or whatever.
Like this doesn't sound some like profound memory kung-fu. It sounds like they're keeping it simple (and maybe better for fragmentation anyway) by always doing the statically defined max, and then they have some (probably generic) validation layer that checks the descriptor before copying.
Like the last time I did any C++, I was playing with statically allocated object pools where the "pointers" you got from allocating out of the pool knew the pool's size and which pool they belonged to at compile time. Their runtime representation was just whatever the smallest int was to be able to index the pool. Literally nothing else. It would no doubt look like dangerous magic if you decompiled because all of the checks were compile time. This, again, was over a decade ago, and I was actually quite inexperienced at the time. I'm sure I could go much fancier these days.
https://gist.github.com/matthewdgreen/38430b5e91780b70fdb5e9...
It got so bad, that in their last global event in Amsterdam in the keynote ( available on YouTube ) their CEO (or CTO.. cant recall) started within the first minutes by apologizing for the past security issues.
Nothing on this is likely to stop this trend.
As a side note, it’s funny that someone who clearly dislikes AI couldn’t get their idea across without using AI
The billion-dollar question is: what things might an LLM see as a bug or candidate for simplification, but supported a unique use case? https://xkcd.com/1172/ as always is prescient.
The thing about "rewrite X in two weeks" - or an entire company's worth of code in 24 months - is that it gives scarcely little time to do incremental rollouts to limit splash radius, or to identify and gather stakeholders to whom one can ask "why was this necessary" (or who can be guided to mine their emails and chat logs for conversations about the same).
Which is not to say this shouldn't be done! Let AI-native people guide the momentum with all the fervency of a hungry founder... but find project leaders who are committed to the empathy towards those who depend on the quirks of their workflows - ones who will create "curiosity loops" that try to bring in stakeholders as rapidly as possible on material questions, not just "continue, follow what you think is best" loops when the LLM brings up an open business question.
The new program won't be any easier on that front. It literally doesn't have an original writer. So the new program will certainly be maintained and updated by a LLM, at which point I wonder if the language switch mattered at all?
If they didn't have a very comprehensive test suite at the end of the exercise, then based on my experience someone is kidding themselves.
A friend had a tiny accounting system - maybe 15k loc, written in a combination of Python 2.7 / bash / LaTeX. I migrated it to Python 3.13 or whatever by hand. I did try using an LLM for that, but it did things like delete chunks of functionality so I had to check every bloody line it changed. Translating Python 2 to 3 is a fairly mechanical process, so traditional tooling worked better. Then I thought I'd explore LLMs by getting one to translate it to Rust.
The original had no tests or doco. As others have noted, the first attempt superficially looked identical to the original near immediately. You think "Wow". But in my case, it was riddled with bugs. When I asked it to fix bugs I noticed, it introduced as many bugs as it fixed.
I eventually figured out the limited context meant it kept forgetting all the lessons it learned when it fixed things, so it reintroduced old bugs and misunderstandings fixed earlier. (Like I said, this was a learning exercise.) Which is how I came to the same realisation a lot of the industry has - the solution to the context problem is you have to define the application via tests, then tell it to make the tests pass.
The tests rapidly became huge and unwieldy, so part of the exercise became DRY'ing the test code out by getting the LLM to build an integration test harness. The harness captured UI responses, files written, emails and printer output; froze time; fed in known input; and compared the outputs with results captured from the original program.
That cured the bug injection problem, but there were still bugs in all sorts of unexpected places. By unexpected I mean places I would be surprised to find them in human-written code. Human devs run the program in the same way human users do. They use the backspace key and the browser back button to navigate as they test their work. AIs don't. They will use APIs to inject test data unless you explicitly tell them to do it differently. The size of the tests exploded to be far, far larger than the code. Rather than coding I spent my time testing, telling the LLM to write tests to catch the errors I found, then writing standing prompts to try and prevent similar errors. It was meant to be a vibe-coded project, but in the end I got worried about code paths I suspected existed, so I looked at the code anyway to see if they were written the wrong way. Sometimes they would have blown up, say when the SMTP server was offline. To be fair, there was no need to look - I could have just asked the LLM. How a programmer without decades of experience would know what to ask is a mystery.
There is no doubt I got a working Rust program faster than if I had written it myself. Several times faster - but not orders of magnitude. If it was a program I had written, so I was very familiar with it, and it had excellent doco, and a high-coverage test suite, it probably would have been orders of magnitude faster. I think Bun's Rust migration might be an example of that - it's a best-case scenario. There were lots of early announcements, but it took over 6 weeks before they were confident enough to ship it in Claude Code.
But an "I migrated a program with hundreds of thousands of lines of code to a new language and ended up with production-ready code in a few days" - sorry, pics or it didn't happen.
Yes, it helped. In fact, it was essential. If you didn’t write a test for a bug, the model would reintroduce it over and over.
As an example, it produced PDF invoices and statements. PDFs are hard to compare, so I didn't bother. As a consequence, the stuff on the PDFs got moved, or was dropped, and the invoice didn't add up. I tried asking it to compare the PDF to a known good one - but it wouldn't / couldn't do that reliably. In the end the only solution was to ensure the PDFs it generated were always the same (so times on them had to be fixed), capture the generated PDF (not easy as it was often emailed or sent to a printer, not saved), and do a byte-for-byte compare, and make a test fail if it was different.
The fundamental difference from a human programmer is that an LLM does not learn. Once a human spots that they keep introducing errors in PDFs, they will remember and devise fixes. The LLM may do that too but then its context window overflows, everything it knows is lost, and it starts repeating all the mistakes it made yesterday. Tests are one of the few ways you can prevent it making the same mistake tomorrow.
The other issue is LLMs make far more mistakes than you realise. If you watch carefully, you'll see many of their bash scripts have syntax errors, code often fails to compile, and tests fail. Running a 2nd model over the work another produced will find many clear violations of style and coding standards. They hide this, in the same way humans hide it. They use tools and tests to find errors, like compilers and `bash -n`. They fix those errors without being asked. Compared to just a year ago, a modern SOTA model is far more aggressive with its integration tests, often going to a lot of effort to set up sandboxes for integration tests.
I suspect this is a major reason for today's models being far better programmers than earlier ones. But as we all know, tests have a fatal flaw - they can only prove the presence of bugs, not their absence. If you are creating code with a large number of random bugs, relying on tests to catch them all won't work. Instead you start using other tricks such as isolation boundaries, simplifying APIs so they are hard to misuse, strict information hiding - all things an experienced programmer learns over time, because they too write code with a lot of bugs. So far, there is little sign the models use the tricks well. Either they don't know them, or their stochastic nature means they can't implement them reliably.
- explore the features and behaviour of the existing code
- first write lots of E2E tests for literally everything
- human-in-the-loop to review the feature list and test cases to see if some obvious or important stuff is missing
- then let implement it and verify by tests
I used this in (way) smaller scenarios and got really good examples with minimal human interaction.
it's weird new reality, but if it works it works I guess. I'm still waiting on the first vibe-coded mass used prod app, not ware of any yet.
LOL. All 27 year-olds looking at a heap of legacy code think they can easily rewrite it better. It's never as easy as it seems on the surface.
On the other hand, models a few years from now will be smarter and more likely to have a track record of success with this task, so being one of the first to do it makes no sense even it had advantages.
So there's no urgency to rewrite; it's all risk with little to no reward for a problem that will probably be much easier to solve if you defer it a little longer. Like, first at least get to 100% AI automatic development for all new products/services, and then for all maintenance, so you know that it actually works.
Perhaps I'm merely feeling my age. But while I suspect Cisco will eventually be disappointed, my emotional state remains a profound void. There is no feeling.
I reserve empathy instead towards those at Cisco who chose rightly in life. They get to work, they do the job, they return home, they feed their family bread. Please endure the 'vision' of your corporate masters. You have already arrived and your destination is good.
When I was young I was told the book of Ecclesiastes was depressing. But with age I see it as a book of hope. Your job is pointless and that will never change regardless of your token burn rate. Enjoy your family as the sun sets.
I don’t believe it has to be this zero sum game where either you use AI and lose understanding or you don’t use AI and gain understanding. There’s a way to do both since, if used properly, AI could be the best teacher we’ve had.
Most engineers should be aware of ADRs. This is in the training data. LLMs know what to do with this.
I feel like that's a pretty standard expectation, even though we all know that the commit history can be 50% "fix bug -> fix bug try 2" and good documentation is often missing.
He had the translation use browser integration to compare the new site to the old site automatically. The result was so good that he had to change background color on the new site because he couldn't be sure that he was testing the new site; it was so close to the original he was paranoid about accidentally being on the old site and not knowing it. The whole thing was done in under 2 weeks, including coding, developer testing, tester testing, and deployment. It's been in production 4-6 weeks now and we've had no reports of problems.
You say "not worth pursuing", which could also be phrased as "too costly to pursue". With much more capacity available, it becomes possible to do more things. This is true for lots of things.
> Productivity is a measure of economic performance that compares the amount of goods and services produced (output) with the amount of inputs used to produce those goods and services.
The inputs here are money and the labor of the engineer. With this new form of automation, it takes far less of both. So much less in fact, that a previously uneconomical project is now positively cheap.
If your point is that the result isn't worth what it would have cost before, that's true, but it's the increase in productivity that changed the value proposition and allowed the project to be accomplished economically.
1. https://www.bls.gov/k12/productivity-101/content/what-is-pro...
Literally the output of the labor dropped to almost zero value?
In this case, is the change of this software from one language to another/framework to another even valuable?
Example: if I manufacture $1,000 worth of some illegal drug that will potentially turn a normal functioning adult into an addicted criminal, how shall we measure my productivity?
This instead is a concern for newer devs. There is now no argument for bringing on the cadre of workers you’d need to do all the hand-jamming, testing, and reviewing, learning how to code in the process.
The remaining 10% of devs that I don't fire. How much software do I really need?
Anyway, I know there are some flaws in my argument, it was just a thought experiment.
Dev teams should be leaner now, with maybe at most a director as their manager up to executive level.
As for ‘how much software do I really need?’ Well, what sort of company is it? Do you have internal applications to support your orgs? If you seriously need to ask ‘how much software do I need?’ Then you should just fire yourself.
Some bean counter will want to replace the principal devs with juniors since Claude can do all the work instead…
Before those 200 people would be occupied digging the tunnel.
Now you get the tunnel plus 200 people doing other work in society…
Ultimately, the economic analysis needs to use monetary units. How much faster can return on investment be realized, and at what scale? This analysis has proven elusive because people believe that the very definition, not just the execution, of a task can be delegated to an LLM (which in some trivial sense must be true because means and ends are never sharply distinguished).
Because everyone in the world doing more of the wrong things faster isn't necessarily a good thing.
On a more serious note, the productivity increase is the time which will no longer be needed to maintain two production systems instead of one.
Yes, that is something you have to be careful of. In this case, as I mentioned I believe, it was done in spare time, not full time, and it was mostly the AI that spent 2 weeks, I'd guess he probably spent a day or two, and the testers probably spent another day (however, I think this ended up coinciding with our quarterly "full site" test, so it would have happened anyway).
The engineer still spent a week there.
I imagine this is a luxury cisco will not have for validation
I've seen reports that say any organisation over 150 people needs to be paying API billing with Anthropic but here in the UK the terms seemed to be very clear when we looked that any sort of professional/commercial use is not permitted on the monthly plans like Pro and Max and you need something like API tokens or maybe some enterprise plan on their commercial terms for that kind work.
My personal observation was that when we looked into setting up those monthly subscriptions for some users at a small business in the UK we stopped at the point where they appeared to be offered only under Anthropic's Consumer Terms of Service because those terms clearly state:
"Non-commercial use only. You agree not to use our Services for any commercial or business purposes and we (and our Providers) have no liability to you for any loss of profit, loss of business, business interruption, or loss of business opportunity."
I can see this wording right now at https://www.anthropic.com/legal/consumer-terms but maybe if you're not in the UK you will see different content at that location.
In contrast the API key route appeared to be specifically offered under Anthropic's Commercial Terms of Service and those obviously don't contain the same restriction.
I would be very happy to find that there's some misunderstanding here and Anthropic do actually allow the use of their (very much cheaper) monthly subscriptions instead of API keys for small businesses like ours. But they aren't exactly known for their transparency in either their legal agreements or their billing policies and pricing plans and so until we know better we've chosen to err on the side of caution.
>Non-commercial use only. You agree not to use our Services for any commercial or business purposes and we (and our Providers) have no liability to you for any loss of profit, loss of business, business interruption, or loss of business opportunity."
that sounds similar to what I'm seeing, but in my case that restriction is limited to their "evaluation-level" services. It says in full:
>Evaluation and Additional Services. In some cases, we may permit you to evaluate our Services for a limited time or with limited functionality. Use of our Services for evaluation purposes are for your personal, non-commercial use only.
Thanks again for pointing this out. I think we're in the clear in the US.
Similar to Pro plan for business, only available up to 150 employees I believe
https://support.claude.com/en/articles/9266767-what-is-the-t...
My understanding is that it's only available under 150 employees.
It seems odd that this apparently leaves anyone working solo on a commercial project without a good option. There must be a lot of freelancers or people working at agencies who get software set up for each client or project that would be excluded by all of these options. Although API pricing appears to be so expensive at present relative to the subscription plans that it's probably still worth buying a "team" subscription with two seats and only using one of them in that kind of situation.
Copied directly from the page you linked as seen from here one minute ago.
Jeetu Patel is the CEO of Cisco, Paddy is actually Padmanabhan Krishnamurthy, a 27yo student/fonder.
We had developed the original app in 2020-2021 when Covid was around, and we were bored sitting at home not having much to do, and wanted to learn Angular. Our employer/company we worked for, wanted all of us to be to be Full-stack and both of us were backend developers. Hence we did our dating app as a side project as my co-founder was single.
Initially it was a $100 Claude Max license, and we were maxxing it at every stage, but once rewritten, $20 is good enough to maintain it.
I'm not trying to be anti-AI snide. Having it write tests, having judge agents, reviewing every PR? I'm legit curious.
We reverse engineered our Angular/Spring test cases to English. After rewriting, used the English test cases to automated tests again. Also released the new app to a niche/specific country audience at a time. We also have a form for users to log complaints. A few came and we fixed it.
The views of these business leaders betray their own lack of vision, they are barking up the wrong tree.
The ability to think differently is not the preserve of the young. Instead of only hiring a bunch of greenhorns they should identify and cultivate people who are idea generators and flexible thinkers, of any age group, starting inside their own organization.
If you, as a business leader, are not stimulating a culture of innovation and instead maintain the status quo of traditional hierarchical roles where managers become gatekeepers of ideas, and everyone is protecting their own little domains, then you are the problem. Shit organizations will quickly turn original-thinking young people into corporate drones.
