Code AI can't write? - I have 6 examples

I was recently at a software development conference where there was a heavy emphasis on getting developers to use AI to write more code, rather than them typing it.

As part of the Q&A, I asked if the panel thought there were any circumstances where AI shouldn't be used to write code or where it wasn't appropriate.

They said that, as AI was being used to write the frameworks that developers are building on top of, they couldn't see a reason not to have AI write the code.

Being nervous of absolutes, and knowing how varied developers and the software they build are, I quickly came up with six scenarios where developers can't (or shouldn't) "just" point an agent at their codebase and tell it what to do:

1. Code is written and exists in an air-gapped environment (so on machines without access to the internet) for security or privacy reasons. This means Claude, ChatGPT, Copilot, or similar can't look at it. [Yes, I acknowledge that as an alternative, you may be able to use an agent with a local model, but you won't have access to the latest tools and may need very high-spec (so expensive) machines to run those models on. Time spent managing and maintaining such tools must also be considered.]
2. Code that you don't want the AI [company] to see. Similar to the above, but where you don't want to, or can't legally, let another company see your code or the data it uses. [Again, a local agent/model may be an option here.]
3. Code that is very hard to test in an automated way. If the code needs complex integrations with other systems that can't easily be mocked or abstracted, especially if it requires specific hardware to interact with, the AI agent is unlikely to be able to do and test everything for you. It may think it's finished, but you wouldn't accept untested code from a person, so why accept it from an AI agent?
4. Code that uses very obscure APIs. In my experience, when the AI hasn't been trained on the APIs you want (and need) to use (because there just aren't very many--or any--examples on the internet for it to learn from), then the AI can easily start hallucinating the methods it wishes existed, or it might even be reasonable to assume should exist. When this happens, it takes a person to correct what the AI has done/tried, and it can be easier to have a person do it in the first place, rather than have to fix the invalid approaches.
5. Code that requires a small and specific change. If the change is small and you know what needs changing and where, it can be quicker and easier to do it by hand rather than trying to describe what the AI should do. (And not have AI make other changes at the same time.)
6. Code you write for fun. Ignoring the challenge of finding an employer to pay you for the things you find fun; if, as a developer, you find writing code fun, then there's no reason you have to stop. It may not be how you earn a living, but that doesn't mean you can't write code by hand at other times.

I'm sure there are more. What have I missed?

Is "QA" just about manual testing?

I've seen recent viral posts that imply it is.

I think differently.

QA is (or, at least should be) a lot more than manually testing software.

What are you (or they) assuring the quality of?

- Software, yes! (Obviously?)

- All customer engagement? (From marketing to ongoing support?)

- Internal processes and interactions? (Even hiring, and onboarding?)

- Every part of the business?

What are the consequences of people who are excused from having to think about, or the responsibilities related to, ensuring that all they do is of an acceptable (& presumably high) standard?

VS [Code] Marketplace error: "An item with the same key has already been added."

 If you're very unlucky and when trying to upload a new extension to the VSCode marketplace, you may get this error: "An item with the same key has already been added."

If you use the website, you may see this (not particularly helpful) error message.

Of course, you really should be using an automated, repeatable process/script to manage your submissions, and if you do that, you'll still get the same result. (I only tried the website after all other attempts to work out what the duplicate key might be.)

Error: An item with the same key has already been added.

Error: Process completed with exit code 1.

Anyway, for me, it turned out that we had content types for both "json" and ".json" defined. (With and without the leading dot.)
Some systems don't mind this, while it turns out the marketplace does.

I hope this helps someone else find a solution to their problems faster than I did. (10 emails, back and forth with support over 11 days!)

Human questions to ask in code reviews

Automated (AI) powered code reviews are becoming increasingly popular and powerful. I use them myself to catch things I miss.

But there are questions I ask as a human reviewer that I've yet to see any AI-powered tooling help with.

 Questions like:

• Does this fix the cause or the symptoms?
• Are there related changes that should be addressed while we're in this part of the codebase?
• Is this just fixing a problem or actively making things better for the user? (They aren't always the same.)
• Do we have related docs or samples that need correcting/updating or adding to?

Is second-order enshittification a thing?

 
Enshittifcation, as a term popularized by Cory Doctrow, has for many become a catch-all term for things generally getting worse. I've found it useful (unsurprising if you know me) to think about it in two categories.
There's the popular definition of how very large, dominant platforms change [to get worse] over time as business/organizational priorities change.

I think there's a second-order version of this that follows as a response.

I'd sum it up like this:

"If [big company] doesn't have to focus on quality, we can cut corners there too."

What happens when "everyone" takes this approach is that things get continually worse. Not all at once and not always in the immediately obvious ways. More in the million paper cuts way.

From a software perspective, it's the increasing number of small bugs that no one seems to care about fixing. This teaches people to not bother reporting them. After all, "what's the point? They're not going to fix it even if I [you] do report it."

With fewer known or reported bugs, the company thinks everything is fine, or worse still, thinks skipping on quality is an approach that works.

And a self-reinforcing cycle continues.

I prefer to think of it another way.

Software exists to help people and to make their lives "better". This could be in one or in many ways.

When people are constantly being interrupted, frustrated, and disappointed by the software they use, it's very hard to argue that it is making them better, happier, or more productive.

This is part of the reason I think that QA, customer service, and fixing bugs are among the most important things that those in software development can focus on.

With AI making copying software easier, it's focusing on how it relates to people that can be the real differentiator. 

Yes, AI may mean you can vibe code a copy of something that exists, but does it include support for all the non-obvious edge cases? Do you even know all the features that exist in what's being copied? How (and?) will you handle any bug reports?

Yes, building software can be fun for you, but high-quality software that works reliably benefits many more people and, in a small way at least, makes the world a better place.

And this can also lead to mediocrity

A small company tries to compete with a large one. They try to do everything because that's what the large company (or their software) does. They try hard, but as they're stretching themselves so thin, everything suffers. "Never mind", they say, "at least we're trying our best." But, for what? Who benefits? The company's staff exhaust themselves producing a mediocre product, and it's no better than the alternatives.
Trying to compete with a large company and basing goals on the perceived [low] standards of another company doesn't produce a great product, happy customers, or happy staff.

_This was written many months before posting--to avoid anyone thinking I'm responding to a single person, company or incident_