All software is change

The cliché says that there are only two certainties in life: death and taxes. 

There's a third: Change. 

It will happen. You can't stop it.

Change is anything new or different. This is core to what a software developer does. A new product, a new feature, and even a bug fix, will all involve changes for the person using the software.

Software developers are also well known for wanting to use the latest product, the newest framework, or the toolset currently being hyped on social media. All these new things will involve changes in what they do and how they do it.

Resistance to change is a natural response amongst most people. Or, to be more precise, unwanted change is resisted by most people. A positive change without the need for a lot of effort or disruption is normally welcomed.

When something changes, you may have to learn how to use the new version. There may be a negative consequence due to the time taken to learn the differences, but these will hopefully be outweighed by the benefits the change provides.

Very little software is written once, used, and then discarded. It's far more common for software to be developed over a period of time and for changes to be made to it once people start using it.

If you're involved in creating software, getting used to, and accepting, change is something that will benefit you in your career.

"There was no runtime pack for Microsoft.NETCore.App available for the specified RuntimeIdentifier 'iossimulator-x64'" - Keep your file and folder names alphanumeric

As part of the testing I do for the MAUI App Accelerator, I have checks that valid (but unexpected) file names won't break anything in the extension.

I probably (based on the things I find) do more testing in this area that the MAUI team.

For example, I just found that you can no longer include an  underscore, at sign, or brackets (opening or closing) in the file paths of any files in you .NET MAUI projects.

Previously these characters weren't an issue.

If you want to know more, there's now a bug about this on the MAUI repo.

However, I expect that none of that is very relevant to you.

So, why did I write this blog post?

Simple, to recommend and encourage you to avoid such issues by keeping you file and directory names simple and only using alpha-numeric characters. (and maybe a dot/period. 😉)

I do the testing that found the change in behavior, not for the sake of discovering weird, outlying bugs, but because I want to avoid anyone using my extension to encounter such a bug and then raise reports that I need to spend a lot of time investigating. Ok, so I may have just spent a lot of time investigating something that no one will encounter (to create the bug report), but at least it didn't mean that a bug was raised on my project when I had no time to investigate. Also, at least to my mind, I'm focusing the limited time I have to support my open source projects on making sure they're robust. If you appreciate such projects and want to help me develop them more, please consider joining the other people who have sponsored me previously.

Open Source Software and "the Church of England's problem"

No, this isn't a religious post.

I was listening to an interview regarding marketing challenges for big brands trying to reinvent themselves. The context was large, older brands that people don't want to go away but whose products they aren't going to buy (any more).

A comparison was made with the Church of England. (It was a very English discussion.) The speakers  said:

• They wanted it to keep existing.
• They weren't going to go.
• And, they certainly weren't going to pay for it.

I immediately saw a comparison with Open Source Software:

• People want it to exist (and be maintained/updated)
• (Most) people are unlikely to participate. (Beyond using it.)
• And, they certainly weren't going to pay for it.

The challenges facing the CofE and OSS have similarities, but I think the solutions each will seek are very different.

That's all, except to say that the impact on anything is highly likely to be the same when people want something but aren't willing to pay or participate.

Even more testing nonsense with generated tests

So, following on from the last post I wrote.

I wrote that post just before I checked in the code.

As a final check before committing the code, I did some manual checks.

I found a[n obvious] scenario that didn't work.

I thought I'd created a test case for it. Turns out I hadn't. So, I added one and it failed. Then I added an extra clause to the function and made sure all tests pass.

I wonder if the Copilot generated tests cover this scenario?

It turns out they do and they don't.

There is a generated test with a name that implies it covers the scenario, but the actual contents of the test method do not do what the name implies.

It's not enough to have the test, it's also necessary that it's valid and correct.

Review generated tests more thoroughly than generated code.
If the test is wrong it leads to code that is wrong.

This also highlights the need to have other people review code (and test plans).

I know I'm not perfect and I make mistakes and omissions.
That's part of the reason for code reviews: to have an extra pair of eyes double check things. (It's just a shame this was all on a personal project.)

This experience really has reinforced my belief that test cases should be written before code.

The future of software development is testing

Or, to be more precise, the future of work for a lot of people doing "software development" depends on the need to test code.

AI is great. You can use it to write code without having to know how to program.

Except, the real value of a good software developer is knowing what code to write.

Yes, you may be able to eventually get AI to generate code that initially appears to produce what you want, but how can you determine if it's good, or efficient, or correctly covers all scenarios you need?

It's not just a case of having code that seems to work, you also (potentially among other things) need to know if the code is correct. 

AI tools can be useful to software developers by saving time generating code more quickly than the developer could have written it themselves. This is true for almost all developers. Even me.

Today, I needed to write a function to see if two numeric ranges overlapped.

It's quite a trivial thing. There are lots of examples of this type of method in existence. I could easily write such a function, but I decided to let Copilot create it for me.

The code it produced looked fine:

public bool Intersects(int start, int length)
{
    return (start >= this.Start && start < this.Start + this.Length) ||
        (start + length >= this.Start && start + length < this.Start + this.Length);
}

All it needed was some boundary checks to return false if either of the inputs was negative.

That should be good enough, I thought.

But, how can I be sure?

Does it matter if it's not?

I know this is just the type of code that: it is easy to have a mistake in; such mistakes are hard to spot; and any mistake is likely to produce annoying bugs.

So, I wanted to make sure the function does all that I need. That means writing some coded tests.

I also know that Copilot can generate tests for a piece of code.

So, I decided on a challenge: Could can I write better or a more complete set of tests?

I went first and I came up with 15 test cases. 

One of them failed.

But, it was easy to spot the issue with the code, make the change, and rerun the tests to see them all pass.

Then I let Copilot have a go. It came up with 5 test cases.

They all passed. First with the unmodified function.

Then, with the modified function, the tests all still passed.

Being able to change the logic within the function and not see any change in test results is a clear sign that there is insufficient test coverage.

Not that this is about the number of tests. This is about the quality of tests. 

When it comes to testing, quality is significantly more important than quantity.

The question should not be "Are there tests?" or "How many tests are there?"

The question you need to ask is "Do these tests give confidence that everything works correctly?"

How do you ensure that AI generated code is 100% what you want and intend? You have to be incredibly precise and consider all scenarios. 

Those scenarios make good test cases.

If you're working with code, whether it's written by a person or AI, you need to test it thoroughly. And the challenging part of thorough testing, and the bit you can't (yet?) totally outsource to AI is determining all the scenarios (test cases) to consider and account for.

What was the case that failed and how did the code need to be changed?

I needed the code to work with input of a length of zero. If this was at the end of the existing range, I need to consider this as intersecting, but the initially generated code considered it not to be.

The fix was to change the final "less than" to be a "less than or equals" and then it did what I needed.

Having now written this up, I also realize there's a potential for an overflow with internal sums that are greater than Int.Max, but as the reality of this code is that it will only be used with maximum input values of a few thousand this shouldn't be an issue...for now! ;)

This wasn't the end of the story: https://www.mrlacey.com/2024/11/even-more-testing-nonsense-with.html