Open Source Software and "the Church of England's problem"

No, this isn't a religious post.

I was listening to an interview regarding marketing challenges for big brands trying to reinvent themselves. The context was large, older brands that people don't want to go away but whose products they aren't going to buy (any more).

A comparison was made with the Church of England. (It was a very English discussion.) The speakers  said:

• They wanted it to keep existing.
• They weren't going to go.
• And, they certainly weren't going to pay for it.

I immediately saw a comparison with Open Source Software:

• People want it to exist (and be maintained/updated)
• (Most) people are unlikely to participate. (Beyond using it.)
• And, they certainly weren't going to pay for it.

The challenges facing the CofE and OSS have similarities, but I think the solutions each will seek are very different.

That's all, except to say that the impact on anything is highly likely to be the same when people want something but aren't willing to pay or participate.

Even more testing nonsense with generated tests

So, following on from the last post I wrote.

I wrote that post just before I checked in the code.

As a final check before committing the code, I did some manual checks.

I found a[n obvious] scenario that didn't work.

I thought I'd created a test case for it. Turns out I hadn't. So, I added one and it failed. Then I added an extra clause to the function and made sure all tests pass.

I wonder if the Copilot generated tests cover this scenario?

It turns out they do and they don't.

There is a generated test with a name that implies it covers the scenario, but the actual contents of the test method do not do what the name implies.

It's not enough to have the test, it's also necessary that it's valid and correct.

Review generated tests more thoroughly than generated code.
If the test is wrong it leads to code that is wrong.

This also highlights the need to have other people review code (and test plans).

I know I'm not perfect and I make mistakes and omissions.
That's part of the reason for code reviews: to have an extra pair of eyes double check things. (It's just a shame this was all on a personal project.)

This experience really has reinforced my belief that test cases should be written before code.

The future of software development is testing

Or, to be more precise, the future of work for a lot of people doing "software development" depends on the need to test code.

AI is great. You can use it to write code without having to know how to program.

Except, the real value of a good software developer is knowing what code to write.

Yes, you may be able to eventually get AI to generate code that initially appears to produce what you want, but how can you determine if it's good, or efficient, or correctly covers all scenarios you need?

It's not just a case of having code that seems to work, you also (potentially among other things) need to know if the code is correct. 

AI tools can be useful to software developers by saving time generating code more quickly than the developer could have written it themselves. This is true for almost all developers. Even me.

Today, I needed to write a function to see if two numeric ranges overlapped.

It's quite a trivial thing. There are lots of examples of this type of method in existence. I could easily write such a function, but I decided to let Copilot create it for me.

The code it produced looked fine:

public bool Intersects(int start, int length)
{
    return (start >= this.Start && start < this.Start + this.Length) ||
        (start + length >= this.Start && start + length < this.Start + this.Length);
}

All it needed was some boundary checks to return false if either of the inputs was negative.

That should be good enough, I thought.

But, how can I be sure?

Does it matter if it's not?

I know this is just the type of code that: it is easy to have a mistake in; such mistakes are hard to spot; and any mistake is likely to produce annoying bugs.

So, I wanted to make sure the function does all that I need. That means writing some coded tests.

I also know that Copilot can generate tests for a piece of code.

So, I decided on a challenge: Could can I write better or a more complete set of tests?

I went first and I came up with 15 test cases. 

One of them failed.

But, it was easy to spot the issue with the code, make the change, and rerun the tests to see them all pass.

Then I let Copilot have a go. It came up with 5 test cases.

They all passed. First with the unmodified function.

Then, with the modified function, the tests all still passed.

Being able to change the logic within the function and not see any change in test results is a clear sign that there is insufficient test coverage.

Not that this is about the number of tests. This is about the quality of tests. 

When it comes to testing, quality is significantly more important than quantity.

The question should not be "Are there tests?" or "How many tests are there?"

The question you need to ask is "Do these tests give confidence that everything works correctly?"

How do you ensure that AI generated code is 100% what you want and intend? You have to be incredibly precise and consider all scenarios. 

Those scenarios make good test cases.

If you're working with code, whether it's written by a person or AI, you need to test it thoroughly. And the challenging part of thorough testing, and the bit you can't (yet?) totally outsource to AI is determining all the scenarios (test cases) to consider and account for.

What was the case that failed and how did the code need to be changed?

I needed the code to work with input of a length of zero. If this was at the end of the existing range, I need to consider this as intersecting, but the initially generated code considered it not to be.

The fix was to change the final "less than" to be a "less than or equals" and then it did what I needed.

Having now written this up, I also realize there's a potential for an overflow with internal sums that are greater than Int.Max, but as the reality of this code is that it will only be used with maximum input values of a few thousand this shouldn't be an issue...for now! ;)

This wasn't the end of the story: https://www.mrlacey.com/2024/11/even-more-testing-nonsense-with.html

Open a web page when a project can't be built

When writing the Uno book, we wanted a way to have the project fail with a useful message if a required dependency hadn't also been set up correctly.

Below is what I came up with.

<Target Name="CheckForSfChart">
    <Exec Condition="!Exists('..\..\..\syncfusion-Uno.SfChart\Src\Syncfusion.SfChart.Uno.csproj')" Command="cmd.exe /c start https:////github.com/PacktPublishing/Creating-cross-platform-applications-with-Uno/tree/main/Chapter6/readme.md" />
    <Error Condition="!Exists('..\..\..\syncfusion-Uno.SfChart\Src\Syncfusion.SfChart.Uno.csproj')" Text="The SfChart project is also needed. See https://github.com/PacktPublishing/Creating-cross-platform-applications-with-Uno/tree/main/Chapter6/readme.md" />
  </Target>

See the real thing in GitHub.

If the required other project is missing, it:

• Opens a webpage with more details.
• Outputs a message with details in the build output.

Points of note:

• Yes, msbuild will fail to compile the solution with missing project anyway. This is just additional info to help in debugging.
• Repeated slashes in the url ("https:////" instead of "https://") due to them assumed to be being escaped
• Because this call cmd.exe, it will only work on Windows.
• Not a perfect solution but better than nothing.

Documenting this now before I forget.

The lack of a visual designer is actually offensive

One of the biggest complaints and requests relates to a lack of a "visual designer" for creating UIs with XAML for WinUI3 and .NET MAUI applications.

For a long time, Microsoft had no plans to build one. Although that may be changing...

There are other, related tools that they are working on but are not the designer people are asking for.

I've even heard it said (by people who are employed by Microsoft) that "these tools are better than a designer so you don't need one".

That's what I find offensive.

The claim that MS know what people want better than they know themselves. And, the implication that people are wrong about what they say they want. More importantly (and concerning) is that I don't think MS are listening to the "real" request. 

Yes, the Hot and Live tools are great for working with existing code when the app is running, but there's also a need to help with a completely blank app (or page, or window...)

There needs to be a better way to help people get started and give them confidence that what is in the source is correct before they start debugging (or even compiling?) the code.

I have worked with people creating drag-and-drop-related tools for XAML UIs, so I know what's possible and how hard it is. That's also why I'm not trying to build one myself.

I think there's another way. It's not necessarily better at first look, but it's certainly very helpful, powerful, and able to offer more productivity than arguing about why something that isn't what people are asking for isn't what they want or need.

Logging in Testing

Once upon a time, I was working with a codebase that contained significant and detailed levels of logging. It supported recording multiple loggers and various levels of details, depending on the required "LogLevel".

While looking at how this code is (and can be) tested, I made a discovery that I hope to bring to many other projects in the future.

Taking a step back, there are a few broad questions to think about in terms of testing and logging:

• Ignoring logging, does the code do everything it is supposed to?
• Are the correct things logged when they should be?
• Does the code do everything it should when there are no loggers registered?
• Does the code do everything it should when there are one or more loggers registered?
• Are we sure that any exceptions in the code are not logged and ignored?
• How does logging code affect the code coverage of tests?

I started thinking about this only when having to deal with a "weird" bug that meant functionality appeared to be working, but the lack of a logger meant a code path failed silently as it was configured to quietly log the exception and then do nothing. :(

My solution: add a wrapper around all tests so that they're run multiple times, with different test-specific loggers attached.

• One test execution had no loggers attached. (To find code that was unexpectedly dependent upon always having at least one logger.)
• One test execution with a logger that listened at all levels but did nothing. (To verify that the presence of a logger had no impact on correct functionality and to ensure that all logging code paths were followed.)
• One test execution with a logger that would cause the test to fail if an error or exception was logged. (Because we shouldn't log errors as part of normal execution.)

This was all on code that ran hundreds of tests in a couple of seconds.

The impact of running multiple times was negligible.

The fact the total number of tests went up considerably was nice. (It also helped people reluctant to write tests feel good as every test they did write was now counted as three, so they felt more productive.)

This gave us more realistic code coverage reports, as the logging paths were now included in the coverage. (Previously, logging code wasn't--mostly--covered, making it harder to identify areas with particularly low coverage.)

Tests for logic, specifically about the loggers or reliant on specific logging behaviour, were put in classes that were excluded from the above.

Adding those different test scenarios found more issues than the one that led me to the original investigation.

I'll definitely do this again.

It's also good to have all these different test and logging combinations configured early in development. Hopefully, by the time you think development is finished, there shouldn't be any exceptions being thrown. It is hard to verify that nothing bad happens when an exception is thrown (and logged). You don't want the handling (logging) of an exception to cause another exception and end up in an infinite loop. These things can be hard to check for, but I've found that having all the different logging combinations available (without me really having to think about them) has made it possible to catch some logging issues I might otherwise have missed.

How do you name "theme" level resources?

This would have been a social media post, but it quickly became too long.

If you have some software that:

a) Has a specific color (or colors) that is (are) used for branding reasons.

b) Adjusts to "light" and "dark" themes.

I have some questions:

• What do you call that (those) color(s)? 
• Do you have different versions of those colors for when in a dark or light theme? And if so, how do you name them?

I've seen lots of variations in the ways the above are answered, but no consensus.

Here's how I wasted $249 on code signing, so you don't have to!

I don't have positive experiences with code signing.

As of today I have another tale of confusion, frustration, anger, disappointment, and expense.

I'll try and keep this to just the facts, so I don't let my emotions carry me away:

• I sign my VSIX (Visual Studio extension) and NuGet packages with a code signing certificate.
• I use a hosted Code Signing solution from DigiCert.
• Their certificates are initially limited to signing 1000 files per year. (Additional issuances are available for $249 for another 1000.)
• I got through my initial thousand MUCH faster than I was expecting.  
• I found out I'd used my allocation when signing suddenly started failing.
• It turns out that, when signing a .vsix file, the default behaviour is to also (first) sign all the files inside the package and then sign the package as a whole.
• Even if the internal files are already signed.
• Regardless of where the files in the package are from.
• So, when I thought I was signing one file, under the hood it was signing many more.
• In some cases it was signing 30-40 files each time.
• In the past I bought the certificate, installed it on my build machine and it didn't matter how many files I signed a file (or multiple packages in a file.)
• Now that everything is a subscription (and especially for expensive ones), it becomes even more important to understand how the things you're using work and you may end up being billed for using them.
• No, the documentation on this is far from extensive, clear, or useful.
• I worked out the solution based on the description here "Some artifacts are containers that contain other signable file types."
• Then I found the setting `--filelist` in the documentation at https://learn.microsoft.com/en-us/visualstudio/extensibility/dotnet-sign-cli-reference-vsix?view=vs-2022#options 
• I've now updated all my signing scripts to only sign what's absolutely necessary. As an example see this commit.

Not mentioned: dealing with DigiCert support and how they made it hard for me to pay them money to buy more "issuances". :'(

I've used future versions of Visual Studio for years

At least, that's how I've convinced myself to think about it. 

What I'm referring to is the large number of extensions I've built for Visual Studio and how, slowly, many of them become built-in features of the product itself.
It's not that my past effort has become unnecessary. It's that everyone else is catching up.

Visual Studio version 17.12 is now out. At the top of the list of new productivity features is the ability to copy just the description from the Error List, by default, rather than the full row.

I have an extension in the marketplace that does this. It's called ErrorHelper & it's been in existence for more than five years!

My extension also does more!

It allows searching for the error message/description (in the search engine of your choice) directly from Visual Studio.
Yes, VS now makes it easier to copy only the text of the description, but without my extension, you still have to go to the browser and search engine and paste it there. Or you could do it in one click.

As some error descriptions include file paths that can impact results or be something you don't want to share, there's also the option to do the search with the path removed. No need to manually remove the path from the text you've copied.

Additionally, as some error descriptions include a URL to get more information, you can directly open the URL in your default browser. No need to manually extract the URL from the other text and then paste that into your browser.

Having the Search and Open URL features has made me feel more productive every time I've used them. Yes, working with error descriptions isn't a massively complicated process but it previously felt slow and frustrating. It was a "paper-cut" that interrupted my flow and felt like more effort than it needed to be. That's why I first made the extension.

It's great that this feature has been added to the product, but it's sad that Microsoft has only gone so far.
They've said that they made this change in response to listening to feedback from developers who have told them that it was harder than necessary to search for error messages with the old functionality.
What has been implemented seems like only part of a solution to that problem. Copying the text to search for is only the first step in searching for an answer. Were they really listening that closely to the real problem if they only implemented half a solution?

They've also said that when choosing which features to add, they avoid adding things that are already possible via existing extensions. However, in their examples of this, they only mention existing extensions that were created by Microsoft staff. I guess my extensions don't count. :(

I also have many other extensions available in the Visual Studio marketplace that you could begin using to improve your productivity now rather than wait for something similar (but possibly not as good) to be added to VS itself.

Naming is important, but consistency possibly moreso

Over on the MAUI repo, I've just started a discussion relating to the naming of code, particularly in relation to what is provided by default and the impact it can have.

Yes, I know that naming is difficult and subjective.

I started the discussion less to debate the names used but more to focus on the consequences of inventing new terms or using terms other than those that the population of likely users is most likely to already be familiar with.

There may be times when new or different terminology is required or preferred, but when that's the case:

• It is essential to explain why the change has been made, or the alternative is being used.
• Need to use the new convention consistently. This includes updating existing documentation, samples, and instructions.
• There needs to be a clear reason/justification for the extra work (& short-term confusion) a change in terminology will create.
• Filtering for personal preferences and justifications based on "how we've always done it" and the potential to "try something new" must also be factored in.

This is probably also a good reminder to me that "those who show up make the difference."

I also need to remember to motivate myself over the things I care about so that the enthusiastic but under-informed squeaky wheels don't cause avoidable issues for others.

cool Vs better

Some people want to make cool things.

But, what if it's only cool for the sake of being cool?

It may be cool, because it's its new but not measurably better (and improvement) in any way.

But what if the cool new thing is different and new and so not easily comparable with what already exists.

What if cool new thing promotes responses like "it looks good, but I'm. It sure what we'd like ever [pay to] use it for".

What if you make a cool new thing but can't justify why someone should use it, beyond it being "cool".

But,

Sometimes the benefits of a cool new thing aren't obvious. Sometimes the benefits aren't seen for a long time, or until someone with very different ideas and experiences finds a new use, or combines it with something else.

So,

How do you balance spending time on cool new things that may not have any short term benefit, and incremental, measurable improvements to what already exists?

Are your programming language criticisms misdirected?

Let's say you use a programming language that you don't like or have other issues with. Is that the language's fault or yours?
Why not use a different language? or technology?

If tooling makes the language easier to work with, is not using that tooling the language's fault?

If using a language that wasn't meant to be written by hand and writing it by hand (typed directly, without other tools) the fault of the language?

If the language is significantly easier to use via a tool and the company responsible for the language don't make good tools, is that the fault of the language or the company?

If a new tool makes directly working with the programming language unnecessary, does that indicate a failure of the language or that it shouldn't have been used that way? Or that the wrong tool/language was being used?

If a language was originally designed as way of storing output from a tool and you're now writing that "output" directly, does it make sense to write it the same way as the tool did? Or would it be better to write it in a way that's easier for you to read and write?

If you are unhappy with a language or how it's used, shouldn't you look at alternatives to that language or other ways of using/writing it?

Are you blaming someone or something else for your choices or lack of action?

Does the above offend you? Or is it a cause for action?

Nick is awesome. Be more like Nick!

Nick shows appreciation to the people that help them.

Nick knows that useful, high-quality software takes time, effort, knowledge, and skill.

Nick encourages people who do things that are beneficial to others.

Nick uses some of the benefits of having a well paid job to help others financially. 

Nick says "thank you".

One of the ways Nick (and many other like them) does this is through sponsoring open source development.

Superstition, placebos, belief, and "playing with code"

Another brief post to help me think by writing.

I recently heard that "there's no merit to length in writing." They were talking about reducing the length of their book from 220K words to 180K, but the idea stands. The value isn't in the number of words, the value is in what it makes you think or do as a response to what you've read.

I've also recently recognized that when I'm thinking about a new idea, I spend a lot of time focusing on how to first express that idea and make little progress in progressing the idea. Once I first start writing and articulating that idea then I make progress in thinking about the consequences and application of that idea.

So, ...

I've recently been struck by the idea that "superstition is a placebo for belief." There's lots to unpick there and maybe that will come in time.

Beliefs are also strong and hard to change. Ironically, this is especially when people think they are very logical and intelligent.

I've recently encountered as lot of developers who are reluctant to embrace significantly new ways of doing things. 

Side note for the irony of developers being responsible for creating change (by producing or altering software) but reluctant to embrace change themselves.

They will quickly try something new (or "play with a new technology") and then quickly decide that it's not as good as what they currently use (or do).

Maybe "doing things the way I've become accustomed to doing them" ("always done them"?) is a superstition about being productive and a believe that it's the best way to do something.

Briefly "playing with something" (Yes, I dislike this term) is unlikely to provide the time or chance to learn the nuances of something dramatically different to what they've used before. It's also likely that it won't enable the opportunity to fully appreciate all the potential benefits.

Or, maybe the time to "ramp-up" on using something new means that it's never adopted because "there isn't the time" now to slow down while learning something new, even if it means being able to save time and move faster in the future. 

Or, maybe it comes down to not appreciating the possibilities of a new technology that requires thinking about usage in a fundamentally (an conceptually?) different way.

I've seen the above happen over and over again as new technologies come along. i.e. Asserting that "the new technology is slow" when using it in a way that an existing technology would be used but that is far from optimal for the new technology.

It's not the technology, it's what you do with it. Unfortunately this isn't always easy to identify without spending a lot of time using it.

And the placebo? - That there can be a perceived performance (or other) benefit by not changing when compared with the short-term impact/cost of slowing down to learn to do something new.

Yes, this applies to AI/LLMs, but to many other things too...

Insanity, LLMs, determination and the future of software development

Insanity Is Doing the Same Thing Over and Over Again and Expecting Different Results

The above quote may or may not be by Albert Einstein. It doesn't really matter.

If you've been around software development for more than about seven minutes, you've probably heard it quoted in some form or another.

It's often used as an argument against blindly repeating yourself, especially when trying to recreate a bug with vague repro steps.

This quote also points to a fundamental difference developers need to consider as the use of LLMs and "AI" become more a part of the software being developed (not just as a tool for developing the code.)

AI (& LLMs in particular) has (have) a level of non-determinism about it (them).

Repeatedly asking the same thing of an LLM shouldn't always produce the same result. 

There's a random element that you almost certainly can't control as an input.

Many software developers likely the certainty of software as (in theory) the same inputs always produce the same outputs. The level of certainty and determinism is reassuring.

Businesses like this too. Although it may not always be as clear.

Looking at the opposite situation highlights the impact on businesses.

Business: "How (or why) did this [bad thing] happen?"

Developer: "We can't say for sure."

Business: "How do we stop it from happening again?"

Developer: "We can add handling for this specific case."

Business: "But how do we stop similar, but not identical, things happening?"

Developer: "We can try and update the training for the AI, or add some heuristics that run on the results before they're used (or shown to the user), but we can't guarantee that we'll never get something unexpected."

Fun times ahead...

not writing about the cutting edge

I think and write slowly - often in combination.

Others are keen to talk/write/speculate about the latest shiny things.

I want to understand how the latest shiny things fit into larger trends and the overall/long term tends.

My first book followed a structure from a presentation first given 6 years earlier and then refined over that time (and the 18 months I spent writing it.)

My next book is in 3 parts.

Part I has its roots in a document I wrote over 20 years before.

Part 2 is based on my own checklist, that I've been building for more than 5 years.

Part 3 is based on a revelation I had and then spent 2 years actively trying to disprove with conversations with hundreds of developers. (Repeatedly asking people to tell me why I'm wrong - was a new, but ultimately enlightening and encouraging experience when no one had an answer.)

Anyway, there's another book coming.

Progress is slow, but I keep discovering new things that must be included and the result wouldn't have been as good if I'd missed them out or if trying to retrofit them into something already written.

Yes, this might be me trying to find excuses for not seeming to have made more progress.

Still, I'm excited for the book and look forward to sharing it's lessons in the (hopefully not too distant) future.

where did the SOCIAL in social media go?

I joined Twitter, and Facebook, and MySpace (although I was a bit late there) on the same day in 2007. Until then I didn't see the point.

MySpace was already pretty much dead by that point. Over the years Facebook became a way of keeping up with family and friends I knew locally. While Twitter became a place to connect with, meet, share, and learn from others with similar interests.

With many people and the friends and acquaintances, I'd made over the years, who had those interests, mostly gathered in one place it made keeping up with announcements, updates, and just general chit -chat possible. I found it reasonably easy to keep up with what was going on in the areas I was interested in. It was, of sorts, a community.

And then a bomb was set off under twitter.

With people leaving at different times and going off in different directions to different alternative apps. It became an impossibility to keep track of everyone who moved to a new platform/app. (Especially with the misinformation about sharing usernames and accounts on other platforms.)

I now have a "presence" in multiple apps (Mastadon, Blue sky, Instagram, Threads, and yes still X -- all profile links 😉) 

But none of them seem a patch on the community that existed before.

In each app there are a few accounts that I used to follow all in one place, but it seems an uncomfortable and unnecessary effort to keep opening and scrolling through each one on the chance of finding something important and/relevant. Plus each now has a terrible signal to noise ratio that is off-putting.

I've tried cross-posting across apps, but the expectations of content on each seems so different. Although I know others treat them as interchangeable--with varying results.

If I just feel the need to say something that I think/hope will get a response I'll go to Twitter/X, but then I'll feel bad because of all the people being vocal elsewhere about why they left and closed their accounts.

Yes, what Elon did to Twitter and what X has become are far from great, but I don't want to be another voice complaining.

How (and) can an online community be created that's anything like what we had in the past?

I know a bit about building communities IRL, but where and how are online communities really built?

Or should I just give up, pick one app, and start making connections again...