Maestro: Design Solution

Findings

Much like my prior experiment with first principles reasoning, LLMs aren’t sophisticated enough to provide granular enough product and architectural design to be immediately actionable by a development team.

While I anticipate that eventually LLM tools could be useful enough to assist in product and engineering tasks, it still requires a human to design something that is ubiquitous, intuitive, scalable, and secure.

Bottom line: don’t spend a lot of time bouncing ideas off the major models, and certainly don’t believe the hype that these tools can magically produce GOOD software with a few AI prompts.

My Experience

I tried to use various LLMs for four main aspects of solution design:

Naming & Branding
Product Design (requirements)
Technical Design (requirements)
UI Design

Overall, the experience with naming was basic. While a couple of responses were novel it wasn’t significant enough to warrant a full model by model breakdown. For UI design LLMs were essentially useless, and I’d rather focus on the two aspects where I think LLMs can help.

In the realm of product design, I found most responses effective enough to write high level feature stories. These would need to be broken down to a more granular level and depending on the development team might need to be more technically prescriptive.

For technical design I found LLM responses to provide a wide variety of technologies. I focused my queries on the Microsoft tech stack that I am most familiar with, and the results were consistent.

However, compared to a skilled and experienced product and engineering team none of the LLMs are a threat. While these tools could be a great brainstorming tool, they lack the specificity to provide anything that would result in a marketable software application.

Findings

My Experience

Test Results