Cool demo from Burke Holland using GitHub Copilot Edits and a preview Vision plugin for image recognition and UI code generation. Burke was using Claude 35 Sonnet with the plug in for the demo.

I gave this a try with OpenAI gpt-4o but couldn’t reproduce the results. While it’s a cool concept to generate boilerplate in another language (Flutter), but I think the more powerful feature is the ability to convert design images into code.

Not something new, especially with a well-built design system, but once mature this technology will certainly make things easier.

Two honorable mentions were Naming & Branding and UI Design. None of the LLM or generative models are great at this. Figma did attempt to create flow charts so it’s possible a different tool set may need to be field tested.

Meta AI Lama 3

Design Report Card

Technical Report Card

CoPilot

Design Report Card

Technical Report Card

Gemini

Design Report Card

Technical Report Card

ChatGPT 3.5

Design Report Card

Technical Report Card

Claude 3 Sonnet

Design Report Card

Technical Report Card

I ran through a human powered excercise in both product and technical design. Here are my brief notes.

Findings

Much like my prior experiment with first principles reasoning, LLMs aren’t sophisticated enough to provide granular enough product and architectural design to be immediately actionable by a development team.

While I anticipate that eventually LLM tools could be useful enough to assist in product and engineering tasks, it still requires a human to design something that is ubiquitous, intuitive, scalable, and secure.

Bottom line: don’t spend a lot of time bouncing ideas off the major models, and certainly don’t believe the hype that these tools can magically produce GOOD software with a few AI prompts.

Grok

CoPilot

Gemini

ChatGPT 3.5

Claude 3 Sonnet

Prompt

What is it?

The idea of first principals is a powerful problem-solving approach that involves breaking down complex issues into its most basic and fundamental parts.

I’ll use this thought exercise as a first step to challenge assumptions and to unlock a creative solution to our problem. This experiment should demonstrate how capable current LLM is at human style reasoning and problem solving.

Findings

While I think LLMs can be used to generate ideas or compile a list of potential features or technologies, the models aren’t sophisticated enough to understand the context of the problem and how to maximize user value.

This leaves a lot of gaps that humans inevitably need to fill. That’s to be expected. Anyone who has built an app from scratch will tell you that gathering feedback early and often is the best way to create the most useful app.

Novel concepts might be inspired by humans interacting with LLMs, but it’s doubtful it could come up with anything compelling on its own. Could you use LLM as a brainstorming tool, absolutely!

My overall experience generating both a newsletter logo and a title / wordmark was satisfactory. While AI accelerated the ideation phase, the tools struggled to produce simplified graphics.

Results

• Ideation and Design
• Image Generation

I needed a title or banner image which Substack calls a “Wordmark”. The models struggled with this concept on a few levels. I used the following request prompt:

Give me ideas for a wordmark for a Substack newsletter called 
"Project Maestro" about using AI to design and develop a 
consumer application.

NOTE: I had to abandon the prompt above and used the following:

Describe the design aesthetics of a stylized image version of 
newsletter title "Project Maestro". The topic of the newsletter 
is using AI to design and develop a consumer application.

Strategy

Stack multiple AI technologies in three phases:

1. Theme Creation - Brainstorm themes using large language models
2. Prompt Optimization - Use LLM to build efficient text-to-image prompts
3. Image Generation - Create images using generative models

I wanted to generate a square logo that I could use to identify the project and I needed something that both captured the intention of the project but would also be iconic when scaled down.

I started the following request prompt:

Give me ideas for a brand logo for a Substack newsletter about using 
AI to design and develop a consumer application.

Strategy

Stack multiple AI technologies in three phases:

1. Theme Creation - Brainstorm themes using large language models
2. Prompt Optimization - Use LLM to build efficient text-to-image prompts
3. Image Generation - Create images using generative models

I settled on Substack for a couple of reasons, popularity, ease of use and discoverability. I’m already active on the platform and I figured it was a great place to start this journey.

For Maestro’s prologue I decided to put the prospective AI tools to the test by creating the branding for a Substack newsletter.

Overview

• The Tech
• Base Prompts

Should AI have unfettered access to money? The Lightning Network does exactly that, and provides a mechanism for AI to create a wallet, earn money and then spend money on other services in order to complete a task.

To paraphrase Lyn Alden’s August newsletter: “An AI agent is purposely set up by the creator to have some funds to spend as needed. AI could start with no money, and be given a task or just “decide” to do a task. It generates a bitcoin/lightning wallet and either convinces its creator to give it funds or otherwise finds a way to earn funds online With access to money, it pays for things it needs.”

The potential is incredible and alarming. Regardless of our reservations it will happen, and it has happened. In a recent video Kody Low (Head of Developer and Product Support at Fedi) asks GPT-4 write code for him, and pays GPT-4 on a per-question basis using Lightning Network. The fact that GPT-4 can write code is not the novelty, it’s that an AI now has access to a payments system and can transfer store of value assets (i.e. money).

Elizabeth Stark, CEO of Lightning Labs when describing L402 as a “network is capable of doing micropayments that are much smaller than what Visa and Mastercard can do. With Lightning, you can send payments worth a fraction of a penny. This opens up new use-cases that aren’t possible with credit cards, for example, such as machine-to-machine payments, the streaming of micro-payments, or the usage of micro-payments as a spam-prevention technique”

All this makes me wonder about the long term impact on on crypto markets as the volume of transactions inevitable experiences hockey stick growth due to growing AI driven microtransactions.

Gerggely Orosz of Pragmatic Engineer dove into Ayhan Fuat Çelik’s analysis of and speculated that coding assistants like Copilot, Cody, ChatGPT are partly to blame. He speculates that engineers are reaching for AI chatbots for code questions, and it’s hurting Stack Overflow’s traffic.

Something I’m very excited about is the ability to run AI, LLM, etc. on local hardware. It addresses a couple of disadvantages to cloud based AI: security & privacy, lack of customization, and high cost of renting GPU cycles.

Stability AI just announced StableCode which is intended to do for code generation what Stable Diffusion did for image creation. Not only can you run it locally, but the model supports a longer context window (16,000 tokens) which allows for more detail instructions.

The present challenge is with hardware as AI requires a very beefy GPU and Nvidia has a lock on most of the market with its proprietary CUDA parallel computing programming model. It is possible to hack your way into running AI on AMD hardware, but it isn’t for the faint of heart.

I’ve been experimenting with Stable Diffusion locally w/ my AMD GPU (RX 580 8GB) for a bit. It’s not the fastest, but b/c I’m since not renting Google Colab I can leave it running long batches without a cost concern. If StableCode is similar it could address some of the privacy concerns about co-mingling code and data inside a public LLM.

Everyone is talking about ChatGPT. If by everyone you mean software developers, students and want to be tech bros. It might seem like a door-to-door scam, but truthfully, it’s just the tip of the iceberg. Or as Tech Unicorn declared: “Have you heard about our lord and savior Chat GPT?”

I’m reminded, every time I pick up the phone or go to a restaurant, that we’re still experiencing widespread labor shortages. Not to sound like a broken record but Boomers retiring, not enough Gen Z to pick up the slack. Birth rates falling… blah, blah. The US is better off than many countries because of Millennials.

It could be gloom and doom, except we forget about the productivity gains of the past 30 years. Largely brought to us by the personal computer and later the Internet, these gains allowed companies to produce more with fewer humans. To be ungracious to the older generations, it allows us to not need to do everything by hand.