I've spent the last 5/6 years in a startup focusing on chatbot development. Before that, a decade and some change doing enterprise Java development. The advances seen in LLM/AI space over the last 18-24 months are honestly phenomenal!
I myself am continually shocked when I dust off hello-worlds from 2023 for specific libraries and then note how far that platform has come since then.
I recall a specific discussion with a client around 2 years back where we spoke about introducing GPT into the existing bot ecosystem but ended up agreeing that the costs were prohibitive. That's not the case today though as prices have come down drastically.
Sure, building a costing model for a solution that utilizes a cloud-hosted LLM (OpenAI for e.g.) and having to thumb-suck token volumes to ensure you don't undercharge is still super frustrating. But at least the model costs are getting cheaper.
I love software development and I loath it when people think you can simply generate an entire point-of-sale or whatever system overnight using an LLM cos they saw a thingie on tikkop about it! Software development done right is an art form. No, I'm not "gatekeeping" the field; I'm stating that there's a lot more to developing software than just writing the code; or in this case, having the LLM write it for you!
Things like source & version control, configuration management, modularization, dependency management are not things that the vibe-coder knows about. And let's not even start talking about building and deployment of said code!
The average non-technical prompt hacker doesn't understand all of these little nuances and therein lies the rub!
It sounds to me that you're relatively new to the development world and you're trying to argue the case from a technical manner.
You're going to lose that battle 10 times out of 10.
Your boss doesn't actually care about what you're telling him about unit testing, conformance to the current tech stack etc. All he's hearing are "excuses". From where he's sitting, it seems to him that "innovation" isn't happening and things are taking too long to happen.
You need to perhaps take a step back and look at your team's development processes a bit more objectively.
Are there a lot of processes for processes sake? What's your turn-around time like on releasing features, bug fixes etc?
You need to find a way to work with him instead of against him. You need to show him that you believe in his vision of using AI and find ways to integrate into your workflow in a manner that is
1) relevant to what you're doing
2) sustainable for the team to keep using
3) valuable to the organization
We must understand that we're probably close to the top of the Generative AI hype cycle; everyone and his dog is doing something with GenAI. Your boss is likely feeling pressure from his peer group; market competitors to get onto the train.
So perhaps what you should do is have a chat and explain to him that in order for the generated code to be usable, it needs to be compatible with the current platform. Therefore, you need him to explain the feature to you and then you'll develop a prompt that takes into consideration your current environment, rules etc and create the artifact that is usable.
LLMs too are a case of garbage-in-garbage-out. Craft yourself a nice handy prompt that specifies things like versions of libraries/dependencies/frameworks, variable naming styles, logging standards, modularization, configuration management.
Then use that prompt to generate his feature, or better yet, arm him with that base prompt for his future endeavours.
He's not going to stop poking around simply cos you told him the code is kak. So, instead of letting him wander, steer him in the direction that works for you.
Step 1 should be to make sure he's using Claude for code generation
Anyway, hope this great wall of text helps you reframe your mind and approach towards this.