I'm no expert and I'm biased because of my current work but fine-tuning (especially LoRA) is one of the most powerful and reliable ways to get LLMs to fit your custom needs.
Better than prompt engineering and even embeddings.

That's all I can say.

Have you considered arguments presented here :

The open web is dead, long live the open web. ⌐◨-◨

Building /warpcast and /farcaster

— gabrielayuso.com

Have you considered arguments presented here :

https://warpcast.com/abhishek1point0/0x2c431a

Taking Design Contracts | Talking about Tools for Thought | Founder metamind.network, Medium/Product Designer, Writing at www.abhishek1point0.com, 📍Bangalore

Agree with this:

"Given the upfront time and computational resources required for fine-tuning, we’d recommend starting with this base first (along with the supplemental techniques mentioned) and seeing how far you get with it!"

It's good to start simple.

However, you can reduce costs and improve quality and performance if you move to a fine-tuned smaller model vs a larger model with few-shot.

It depends on your needs and resources.