Product

Over the past few weeks, Ilja and I have been working on a tool that simplifies the process of creating tutorials for your platform.

Here's our product journey so far 🧵

Product @crossmint | prev @manifoldxyz @spotify

The tool works like this: you prompt it with what you want to teach (e.g., how to create a new collection of products on Shopify) and it gives you back a step-by-step guide that includes screenshots of each step (eventually also screen recordings); all powered by AI

It’s funny how one would think that with the current AI demos out there, it should be trivial to implement something like this, but there are actually many things to take into consideration 😄

Like with any product out there, the devil is in the details, which is also what makes developing products so fun (and sometimes painful!)

Here’s how it went for us:

The task was straightforward: you prompt it with a flow to teach, and you get a step-by-step guide on how to execute that flow with screenshots of each step

1.  A human takes the screenshots of each step, and you embed these as docs that you can use in a RAG pipeline to generate the guide.
• Pros: Straightforward
• Cons: Doesn’t scale for any flow. You can’t generate variants with different inputs, paths taken, screens, etc.

2.  You use a browsing agent (like Voyager) to go through the flow and take the screenshots for you while explaining each step it takes when creating the guide
• Pros: Scales for any flow

• Cons: Browsing agents out there get stuck very often. You need to update them to now take screenshots at each step and explain what they’re doing

To get the best of both worlds, we went with a hybrid approach. A human would record themselves doing the flow once, and that flow would be used to teach a custom browsing agent how to navigate the site and where to take screenshots

Following this plan, we started by creating a Chrome Plugin that would allow you to record yourself going through a flow

Each action taken (e.g., clicks, writing text, scrolling, etc.) would be saved, including a screenshot of the page at that time and a few other metadata such as the element it clicked on and the URL it was on

Here you are already facing a lot of small details you need to take care of. What metadata is useful to describe what action you are taking to an LLM? When do you take screenshots of events that are delayed by animations such as opening a modal?

How do you handle knowing where someone clicked based on the page position? How do you signal to the user that the plugin is recording but avoid showing that part of the UX in the screenshots you take?

These might sound trivial (and some of them are) but these are things you need to solve for, and probably didn’t account for when you first thought about the problem 😄 It’s not only about discovering the problem but deciding on a good solution for them!

Following up, we would use GPT-4 to summarise all this data recorded into a single step-by-step guide of the flow

This would serve as instructions that we could use to “teach” our custom browser agent how to navigate the platform (first navigate here, then click here, then upload this file, etc.)

Next step was building the agent. We decided to follow the approach that other browser agents follow but adding the relevant step-by-step guide produced in the step before

Even for flows that were not exactly what was recorded, the agent would still be able to go through those paths since a lot of the steps were shared

Yet, there’s another set of problems you still need to solve for! Like, how do you handle authentication? Which user should be logged in to take screenshots and what state should that account be in? What text, files, etc., should it use when filling in inputs?

Is that provided in the prompt? How do you feed it to the agent? Does it still get stuck even if it has the instructions on how to go through the flow? How do you handle those edge cases? Should we experiment with different prompts? Models? Fine-tuning?

Each of these challenges can go even deeper and you need to balance between solving some of these and actually shipping something that works, even if it fails on some cases or certain cases need to be treated manually

A lot of times you won’t even find out about these challenges until you are deep into developing the product and learning how people are actually using them

It’s all part of the craft of developing products, being playful with it, understanding how it needs to work, and figuring out creative ways to get there.

Makes you think so much about the decisions and problems solved behind each product we use on a daily basis!

Anyway, if you are interested in creating personalised guides for your platform or think this tech could be used in other ways to solve a problem you have, do let me know! Happy to chat!

Anyway, if you are interested in creating personalised guides for your platform or think this tech could be used in other ways to solve a problem you have, do let me know! Happy to chat!

https://mordisco.xyz

The main question is, how will an LLM get this knowledge and obtain the screenshots of each step to generate the guide?

There are two options here: