Nadav pfp
Nadav
@nadav
say you had to measure your organization's effective usage of AI as a KPI. how would you do it? particularly for engineers
14 replies
13 recasts
119 reactions
Brenner pfp
Brenner
@brenner.eth
If you can get Claude Code to commit each time it does the work, then % of code written by Claude vs manually
0 reply
0 recast
0 reaction
shazow pfp
shazow
@shazow.eth
if AI empowered people to take up more ambitious projects. i worry that sometimes the opposite is true (easier to pump out higher quantity of superficial things)
0 reply
0 recast
2 reactions
christopher pfp
christopher
@christopher
Value to cost, maybe merged PRs divided by billable token usage.
0 reply
0 recast
1 reaction
avi pfp
avi
@avichalp
size of context windows
0 reply
0 recast
0 reaction
Garrett pfp
Garrett
@garrett
usage on their respective accounts for the models they’re using
0 reply
0 recast
0 reaction
ted (not lasso) pfp
ted (not lasso)
@ted
this is a really good question and i’m gonna think about it for non-engineers
0 reply
0 recast
0 reaction
RoboCopsGoneMad pfp
RoboCopsGoneMad
@robocopsgonemad
By how many humans you can replace. Just skip to the end
0 reply
0 recast
0 reaction
keit45 pfp
keit45
@keit45
Nice. I will give you. 20 $degen
0 reply
0 recast
0 reaction
Fly pfp
Fly
@flys
Great like
0 reply
0 recast
0 reaction
Butterfly pfp
Butterfly
@sdq
That's a tough one, but probably I would focus on something like requests/user, it feels reasonable to use as a KPI because the more the user likes your AI, the more requests he will make
0 reply
0 recast
0 reaction
Steve Pederzani pfp
Steve Pederzani
@pederzani.eth
By how many employees using it have had overall metric improvements versus number of employees using it that are performing about the same or worse. Buy-in isn’t just about buying.
0 reply
0 recast
0 reaction
Thomas Walsh pfp
Thomas Walsh
@thomaswalsh
I have no clue how is what does KPI even mean?
0 reply
0 recast
0 reaction
zayd pfp
zayd
@zayd
API usage to commits/PRs seems reasonable but that can still be gamed. API usage feels like part of the equation though
0 reply
0 recast
0 reaction
Jon Commstark pfp
Jon Commstark
@commstark
Pretty high 80%
0 reply
0 recast
0 reaction