maurelian
@maurelian.eth
I want to be able to have a conversation with an LLM that has full context on a large piece of text, that is too large to copy/paste into ChatGPT. For example I might want to give it the text of a book, or the contents of a repo, and (I think the term is) fine-tune it. Does anyone know how I'd go about that?
9 replies
1 recast
5 reactions
Petra ⊙
@0xpetra
Run into the same question. Basically what you’re looking for are embeddings. tl;dr: What you do is traslate the text into LLM language and store it. Later you can retrieve relevant info, and feed it to the model to use it as input.
1 reply
0 recast
0 reaction
Peter Ferguson
@peterferguson.eth
https://github.com/fraserxu/book-gpt
0 reply
0 recast
1 reaction
notdevin
@notdevin.eth
https://brain-search.vercel.app This is basically my cms being embedded into gpt which lets you then talk to the site about the corresponding projects
0 reply
0 recast
1 reaction
Sanch
@sanchitram.eth
Upload it to some GitHub repo as a pdf, and using the pdf reader plugin, if you pay for GPT plus. If you have time, https://github.com/openai/openai-cookbook/blob/297c53430cad2d05ba763ab9dca64309cb5091e9/examples/Question_answering_using_embeddings.ipynb
0 reply
0 recast
0 reaction
moreReese
@morereese
Not sure what the size limit is with chatpdf but could be worth a try https://www.chatpdf.com
0 reply
0 recast
0 reaction
Cameron Armstrong
@cameron
@moar LLMs w big inputs
1 reply
0 recast
0 reaction
Sam (crazy candle person) ✦
@samantha
Maybe this isn’t the exact answer you’re looking for, but legal software exists like this to parse large bodies of info and summarize it for lawyers. Just heard about it from some friends, not sure how good it is!
0 reply
0 recast
0 reaction
clun.eth
@clun.eth
I believe Google will give you the ability to do this pretty soon, at least inside their ecosystem. If you have some Google docs with your stuff then you can have them train a network specifically on that. They’ve done similar things already: MedPalm trained on medical stuff, SecPalm trained on infosec stuff.
1 reply
0 recast
0 reaction
Jeadie
@jeadie
Try marqo.ai, here's a demo https://www.marqo.ai/blog/from-iron-manual-to-ironman-augmenting-gpt-with-marqo-for-fast-editable-memory-to-enable-context-aware-question-answering
0 reply
0 recast
1 reaction
