Create RESEARCH.MD

[3th-Enjay]

found models that claim to have been trained in Cairo coding, will test and attach results and finalize research on what model is better and why

🚀 Pull Request Description

[Provide a clear and concise description of the changes in this pull request.]

🔗 Linked Issues

Fixes #202

📸 Screenshots

[Include relevant screenshots or screen recordings demonstrating the changes]

✅ PR Checklist

• I have tested these changes locally
• Relevant documentation is updated
• My code follows the repository's coding guidelines
• I have added/updated tests that prove my fix/feature
• No breaking changes introduced
• Code is well-commented and readable

[vercel]

@3th-Enjay is attempting to deploy a commit to the poulavbhowmick03's projects Team on Vercel.

A member of the Team first needs to authorize it.

[PoulavBhowmick03]

The details seem good. Awaiting @tosoham 's review

[tosoham]

Looks Good but I want some more details for data collection for Fine-tuning. Also exploration of Deepseek r1 model and Grok 3 is requested.

[Birdmannn]

Well, for Starknet Agent, it has good cairo data, but has problems understanding what you say to it. That said, 90 percent of the commands you send it won’t go through. Seems it was trained only with keywords, but it tries it best.

Llama, I believe it’s the one by Zuckerberg (if it is), GPT-4 is better than it in logic and correctness, and gpt-4 doesn’t even know cairo.

Github copilot is the AI that is usually meant to be switched off when coding because it feels it knows something. It has very poor code and error analysis, so even if it finally knows the cairo code, you’ll have an extra overhead of work to perform on your LLM other than just training it with cairo data.
copilot is better off building front end typescript code and generating comments

Deepseek is the LLM that comes this close… at least it’s the first to start a contract with #[starknet::contract] and understands the english you feed it. It’s more of a copycat (except the deepthink — which is a talkative good for word projects and articles). It has better logic and correctness than gpt painfully, and much better cairo syntax (with a few syntax errors that can be corrected in time), so the workload of training this might be lower.

Heard about Claude. Heard it’s the best LLM for devs, but since github has access to it, I feel it’s overrated. This is one i haven’t used before due to its overhead in creating an account. I feel it’s better off typing frontend code.

I haven’t tried Grok. Elon sure took his time.

i’ve leaned to LLMs during desperate times of cairo logic, and there’s one to research, and there’s one to have hands on experience, desperate with frustration, and giving out info due to the outcomes.

starknet agent is fed with accurate cairo code. But it’s useless if it can’t even understand my problems. I respect the devs, but from that desperate standpoint, the chances i lean towards it is 1%. It tells you “You’re better of just googling and searching through the available docs, if not, just give up, man”

Llama, so far it’s Zuckerberg, I can never use it so far gpt exists. AI can take over the world, but Llama won’t be among them.

GPT has good understanding and breaking down skills, but poor cairo syntax. In short, it still uses cairo 1 syntax. If it doesn’t use it anymore, then it was changed recently. Anything less than GPT4 is crossed out the list. But GPT4 can’t be hosted, the API must be read…. please don’t think of using Hugging face to host gpt2 or 3, you’d have a greater workload.

Deepseek is close to it, and it’s open sourced I guess, so it can be hosted no? It’s like a copy of gpt4, but with good cairo skills.

I might just go and use Grok in desperate times, and see how it’d play out. Heard it’s the best LLM so far, but that’s where it ends for me.

Sorry for the long comment 😅 @PoulavBhowmick03 @tosoham

[tosoham]

sure liked your input @Birdmannn