{isPending ? ( <> {" "} Generating Component... ) : generatedCode.length === 0 ? (

No Component Generated yet

) : ( Code Component


              {generatedCode.split("\n").map((item) => (
                {item}
              ))}

)} ); }; export default UserInput; ``` ## Takeaways This was a very fun project to implement as a quick weekend hack and again, it is not production level code at all, not even close. ### Main Problems I think there are a few main things that plague this project 1. A lack of bundler validation 2. Insufficent Data 3. Hacky way of rendering components One of the best ways to validate the react code is to have access to a javascript bundler which can deterministically tell you if a given chunk of code is valid or not. However, I'm not too sure how to do that at all using python. As a result, sometimes we would end up with code that was almost correct but was missing a few imports. This would have been easy to catch if we had access to a bundler Another thing that my model suffers from is that of a simple dataset. The embedding search and lookup is as good as the quality of examples that we feed into it. However, since the tasks are being manually generated by GPT-4 and we have no easy way of evolving the given examples into more complex ones without spending more time on it, this limits the ability of our model to generate complex examples. I personally found that when we gave it more complex examples, it started making references to objects it didn't know how to mock. Code like the following would be generated ```js {user.email} // user is not mocked at all, throwing an error ``` Lastly, rendering components using a dynamic import is very hacky. I personally don't think it's the best way but I was unable to think of an easy way to bundle the generated code and injecting it as html to my endpoint. ### Future Improvements If I had more time to work on this, I'd probably work on the following things 1. Fine Tuning : A large chunk of these tasks can probably rely on a well tuned gpt3.5 model. Examples could be generating small and specific task files (Eg. helper functions, simple input and label forms) 2. Using a LLM Validator : I recently learnt that `instructor` also provides a `llm_validator` class that you can validate outputs against. That might have been a better place to use the rules validation that we defined. 3. Using multiple agents : I think that this project is best served if we could look towards more specialised agents for specific tasks (Eg. One for design, one for planning, another for writing tests etc). Lindy does something similiar with calendar and emails. 4. Dataset Improvements : It would be interesting to use something like evol-instruct to come up with either more complex prompts in our database or more complex code examples that show how to build increasingly difficult schemas. I strongly believe this would have improved the quality of the generatde output. 34:T3d2d, ## Introduction RWKV is an alternative to the transformer architecture. It's open source and has it's own [paper](https://arxiv.org/abs/2305.13048) over here. I found out about it sometime back in a paper club and thought i'd

Writing Better Extraction Evals

Looking At Your Data