sharing cool new stuff related to codegen ai workflows…
…this is a cool implementation of (some of the ideas from) the alphacodium paper:
1 Like
similar idea to the one that’s topping the HumanEval chart at the moment, but that one uses actual interpreter during getting response to query to the model, which would only work for python/JS something that’s not compiled. Imagine trying that approach with C++ and waiting for the build each time while the model iterates.
but in this case as I see he doesn’t run any interpreter and everything is simply to prepare a good quality dataset for fine-tuning or what? will the model itself “think in a flow” by steps and iterating on possible solutions internally, or it’s a “software” around it which will make it work in a specific flow?
I watched briefly but didn’t really understand, seems like it’s all about preparing a data set, well it’s a good approach to make a high quality data set I guess?
by the way I’ve read this and realized we can’t trust the HumanEval and other synthetic tests anymore, because their data got into the training sets https://arxiv.org/pdf/2311.04850.pdf so the only way to actually test models is using our own evaluation sets, to see if it works for us or not 
. I’m using a few simple questions combined with a chunk of documentation uploaded in the context, to see if a model is good or not.