Sat down and did a crude experiment: try to construct a NSF Project Summary, using a pair of rules files, one for the NSF requirements, and the other for what makes for good academic writing. I had a main directory containing the section file (as markdown). A reference directory contained text version of two papers that should be used to construct the summary. I used Agent mode and gave the references as context. I used claude-3.7-sonnet (w/o thinking)
I added the writing rule (probably should be just writing) later on and I’m not sure it picked up on it, so I specifically @'ed the file during my final revision, which improved the draft a bit.
The draft wasn’t bad but it’s not that good, compared to other accepted NSF Project Summaries that I’ve seen. I have a couple of avenues at this junction:
a) Draw on good how-to-write-a-grant-proposal references to construct another rule and try again
b) Put on my academic critique hat together with a how-to guide to heavily critique the output
I get the sense I’ll have to use a thinking mode to actually get things over the hump. But so far this is less painful than single stream chatbot modes that Claude and other’s largely present. If I return back to this experiment I’ll update with more.
EDIT - Yeah, I had to use Claude thinking mode; it was helpful but started hallucinating/misinterpreting. Not sure where exactly to go next but this was useful to consider so far. I may need more rules and let the agent figure out where and when to apply them.