Like I said, when you need more you'll know and you won't need to ask how to do it. Let me give you an example. I have been building an inference engine, I started with a handful of model integrations that I handheld through implementation to get structure, style etc., then I put the 800+ models I wanted into the backlog, this was a GitHub project, for each model in the backlog it listed the model's source, checkpoints, etc. I wrote a prompt that described the integration process, that it should write the model graph, compile it with profiling, run inference and compare to the reference output, and write a Rust crate that can use the compiled module with all the tokenizing or pre/postprocessing etc. in Rust, I was then able to just queue up, it would take batches of 4, and I was able to just queue up "Continue with the next batch", this of course took weeks considering the compilation, profiling and inference times. It worked fine, all the models eventually got integrated, passed reference validation, etc. However, upon inspection, I found that it had drifted from the initial requirements, there were a few different code styles, and various other issues, only the core part of a model/pipeline implemented like text-to-image not image-to-image for example. This is when I knew I needed something more. So I developed a custom harness, something using the codex sdk, this would again work from a backlog, this used a combination of prompts and enforced hard gates, meaning something that physically checks the code has followed the style, checks that there are no random helper functions, checks that it hasn't modified anything outside of the permitted scope, etc. if a hard check fails it gets sent back to the queue. I then ran that orchestrator with 8 workers, basically the equivalent of having 8 sessions but headless. I also ran an actual Codex session as a coordinator, this monitored the queue for stalls 1/?

⬆ upvote 0