On Oct 2, three Grok Bot scheduled routines on my account failed with only “Activity task failed” (6:50 PM, 7:00 PM and 8:05 PM ET, on different agents). The box looked healthy at those times: no host restarts, normal CPU, free disk, and a manual rerun of one routine on the same box succeeded a little later. Run records aren’t on the box, so I can’t see why they failed.
Questions:
Where can I see the real failure reason for a routine run?
Is there an automatic retry for “Activity task failed”?
Feature request: rotate the box telemetry log instead of stopping at its size cap, and add timestamps to its lines.
Feature request: swap or more RAM on the box. Chrome renderers get OOM-killed regularly with 16 GB and no swap.
Hey @TheAviv, thanks for your patience, and sorry for the slow follow-up here.
Your routines are running normally again. Since October 6, every routine run on your account has completed successfully.
Two things happened in between. The model capacity issue behind your original failures caused a couple more failed runs on October 5, then eased. Separately, on October 8 scheduled routines across Grok Bot were held back for most of the day during a period of very high load, so several of your daily routines didn’t run that day. That has been resolved. Missed runs weren’t replayed automatically, so if any of those mattered, you can run them manually.
On your earlier questions: in some cases the run history still shows the generic “Activity task failed” text instead of the underlying reason, and that’s being worked on. There’s no automatic delayed retry of a whole routine yet, so if a run fails, running it again manually is still the quickest fix. I’ve also passed on your requests for log rotation with timestamps and for swap or more memory on the computer.
If you see more failures, reply with the time and the routine name and I’ll take a look.