I subsequently built a workaround (it still eats tokens, but it prevent code contamination). Here was my prompt to pull all the information on what i did and the scripts and sequence that i have Cursor run before it saves or commits files:
look up the project’s UTF-8 validation path from the indexes and the scripts/hooks that actually run before save and commit.
I have the hook layout. Next I’ll read the validator, pre-commit/pre-push, and the encoding notebook so the process matches what actually runs.
There is no automatic convert-on-save. Cursor edit hooks that used to rewrite files after save are disabled. Encoding is prevented in the editor and PowerShell, then checked (fail-closed) at commit, push, and CI. Conversion is a separate, explicit --fix / normalizer step.
Incident log: Scripts/Diagnostic Bundle/Documentation/FILE_ENCODING_VIOLATIONS_NOTEBOOK.md. Policy: .cursor/rules/utf8-only-source.mdc.
1. Before save (prevention, not a gate)
These stop new files from landing as UTF-16. They do not scan or rewrite the tree.
| Mechanism |
Path |
What it does |
| Workspace editor defaults |
.vscode/settings.json |
files.encoding = utf8, files.autoGuessEncoding = false for PowerShell, Python, Markdown, JSON |
| EditorConfig |
.editorconfig |
charset = utf-8 |
| PowerShell session |
Scripts/Robust Automated Processes/PowerShell/Initialize-Utf8Session.ps1 |
Console UTF-8; Out-File / Set-Content / Add-Content default to UTF-8, not UTF-16 |
| Profile install (once per clone) |
Install-RepoGitHooks.ps1 → Install-RepoPowerShellUtf8Defaults.ps1 |
Dot-sources the session script in CurrentUserAllHosts |
| Fleet Windows hosts |
Services/AWS/Scripts/PowerShell/Deploy-Utf8PowerShellProfiles.ps1 |
Same defaults on EC2 |
Cursor save-time hook is off. .cursor/hooks.json is "hooks": {}. Enforce-RepoTextUtf8.ps1 still exists under .cursor/hooks/ for manual use, but it is not run on afterFileEdit / postToolUse (Norton was interrupting every pwsh spawn).
If a buffer was already opened as UTF-16, the editor keeps that encoding until you use Save with Encoding → UTF-8.
2. Canonical checker (the one gate that matters)
Scripts/Robust Automated Processes/Python/validate_repo_sources_utf8.py
Detects UTF-16 (BOM or NUL-between-ASCII) and other non-UTF-8. Default scope is all tracked text-like files (.py, .ps1, .mq5, .mqh, .md, .json, .ini, .set, etc.).
| Flag |
When used |
--git-staged |
Pre-commit and ROBUST_GIT_COMMIT |
--full-repo (or no mode flag) |
Pre-push and CI |
--fix |
Rewrite those files as UTF-8 without BOM (manual; review the diff) |
--paths REL … |
One-off cleanup |
--policy-sources-only |
Legacy narrow scan (Scripts/Services/Experts/Include) |
Check (no write):
py -3 “Scripts\Robust Automated Processes\Python\validate_repo_sources_utf8.py” --root . --git-staged
py -3 “Scripts\Robust Automated Processes\Python\validate_repo_sources_utf8.py” --root . --full-repo
Fix, then re-check without --fix (must exit 0):
py -3 “Scripts\Robust Automated Processes\Python\validate_repo_sources_utf8.py” --root . --fix --paths “relative\path\file.py”
3. Before commit (hard gate — check only, no auto-fix)
Install once per clone:
pwsh -NoLogo -File “Scripts\Robust Automated Processes\PowerShell\Install-RepoGitHooks.ps1”
That sets core.hooksPath=.githooks.
| Layer |
Script |
UTF-8 action |
| Pre-commit |
.githooks/pre-commit → .githooks/pre-commit.ps1 |
Runs validate_repo_sources_utf8.py --git-staged. Fails the commit on UTF-16 / invalid UTF-8. Does not convert. |
| Robust wrappers |
ROBUST_GIT_ADD.ps1, ROBUST_GIT_COMMIT.ps1 |
Same staged UTF-8 check before git add / git commit |
| EA-only helper |
ROBUST_FILE_ENCODING_VALIDATOR.ps1 + _EncodingValidation.ps1 |
Test-FileEncoding / Convert-FileToUTF8 for .mq5 / .mqh |
Pre-commit also runs PowerShell parse (Assert-StagedPowerShellParses.ps1) and the new-script catalog gate. Those are not encoding converters.
git commit --no-verify skips this. Policy is not to use it.
4. Before push and in CI (full tree)
| Layer |
Script |
UTF-8 action |
| Pre-push |
.githooks/pre-push → .githooks/pre-push.ps1 |
validate_repo_sources_utf8.py --full-repo |
| CI |
.github/workflows/regression-manifest-validate.yml step “UTF-8 policy” |
Same Python validator on the full checkout |
5. How files actually get fixed
Conversion is operator/agent-run, not hook-run. Hooks only refuse the commit.
| Tool |
Use |
validate_repo_sources_utf8.py --fix |
Preferred for known paths or staged files |
Scripts/Tools/normalize_repo_text_utf8.py |
Bulk convert (UTF-16 LE/BE, UTF-8 BOM strip). --dry-run first |
Scripts/PowerShell/Run-NormalizeRepoTextUtf8.ps1 |
PowerShell wrapper around that Python normalizer |
Services/AWS/Scripts/PowerShell/_ConvertOneFileToUtf8NoBom.ps1 |
Single-file rewrite |
_EncodingValidation.ps1 → Convert-FileToUTF8 |
Used by the EA encoding validator |
Typical repair:
- Convert with
--fix or the normalizer
- Re-run the validator without
--fix (exit 0)
- Stage and commit through
ROBUST_GIT_* / hooks
Flow in one line
Editor/PS defaults try to write UTF-8 → optional --fix / normalizer if a file is already wrong → pre-commit scans staged text and blocks → pre-push and CI scan the whole tracked tree and block. Nothing in the live Cursor hook path rewrites files on save.
