RBC Payment Processor — Documentation

Upload an RBC Corporate Creditor Bill Payment flat file, review every payment, then submit them to Sonar via Hasura — with full audit trail and multiple layers of protection against duplicates and accidental double-submission.

The five-step workflow

1. Upload — drag an RBC .txt/.dat file onto the drop zone or click to browse. The backend computes the file’s SHA-256, parses every G-record, and persists everything to the database in a ready state. If this exact file has already been successfully processed, you’ll see a red duplicate banner with details of the previous run. Only an admin can force a re-upload.
2. Precheck — summary cards show total payments, total amount, anomaly count, and the CCIN/file date from the A-record. File-level issues (missing header/trailer) and skipped G-records (anomalies the regex rejected) are listed. Download buttons produce a human-readable review.txt and a machine-readable review.jsonl (the exact payloads that will be POSTed to Hasura).
3. Review — the full payment table. Every row is pre-selected; uncheck anything you want to exclude. Rows with issues (missing account, zero amount, missing trace) are highlighted amber. The Dry Run toggle defaults ON (green) and must be explicitly unticked to go live (turns red). Selected count and total update live.
4. Process — clicking "Submit selected payments" opens a confirmation modal. Live submissions require you to type RUN to enable the confirm button. Once confirmed, the backend streams Server-Sent Events as each payment is submitted: a live terminal console shows every request, running counters update, and a progress bar fills in. Dry runs use 100 ms pacing; live runs use 200 ms.
5. Report — final summary cards, tabbed results (All / Succeeded / Failed / Duplicate / Skipped), and download buttons for the decorated audit report (audit.txt) and the full JSON export. If any payments failed, a "Retry failed" button appears — admin-only, re-streams ONLY the failed rows.

Safety rails

• File-hash duplicate block. SHA-256 of every uploaded file is stored. Re-uploading a previously-processed file returns HTTP 409 with the prior run’s metadata. Admin can bypass with ?force=true.
• Singleton lock. At most one run may be in running state at any time. A second accountant trying to start a run while one is in progress gets a clear 409 error.
• Trace-level de-dupe (belt). Before each Hasura POST, the backend checks whether the payment’s trace number has already been successfully submitted in a different run. If it has, the payment is marked duplicate and skipped — no Hasura call is made.
• Trace warnings at precheck (suspenders). Same check runs informationally at upload time so the accountant sees it before submitting.
• Dry-run default. Every new run starts with Dry Run ON. Live submission requires an explicit tick plus typing RUN in the confirmation modal.
• Interrupted-run recovery. If the backend crashes mid-stream, any run stuck in running state is marked interrupted at startup. It will never auto-resume — an admin must explicitly resume (re-streams only still-pending rows) or abort it.
• No retries, no loops, no polling. No file watchers, no cron jobs, no automatic retries of failed rows. Every action requires a human click. The legacy SAK2 10-minute polling loop is gone for good.
• Single-pass iteration. Each run invocation iterates its payment list exactly once. Retry-failed creates a new iteration with only the failed rows (never touches success rows).

Where files live

What	Where
Application root	/home/wavedirect/sak3/
Backend (FastAPI)	/home/wavedirect/sak3/backend/app/
Payment routes	backend/app/routes/payments.py
Database	backend/data/sak3.db (SQLite)
Sonar debug log	backend/logs/sonar_debug.log — tail -f-able, legacy format preserved
Frontend (single SPA)	frontend/index.html — no build step
systemd service	sak3.service → uvicorn on 127.0.0.1:8000
nginx reverse proxy	Port 443 with Let’s Encrypt SSL
Timestamped backups	Siblings of the original (*.bak.YYYYMMDD-HHMMSS)

Admin actions

• Force re-upload — bypass the SHA-256 duplicate block. Available only to admins, surfaced as a button on the duplicate banner.
• Abort a run — cancel a running or interrupted run. Currently backend-only: POST /api/payments/runs/{id}/abort.
• Resume an interrupted run — re-streams only still-pending rows. Backend-only: POST /api/payments/runs/{id}/resume.
• Retry failed payments — button on the Report step. Resets failed rows to pending and re-streams only those.
• Clear debug log — button on the API Log tab. Truncates sonar_debug.log on disk.
• Delete a run — admin-only via DELETE /api/payments/runs/{id}.

Hasura / Sonar integration

Payments are submitted as query-string parameters to Hasura’s REST endpoint: https://hasura.wavedirect.net/api/rest/submitcustomerpayment. Hasura wraps the call as a GraphQL mutation into Sonar. Authentication is via the x-hasura-admin-secret header, read from the backend .env and never exposed to the browser. Success is detected by the presence of a createPayment key in the response.

The payload sent to Hasura for each payment has four fields: account_id (10-digit zero-padded), amount (integer cents), reference (trace number from the G-record), payment_datetime (ISO timestamp at upload time).

The submitter is encapsulated in submit_to_sonar() in payments.py so it can be replaced with a direct GraphQL mutation later without touching the workflow code.

File format (RBC G-record parser)

140-byte fixed-width records. The parser uses the exact regex from the legacy SAK2 main2.py:

^G\d{9}(\S{1,30})\s+(\d{4,10})\s+(\d{13,15})\s+(\d{8})(\d{8})\d{6}(.{1,35})

Field order: G + 9-digit sequence + trace (up to 30 chars) + account number + amount + pay date + (filler) + payer name. Any G-record the regex doesn’t match is reported as a "skipped line" in the precheck — these are real anomalies the regex was hand-tuned to reject, and they must NOT be auto-fixed or included. Parity between this parser and the legacy script is verified by backend/tests/test_rbc_parser_parity.py.

Endpoints reference

Method	Path	Purpose
POST	/api/payments/upload	Upload + parse → ready run
GET	/api/payments/runs	List all runs
GET	/api/payments/runs/{id}	Run detail + payments
GET	/api/payments/runs/{id}/review.txt	Human-readable review (pre-submit)
GET	/api/payments/runs/{id}/review.jsonl	Exact Hasura payloads (JSON Lines)
GET	/api/payments/runs/{id}/audit.txt	Final decorated audit report
GET	/api/payments/runs/{id}/export	Full JSON export
POST	/api/payments/runs/{id}/start	Begin processing (SSE stream)
POST	/api/payments/runs/{id}/abort	Admin: cancel a run
POST	/api/payments/runs/{id}/resume	Admin: resume interrupted
POST	/api/payments/runs/{id}/retry-failed	Admin: re-run failures only
DELETE	/api/payments/runs/{id}	Admin: delete a run
GET	/api/payments/errors	All failed payments across runs
GET	/api/payments/sonar-log?tail=N	Last N lines of sonar_debug.log
DELETE	/api/payments/sonar-log	Admin: clear the debug log

Troubleshooting

• 502 Bad Gateway — the sak3 service crashed on startup. Check sudo journalctl -u sak3 -n 50 for the traceback, fix the code, then sudo systemctl restart sak3.
• Upload returns 409 file_already_processed — this file’s SHA-256 matches a previously-completed run. Contact an admin to force re-upload, or (more likely) use a different file.
• "Another run is already in progress" — the singleton lock is engaged. Check the Run History tab for a run in running state. If the backend crashed, the run will be marked interrupted at startup instead — an admin must resume or abort it.
• Payments marked duplicate — their trace number has already been successfully submitted in a previous run. This is the safety belt preventing double-submission. If you believe it’s wrong, investigate the trace in the Run History.
• Can’t find my old file in "Review Complete" — the new SAK3.0 system does not move files anywhere. The uploaded file content is stored directly in the database on each run row (file_content column). Use the JSON export to retrieve it.
• Live tail of the debug log — tail -f /home/wavedirect/sak3/backend/logs/sonar_debug.log from any shell on this server. Same format as the legacy SAK2 debug log.

Version & credits

SAK3.0 — WaveDirect internal tools platform. FastAPI + SQLite backend, single-file HTML/JS/CSS frontend, JWT auth, systemd-managed uvicorn process, nginx reverse proxy with Let’s Encrypt. Replaces the SAK2 directory-watcher + regex scripts.

Read ~/sak3_context.md for the broader project overview, and ~/rbc_payment_processor_ui_summary.md for the UI spec this tool was built against.