Night shift in Batch taugh me "ownership" The first time I truly understood “ownership” on a mainframe wasn’t during a project delivery or a successful migration. It was at midnight, during a batch window. I was still relatively new to the team, comfortable enough to navigate SDSF and read job logs, but not experienced enough to feel calm when production started turning red. It started with a single job failure. Within minutes: - The scheduler queued downstream jobs into WAIT because their prerequisite datasets weren’t produced - Multiple applications started failing SLA checkpoints I opened SDSF, jumped into the failing job’s spool, and saw the abend: <b>S0C7 => data exception</b> If you’ve worked on z/OS long enough, you know S0C7 isn’t “one problem.” It’s a category of problems. It often means a program tried to treat non-numeric bytes as numeric i.e. decimal fields containing spaces, invalid signs, unexpected characters, or misaligned layouts. My first instinct was to blame the COBOL program. The module name was old-school and the logic was untouched for years. That made it feel like “legacy code breaking again.” But my senior engineer asked one question that cut through the noise: “What changed?” That’s when I stopped trying to be fast and started trying to be correct. I worked backwards from the failing step: > Identify the input dataset(s) for the abending step (from DD statements in JCL) > Locate the upstream job that created those datasets (from scheduler flow / dataset naming conventions) > Validate dataset attributes and sample records > Compare against the copybook expectations (at least record length and key numeric fields) A quick check revealed subtle red flags, The dataset creation timestamp looked unusual for where it sat in the stream. A key numeric field occasionally contained blanks where a packed decimal value should have been. The upstream job had ended with RC=0. That’s what made it dangerous: it looked “successful” to the scheduler and to humans scanning dashboards. Digging into the run history, I found that the upstream job had been restarted earlier in the evening. Someone had used a nonstandard parameter/control-card setup (intended for a past one-off run) and the configuration had never been fully reverted. So the upstream job behaved correctly according to its parameters, but it produced output that violated downstream assumptions. The downstream COBOL program hit that unexpected input and threw S0C7. Not because it suddenly became “bad,” but because the data contract was broken. The recovery: At 2 a.m., the goal isn’t elegance. The goal is restoring the stream without introducing new risk. We discussed two options: - Patch downstream logic/JCL to tolerate blanks (high-risk, slow, hard to validate under pressure) - Regenerate the upstream dataset using standard parameters (low-risk, reversible) We chose the second. The steps were careful and explicit: - Verify dataset disposition and catalog entries to avoid deleting the wrong generation - Regenerate the dataset using the correct control cards - Rerun the downstream job and watch the scheduler drain the backlog - Document every action so the morning shift didn’t inherit guesswork Once the corrected input was produced, the abending job ran clean. Downstream dependencies recovered. The stream went green again. What we changed?(so it wouldn't repeat) The real win wasn’t getting the batch done that night. it was preventing the same class of failure. We implemented small but effective guardrails: - Restricted/controlled edits to production control cards (change process + review) - Added a lightweight pre-check step to validate numeric sanity on critical fields before running the heavy COBOL program - Documented restart procedures clearly: what is safe to restart, what must be regenerated, what parameters must be reset - Treated “RC=0” as a completion signal, not a correctness guarantee What it taught me about mainframes? That night changed how I think about production systems: - Mainframes aren’t “old.” They’re dependable systems that punish sloppy assumptions - Most midnight incidents aren’t mysterious they’re dependencies revealing hidden contract breaks - Being an engineer isn’t just writing code; it’s protecting data integrity, process integrity, and operational clarity If I had to reduce the lesson to one line, it’s this: When you see an abend, don’t ask “Who changed the code?” first. Ask “What changed in the data, parameters, restarts, or assumptions?” That question has saved me more hours than any debugging trick.
