JEREMY PANASUK | THE CLOSER
Operator
Consulting artifacts · 2017 floor

THE
PLAYBOOK

Real-world documentation standards.
Real-world performance.
Customer coaching that leaves
the team running it.

Environment Summaries, Upgrade Roadmap v3,
a Phase 2 architecture review, and the PC 10 deck. Hostnames and customer names stay off this site.

Read the standard Open the floor
Three surfaces. One closer.

What got left on the table

The parachute was never a hero hour. It was a written operating path: what is running, what is on fire, what to change, how to prove the upgrade, and who on the customer team can run it after you leave. Same diagnostic loop the RTX lab uses now — measure, isolate, document, hand off the boundary.

01

Documentation standards

Every burning platform got the same three artifacts before anyone touched config: an Environment Summary, an Upgrade Roadmap, and a Phase / architecture review. The v10 deck taught the room how to choose an upgrade type and how to test it.

02

Real-world performance

Baseline. Goal. Bottleneck. Eliminate. Repeat. Thread statistics before folklore. Do not partition until a single partition is understood. Cache in RAM. The filesystem is often the real Grid/HA product.

03

Customer coaching

Knowledge transfer is a deliverable. Skill-up is named in the architecture review. The whole stack sits in one room with GCS. The team is praised in writing. Loyalty is the runbook they keep.

The artifacts

What a real consulting document looks like

Not a slide with a logo. Owner, date, environment evidence, what was already changed, what is still open, and the recommendation the customer can execute without you in the building.

Artifact · Environment Summary

PowerCenter Environment Recommendations

Author and owner on the cover. Overview of how mixed internal / consultant / offshore developers actually use the platform. Then a numbered review of what was measured and changed — not a wish list.

  • Utilization first: CPU, RAM, disk wait, database wait. Low utilization is a finding. 100% CPU is a finding.
  • Config actually applied: ulimit, heap, concurrent jobs, sorter/lookup cache, stale session-log cleanup.
  • Incidents closed with GCS: P1 start-delay, bulk-load sorter RAM, dispatch that would not start.
  • Grid/HA and TCO path: Gold/Platinum SLA tiers, QA twin of production, repository off the shared DB host.
Artifact · Upgrade Roadmap v3

Upgrade Roadmap

Current-state performance review, then the product recommendation: PowerCenter with Grid and HA. Internal domain/gateway/node design. External clustering and the shared filesystem that makes or breaks Session on Grid.

  • Graphs over a month, not a hallway anecdote. Round-robin dispatch vs adaptive. RAM sitting idle while cache spills to disk.
  • Filesystem matrix: UNC, DFS, CIFS/NAS, NFS v3, GFS2, Veritas CFS, GPFS. POSIX + I/O fencing or you do not have HA.
  • Database HA: Oracle RAC / Data Guard, SQL Server mirroring. Source and target availability called out separately from the domain.
  • Admin of the grid: logging, restarting services, installing new components. The customer can operate it.
Artifact · Phase review

Architecture Review — next increment

Phase 2 is not “more of Phase 1.” Name the product features the next workload actually requires, the topology that will hang, the filesystem that will plateau partitioning, and the skill-up the team still needs.

  • Example: advanced XML / real-time web-service consumers need Model Repository + Data Integration Service, cores, and Developer-tool skill.
  • 30+ repositories and 30+ integration services is TCP/IP overhead. Upgrades become a migration factory.
  • Outstanding deliverables are demos the customer can see: Proactive Monitoring, Advanced XML on their schema.
  • Write down that the team is good. Openness is the coaching surface.
Deck · PC 10 Upgrade Overview

How the room chooses an upgrade

Jeremy Panasuk / professional services. Why v10, PAM, three upgrade types, six classes of test, Data Validation Option to make same-data honest.

  • In-place — short cycle, no extra schemas, outage for restore. Cannot rename domain or nodes.
  • Parallel — snapshot the repo, install fresh, change names, no domain-install outage. Migrate users/groups.
  • Clone — snapshot the domain, fastest like-for-like go-live, still an outage for restore.
  • DVO: 50% source-to-target, 80% regression, 90% upgrade testing. Audit trail. No extra server.
Artifact · TDM 9.5.2 · financial services

PHI masking development guide

Test Data Management 9.5.2 HF1 in Shared Analytics on AIX. Persistent masking and subset so non-production is not production. Cover prepared by the TDM practice. Jeremy’s 3,000-hour V1 is the platform this baseline sits on. Naming standards as a deliverable — the same motion as an Environment Summary cover.

  • Discovery first: sensitive-field profiles before anyone guesses which columns are PHI. Then subset. Then mask.
  • Reusable generic rules named by type + technique (Num_Key, Str_Sub_Firstname). Exception rules get a qualifier, not a new dialect per application.
  • Wrapper policies POL_PHI / POL_PII. Assigning policies one table per plan so PowerCenter masking workflows can run concurrently.
  • LDAP groups for authorization. Standard seed values and dictionaries so substitution is repeatable across applications. Hostnames and production inventory stay off this copy.

Sample Documentation · xyz teaching templates

Five files the practice used to teach the next consultant

Google Drive folder Sample Documentation. Same author as the named floors. Cover says xyz. The operating path is identical. Public copy still does not publish hostnames, UNC aliases, or raw capacity tables.

xyz- PowerCenter Environment Summary.doc · Jun 16, 2017

Teaching copy of an auto-finance Environment Summary

Author and owner: Jeremy Panasuk. Customer name on the original stays off this site. This is the file with the performance cheat sheet inside.

  • ALFA→ODS at 100% CPU. Cache RAM raised so large transformations stop spilling — 20–30% off job runtime.
  • PWX CDC / batch POC vs the bundled bulk loader. GFS2 has no flock — keep-persistence-in-DB is on, and it costs performance.
  • Custom Integration Service property so RHEL reports free memory correctly. Model Office + production domains to best practice.
  • When dispatch will not start: entire technology stack plus vendor support in the same room. Not a ticket queue.
xyz- PowerCenter Environment Summary (2).doc · Apr 14, 2017

Teaching copy of an insurer Environment Summary

Author and owner: Jeremy Panasuk. Low utilization is a finding. Config that was actually applied is listed by number.

  • ulimit 64k, DB2 repo settings, concurrent jobs 500, heap, stale NFS session logs cleaned so I/O stops lagging.
  • P1 start-delay: GCS escalation. DB2 semaphore on the client tool. Resolved.
  • Gold / Platinum parent GRID: dedicated nodes for the high SLA, then spill to shared nodes. LOB-split Integration and Repository services with a backup RS on another node.
  • QA domain identical to production so the next incident does not take prod down. Tidal for schedule. Proactive Monitoring reviewed. 4- or 6-core VMs to right-size TCO.
xyz_Upgrade_Roadmap_v3.doc · Jun 27–30, 2017

Teaching copy of an asset-manager Upgrade Roadmap

Author: Jeremy Panasuk. Two-node Windows 2008 VMs, 4 cores / 16 GB each. PowerCenter 9.1.0 plus Data Quality 9.6.1. A month of graphs, not a hallway anecdote.

  • Round-robin dispatch made the two nodes uneven. 80–90% CPU at peak. RAM only half used — cache still spilling to disk.
  • Disk wait 30–90 ms (Microsoft’s published max is 10–12). SQL wait past 200 ms with RAM sitting idle. UNC as the file share is not HA.
  • HA is resilience, failover, and recovery. Internal domain/gateway/nodes vs external clustering and the shared filesystem.
  • Filesystem matrix: UNC, DFS, CIFS/NAS, NFS v3, Veritas CFS, GFS2, GPFS. Session on Grid vs Workflow on Grid. Adaptive dispatch. Admin of the grid: logging, restart, install.
xyz_2_Summary - Copy.doc · Oct 24–30, 2017

Teaching copy of a P&C insurer Phase 2 Architecture Review

Author: Jeremy Panasuk. Phase 2 is not more of Phase 1. Guidewire Advanced XML and real-time web-service consumers.

  • Model Repository + Data Integration Service, extra cores, Developer-tool skill-up named in the write-up.
  • 30+ repositories and 30+ integration services is TCP/IP overhead. Domains hang. Upgrades become a migration factory.
  • NFS on NAS is the vendor’s least recommended HA filesystem: extra hops, weak cache coherency, lock-manager bounce when a host dies. Partitioning plateaus at one inode per logical node. Expect 35–100% slower than a proper CFS with host-side striping. Supported NAS: NetApp WAFL or EMC — not a RAID export called a share.
  • Veritas CFS or GPFS: I/O fencing, concurrent reads/writes, no keep-persistence-in-DB flag (that flag can cost ~20%). Outstanding demos: Proactive Monitoring / Operational Insight, Advanced XML on their Guidewire schema. Team praised for openness.
PC_V10_Upgrade_Overview.pptx · 28 slides · Jeremy Panasuk

The coaching deck that stops “just run the installer”

Why v10, PAM first, three upgrade types, six classes of test, Data Validation Option so same-data is honest.

  • Editions: Standard (batch ETL, partitioning), Advanced (Metadata Manager / Business Glossary, real-time engine), Premium (DVO, Proactive Monitoring, advanced transformation).
  • In-place: short cycle, no extra schemas, users/groups stay, outage for restore, cannot rename domain or nodes.
  • Parallel: snapshot the repo, install fresh, can rename, no domain-install outage, migrate users/groups, small settings-drift risk.
  • Clone: snapshot the domain, fastest like-for-like, still an outage, extra HTTPS caveats. Six-class test: operability (smoke), same-data, security, performance, 3rd-party (schedulers, SVN, encrypt, file transfer, customer monitors), HA vs DR. DVO: 50% source-to-target, 80% regression, 90% upgrade testing.

These five files are the kit. Named customer copies are the work. xyz copies are how the next consultant was taught to write the same document. Hostnames, UNC aliases, and capacity tables stay off this site.

Performance methodology

The cheat sheet that stayed with the team

Left inside the auto-finance Environment Summary. Same loop as the GPU routing bench: a baseline that is allowed to be slow, a goal, one change between runs, notes that make the next person dangerous.

1Establish baseline
2Set a goal
3Find bottleneck
4Eliminate it
5Repeat

Performance dos

  • Cache as much as possible in memory. Spilling lookup / aggregator / sorter / joiner index to disk is the silent 20–30% tax.
  • Spread I/O across as many devices as possible. The shared filesystem is a product decision, not a mount.
  • Round-robin a single input file until you prove that file is the bottleneck.
  • Key-range partition to align the target with database partitioning. Pass-through to add CPU. Then stop.
  • Thread statistics first. Transformation ~100% busy → repartition point. Reader/writer ~100% busy → maybe the thread, maybe the system.

Performance don’ts

  • Do not add multiple partitions until you have tested and tuned a single partition. Written three times on purpose.
  • Do not add partitions or repartition points if the session is already source- or target-constrained.
  • Do not add partitions if CPUs are already maxed (%idle under ~5%).
  • Do not optimize a sub-two-minute session. Why are you here?
  • Do not change three knobs between runs. You will not know what helped.

Isolation tests when thread statistics lie

  • Target: write to a flat file instead of the database. Filter FALSE immediately before the target. Compare to a stand-alone bulk loader. Faster = target-constrained.
  • Source: filter FALSE after the Source Qualifier. Read from a flat file instead of the database. FTP the remote file outside PowerCenter. Same speed = source-constrained.
  • Simple before complex: analyze table / runstats before optimizer hints. Source-only mapping to take transformation and target out of the experiment.
  • OS evidence: vmstat and sar. Low %idle = CPU. Non-zero si/so or page-outs = memory. Sustained %wio = disk. Notes: session time, rows/sec, config delta, CPU and disk.
Upgrade operating path

Pick the type. Then test like go-live depends on it.

The v10 overview is the coaching deck. The roadmap is the customer-specific evidence. Together they stop an account from “just running the installer.”

When Type Why
Very short cycle, many installs already certified in a sandbox In-place No extra schemas. Users/groups stay. Outage for domain restore. Cannot change domain or node names.
Longer cycle, tight outage, want new names Parallel Fresh server + repo snapshot. No domain-install outage. Migrate users/groups. Small settings-drift risk.
Pre-prod go-live, minimize settings drift Clone Identical domain snapshot. Fast like-for-like. Still an outage. Cannot rename. Extra HTTPS caveats.

Operability

Smoke: installer, domain start, repo wizard, services, connectivity, a workflow runs, client login.

Same data

Full, partial, or automated regression. The longest bar on the plan. DVO exists so you test all rows, not a sample.

Security

External: injection, SSL, DB grants, open ports. Internal: admin passwords, folder / connection / tool rights.

Performance

Client, integration service, server, network. Equal or better than the legacy environment — with numbers.

3rd-party

Schedulers, SVN, deploy frameworks, encryption, file transfer, customer monitors. The item people skip.

HA / DR

HA = single-instance failover, measured in uptime. DR = data-center loss, measured in recovery time and stagnation.

Customer coaching

Leave the team dangerous

“Customer satisfaction is worthless; customer loyalty is priceless.”
The operating line · Gitomer / Carnegie / Greene

How coaching showed up in the documents

  • Named skill-up. P&C insurer: Developer tool and Advanced XML for Guidewire. Credit union: install, services, naming standards, go-live tests, plus TDM reusable POL_PHI rules so the next developer does not invent a masking name per application.
  • Walked loads. Auto-finance knowledge transfer on ALFA→ODS, ODS→flat file, ODS→warehouse — not a PDF dump after the fact.
  • One room. When production would not dispatch, the recommendation was the entire technology stack plus vendor support in the same room.
  • Praise in the deliverable. P&C insurer: the team was knowledgeable, open, and could explain why the current architecture existed. That sentence is coaching.
  • Runbook stays. Auto-finance tuning docs adopted by the follow-on integrator. Food-manufacturer Type-2 / security / upgrade path locked. QA twin of production so the next incident does not take prod down.
The receipts in these files

Five floors. Same playbook.

Anonymized “xyz” copies were the teaching templates. The named copies are the work. Public site carries the operating findings, not hostnames.

P1
Insurer · Apr 2017

QA/prod domains brought to best practice (ulimit 64k, DB2 repo settings, concurrent jobs 500, heap, stale NFS logs). P1 workflow-start delay: GCS escalation, DB2 semaphore on the client tool. Grid/HA TCO path and a QA twin of production.

20–30%
Auto finance · Jun 2017

ALFA→ODS at 100% CPU. Cache RAM raised so large transformations stop spilling to disk — 20–30% runtime drop. PWX CDC POC vs bulk load. GFS2 flock gap. Performance cheat sheet and load walkthroughs left with the team.

v9→v10
Asset manager · Jun 2017

2-node 9.1.0 grid measured for a month: 80–90% CPU, RAM under-used, disk wait 30–90 ms, SQL wait past 200 ms. UNC is not an HA filesystem. Upgrade types + six-class test plan. Adaptive dispatch, homogeneous nodes, Linux CFS.

Phase 2
P&C insurer · Oct 2017

Guidewire advanced XML and real-time consumers. 30+ repos / integration services as domain risk. NFS vs Veritas/GPFS. Skill-up named. Outstanding demos: Operational Insight and Advanced XML on their schema. Team praised in the write-up.

PHI
Credit union · TDM 9.5.2 baseline

TDM Workbench + PowerCenter for PHI masking and subset in Shared Analytics. LDAP security. POL_PHI wrappers. One-table assigning plans so masking can run concurrently. 3,000-hour V1 with the vendor resource office plus customer HR: PowerCenter, Metadata Manager, Data Quality, PWX CDC for DB2 on mainframe, Web Services, go-live tests. Cover: TDM practice expert. Platform: Jeremy’s V1.

Source artifacts: insurer PowerCenter Environment Summary (Apr 2017, Jeremy Panasuk); auto-finance PowerCenter Environment Summary with performance cheat sheet (Jun 2017, Jeremy Panasuk); asset-manager Upgrade Roadmap v3 (Jun 2017, Jeremy Panasuk); P&C insurer Phase 2 Architecture Review (Oct 2017, Jeremy Panasuk); PowerCenter 10.0.0 Upgrade Overview (28 slides, Jeremy Panasuk); credit-union TDM-PHI masking Development Guide, TDM 9.5.2 HF1. Customer names, internal hostnames, credentials, and production table inventory are not published.

The closer they send in alone

SEND THE PLAYBOOK

Same operating path on Fortune-100 Grid/HA and on a private RTX 4070: read the evidence, isolate the bottleneck, write the standard, coach the team, leave.

Open the operator floor Back to the campaign
715-970-0204
jpanasuk@gmail.com
GitHub · Hugging Face · github.io/the-closer/playbook