OSIsoft PI Export¶

Use historian quality codes from an OSIsoft PI (or similar) export alongside — or instead of — internal tsqc rules.

When to use this tutorial¶

You already have PI / Aspen / Wonderware / OPC UA quality integers on each row and want tsqc to:

• Respect those codes as first-class quality, and/or
• Layer flatline / range / outlier checks on top

Expected export shape¶

Rename columns if needed (time_col, tag_col, value_col) so they match what you pass to tsqc.check.

Quality modes¶

Step 1 — Define a quality map¶

Common PI System status examples (adjust to your site’s dictionary):

In YAML (preferred)¶

# pi_rules.yaml
quality_map:
  0: good
  192: bad
  193: sus
  194: bad
  195: bad

default_rules:
  - check: null
    level: bad
  - check: flatline
    window: 1h
    min_delta: 0.001
    level: sus

tag_rules:
  "GENERATOR.*":
    - check: range
      min: 0
      max: 200
      level: bad

As a Python dict¶

quality_map = {
    0: "good",
    192: "bad",
    193: "sus",
    194: "bad",
    195: "bad",
}

If both YAML and quality_map= are provided, YAML wins.

Step 2 — Combined mode (recommended default)¶

Historian codes and internal rules both contribute. Worst level wins; reasons are pipe-delimited.

import tsqc
import pandas as pd

df = pd.read_csv("pi_export.csv", parse_dates=["timestamp"])

result = tsqc.check(
    df,
    rules="pi_rules.yaml",
    external_quality_col="pi_quality",
    quality_mode="combined",
    assume_tz="America/Chicago",  # plant wall-clock zone for tz-naive exports
)

print(result.df[["timestamp", "tag_name", "value", "quality", "quality_reasons"]].head())

Example reasons:

source_data_quality: 192
flatline @ 45.2000
source_data_quality: 192|flatline @ 45.2000

Unmapped codes (for example 99) become bad with:

source_data_quality: 99

Step 3 — Exclusive mode (trust the historian)¶

Skip internal rule evaluation; only the mapped external column drives quality.

result = tsqc.check(
    df,
    external_quality_col="pi_quality",
    quality_mode="exclusive",
    quality_map={
        0: "good",
        192: "bad",
        193: "sus",
        194: "bad",
        195: "bad",
    },
    assume_tz="America/Chicago",
)

Use exclusive mode when you only need a uniform good/sus/bad timeline and HTML report over existing PI quality — without re-running flatline/range logic.

Step 4 — Escape hatch (quality_mode="none")¶

Temporarily ignore the status column without dropping it from the DataFrame:

result = tsqc.check(
    df,
    rules="pi_rules.yaml",
    external_quality_col="pi_quality",
    quality_mode="none",
    assume_tz="America/Chicago",
)

No quality_map is required in this mode.

Column name conflicts¶

If your input already has a column named quality (or quality_reasons), tsqc auto-renames the output columns to qc_quality / qc_quality_reasons and preserves the input column. Prefer renaming the historian column (e.g. pi_quality) up front for clarity.

Inspect and report¶

summary = result.summary()
print(summary.sort_values("pct_bad", ascending=False).head())

# Rows driven by historian codes
ext = result.df[result.df["quality_reasons"].str.contains("source_data_quality", na=False)]
print(ext[["tag_name", "quality", "quality_reasons"]].value_counts())

result.plot(title="PI Export QC").show()
result.export_report("pi_qc_report.html", title="PI Historian QC")

From PI Web API to DataFrame (sketch)¶

# Pseudocode — adapt to your PI Web API client
# Fetch recorded values + status, then:
df = pd.DataFrame({
    "timestamp": timestamps,       # often tz-naive local wall clock
    "tag_name": tags,
    "value": values,
    "pi_quality": status_codes,    # integers from PI
})

result = tsqc.check(
    df,
    rules="pi_rules.yaml",
    external_quality_col="pi_quality",
    quality_mode="combined",
    assume_tz="America/Chicago",
)

Always set assume_tz to the historian’s wall-clock zone for naive timestamps.

Next steps¶

• SCADA Integration — broader historian patterns
• YAML Rules From Scratch — author internal rules
• CI Gate on Data Quality — fail builds on pct_bad
• User Guide — external quality reference