Dataset report
{{ source_stem }}{{ source_suffix }}
- {{ report.source.format | upper }} {% if report.source.format in ('json', 'jsonl') %}
- Dataset
{{ ds.id }} - {{ ds.summary.row_count | count }} records
- {{ ds.summary.column_count | count }} fields {% else %}
- {{ ds.summary.row_count | count }} rows
- {{ ds.summary.column_count | count }} columns {% endif %}
- {{ report.source.size_bytes | size }} {% if report.source.encoding %}
- {{ report.source.encoding }} {% endif %}{% if report.source.csv %}
- Delimiter
{{ report.source.csv.delimiter }}{% endif %}
01 / Overview
Dataset overview
Size, completeness and the main signal to check first.
- Rows
- {{ ds.summary.row_count | count }}{% if ds.summary.empty_row_count %}{{ ds.summary.empty_row_count | count }} empty rows{% else %}No empty rows{% endif %}
- Columns
- {{ ds.summary.column_count | count }}{{ ds.summary.cell_count | count }} cells analyzed
- Missing cells
- {{ ds.summary.missing_percent }}%{{ ds.summary.missing_count | count }} cells
- Duplicate rows
- {% if ds.summary.duplicate_row_status == 'disabled' %}–Not checked{% elif ds.summary.duplicate_row_status == 'limited' %}≥ {{ ds.summary.duplicate_row_count | count }}At least, beyond the first occurrence{% else %}{{ ds.summary.duplicate_row_count | count }}Beyond the first occurrence{% endif %}
{{ label }} {{ n | count }} in {{ cols | count }} {{ 'column' if cols == 1 else 'columns' }}
{{ kpi_columns(items, label) }}{% else %}Affected columns
Affected columns {{ qv.top | length | count }} of {{ qv.columns | count }}
- {% for c in qv.top %}{% set n = c.errors + c.issues %}
- {{ c.name or '(unnamed)' }}{{ c.percent_text or '–' }}{{ c.labels | join(' · ') }} {% endfor %}
Bar = largest finding share in the column
Uses a log scale to keep small and large rates readable together.
No Error and no Issue.
{{ qv.columns | count }} {{ 'column was' if qv.columns == 1 else 'columns were' }} checked and the checks that ran found nothing to report.
{% endif %}Errors and Issues
What the checks found, by kind
{% if sg.shown or dataset_notes %}- {% for s in sg.shown %}{% set pid = px ~ 'pop-s' ~ loop.index %}{% set nc = s.columns | length %}
- 8 %} class="ov-x"{% endif %}>
{{ s.title }} {{ s.total | count }} in {{ nc | count }} {{ 'column' if nc == 1 else 'columns' }}
{% if s.note %}{{ s.note }}
{% endif %}- {% for item in s.columns %}
- {{ item.name or '(unnamed)' }}{{ item.count | count }}{% if item.spellings %}{% for sp in item.spellings %}
{{ sp.value }}×{{ sp.count | count }}{% if not loop.last %} · {% endif %}{% endfor %}{% endif %} {% endfor %}
{% endfor %}{% for item in dataset_notes %} - {{ item.name or '(unnamed)' }}{{ item.count | count }}{% if item.spellings %}{% for sp in item.spellings %}
- 8 %} class="ov-x"{% endif %}>{{ item.title }}{{ item.observation.count | count }}Dataset · {{ item.noun }} {% endfor %}
+ 0 more, listed in Observations
{% endif %}{% else %}No Error and no Issue was detected by the checks that ran.
{% endif %}{% if sg.incomplete_columns %}{{ sg.incomplete_columns | count }} of {{ ds.summary.column_count | count }} columns were not completely checked.
{% endif %}Column types
Classification: family, then its types
- {% for name, count in ds.summary.type_family_counts.items() %}
{{ family_labels[name] }}{{ count }}
- {% for key, kind_count in ds.summary.type_counts.items() if key.split('/')[0] == name %}
- {{ type_labels[key] }} {{ kind_count }} {% endfor %}
{% endfor %}
Technical type: what the values are made of
- {% for name, count in ds.summary.inferred_type_counts.items() %}
- {{ name | replace('_',' ') | title }} {{ count }} {% endfor %}
Mixed: fewer than {{ (report.config.types.minimum_confidence * 100) | round(1) }}% of present values agree on one type.
{% endif %}02 / Summary
Column summary({{ ds.summary.column_count }})
Type, overview and findings per column. Errors: values a rule proves invalid. Issues: ambiguities and deviations to inspect. Missing values are counted apart. Remarks are under Observations.
| {{ row.position }}{% if column_links.get((row.dataset_id, row.field_id)) %}{{ row.name or '(unnamed)' }} {% else %}{{ row.name or '(unnamed)' }}{% endif %}{% if row.path != row.name %}{{ row.path }}{% endif %} | {{ row.family_label }} | {{ row.type_label }} | {{ row.overview }} | {{ percent_cell(row.missing.percent, row.missing.count) }}{{ finding_cell(row.errors, 'error', px ~ 'pv' ~ loop.index ~ 'e') }}{{ finding_cell(row.issues, 'issue', px ~ 'pv' ~ loop.index ~ 'i') }}{{ details_cell(row, px ~ loop.index) }}
|---|
03 / Findings
Observations({{ ds.quality.items | length }})
What the scan found, by rule, from its facts and never from a guess. A candidate anomaly is worth a look, a remark may be perfectly valid. Counts are exact unless marked.
Not computed: the scan has no observations (scan.observations.enabled is false, or building them failed).
No observations: the scan found nothing to report.
{% else %}{% set qc = ds.quality.counts %}- Confirmed issues
- {{ qc.confirmed_issue | count }}Defects of the source
- Candidate anomalies
- {{ qc.candidate_anomaly | count }}Worth a look
- Remarks
- {{ qc.informational | count }}May be valid
- Limits
- {{ ds.quality.summary.limits | count if ds.quality.summary else '–' }}What the scan could not establish
| {{ status(item.status_word, item.status_class, item.status_title) }} | {{ item.title }}{{ o.rule }} | {% if item.field_name %}{{ item.field_name }}{% else %}dataset{% endif %} | {% if item.kind == 'at_least' %}≥ {% endif %}{{ o.count | count }} | {% if o.of is not none %}{{ o.of | count }}{{ item.noun }}{% else %}–{% endif %} | {{ acc(item.kind) }} | {% set vd = finding_details.get((o.field, o.rule)) %}{% set fv = finding_views.get((o.field, o.rule)) %}{% if vd %}{{ variant_preview(vd) }} Spelling variants {{ item.field_name }} {{ variant_list(vd) }}{{ fv.title }} {{ item.field_name }} {{ detail_list(fv) }} |
Showing the first {{ display_limits.findings | count }} of {{ ds.quality.items | length | count }} findings; the report JSON lists them all.
{% endif %}{% if ds.quality.truncated %}The scan listed only its first scan.observations.max_items findings; its summaries count all of them.
Accuracy: exact counted on every value of the file; at least a scan limit stopped the count; sample counted on a sample of the distinct values. Remarks such as missing values are not quality issues.
{% endif %}04 / Structure
Columns({{ ds.summary.column_count }})
Inferred type and classification per column. Issues: confirmed issues and candidate anomalies found by the scan; remarks such as missing values are under Observations.
| {{ column.inferred_type }} | {{ column.type_label }} | {{ percent_cell(column.type_error_percent,column.type_error_count or 0,'var(--attn)',column.type_error_count != 0) }}{% set nq = column.quality | rejectattr('observation.status','equalto','informational') | list | length %}{% set nr = column.quality | length - nq %}{% if nq %}{{ nq | count }}{% else %}0{% endif %}{% if nr %}+{{ nr | count }} remarks{% endif %} | {{ value_examples(column, px=px) }}
Distinct values are exact unless marked: ≥ at least (a scan limit stopped the count), ≈ estimated from a sample with its error.
05 / Shape
JSON structure({{ st.fields | length }})
Every path of the records, containers included, with presence per parent and array lengths.
- Records
- {{ ds.summary.row_count | count }}{% for name, count in st.record_types.items() %}{{ count | count }} {{ name }}{% if not loop.last %}, {% endif %}{% else %}No records{% endfor %}
- Paths
- {% if st.path_count is none %}limited{% else %}{{ st.path_count | count }}{% endif %}{{ ds.summary.column_count | count }} with values
- Maximum depth
- {{ st.max_depth_seen }}Limit {{ report.config.limits.max_depth }}
{{ f.path }} | {{ f.depth }} | {% for name, count in f.native_types.items() %}{{ name }}{% endfor %} | {% if f.present_percent is none %}items {{ f.present_count | count }} | {% else %}{{ f.present_percent }}%{{ f.present_count | count }} | {% endif %}{% if f.absent_count is none %}– | {% else %}{% if f.absent_count %}{{ f.absent_count | count }}{% else %}0{% endif %} | {% endif %}{% if f.arrays %}{% if f.arrays.minimum_length == f.arrays.maximum_length %}{{ f.arrays.minimum_length | count }}{% else %}{{ f.arrays.minimum_length | count }}–{{ f.arrays.maximum_length | count }}mean {{ f.arrays.mean_length | number }}{% endif %} | {{ f.arrays.empty_count | count }} | {% else %}– | – | {% endif %}{% set role = 'collection' if f.collection else ('column' if f.column else 'container') %}{% if f.collection %}collection {{ f.collection }}{% elif f.column %}column{% else %}container{% endif %} |
|---|
No paths: the dataset has no records.
{% endif %}06 / Cleaning
Transformations({{ ds.summary.column_count }})
Occurrences changed by each normalization stage, and the spellings it groups. Raw preview values remain unchanged.
Variant groups({{ variant_rows | length }})
Raw spellings that every normalization stage compares as equal: an analytical equivalence, not proof of identity. Whitespace is shown as written.
{{ group.key }} | {{ group.count | count }} | {{ group.distinct_count | count }} | {% for item in group.variants %}{{ item.value }}{{ item.count | count }}{% endfor %}{% if group.truncated %}+{{ group.distinct_count - group.variants | length }}{% endif %} |
Only the largest groups of each column are listed (scan.limits.max_variant_groups).
07 / Numeric
Numeric analysis({{ ds.summary.numeric_column_count }})
Range and distribution statistics for accepted numeric values.
| {{ column.numeric.range | number }} | {{ column.numeric.minimum | number }} | {{ column.numeric.maximum | number }} | {{ column.numeric.mean | number }} | {% if column.numeric.median is none %}{% set h = column.deep.histogram %}{% if h %}{{ h.median.value | number }}{{ 'exact' if h.median.accuracy == 'exact' else 'est.' }} | {% else %}limited | {% endif %}{% else %}{{ column.numeric.median | number }} | {% endif %}{{ distinct_cell(column) }}{{ value_examples(column,'num',px) }}
08 / Dates
Date analysis({{ ds.summary.date_column_count }})
Strict date parsing keeps ambiguous, invalid and non-date values separate. Ambiguous values are never resolved from the other values of the column.
| {{ column.date_profile.status | replace('_',' ') }} | {{ column.date_profile.format_count }} variant{{ '' if column.date_profile.format_count == 1 else 's' }} {{ column.date_profile.format_count }} variant{{ '' if column.date_profile.format_count == 1 else 's' }} {% set evidence = column.date_profile.ambiguity_evidence %}{% if column.date_profile.ambiguous_count and evidence %}Unambiguous values: {% for order, count in evidence.items() %}{{ order }} {{ count | count }}{% if not loop.last %} · {% endif %}{% endfor %}.{% set attested = evidence | dictsort | selectattr(1) | list %}{% if attested | length == 1 %} Only {{ attested[0][0] }} is attested: set {% for item in column.date_profile.breakdown %} {{ item.label }}{{ item.count | count }} ({{ item.percent }}%) {% endfor %} |
Distribution over time exact
Valid dates counted by the second pass per calendar period, as written: the offset of a datetime is not converted, and ambiguous dates are counted apart and never placed, unless the unambiguous dates of the column prove one order and the second pass reads them with it.
{% for column in period_columns %}{% set pc = column.deep.periods %}{% set top = (pc.periods | map(attribute='count') | max) %}{{ column.name }} · {{ pc.count | count }} dates by {{ pc.granularity }}, {{ pc.first }} to {{ pc.last }}{% if pc.ambiguous %} · {{ pc.ambiguous | count }} ambiguous not placed{% endif %}{% if pc.resolution %} · {{ pc.resolved | count }} ambiguous read as {{ pc.resolution.order }} (proved by {{ pc.resolution.evidence.get(pc.resolution.order, 0) | count }} unambiguous dates){% endif %}{% if pc.unplaced %} · {{ pc.unplaced | count }} not dates{% endif %}
| Period | Dates |
|---|---|
| {{ period.period }} | {{ period.count | count }} |
09 / Text
String analysis({{ ds.summary.string_column_count }})
Length classes, fixed widths and representative values.
| {{ column.string_profile.status | replace('_',' ') }} | {% if column.string_profile.fixed_length is not none %}{{ column.string_profile.fixed_length }}{% else %}–{% endif %} | {% if column.string_profile.fixed_length is not none %}– | – | – | – | {% else %}{{ column.string_profile.minimum_length }} | {{ column.string_profile.maximum_length }} | {{ column.string_profile.mean_length }} | {{ column.string_profile.median_length }} | {% endif %}{% for item in column.string_profile.length_distribution %}{% endfor %} | {{ distinct_cell(column) }}{% for value in column.string_profile.representative_examples %}{{ value }}{% endfor %} Lengths {{ column.string_profile.distinct_length_count }} {% for item in column.string_profile.length_distribution | sort(attribute='count', reverse=true) %} {{ item.length }} character{{ '' if item.length == 1 else 's' }}{% for example in item.examples[:3] %}{{ example.value }}{% if not loop.last %} / {% endif %}{% endfor %} {{ item.percent }}%{{ item.count | count }} |
10 / Detectors
Detectors and formats({{ detector_rows | length }})
What each detector recognized per column, with the formats it found. Primary: the interpretation the classification starts from.
| {{ detector.id }}{% if detector.primary %} primary{% endif %} | {% if detector.status == 'failed' %}failed | – | – | – | {% else %}{{ percent_cell(detector.matched_percent, detector.matched_count, 'var(--ok)') }}{{ detector.ambiguous_count | count }} | {{ detector.invalid_count | count }} | {% if detector.formats %}{% for item in detector.formats %}{{ item.format }}{% endfor %} {{ detector.formats | length }} format{{ '' if detector.formats | length == 1 else 's' }} {% for item in detector.formats %} {{ item.format }}{{ item.count | count }} ({{ item.percent }}%) {% endfor %} | {% else %}– | {% endif %}{% endif %}
11 / Shapes
Patterns({{ pattern_columns | length }})
Structural patterns inferred from the distinct values the scan sampled, grouped by family (the values of one structure); the column page shows each family with its nodes and tables. 9 a digit, A a capital, a a lowercase letter, @ any letter, X a capital or a digit, x a lowercase letter or a digit, # a letter or a digit, other characters as written. Shares count distinct values, never occurrences in the file, and are not validated against the whole file.
- Columns with patterns
- {{ pattern_columns | length | count }}of {{ ds.summary.column_count | count }} columns
- Without patterns
- {{ pattern_missing | count }}Numbers, dates, masked or too few distinct values
- Grammar
- {{ pattern_grammar }}Version of the pattern rules
| – | unmatched · {{ o.outside_structures | count }} structures | other | {{ o.outside_distinct | count }} | {{ percent_cell((o.outside_share * 100) | round(2), o.outside_distinct, 'var(--d3)') }}{{ acc(kind) }} | {% set vp = column.deep.patterns %}{% if vp %}{{ percent_cell(((vp.outside_patterns() * 100 / vp.values) | round(2)) if vp.values else 0, vp.outside_patterns(), 'var(--d3)') }}{% else %}– | {% endif %}– | – | |
{{ c.family_key }}{{ c.family }} | {{ c.render }} | {{ c.role }} | {{ c.distinct | count }} | {{ percent_cell((c.share * 100) | round(2), c.distinct, 'var(--d2)') }}{{ acc(kind) }}{{ 'of every distinct value' if kind == 'exact' else 'of sampled distinct values' }} | {% set vp = column.deep.patterns %}{% if vp %}{% set tally = (vp.all_patterns() | selectattr('id', 'equalto', c.id) | first) %}{{ percent_cell(((tally.occurrences * 100 / vp.values) | round(2)) if vp.values else 0, tally.occurrences, 'var(--d2)') }}{% else %}– | {% endif %}{{ c.length }} | {% if shown %}{% for value in shown[:3] %}{{ value }}{% endfor %} Examples {{ shown | length }} {% for value in shown %} {{ value }} {% endfor %} | {% else %}not kept | {% endif %}
Showing the first {{ display_limits.patterns | count }} of {{ pattern_rows | length | count }} patterns; the report JSON lists them all.
{% endif %}A share is the part of the distinct values a pattern takes: exact when the scan held every distinct value, sample when it held a sample of them. A value counts for one pattern: the one of its family that has its lengths. Quoted text is a constant the pattern folds. Validated on every value is the second pass: how many occurrences of the whole file the pattern reads, exactly; a value no pattern reads is an observation of structure, not an error.
12 / Relationships
Relationships({{ rel.count }})
How one column goes with another, counted on every row by the second pass. The counts and the percentages of each category come first; a score only summarizes them. An association is not a cause, and a candidate rule is learned from the data and validated by nobody.
The same pair the other way round: {{ g.category }} to {{ g.column }}
{% endif %}{{ g.category }} to {{ g.column }}({{ g.tables | length }} {{ 'feature' if g.tables | length == 1 else 'features' }})
{% for t in g.tables %}{% if t.leading %}{{ t.feature_label | capitalize }}: {% if t.score is not none %}score {{ '%.2f' | format(t.score) }}{% else %}no score{% endif %}, purity {% if t.purity is not none %}{{ '%.1f' | format(t.purity * 100) }}%{% else %}n/a{% endif %}
{% endif %}By {{ t.feature_label }} {{ acc('exact') }} {{ t.sentence }}
- Classified rows
- {{ t.counted | count }}of {{ t.examined | count }}{% if t.left_out %}; {{ t.left_out }}{% endif %}
- Purity
- {% if t.purity is not none %}{{ '%.1f' | format(t.purity * 100) }}%{% else %}–{% endif %}Dominant class of each category{% if t.reverse_purity is not none %}; the other way round {{ '%.1f' | format(t.reverse_purity * 100) }}%{% endif %}
- Association
- {% if t.score is not none %}{{ '%.2f' | format(t.score) }}{% else %}–{% endif %}{% if t.strength == 'unstable' %}Unstable: low counts{% elif t.strength %}Cramér's V, bias-corrected: {{ t.strength }}{% else %}Not defined{% endif %}
- Exceptions
- {{ t.exception_count | count }}Rows outside the dominant class, in {{ t.exception_cells | count }} cells
{{ t.note }}
{% endif %}{% if t.feature == 'pattern' and c != 'other' %} {{ c }}{% else %}{{ c }}{% endif %} | {% endfor %}{{ table_head('Rows', align=true, filter=false) }}{{ table_head('Coverage', align=true, filter=false) }}{{ table_head('Purity', align=true, filter=false) }}||||
|---|---|---|---|---|
| {{ r.label }} | {% for cell in r.cells %}{% if cell.count %}{{ cell.count | count }}{{ '%.1f' | format(cell.share * 100) }}%{% else %}0{% endif %}{% if cell.dominant %} dominant class{% endif %} | {% endfor %}{{ r.classified | count }} | {% if r.coverage is not none %}{{ '%.1f' | format(r.coverage * 100) }}%{% else %}–{% endif %} | {% if r.purity is not none %}{{ '%.1f' | format(r.purity * 100) }}%{% else %}–{% endif %} |
| All | {% for n in t.column_totals %}{{ n | count }} | {% endfor %}{{ t.total | count }} |
Exceptions({{ t.exception_count | count }} rows)
| {{ e.category }} | {% if t.feature == 'pattern' and e.label != 'other' %}{{ e.label }}{% else %}{{ e.label }}{% endif %} | {{ e.count | count }} | {% if e.examples %}{% for record, value in e.examples %}{{ value }}{% endfor %}Records {{ e.examples | map(attribute=0) | join(', ') }}{% else %}–{% endif %} |
|---|
Showing the {{ t.exceptions | length | count }} largest of {{ t.exception_cells | count }} cells; the JSON lists them in its analyses.
{% endif %}{% endif %}{% if t.rules %}Candidate rules({{ t.rules | length }})
- {% for r in t.rules %}
- candidate When
{{ g.category }}is {{ r.category }},{{ g.column }}reads {% if t.feature == 'pattern' %}{{ r.label }}{% else %}{{ r.label }}{% endif %} in {{ (r.support - r.exceptions) | count }} of {{ r.support | count }} rows ({{ '%.1f' | format(r.confidence * 100) }}%).Learned from the data, not validated.{{ r.exceptions | count }} {% endfor %}
Showing the first {{ display_limits.relations | count }} of {{ rel.groups | length | count }} relations; the JSON lists them all.
{% endif %}{% for note in rel.notes %}{{ note }}
{% endfor %}{% if rel.correlations %}Correlations between numeric columns({{ rel.correlations | length }})
{% set l = column_links.get((c.dataset_id, c.field_id)) %}{% set r = column_links.get((c.dataset_id, c.partner_id)) %}{% if l and r %}{% for fid, href in [(c.field_id, l), (c.partner_id, r)] %}{{ (ds.columns | selectattr('id', 'equalto', fid) | first).name }} {% if not loop.last %} and {% endif %}{% endfor %}{% else %}{{ c.pair }}{% endif %} | {% if c.coefficient is not none %}{{ '%+.3f' | format(c.coefficient) }}{% else %}–{% endif %} | {{ c.pairs | count }} | {{ c.note }} | {{ acc('exact') }} |
|---|
Counts are exact: every row of the file is placed in one category and one class, or counted as left out. The dominant class of a category is the one that holds most of it; a tie goes to the first class listed. Purity is the share of the classified rows that sit in the dominant class of their category. The score is Cramér's V corrected for the number of rows and classes; when the expected counts are low it is marked unstable and given no strength. An empty other row or column is not printed. Correlations show a linear association between two columns, over the rows where both have a number.
13 / Insights
Insights({{ ds.insights | length }})
What the second pass found in the values, with the accuracy of each figure: exact counts every value of the file, estimated comes with certain bounds. The same cards appear on the page of each column they concern.
{{ family_titles[card.family] }}
{% endif %}{{ insight_card(card, px) }}{% endfor %}{% if v.insight_cards | length < ds.insights | length %}Showing {{ v.insight_cards | length | count }} of {{ ds.insights | length | count }} cards, at most {{ display_limits.insights | count }} of each family; the report JSON holds every result.
{% endif %}14 / Second pass
Second pass({{ sp.rows | length }})
Exact analyses added by reading the source again. {{ mode_words[sp.mode] }} {{ outcome_words[sp.outcome] }}
{{ sp.reason }}
{% endif %}{% if sp.metrics %}{% set m = sp.metrics %}- Computed
- {{ m.computed | count }}Analyses with a result
- Not computed
- {{ (m.skipped + m.failed) | count }}{{ m.skipped | count }} skipped, {{ m.failed | count }} failed
- Already answered
- {{ m.already_answered | count }}By the scan, no reading needed
- Not selected
- {{ m.not_selected | count }}Left out by the planner
{% set href = column_links.get((row.dataset_id, row.field_id)) %}{% if href %}{{ row.column }} {% else %}{{ row.column }}{% endif %} | {{ row.analysis }} | {{ row.status }} | {{ row.summary }} | {{ acc(row.kind) }} |
|---|
No analysis was run for this dataset.
{% endif %}{% if sp.left_out %}Not selected, by reason: {% for reason, count in sp.left_out %}{{ count | count }} {{ reason }}{% if not loop.last %}; {% endif %}{% endfor %}.
{% endif %}{% if sp.source_read %}The source was read again{% if sp.source_verified %} and checked against the digest of the scan, so no result mixes two versions of the data{% endif %}{% if sp.seconds is not none %}, {{ sp.seconds | seconds }} s{% endif %}. This run computed {{ sp.executed | count }} analyses and reused {{ sp.reused | count }} stored ones.
{% elif sp.metrics %}The source was not read for this report. Stored results and the facts of the scan were used.
{% endif %}15 / Limits
Limits and diagnostics
What the scan could not measure completely, and the technical events it recorded. Counters and statistics continue past a limit.
- Untracked observations
- {{ lim.untracked_observations | count }}Paths beyond
scan.limits.max_fields - Truncated by depth
- {{ lim.depth_truncated_observations | count }}Deeper than
scan.limits.max_depth
The limited measures are not computed: the scan has no observations (scan.observations.enabled is false, or building them failed).
Limited measures({{ lim.items | length }})
{% if m.field_path is none %}dataset{% else %}{{ m.field_path }}{% endif %} | {{ m.label }}{{ m.measure }} | {{ m.reason }}{% if m.setting %}{{ m.setting }}{% elif m.note %}{{ m.note }}{% endif %} | {% if m.limit is none %}– | {% else %}{{ m.limit | count }} | {% endif %}{% if m.lower_bound is none %}– | {% else %}{{ m.lower_bound | count }} | {% endif %}
|---|
Showing the first {{ display_limits.limits | count }} of {{ lim.items | length | count }} limits; the report JSON lists them all.
{% endif %}{% endif %}{% if lim.diagnostics %}Diagnostics({{ lim.diagnostics | length }})
- {% for d in lim.diagnostics %}
- {{ d.level }}
{{ d.code }}{{ d.message }}{% if d.dataset is none %} whole scan{% endif %}{% if d.locations %}Records {% for loc in d.locations %}{{ loc.record }}{% if loc.line is not none %} (line {{ loc.line }}){% endif %}{% if not loop.last %}, {% endif %}{% endfor %}{% if d.count > d.locations | length %}…{% endif %}{% endif %}{{ d.count | count }} {% endfor %}
16 / Raw values
Data sample({{ ds.rows.rows | length }})
{% set rs = ds.rows.sample %}{% if rs %}Rows of the scan's row sample, with original row numbers and raw values: the first records, records drawn at random from the interior and the last records. The sampled rows are not uniform over the whole file.
{% else %}Rows with original row numbers and raw values.
{% endif %}No row sample: scan.records.sample.rows is 0.
{% if rs.of %}This report keeps no row (report.row_sample asks for none).{% else %}No data records.{% endif %}
{{ ds.rows.rows | length | count }} of the {{ rs.of | count }} rows the scan sampled ({{ rc.start | count }} at the start, {{ rc.middle | count }} sampled, {{ rc.end | count }} at the end) out of {{ rs.population.records | count }} records. The sampled rows come from the interior, records {{ (rs.requested.edge + 1) | count }} to {{ (rs.population.records - rs.requested.edge) | count }}, where each record had the same chance by a seeded hash.{% if rs.reason %} The scan cut its sample to stay within its memory budget.{% endif %}
Row | Part | {% for column in ds.columns %}{{ (column.name or '(unnamed)') | breakable }}{{ column.inferred_type }} | {% endfor %}||
|---|---|---|---|---|
| {{ row.record }} | {% for origin in row.origins %}{{ origin_words[origin] }}{% if not loop.last %} · {% endif %}{% endfor %} | {% for column in ds.columns %}{% if column.id not in row.values %}absent | {% elif row.values[column.id] is none %}hidden | {% else %}{% set cell = row.values[column.id] | cell_text %}{% set cut = column.id in row.truncated %}{% if cell == '' %}empty{% else %}{{ cell }}{% endif %}{% if cut %}… cut{% endif %} | {% endif %}{% endfor %}
Showing the first {{ display_limits.rows | count }} of {{ ds.rows.rows | length | count }} rows kept; the report JSON lists them all.
{% endif %}{% endif %}17 / Method
Analysis settings
Rules used for this analysis, so the result can be reproduced.
- Scope
- All {{ ds.summary.row_count | count }} records {% set scan = report.config %}
- Missing values
- {{ scan.values.missing | join(', ') }}; markers: {% for marker in scan.values.null_markers %}
{{ marker | tojson }}{% else %}none{% endfor %} - Marker comparison
- Whitespace trimmed; {{ 'case-sensitive' if scan.values.null_markers_case_sensitive else 'case-insensitive' }}
- Value normalization
- Unicode composition (NFC): {{ 'enabled' if scan.normalization.nfc else 'disabled' }}; trim: {{ 'enabled' if scan.normalization.trim else 'disabled' }}; collapse internal whitespace: {{ 'enabled' if scan.normalization.collapse_whitespace else 'disabled' }}; comparison also with case folding: {{ 'enabled' if scan.normalization.casefold else 'disabled' }}, accents removed: {{ 'enabled' if scan.normalization.strip_accents else 'disabled' }}. Raw preview values are preserved.
- Duplicate comparison
- {% if scan.records.duplicates %}Exact raw values across every column; up to {{ scan.limits.max_tracked_records | count }} distinct records compared{% else %}Disabled{% endif %}
- Type inference
- At least {{ (scan.types.minimum_confidence * 100) | round(1) }}% agreement across present values; numbers with {{ scan.detectors.number.conventions | join(' or ') }} decimals, configured dates and true/false. Leading-zero identifiers remain text.
- Date detection
- {% if scan.detectors.date.enabled %}Orders: {{ scan.detectors.date.orders | join(', ') }}; separators: {% for separator in scan.detectors.date.separators %}
{{ separator | tojson }}{% endfor %}; month names: {{ scan.detectors.date.month_languages | join(', ') or 'none' }}; ambiguous order: {{ scan.detectors.date.ambiguous_order or 'unresolved, column evidence shown but never applied' }}{% else %}Disabled{% endif %} - Column types
- The technical type, a detector matching at least {{ (scan.detection.minimum_share * 100) | round(1) }}% of present values, the length of the texts, the name of the column and, when the second pass counted them, the exact distinct values
- Adaptive detection
- {% if scan.detection.warmup_values %}Every detector on the first {{ scan.detection.warmup_values | count }} distinct values of each column; detectors that recognized none of them{% if scan.detection.rare_share %}, or at most {{ (scan.detection.rare_share * 100) | round(2) }}% of them and none of the last {{ (scan.detection.warmup_values - scan.detection.warmup_values // 2) | count }},{% endif %} then skip the others{% if scan.detection.probe_interval %}, except about 1 value in {{ scan.detection.probe_interval | count }}{% endif %}. Number and date detection always test every value{% else %}Disabled: every detector tests every value{% endif %}
- Enumerations
- {% if scan.detectors.enumeration.enabled %}At most {{ scan.detectors.enumeration.maximum_distinct }} distinct values among at least {{ scan.detectors.enumeration.minimum_values | count }} present values{% else %}Disabled{% endif %}
- Sensitive values
- {{ {'mask': 'Masked', 'hide': 'Hidden', 'show': 'Shown'}[scan.exposure.sensitive_values] }} in examples and the data sample
- String lengths
- Very short through {{ report.config.classification.very_short_max_length }}; short through {{ report.config.classification.short_max_length }}; medium through {{ report.config.classification.medium_max_length }}; long through {{ report.config.classification.long_max_length }}; otherwise very long. Up to {{ report.report.config.string_analysis.examples_per_length }} examples per length are retained when the maximum is {{ report.report.config.string_analysis.length_distribution_max_length }} or less.
- Value representation
- Complete through {{ report.report.config.value_examples.full_distribution_max_distinct }} distinct values; otherwise a reproducible sample of up to {{ v.sample_size }} distinct values, the ones with the smallest hash priorities
- Row numbering
- Data records start at 1, excluding the header. Quoted multiline values count as one record.
- Row sample
- {% if ds.rows.sample %}The scan kept the first {{ ds.rows.sample.requested.edge }} records, up to {{ ds.rows.sample.requested.middle }} records drawn from the interior by a seeded hash, and the last {{ ds.rows.sample.requested.edge }}; this report shows {{ report.report.config.row_sample.start }} at the start, {{ report.report.config.row_sample.middle }} sampled and {{ report.report.config.row_sample.end }} at the end; raw values preserved{% else %}Disabled{% endif %}
- Patterns
- {% if scan.pattern_inference.enabled %}Inferred from the sampled distinct values: up to {{ scan.pattern_inference.max_patterns }} per column, values up to {{ scan.pattern_inference.max_value_length }} characters, at least {{ scan.pattern_inference.min_distinct }} distinct values per pattern{% else %}Not computed: disabled by
scan.pattern_inference.enabled{% endif %} - Observations
- {% if scan.observations.enabled %}Rules version {{ report.observations.version if report.observations else '–' }}; at most {{ scan.observations.max_items | count }} findings and {{ scan.observations.max_items | count }} limits listed{% else %}Not computed: disabled by
scan.observations.enabled{% endif %} - JSON format
- {{ report.format_version }} - revision {{ report.format_revision }} (experimental)
{{ report.source.sha256 }}