+
+ {{ title }}
+
+
+
+ {% include './components/usa-banner.njk' %}
+ {% include 'header.njk' %}
+
+
+
+
+ {{ metadata.title }}
+ {{ metadata.description }}
+
+
+ {{ collections.posts | getNewestCollectionItemDate | dateToRfc3339 }}
+ {{ baseUrl | htmlBaseUrl(baseUrl) }}
+
+ {{ metadata.author.name }}
+
+ {%- for post in collections.known_errors | reverse %}
+ {%- set absolutePostUrl %}{{ post.url | htmlBaseUrl(baseUrl) }}{% endset %}
+
+ {{ post.data.title }}
+
+ {{ post.date | dateToRfc3339 }}
+ {{ absolutePostUrl }}
+ {{ post.content | renderTransforms(post.data.page, baseUrl) }}
+
+ {%- endfor %}
+
\ No newline at end of file
diff --git a/src/data/concerns/index.md b/src/data/concerns/index.md
new file mode 100644
index 00000000..bc4dcf60
--- /dev/null
+++ b/src/data/concerns/index.md
@@ -0,0 +1,36 @@
+---
+layout: sidenav.njk
+title: Known errors in FAC data
+meta:
+ name: Historic data curation
+ description: Documentation about the data migration from Census to GSA.
+---
+
+# {{title}}
+
+The data contained within the FAC is a living thing. Whether it is an error in how data is validated, errors introduced as a result of system changes, or mistakes made on the part of auditors and grantees, the data needs to be tended and cared for.
+
+Here, we document the errors we know about in the FAC data. We follow our [curation process](../curation/) when repairing data.
+
+This page summarizes the known issues, and links to our strategies, timelines, and outcomes.
+
+You can subscribe to an [RSS]({{"/feeds/rss/concerns.xml" | htmlBaseUrl(baseUrl)}}) or [Atom]({{"/feeds/atom/concerns.xml" | htmlBaseUrl(baseUrl)}}) feed to learn about new concerns as they are discovered. See [Microsoft's documentation for subscribing to feeds](https://support.microsoft.com/en-us/office/what-are-rss-feeds-e8aaebc3-a0a7-40cd-9e10-88f9c1e74b97) as a starting point.
+
+## Known concerns
+
+We are currently tracking {{ collections.known_errors | length }} concerns in the data.
+
+We include a brief, one-line sumamry of each error here, and full explanations can be found on the linked pages.
+
+{% for item in collections.known_errors | sort(false, false, 'data.discovered_date') %}
+ {% if item.data.title | length %}
+
+
+ {{ metadata.title }}
+ {{ baseUrl | htmlBaseUrl(baseUrl) }}
+ {{ metadata.description }}
+ {{ metadata.language or page.lang }}
+ {%- for post in collections.known_errors | reverse %}
+ {%- set absolutePostUrl = post.url | htmlBaseUrl(baseUrl) %}
+ -
+
{{ post.data.title }}
+ {{ absolutePostUrl }}
+ {{ post.content | renderTransforms(post.data.page, baseUrl) }}
+ {{ post.date | dateToRfc822 }}
+ {{ metadata.author.name }}
+ {{ absolutePostUrl }}
+
+ {%- endfor %}
+
+
\ No newline at end of file
diff --git a/src/data/curation-log/_2024-07-test-long.md b/src/data/curation-log/_2024-07-test-long.md
new file mode 100644
index 00000000..0bb9f9da
--- /dev/null
+++ b/src/data/curation-log/_2024-07-test-long.md
@@ -0,0 +1,16 @@
+---
+layout: sidenav.njk
+tags: _curation_log
+concern: 2024-03-fy-off-by-one
+slug: Repair of FY end dates in migrated data
+adr: https://asdfasdf.com/asdfasdf.html
+repair_date: 2024-08
+github_pr: https://github.com/pr/asdfasdf
+---
+
+{% from "components/macros.njk" import curationSummary %}
+
+{{ curationSummary(slug, discovered_date, timeline_to_repair, github, log) }}
+
+asdfasdfasdf
+
diff --git a/src/data/curation-log/atom.njk b/src/data/curation-log/atom.njk
new file mode 100644
index 00000000..2eb737ad
--- /dev/null
+++ b/src/data/curation-log/atom.njk
@@ -0,0 +1,33 @@
+---
+permalink: "/feeds/atom/curation-log.xml"
+eleventyExcludeFromCollections: true
+metadata:
+ title: "Log of Federal Audit Clearinghouse curation actions"
+ description: "Log of Federal Audit Clearinghouse curation actions"
+ language: en
+ base: "https://fac.gov/data/curation-log"
+ author:
+ name: "Clearinghouse staff"
+---
+
+
+ {{ metadata.title }}
+ {{ metadata.description }}
+
+
+ {{ collections.posts | getNewestCollectionItemDate | dateToRfc3339 }}
+ {{ baseUrl | htmlBaseUrl(baseUrl) }}
+
+ {{ metadata.author.name }}
+
+ {%- for post in collections.curation_log | reverse %}
+ {%- set absolutePostUrl %}{{ post.url | htmlBaseUrl(baseUrl) }}{% endset %}
+
+ {{ post.data.title }}
+
+ {{ post.date | dateToRfc3339 }}
+ {{ absolutePostUrl }}
+ {{ post.content | renderTransforms(post.data.page, baseUrl) }}
+
+ {%- endfor %}
+
\ No newline at end of file
diff --git a/src/data/curation-log/index.md b/src/data/curation-log/index.md
new file mode 100644
index 00000000..2acf41ca
--- /dev/null
+++ b/src/data/curation-log/index.md
@@ -0,0 +1,31 @@
+---
+layout: sidenav.njk
+title: Log of curation actions
+meta:
+ name: Historic data curation
+ description: Documentation about the data migration from Census to GSA.
+---
+
+
+{% from "components/macros.njk" import getKnownErrorTitle %}
+
+# {{title}}
+
+
+## Curation actions
+
+This page documents the actions we have taken to curate the data contained within the FAC. We include a brief, one-line sumamry of each curation action here, and full explanations can be found on the linked pages.
+
+Curation actions are often in response to concerns that are identified by the FAC team or members of the community. We are currently tracking {{ collections.known_errors | length }} concerns in the data. Data concerns are [documented separately](../concerns/).
+
+You can subscribe to an [RSS]({{"/feeds/rss/curation-log.xml" | htmlBaseUrl(baseUrl)}}) or [Atom]({{"/feeds/atom/curation-log.xml" | htmlBaseUrl(baseUrl)}}) feed of our data curation actions. See [Microsoft's documentation for subscribing to feeds](https://support.microsoft.com/en-us/office/what-are-rss-feeds-e8aaebc3-a0a7-40cd-9e10-88f9c1e74b97) as a starting point.
+
+{% for item in collections.curation_log | sort(false, false, 'data.repair_date') %}
+ {% if item.data.concern | length %}
+
+
+ {{ metadata.title }}
+ {{ baseUrl | htmlBaseUrl(baseUrl) }}
+ {{ metadata.description }}
+ {{ metadata.language or page.lang }}
+ {%- for post in collections.curation_log | reverse %}
+ {%- set absolutePostUrl = post.url | htmlBaseUrl(baseUrl) %}
+ -
+
{{ post.data.title }}
+ {{ absolutePostUrl }}
+ {{ post.content | renderTransforms(post.data.page, baseUrl) }}
+ {{ post.date | dateToRfc822 }}
+ {{ metadata.author.name }}
+ {{ absolutePostUrl }}
+
+ {%- endfor %}
+
+
\ No newline at end of file
diff --git a/src/data/curation.md b/src/data/curation.md
new file mode 100644
index 00000000..b61e271b
--- /dev/null
+++ b/src/data/curation.md
@@ -0,0 +1,43 @@
+---
+layout: sidenav.njk
+title: The curation process
+meta:
+ name: General migration transforms
+ description: Documentation about the data migration from Census to GSA.
+---
+
+# {{title}}
+
+The Federal Audit Clearinghouse holds the record of Single Audit submissions from 2016-present. It is easy to think that data collected in the past "never changes," but as systems evolve, so must the data and the tools that access it. And, sometimes, the data itself must change or be cared for.
+
+## What is data curation?
+
+**Curation** is the act of looking after a collection. We use this word intentionally to describe how we manage the data contained within the FAC, as it implies that we do not simply *collect* and *store* data, but we must also *monitor* and *improve* the quality of that data over time, where possible. Sometimes we engage in curatorial acts because of errors discovered by users (or ourselves), and sometimes because of changes to the data collection or system itself.
+
+
+Our curation process follows 6 high-level steps.
+
+## 1. Identify an issue in the data
+
+A member of the public, grantee, Federal resolution official, IG, or a member of the Clearinghouse may identify some issue in the data. It might involve a single record, or it might be systemic across many, many records.
+
+## 2. Document the issue
+
+We document our [data concerns]({{"../concerns/" | absoluteUrl}}) publicly. The FAC data is public, and it is important for users and systems downstream to know about any concerns we have about the data.
+
+## 3. Implement a data quality regression test
+
+A regression test is a test that *fails* because of a known problem. Once the problem is fixed, the test *passes*, indicating the problem no longer exists. That test is then run regularly, in the future, to ensure that the problem does not happen again. Not all data errors can have regression tests applied, but we will strive to do so to ensure the quality of data is ever-improving.
+
+## 4. Communicate a solution
+
+The Clearinghouse, in consultation with its Advisory Committee, will propose a solution via our [Decision Record process](https://github.com/GSA-TTS/FAC/blob/main/docs/architecture/decisions/0001-record-architecture-decisions.md). We will then announce that proposed solution, and a timeline to resolution, via our [updates page]({{"/updates/" | htmlBaseUrl(baseUrl)}}).
+
+## 5. Implement the solution
+
+The solution will be implemented and executed on the timeline communicated.
+
+## 6. Log the change
+
+The curation process generates an internal audit log; we will also publish, publicly, the results of the action to our [curation log]({{"../curation-log/" | absoluteUrl }}). This provides a public record of the completion of the curatoral act and the results of our work.
+
diff --git a/src/data/general-transforms.md b/src/data/general-transforms.md
new file mode 100644
index 00000000..0b100ecf
--- /dev/null
+++ b/src/data/general-transforms.md
@@ -0,0 +1,60 @@
+---
+layout: sidenav.njk
+title: General migration transforms
+meta:
+ name: General migration transforms
+ description: Documentation about the data migration from Census to GSA.
+tables:
+ - general
+ - federal_awards
+ - notes_to_sefa
+ - findings
+ - findings_text
+ - corrective_action_plans
+ - passthrough
+ - secondary_auditors
+ - additional_ueis
+ - additional_eins
+---
+
+# {{title}}
+
+The Census data from 2016 to 2022 saw multiple changes to the SF-SAC and validations that were applied. For example, there are entire sections of the form that did not exist before 2018.
+
+The tables below describe the transformations that we applied to Census data in migrating it to the GSA intake. For example, Census stored zip codes with hyphens; we remove those hyphens. Further, we found that many zip code records were only 4 or 8 digits; in those cases, we front-padded the zip code with a zero.
+
+Where we say `No transformation`, this means that we passed data through without any modification or further validation. This is because (for example) we have no way of assessing the correctness or otherwise validating an entity name.
+
+There are additional
+
+{%- for tab in tables -%}
+ Table:
+
These are transformations that were applied wholesale to files as data was migrated from Census to GSA. +* [Specific field transformations]({{"/data/specific-transforms/" | absoluteUrl }})
These are transformations applied to specific portions of the data. For example, a Census record that is missing an entire portion of the form required specialized handling. +#} \ No newline at end of file diff --git a/src/data/specific-transforms.md b/src/data/specific-transforms.md new file mode 100644 index 00000000..eb50710f --- /dev/null +++ b/src/data/specific-transforms.md @@ -0,0 +1,9 @@ +--- +layout: sidenav.njk +title: Specific migration transforms +meta: + name: Specific migration transforms + description: Documentation about the data migration from Census to GSA. +--- + +# {{title}} diff --git a/src/data/validations/cross-validations.md b/src/data/validations/cross-validations.md new file mode 100644 index 00000000..e1367994 --- /dev/null +++ b/src/data/validations/cross-validations.md @@ -0,0 +1,61 @@ +--- +layout: sidenav.njk +title: SF-SAC cross-validations +meta: + name: SF-SAC cross-validations + description: Documentation about the data migration from Census to GSA. +baseurl: https://github.com/GSA-TTS/FAC/tree/414f9932b3797de68c7271ba0b471c9b834062ce/backend/audit/cross_validation +--- + +# {{title}} + +These validations are the penultimate line of defense before a submission is considered complete. They are the validations that look across all of the form sections, and attempt to help auditors and auditees submit consistent, correct data. + +## Why cross-validations? + +The SF-SAC expects grantees and auditors to report on data that is interconnected. For example: + +1. A grantee might have an award from an agency. They need to report ont that award. +2. If the auditor found something of concern, then there are findings that have to be described. Findings are associated with awards. +3. A finding will have a corrective action plan. That plan is described in reference to both an award and a finding. + +These connections are critical to oversight officials; for that reason, we do our best to help auditees and auditors describe these connections in the SF-SAC correctly. The cross-validations are how we support users in this kind of inter-form data entry. + +## What are the cross-validations? + +There are fewer cross-validations than intake validations, but they play a role every bit as critical. For example: + +* The check `auditee_ueis_match` makes sure that every form section has the same UEI in it. This way, if an auditor is supporting multiple submissions, we don't actually end up with form sections from different grantees. +* The check `check_certifying_contacts` make sure that one person did not attempt to certify as both the auditor and auditee certifying officials. +* The check `tribal_data_sharing_consent` makes sure that the consent to distribute the audit report was given if the entity is marked as being a Tribe or Tribal Organization. + +Below is a list of all of the cross validations applied to the SF-SAC as of July 2024. + +
+Valid and free from failure
+Except when it's not.
+ + +The dissemination of data serves as one final check that we have all of the data we expect, and that it is of the correct type. It is unlikely at this point that we will detect an error. However, if we do, the dissemination of the data fails, allowing the FAC team and grantee/auditor understand why this happened, and resolve the error before passing data on to agencies and the public. + + +## From intake to validation + +The last step in an SF-SAC submission is to go from these [internal tables to our external dissemination tables](https://github.com/GSA-TTS/FAC/blob/44ae639f70c4552fa584f7042e0c116b0cd93d29/backend/audit/intake_to_dissemination.py). From a systems perspective, we continue to proceed defensively, making no assumptions that all is well with the submission. + +The FAC has an internal database for storing a submission; this is the JSON representation that reflects the shape of the SF-SAC. Once a submission is complete, we convert this into another shape for dissemination. Long-time users of the FAC will recognize this as the `general` table, the `federal_awards` table, the `findings` table, and so on. Our [API]({{"/api" | absoluteUrl}}) is structured around this shape, for consistency of experience for our users. + + +In converting data from our internal to external representation, we make sure all of the data is where we expect it. And, if the wrong *kind* of data is somewhere, it will fail to be written to our dissemination tables. In this regard, we make sure that one of two things is true: + +1. Everything is in the correct place, and can be written out for agencies to begin their work, or +2. Everything isn't, and the submission fails. + +We have not seen this backstop triggered for a very long time. However, it is the final set of checks that are performed on the SF-SAC before we consider it a complete submission. + +*Back to the [FAC data overview]({{"/data" | absoluteUrl}})*. \ No newline at end of file diff --git a/src/data/validations/index.md b/src/data/validations/index.md new file mode 100644 index 00000000..617632c6 --- /dev/null +++ b/src/data/validations/index.md @@ -0,0 +1,51 @@ +--- +layout: sidenav.njk +title: FAC data validations +meta: + name: FAC data validations + description: Documentation about the data migration from Census to GSA. +--- + +# {{title}} + +The data coming into the FAC is validated in five distinct ways. + +## In-form validations + +The SF-SAC is made up of a collection of web forms and spreadsheets that are filled out by grantees and auditors. Within both the web forms and spreadsheets, we have input restrictions and validations that help people complete the forms correctly. + +Because it is difficult to add validations to spreadsheets in a way that works everywhere, this is one of the simplest validation layers in the FAC. + +## Intake validations + +The second layer of data validation is made up of many {# [intake validations]({{"intake-validations/" | absoluteUrl }}) #} intake validations. Here, we transform the spreadsheet or form a user submits into a shape that is more manageable for valiation. We then apply many validations, one after another, to help ensure data quality. Where we find errors, we do our best to provide error messages that guide people to a correct submission. + +{# +Read more about our [intake validations]({{"intake-validations/" | absoluteUrl }}). +#} + +## Schema validations + +A *schema* describes something's *form* or *shape*. For example, we might say the schema for a zip code is five digits followed by an optional hyphen and four more digits. A *schema validation* is where we look at a piece of data, and ask if everything about it has the right shape. + +Every form section in the SF-SAC has a shape. That shape is made up of the fields (names, zip codes, email addresses), and each piece of data in each field has a shape of its own. The SF-SAC schema validations make sure that a submission, after going through our intake validations, ends up in the right shape for storage. + +{# +Read more about our [schema validations]({{"schema-validations/" | absoluteUrl }}). +#} + +## Cross-validations + +Because the SF-SAC is made up of multiple sections, we have to make sure the sections make sense with respect to each-other. Cross-validations look at the submission as a whole, and make sure everything is consistent *between* the SF-SAC sections. + +{# +Read more about our [cross-validations]({{"cross-validations/" | absoluteUrl }}). +#} + +## Dissemination validations + +The final step is to make sure that everything is ready for sharing with agencies and the public. This means we make sure every piece of data we're expecting as part of an SF-SAC submission is present and accounted for. Our goal is to block as many errors as possible; if we don't, then we'll have to [document the errors that get past us and fix them]({{"../curation/" | absoluteUrl}}). + +{# +Read more about our [dissemination validations]({{"dissemination-validations/" | absoluteUrl }}). +#} \ No newline at end of file diff --git a/src/data/validations/intake-validations.md b/src/data/validations/intake-validations.md new file mode 100644 index 00000000..853ea7a2 --- /dev/null +++ b/src/data/validations/intake-validations.md @@ -0,0 +1,76 @@ +--- +layout: sidenav.njk +title: SF-SAC intake validations +meta: + name: SF-SAC intake validations + description: Documentation about the data migration from Census to GSA. +# baseurl: https://github.com/GSA-TTS/FAC/tree/44ae639f70c4552fa584f7042e0c116b0cd93d29/backend/audit/intakelib/checks +baseurl: https://github.com/GSA-TTS/FAC/blob/414f9932b3797de68c7271ba0b471c9b834062ce/backend/audit/intakelib/checks/ +--- + +# {{title}} + +Intake validations are the first layer of validation that the SF-SAC undergoes as it is uploaded to the FAC. In order to validate SF-SAC form sections, we do three things: + +1. Convert the spreadsheet into JSON +2. Validate the data in this intermediate JSON form +3. Convert the data to a meaningful semantic representation + +Each of these steps helps ensure a better submission to the FAC. + + +### Reporting errors + +Intake validations are critical for several reasons. + +1. **They are user-centered**.
Each validation produces errors that are human-friendly. They are what help auditors and grantees fix errors they may have made. Our messages specify errors down to the specific cell, and are refined for usability based on user feedback. +2. **They are self-contained**.
Each validation does only one thing. This helps the GSA team with maintaining the validations. +3. **They are comprehensive**.
The intake validations are intended to cover all possible error conditions we might encounter. In the event that an auditor or grantee finds a new, surprising way to enter data into our forms... we add a new intake validation. + +Ultimately, our goal is for the intake validations to provide a good user experience when part of the SF-SAC is completed incorrectly while simultaneously ensuring that we colelct quality data via the form. + +## Step 1: Convert the spreadsheet into JSON + +Spreadsheets are first converted into JSON, or [Javascript Object Notation]({{glossary.sites.json.url}}). Internally, this is referred to as the *intermediate representation*. This intermediate representation is a small language that uses named ranges to define regions of data. We describe this representation more fully as part of our [intakelib](https://github.com/GSA-TTS/FAC/blob/414f9932b3797de68c7271ba0b471c9b834062ce/backend/audit/intakelib/README.md) documentation. + +This conversion is a validation of its own kind; if we cannot convert the input to our intermediate representation, it is obviously not an SF-SAC upload! The SF-SAC forms must be saved in an Excel 2007 XML-based format (or `xlsx`). We chose this because many current spreadsheet manipulation tools (e.g. [LibreOffice]({{glossary.sites.libreoffice.url}})) are capable of saving data in this format. This means that it is possible to prepare an SF-SAC submission using free and open source software. + +## Step 2: Validate the data + +The intermediate form makes the data validation easier. Every section of the SF-SAC has a common set of validations applied. For example, we make sure every form section has a UEI defined. Then, we apply section-specific validations. Federal Awards is the most complex form section, and therefore many of our data validations are applied to that portion of the submission. + +
+
+
+## Challenges
+
+We encountered a bug on Friday, October 27th. This is not a bug in our software, but in one of our support systems. We are working with our systems providers to understand this bug and resolve it. Until it is resolved, we have to slow releases into our production environment. **This bug does not impact data quality or the current operation of the FAC.**
+
+## What's next?
+
+We are currently working on [search](https://github.com/GSA-TTS/FAC/issues/2236). We have a version ready for release, but are unable to promote it to production due to underlying system issues. Search includes the ability to download PDFs, both via the website and API.
+
+After releasing search, our priorities are:
+- Adding the ability to add, remove, or change auditors and certifying officials who are attached to the audit.
+- Adding the ability to search for and download Tribal audits.
+- Adding the ability to resubmit a previously submitted and completed audit.
+- Adding email notifications to alert auditees and auditors that a step is ready for them to complete.
+- Adding the ability to bulk download search results.
+
+## Opportunities
+
+Do you have an idea that could transform and improve grants management and oversight? 10x, a federal idea accelerator, is [accepting proposals for new investment opportunities](https://10x.gsa.gov/posts/2023-submission-period/). All Federal employees are invited to submit ideas by November 30th.
+
\ No newline at end of file
diff --git a/src/updates/2023-10-30-updates.md b/src/updates/2023-10-30-updates.md
new file mode 100644
index 00000000..1277b003
--- /dev/null
+++ b/src/updates/2023-10-30-updates.md
@@ -0,0 +1,163 @@
+---
+layout: home.njk
+title: FAC Updates - Week of Oct 30, 2023
+date: 2023-10-30
+tags: updates
+---
+
+# {{title}}
+
+We launched search and are ready for users to begin searching single audit reports. This initial iteration of search leverages six different filters for sorting data.
+
+## What we delivered
+
+We work in [an agile manner](https://asana.com/resources/agile-methodology). That means we have a long-term strategy, medium-term features we work to deliver, and make continuous improvement and bug fixes to the existing product. We update our system through pull requests, or PRs.
+
+- Features add new capabilities or documentation to the FAC.
+- Improvements take existing code or text and increase its quality.
+- Validations increase the integrity of the data collected or disseminated by the FAC.
+- Decision records chart our product planning and decision making process.
+- Debt payment removes, cleans up, or otherwise improves existing code.
+
+
+
+
+## What's next?
+
+We’ve released search, and will be continuing to make improvements. We’ll also be focusing on the migration of historical data, and doing some preparatory work to make future work easier.
+
+Our priorities are:
+- Adding the ability to add, remove, or change auditors and certifying officials who are attached to the audit.
+- Adding the ability to search for and download Tribal audits.
+- Adding the ability to bulk download search results.
+- Adding the ability to resubmit a previously submitted and completed audit.
+- Adding email notifications to alert auditees and auditors that a step is ready for them to complete.
+
+## Opportunities
+
+Do you have an idea that could transform and improve grants management and oversight? 10x, a federal idea accelerator, is [accepting proposals for new investment opportunities](https://10x.gsa.gov/posts/2023-submission-period/). All Federal employees are invited to submit ideas by November 30th.
+
\ No newline at end of file
diff --git a/src/updates/2023-11-20-updates.md b/src/updates/2023-11-20-updates.md
new file mode 100644
index 00000000..19dcb44c
--- /dev/null
+++ b/src/updates/2023-11-20-updates.md
@@ -0,0 +1,67 @@
+---
+layout: home.njk
+title: FAC Updates - Week of Nov 20, 2023
+date: 2023-11-20
+meta:
+ name: FAC Updates - Week of Nov 20, 2023
+ description: Read historical updates from the Federal Audit Clearinghouse.
+tags: updates
+---
+
+# {{title}}
+
+*This update is a bit richer given the cancellation of the November office hour for Federal partners.*
+
+We completed the development work necessary to make it possible to unlock audits that had been locked for certification. Auditors and auditees are now able to unlock these audits to make further edits before re-locking for certification.
+
+## What we delivered
+
+We work in [an agile manner](https://asana.com/resources/agile-methodology). That means we have a long-term strategy, medium-term features we work to deliver, and make continuous improvement and bug fixes to the existing product. We update our system through pull requests, or PRs.
+
+### Tribal data access controls
+
+We have implemented the data access controls that are required to provide data access to Federal users who have a need-to-know for audits from tribes and tribal organizations. We are now loading users into the system, and we will notify NSACs and KSMLs when that is complete.
+
+## What's next?
+
+### Historic data migration
+
+We're charged with improving the quality of the data the FAC collects and disseminates. This is challenging but critical work that is necessary if we want the SF-SAC data to be meaningful for oversight. Because of this, we're approaching the migration of historical data from Census as an exercise in data improvement.
+
+We believe it is important for users of FAC data to understand how we're carrying out this work. For today, this is the summary of our process.
+
+1. **Copy.** We have to transfer terabytes of databases and more than 650,000 files from Census to GSA.
+
+1. **Analyze.** The data is not perfect. The GSA team needs to understand all the ways that the data does, and doesn’t, meet our new validation standards.
+
+1. **Migrate.** We’ll begin by migrating 2022 data. Our team will use the Census data to construct “virtual submissions” to the FAC. Doing this will apply all of our integrity checks, confirming the data coming in is of the highest quality.
+
+1. **Record Changes.** As we migrate the data, some things might have to be changed. For example, date formats might change. Anything that changes as part of the migration will be recorded.
+
+1. **Record Failures.** Many of these “virtual submissions” will fail to pass our new integrity checks. We'll keep track of them, and set them aside.
+
+1. **Adjust.** We'll go back to step 2, looking at only the records that failed to migrate. We'll then adjust how we are migrating the data so that we can transfer more historical submissions.
+
+1. **Repeat.** We'll repeat steps 2-6 until we have migrated the submissions from 2022.
+
+Our initial goal is to migrate the last 5 years of data with the highest possible integrity. This will improve the quality of the FAC data and create a public record of the migration.
+
+We're highlighting this process for two reasons:
+
+1. **The data migration is about data integrity.** This work is about improving the integrity of historical FAC data.
+
+1. **The data migration is iterative.** This is not a singular, massive migration of data. It's a sequence of small migrations.
+
+We'll write more about the migration in the future, and documentation about the migration as a whole is part of what we'll deliver with this work.
+
+### Other priorities
+
+The historical data migration occupies a lot of our team’s time. However, we are also working on bug fixes, infrastructure, and a few features that we know are important to public and Federal users. Our current feature priorities as we close out 2023 are:
+
+- Adding the ability to add, remove, or change auditors and certifying officials who are attached to the audit.
+
+- Adding the ability to search for single audits by auditee state.
+
+- Improving the visibility of findings on audits.
+
+- Providing summary reports for both single audit submissions and for all results of a search.
diff --git a/src/updates/2023-12-18-updates.md b/src/updates/2023-12-18-updates.md
new file mode 100644
index 00000000..5a882e4b
--- /dev/null
+++ b/src/updates/2023-12-18-updates.md
@@ -0,0 +1,45 @@
+---
+layout: home.njk
+title: FAC Updates - Week of Dec 18, 2023
+date: 2023-12-17
+meta:
+ name: FAC Updates - Week of Dec 18, 2023
+ description: Read historical updates from the Federal Audit Clearinghouse.
+tags: updates
+---
+
+# {{title}}
+
+This week we've made improvements to user access roles, search results, and Tribal data access.
+
+## What we delivered
+
+We work in [an agile manner](https://asana.com/resources/agile-methodology). That means we have a long-term strategy, medium-term features we work to deliver, and make continuous improvement and bug fixes to the existing product.
+
+### Audit access management
+
+We improved [the user access experience]({{ config.baseUrl }}resources/instructions/user-access) for single audits. Users can now change both the auditor and auditee certifying official for a submission and add new audit editors.
+
+Future improvements will allow for the removal of other contributors to a submission.
+
+### Search improvements
+
+We added the ability to filter search results by auditee state. We also enhanced the results returned by the ALN filter, making it possible to see if a submission includes audit findings.
+
+### Tribal data access controls
+
+We have implemented [access controls for Tribal audit data]({{ config.baseUrl }}data/tribal), and notified NSACs and KMSALs. This data will now appear in searches for logged in Federal users who have access.
+
+### Summary reports
+
+We added the ability to download a summary report for an individual single audit submission. This combines all data from a single SF-SAC into one Excel document.
+
+### Pre-certification review
+
+Auditors and auditees can now review a submission after it is validated and locked, but before it is certified.
+
+## What's next?
+
+### Historic data migration
+
+We continue to work with Census on the migration of historic data.
diff --git a/src/updates/2024-02-05-updates.md b/src/updates/2024-02-05-updates.md
new file mode 100644
index 00000000..87da756c
--- /dev/null
+++ b/src/updates/2024-02-05-updates.md
@@ -0,0 +1,82 @@
+---
+layout: home.njk
+title: FAC Updates - Week of Feb 5, 2024
+date: 2024-02-05
+meta:
+ name: FAC Updates - Week of Feb 5, 2024
+ description: Read historical updates from the Federal Audit Clearinghouse.
+tags: updates
+---
+
+# {{title}}
+
+The team has been focused on completing a migration of data from Census to GSA. This update covers mid-December to the current week.
+
+## Historic data transfer
+
+**Single Audit submissions from 2016–present are now available from the GSA FAC, and the Census Bureau is shutting down their data distribution.** Single Audits from prior years will appear in our search results. This is a major milestone, and future work will involve documenting the data migration, our continued curation of this data, and improvements to how this data is queried and used.
+
+The migration caused some disruptions to service. With 10 times more data, our system has struggled to keep up with volume for some of our search filters. The most affected filter is the ALN search. Results for this filter show whether an audit contains findings (“Findings My ALN/Findings Other ALN”). Before the migration, this filter applied to small amounts of data, but after adding prior years, this began crashing the system. We've removed this feature to resolve the system strain. Our team is working on a replacement feature to handle the larger amount of data.
+
+## What we delivered
+
+We work in [an agile manner](https://asana.com/resources/agile-methodology). That means we have a long-term strategy, medium-term features we work to deliver, and make continuous improvement and bug fixes to the existing product.
+
+### Customer experience
+We achieved “helpdesk zero” in the last week of January. We had no questions come in on a workday, and we emptied our queue so we had no support questions left to answer. We have continued to empty our question queue on a near-daily basis.
+
+The team currently replies to a ticket within 1.5 hours of receipt, and closes 75% of tickets within 8 hours. The helpdesk remains an excellent place to get timely support.
+
+### UX/Design
+
+We completed user testing on navigation changes for [FAC.gov]({{ config.baseUrl }}). This validated a new site structure that will be easier for users to navigate.
+
+The team finalized a research plan around audit resolution work. We began interviews with audit resolution officials, IG offices, and other users.
+
+### Submission improvements
+
+We made multiple improvements to the submission experience. We use research and interviews with our users, as well as issues reported in the helpdesk, to guide this work.
+
+- Added an SF-SAC summary report, allowing users to review the form data for in-progress and completed audits (PR [3025](https://github.com/GSA-TTS/FAC/pull/3025)).
+
+- Significantly sped up PDF validation (PR [3057](https://github.com/GSA-TTS/FAC/pull/3057)).
+
+- Submitters can now unlock an in-progress audit anytime up to the time of final submission (PR [3136](https://github.com/GSA-TTS/FAC/pull/3136)).
+
+- Synchronization fixes between the audit state and submission checklist (PR [3144](https://github.com/GSA-TTS/FAC/pull/3144)).
+
+- Users no longer have to log out and back in for changes to user access (e.g. changing the Auditee Certifying Official email address) to take effect. (PR [3194](https://github.com/GSA-TTS/FAC/pull/3194)).
+
+- Simplified the submission checklist after an audit is validated and locked for certification. (PR [3197](https://github.com/GSA-TTS/FAC/pull/3197)).
+
+- Improved validations regarding reference numbers (PR [3221](https://github.com/GSA-TTS/FAC/pull/3221)).
+
+- Added the ability to download Audit Report Packages and the SF-SAC for in-progress review (PR [3212](https://github.com/GSA-TTS/FAC/pull/3212)).
+
+- Many search performance improvements (PR [3283](https://github.com/GSA-TTS/FAC/pull/3283)).
+
+- Fixes around our new “testing” banner and pre-certification report generation (PRs [3261](https://github.com/GSA-TTS/FAC/pull/3261), [3326](https://github.com/GSA-TTS/FAC/pull/3326), [3333](https://github.com/GSA-TTS/FAC/pull/3333)).
+
+- Improved validation error reporting around ALNs (PR [3230](https://github.com/GSA-TTS/FAC/pull/3230)).
+
+- Improved validation and error reporting around empty/zero loan balances (PR [3335](https://github.com/GSA-TTS/FAC/pull/3335)).
+
+### Intake, Search, and API
+
+We also made multiple improvements to the audit search experience.
+
+- A small update to SF-SAC summary exports (PR [3296](https://github.com/GSA-TTS/FAC/pull/3296)) that adds ALNs to the Federal Awards sheet.
+
+- Fixed a performance bug within the FAC API (PR [3355](https://github.com/GSA-TTS/FAC/pull/3355)).
+
+## What's next?
+
+We're anticipating March to be a busy month. To support this, we'll be:
+
+- **Focusing on performance improvements** and growing our team’s capacity for user support so that the auditors and auditees are well-supported during this time.
+
+- **Providing documentation** regarding the migration of Census data to GSA and how to use it effectively.
+
+- **Improving search**, both to support Federal oversight work as well as the public’s experience with FAC data.
+
+- **Adding Tribal data access to the API** (PR [3359](https://github.com/GSA-TTS/FAC/pull/3359)) and writing documentation and legal agreements to accompany it.
\ No newline at end of file
diff --git a/src/updates/2024-02-26-updates.md b/src/updates/2024-02-26-updates.md
new file mode 100644
index 00000000..af5c7a39
--- /dev/null
+++ b/src/updates/2024-02-26-updates.md
@@ -0,0 +1,68 @@
+---
+layout: home.njk
+title: FAC Updates - Week of Feb 26, 2024
+date: 2024-02-26
+meta:
+ name: FAC Updates - Week of Feb 26, 2024
+ description: Read historical updates from the Federal Audit Clearinghouse.
+tags: updates
+---
+
+# {{title}}
+
+Our work through the end of February and into March is focused in three areas:
+
+1. Improving search.
+2. Improving access to Tribal data.
+3. Preparing for the March submission deadline.
+
+## Improving search
+
+We added new search filters, covered in ticket [#3407](https://github.com/GSA-TTS/FAC/issues/3407). These new filters allow for:
+
+1. Searching audits by finding type.
+2. Filtering by direct or passthrough funding.
+3. Filtering by whether an audit is on a major program or not.
+
+We're continuing to improve search performance and are tracking our plans in story [#3388](https://github.com/GSA-TTS/FAC/issues/3388).
+
+ If you encounter issues with any of these new features, please let us know [via the FAC helpdesk](https://support.fac.gov/hc/en-us/requests/new).
+
+## Improving access to Tribal data
+
+We're reviewing ticket [#2298](https://github.com/GSA-TTS/FAC/issues/2298). This will make Tribal data available to Federal partners via the FAC API. The team is also close to completing ticket [#3331](https://github.com/GSA-TTS/FAC/issues/3331), which will make the public portion of Tribal audits appropriately visible in search.
+
+## March submission deadline
+
+It's possible to [use the FAC API](https://github.com/GSA-TTS/fac-api-examples/tree/main/count-march-submissions) to predict what is coming in March. Here are plots of the submissions-per-day to the FAC from March of 2016 through 2022.
+
+
+ {{ metadata.title }}
+ {{ metadata.description }}
+
+
+ {{ collections.posts | getNewestCollectionItemDate | dateToRfc3339 }}
+ {{ baseUrl | htmlBaseUrl(baseUrl) }}
+
+ {{ metadata.author.name }}
+
+ {%- for post in collections.updates | reverse %}
+ {%- set absolutePostUrl %}{{ post.url | htmlBaseUrl(baseUrl) }}{% endset %}
+
+ {{ post.data.title }}
+
+ {{ post.date | dateToRfc3339 }}
+ {{ absolutePostUrl }}
+ {{ post.content | renderTransforms(post.data.page, baseUrl) }}
+
+ {%- endfor %}
+
\ No newline at end of file
diff --git a/src/updates/index.md b/src/updates/index.md
index a639f2ee..7cc17987 100644
--- a/src/updates/index.md
+++ b/src/updates/index.md
@@ -13,30 +13,29 @@ To help you understand our work, we’ll provide regular updates on the work we
The Federal Audit Clearinghouse team works in the open. Our day-to-day task board can be found on [Github](https://github.com/orgs/GSA-TTS/projects/11/views/2) and prior updates are available in [our archive]({{ config.baseUrl }}updates/archive).
-## Jump to
+You can subscribe to an [RSS]({{"/feeds/rss/updates.xml" | htmlBaseUrl(baseUrl)}}) or [Atom]({{"/feeds/atom/updates.xml" | htmlBaseUrl(baseUrl)}}) feed of our updates. See [Microsoft's documentation for subscribing to feeds](https://support.microsoft.com/en-us/office/what-are-rss-feeds-e8aaebc3-a0a7-40cd-9e10-88f9c1e74b97) as a starting point.
-* [General updates](#general)
-* Updates for [grantees and auditors](#grantees-and-auditors)
-* Updates for [agencies](#agencies)
-## Week of June 10
-
-### Security testing
-
-The FAC completed our compliance-required security testing earlier this month and received a positive report from our oversight body. With this testing complete, the FAC has now done the majority of our Authorization to Operate (ATO) assessment. We're now working with our assessment team to complete the remaining documentation requirements. The ATO process identifies areas of potential security risk and ensures we've taken steps to reduce or eliminate those risks. It also documents our policies for keeping the FAC up-to-date. You can read more about the ATO process on [digital.gov](http://digital.gov/).
-
-### Historical data migration
+{% set count = 2 %}
+{% for item in collections.updates | reverse %}
+ {% if loop.index0 <= count %}
+ {{ item.content | safe }}
+ {% endif %}
+{% endfor %}
-The team is closing in on final remediation for the historical audits that failed initial migration. These audits have a variety of data quality issues that don't meet the FAC's validation standards. For example, some historical audits are missing contact information for certifying officials; others incorrectly report total federal program expenditures. We follow a standardized process to address each of these issues. First, we describe the issue and analyze its prevalence in the dataset. Next, the team brainstorms ways to fix the issue, such as calculating the correct total for a summary field or using a placeholder value where information is missing. Finally, we apply the remediation to the problematic data. If you would like to learn more, visit [this GitHub issue](https://github.com/GSA-TTS/FAC/issues/3364) where we are tracking the work.
+## Previous updates
-### Fixes and improvements
+
+
+ {{ metadata.title }}
+ {{ baseUrl | htmlBaseUrl(baseUrl) }}
+ {{ metadata.description }}
+ {{ metadata.language or page.lang }}
+ {%- for post in collections.updates | reverse %}
+ {%- set absolutePostUrl = post.url | htmlBaseUrl(baseUrl) %}
+ -
+
{{ post.data.title }}
+ {{ absolutePostUrl }}
+ {{ post.content | renderTransforms(post.data.page, baseUrl) }}
+ {{ post.date | dateToRfc822 }}
+ {{ metadata.author.name }}
+ {{ absolutePostUrl }}
+
+ {%- endfor %}
+
+
\ No newline at end of file
+
+
\ No newline at end of file
diff --git a/src/_includes/sidenav.njk b/src/_includes/sidenav.njk
new file mode 100644
index 00000000..1d562e27
--- /dev/null
+++ b/src/_includes/sidenav.njk
@@ -0,0 +1,69 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ {% include 'footer.njk' %}
+
+
+
+
+
+
+
+
+
+
+ {% if include_survey %}
+
+ {% endif %}
+
diff --git a/src/data/concerns/2023-11-incorrect-removal.md b/src/data/concerns/2023-11-incorrect-removal.md
new file mode 100644
index 00000000..29ea4509
--- /dev/null
+++ b/src/data/concerns/2023-11-incorrect-removal.md
@@ -0,0 +1,44 @@
+---
+layout: sidenav.njk
+tags: known_errors
+order: 4
+title: Removal of incorrect audits
+slug: Some audits were submitted with completely incorrect information.
+discovered_date: 2023-11
+timeline_to_repair:
+github:
+log:
+---
+
+{% from "components/macros.njk" import curationSummary %}
+
+{{ curationSummary(slug, discovered_date, timeline_to_repair, github, log) }}
+
+## Curation challenge
+
+A user looked up their UEI in SAM.gov. It was for the [Newark Housing Authority](https://app.fac.gov/dissemination/summary/2023-03-GSAFAC-0000012783) They submitted. They then realized they meant to submit for the [Newark Housing Authority](https://app.fac.gov/dissemination/summary/2023-03-GSAFAC-0000014267), which has a different UEI, and submitted again.
+
+The first of these two is simply incorrect, and needs to be removed.
+
+This is not the only example of this kind of problem within the system, and a mechanism by which the FAC can remove audits from circulation that are simply *in error* is needed.
+
+### Example
+
+See above.
+
+## Consequences
+
+There is no indication to a resolution official which audit is correct or incorrect. It is likely to be resolved through the resolution process, but it would be better if the FAC curated the data and removed the incorrect audit submissions.
+
+## Mediating the error
+
+There is no clear mediation pathway for users.
+
+## Possible resolution
+
+We will want a documented process for removing audits we believe to be in error. This process may involve a positive assertion (documented) from the auditor and auditee certifying officials that the audit in question should be removed. Consultation with OMB and OIGs will be necessary before implementing an SOP regarding this kind of error.
+
+We will likely maintain the record internally, but flag it as `in-error`. This way, if we are wrong in our assessment, we do not lose the data.
+
+
+
diff --git a/src/data/concerns/2024-02-duplicate-rows.md b/src/data/concerns/2024-02-duplicate-rows.md
new file mode 100644
index 00000000..18844f35
--- /dev/null
+++ b/src/data/concerns/2024-02-duplicate-rows.md
@@ -0,0 +1,50 @@
+---
+layout: sidenav.njk
+tags: known_errors
+order: 1
+title: Duplicated data rows
+slug: Some rows in some tables from the 2023 data collection are duplicated.
+discovered_date: 2024-02
+timeline_to_repair:
+github:
+log:
+---
+
+{% from "components/macros.njk" import curationSummary %}
+
+{{ curationSummary(slug, discovered_date, timeline_to_repair, github, log) }}
+
+## Curation challenge
+
+In February of 2024, we discovered a bug where, if users pressed the "submit" button twice in rapid successesion, some data in the FAC database could be duplicated. Investigation determined that this did not impact the internal records of the SF-SAC collection, but it did impact how the data was copied out to our dissemination data tables. The dissemination tables power our search and API, and therefore impact all users of the data.
+
+### Example
+
+Some records in the `federal_awards` and `findings` tables can be replicated because of this bug. For example, report [2023-06-GSAFAC-0000020785]() encountered this bug. As a result, its award rows were duplicated. Below is a subset of rows from the `federal_awards` data table demonstrating how a set of awards were listed twice.
+
+|
+
+
+
+
+ {{ content | safe }}
+
+
+ report_id
| reference | prefix | extension |
+| -- | -- | -- | -- |
+| 2023-06-GSAFAC-0000020785 | AWARD-0001 | 93 | 959 |
+| 2023-06-GSAFAC-0000020785 | AWARD-0002 | 93 | 959 |
+| 2023-06-GSAFAC-0000020785 | AWARD-0003 | 93 | 778 |
+| 2023-06-GSAFAC-0000020785 | AWARD-0001 | 93 | 959 |
+| 2023-06-GSAFAC-0000020785 | AWARD-0002 | 93 | 959 |
+| 2023-06-GSAFAC-0000020785 | AWARD-0003 | 93 | 778 |
+
+Our data validation process would not allow a user to enter this data this way; each `AWARD-####` can only appear once. This error occurred when the data was copied into our dissemination tables (used by search and the API). Anyone requesting the federal awards for report `GSA-20785` will get two `AWARD-0001` rows, two `AWARD-0002` rows, and so on.
+
+## Consequences
+
+Searches for audits are not impacted, but calculations over the awards may yield incorrect results.
+
+## Mediating the error
+
+* In the `federal_awards` table, a given `AWARD` reference should only occur once. Filtering this down to a unique set will yield correct results.
+* In the `findings` table, a given `AWARD` and `REF` should only appear once in combination; filtering this combination of identifiers down to a unique set will yield correct results.
+
+## Possible resolution
+
+Remove the records from dissemination that contain this error, and re-disseminate the original record. This will yield correct data in the public-facing record.
+
+
+
diff --git a/src/data/concerns/2024-03-fy-off-by-one.md b/src/data/concerns/2024-03-fy-off-by-one.md
new file mode 100644
index 00000000..d9de7626
--- /dev/null
+++ b/src/data/concerns/2024-03-fy-off-by-one.md
@@ -0,0 +1,56 @@
+---
+layout: sidenav.njk
+tags: known_errors
+order: 3
+title: Migrated fiscal year dates off-by-one
+slug: The fiscal year start and end date in migrated records are one day early.
+discovered_date: 2024-03
+timeline_to_repair:
+github:
+log:
+---
+
+{% from "components/macros.njk" import curationSummary %}
+
+{{ curationSummary(slug, discovered_date, timeline_to_repair, github, log) }}
+
+
+## Curation challenge
+
+4. Our server records time in GMT.
+5. We treat all submissions as if they were in [SST](https://www.timeanddate.com/time/zones/sst) (Samoa Standard Time)
+
+What this means is that when someone submits data to the FAC, we look at the timestamp (in GMT), and subtract 11 hours (because we pretend the submission is coming from American Samoa). We do this to make sure that submissions from *everywhere* that might submit a Single Audit is considered "on time" as long as the grantee completes the submission on or before the day it is due.
+
+This transformation makes sense for new submissions to the GSA FAC. It did *not* make sense for data we were migrating from Census. Because we subtracted 11 hours from every acceptance date coming from Census, it had the effect of moving those dates back one day.
+
+The source of the error can be found in [`sac_creation.py` in the code](https://github.com/GSA-TTS/FAC/blob/c59eac9be7f6ca97f38fa66829d61f2a1144557a/backend/dissemination/workbooklib/sac_creation.py#L114).
+
+## Example
+
+Any record migrated from Census will demonstrate this issue. For example:
+
+| Report ID
| Entity | Incorrect date | Correct date |
+| -- | -- | -- | -- |
+| 2022-06-CENSUS-0000212928 | Gulf of Maine Research Institute | Dec. 27, 2022 | Dec. 28, 2022 |
+| 2022-06-CENSUS-0000091651 | Berea College | Nov. 22, 2022 | Nov. 23, 2022 |
+| 2022-06-CENSUS-0000211679 | Franklin W. Olin College of Engineering, Inc. | Nov. 16, 2022 | Nov. 17, 2022 |
+
+
+## Consequences
+
+
+This impacts searches for audits in past years when using the acceptance date as a criterion. An audit that was submitted one day late will now appear on-time.
+
+## Mediating the error
+
+
+End users can read fiscal year dates as one more than they are.
+
+## Possible resolution
+
+
+Update the records containing incorrect dates, and re-disseminate the record. This will yield correct data in the public-facing tables.
+
+
+
diff --git a/src/data/concerns/2024-04-incorrect-identifiers.md b/src/data/concerns/2024-04-incorrect-identifiers.md
new file mode 100644
index 00000000..e7e63f1d
--- /dev/null
+++ b/src/data/concerns/2024-04-incorrect-identifiers.md
@@ -0,0 +1,44 @@
+---
+layout: sidenav.njk
+tags: known_errors
+order: 5
+title: Incorrect report identifiers
+slug: Report IDs are based in part on (possibly incorrect) user-entered data.
+discovered_date: 2024-04
+timeline_to_repair:
+github:
+log:
+---
+
+{% from "components/macros.njk" import curationSummary %}
+
+{{ curationSummary(slug, discovered_date, timeline_to_repair, github, log) }}
+
+## Curation challenge
+
+Because of how/when we construct record ids, some of them are incorrect. For example, if a user creates an audit that (incorrectly) indicates a FY22 end date, we will (forever) report that audit submission as `2022-MM-GSAFAC-##########`.
+
+If we are going to assign semantics to this identifier (e.g. the first four digits are the audit’s fiscal year), then we should enforce that. Or, we should disclaim the semantics.
+
+
+### Example
+
+See above.
+
+## Consequences
+
+There is no indication to a resolution official what is the correct fiscal year for the audit in question when looking at the SF-SAC data. However, the report will contain the correct information.
+
+Fiscal year is not a common search term for resolution officials (from our research), and this is unlikely to have a significant impact on resolution/oversight.
+
+## Mediating the error
+
+There is no clear end-user mediation, other than looking at the Single Audit Report for the correct data.
+
+## Possible resolution
+
+We need a way to associate more than one report id with a record, and indicate which is “primary.” In this way, we can retain old (incorrect) ids with some records, and then associate a new (correct) id with it. The primary id should be used as the default in all views.
+
+This is difficult in dissemination, and may require changes to our model.
+
+
diff --git a/src/data/concerns/2024-07-acceptance-dates.md b/src/data/concerns/2024-07-acceptance-dates.md
new file mode 100644
index 00000000..3c8e7636
--- /dev/null
+++ b/src/data/concerns/2024-07-acceptance-dates.md
@@ -0,0 +1,54 @@
+---
+layout: sidenav.njk
+tags: known_errors
+order: 2
+title: Migrated accepted dates are incorrect
+slug: The accepted date in migrated records used the wrong historical date field.
+discovered_date: 2024-07
+timeline_to_repair:
+github:
+log:
+---
+
+{% from "components/macros.njk" import curationSummary %}
+
+{{ curationSummary(slug, discovered_date, timeline_to_repair, github, log) }}
+
+## Curation challenge
+
+As part of the move from Census to GSA, we migrated the SF-SAC records from 2016-2022. To complete this migration, we treated every record from Census as a submission to GSA. Unfortunately, Census record *multiple* acceptance dates for submissions:
+
+1. There was a `DATERECEIVED`, which was when an audit was submitted to the Census Clearinghouse.
+2. There was a `DATEACCEPTED`, which is when the audit was deemed to be acceptable for submission.
+3. There was a `DATEFIREWALL`, which is when the audit was published to agencies and the public.
+
+The GSA Clearinghouse only records one acceptance date, which we call `fac_accepted_date`. When bringing historical records forward, we used the Census field `FACACCEPTED`, believing it was equivalent to our `fac_accepted_date`. It is not; we should have, instead, used `DATEFIREWALL`. To further complicate the issue, we applied the [same transformation to this date as all other dates](migrated-acceptance-dates-incorrect), which means that we not only chose the wrong date, but we subtracted one from it.
+
+The error is in [end_to_end_core.py on line 69](https://github.com/GSA-TTS/FAC/blob/dda11bdfd31a000601e427379a3fac6ee9e7f1f8/backend/census_historical_migration/end_to_end_core.py#L69). This impacts all records migrated from Census.
+
+### Example
+
+Any record migrated from Census will demonstrate this issue. For example:
+
+| Report ID
| Entity | Incorrect date | Correct date |
+| -- | -- | -- | -- |
+| 2022-06-CENSUS-0000212928 | Gulf of Maine Research Institute | Dec. 27, 2022 | Jan. 3, 2023 |
+| 2022-06-CENSUS-0000091651 | Berea College | Nov. 22, 2022 | Nov. 30, 2022 |
+| 2022-06-CENSUS-0000211679 | Franklin W. Olin College of Engineering, Inc. | Nov. 16, 2022 | Nov. 28, 2022 |
+
+
+## Consequences
+
+
+This impacts searches for audits in past years when using the acceptance date as a criterion. In theory, an audit that was late may now appear as if it was on time.
+
+## Mediating the error
+
+If an agency has stored historical data from Census, those records could be consulted. Otherwise, there is no end-user mediation for this error. The FAC must correct the record.
+
+## Possible resolution
+
+Update the records containing incorrect dates, and re-disseminate the record. This will yield correct data in the public-facing tables.
+
+
+
diff --git a/src/data/concerns/atom.njk b/src/data/concerns/atom.njk
new file mode 100644
index 00000000..5e43c1b8
--- /dev/null
+++ b/src/data/concerns/atom.njk
@@ -0,0 +1,33 @@
+---
+permalink: "/feeds/atom/concerns.xml"
+eleventyExcludeFromCollections: true
+metadata:
+ title: "Known concerns in the FAC data"
+ description: "Known concerns int he FAC data"
+ language: en
+ base: "https://fac.gov/data/concerns"
+ author:
+ name: "Clearinghouse staff"
+---
+
+{{loop.index}}. {{item.data.title}}
+{{item.data.slug}}
+Details: {{item.data.title}}
+Discovered: {{item.data.discovered_date}}
+Timeline to repair: {% if item.data.timeline_to_repair %}{{item.data.timeline_to_repair}}{% else %}TBD{%endif%}
+Github issue: {% if item.data.github %}{{item.data.github}}{% else %}TBD{%endif%}
+Log: {% if item.data.log %}{{item.data.log}}{% else %}TBD{%endif%}
+ {% endif %} +{% endfor %} + diff --git a/src/data/concerns/rss.njk b/src/data/concerns/rss.njk new file mode 100644 index 00000000..cbca019b --- /dev/null +++ b/src/data/concerns/rss.njk @@ -0,0 +1,32 @@ +--- +permalink: "/feeds/rss/concerns.xml" +eleventyExcludeFromCollections: true +metadata: + title: "Known concerns in the FAC data" + description: "Known concerns int he FAC data" + language: en + base: "https://fac.gov/data/concerns" + author: + name: "Clearinghouse staff" +--- + +{{item.data.slug}}
+Related curation issue: {{- getKnownErrorTitle(collections.known_errors, item.data.concern) }}
+Decision record: {% if item.data.github %}{{item.data.github}}{% else %}TBD{%endif%}
+Repair date: {{item.data.repair_date}}
+ {% endif %} +{% endfor %} + diff --git a/src/data/curation-log/rss.njk b/src/data/curation-log/rss.njk new file mode 100644 index 00000000..e3662dca --- /dev/null +++ b/src/data/curation-log/rss.njk @@ -0,0 +1,32 @@ +--- +permalink: "/feeds/rss/curation-log.xml" +eleventyExcludeFromCollections: true +metadata: + title: "Log of Federal Audit Clearinghouse curation actions" + description: "Log of Federal Audit Clearinghouse curation actions" + language: en + base: "https://fac.gov/data/curation-log" + author: + name: "Clearinghouse staff" +--- + +Table: {{tab}}
+
+
+
+ {% if tab == "general" %}
+
+{%- endfor -%}
diff --git a/src/data/index.md b/src/data/index.md
new file mode 100644
index 00000000..94cfcdcc
--- /dev/null
+++ b/src/data/index.md
@@ -0,0 +1,58 @@
+---
+layout: sidenav.njk
+title: Using FAC data
+meta:
+ name: Historic data curation
+ description: Documentation about the data migration from Census to GSA.
+---
+
+# About our data
+
+The Federal Audit Clearinghouse (FAC) collects and shares data collected under the Single Audit Act. The Single Audit Act requires an annual audit of recipients of Federal Financial Assistance that expend ${{glossary.variables.threshold}} or more in a fiscal year. If you want to learn more, we recommend the [excellent overview of Single Audits]({{glossary.sites.hhs_oig_sa.url}}) developed by the Office of the Inspector General at Health and Human Services.
+
+There are two pieces to a Single Audit submission to the FAC: the **form** and the **report**. We provide a way for you to [search the data on the WWW]({{glossary.sites.basic_search.url}}), and you can also [write code to download and analyze the data using our API]({{glossary.sites.api.url}}).
+
+The data collected by the FAC is **free to use and in the public domain**.
+
+## Who uses this data?
+
+Many people use this data in many ways. For example:
+
+1. **Members of the public** explore how Federal dollars are spent locally and nationally
+2. **Reporters** report on Federal spending to local, national, and global audiences
+3. **Grants officers** review grantee performance as part of the allocation of funds
+4. **Resolution officials** provide oversight of Federal spending from a programmatic and agency perspective
+5. **Inspectors General** oversee the quality of Single Audits
+
+## How do I access the data?
+
+
+There are many audiences for the data provided by the FAC. If you're exploring the data for the first time, we recommend you start with our web search. If you're a programmer or data scientist looking to dive deep, we recommend our API. If you rely on this data, you will also want to look at how we care for, or **curate**, the data year-over-year.
+
+### Access via the WWW
+
+There are two search interfaces:
+
+* [Basic search]({{glossary.sites.basic_search.url}}): Use this if you are just starting out. Look for your hometown, or a school you've attended.
+* [Advanced search]({{glossary.sites.advanced_search.url}}): Use this if you are starting to dig deeper; perhaps look for your home state, and discover what entities had audit findings in your state.
+
+### Access via code (API)
+
+API stands for **Application Programming Interface**. APIs make it easy for computer programs to request and receive information in a useable format. The FAC provides data that is easily computer-readable so that you can search and incorporate SF-SAC data directly into your own analyses.
+
+[Get started]({{glossary.sites.api.url}}) with our API.
+
+## How is the data curated?
+
+Caring for a data collection of this size requires ongoing care. We, or our users, discover errors (and, sometimes, introduce them). When this happens, we follow a process to document, schedule, and repair those inconsistencies or errors in the data. Museums curate arts and antiquities; at the FAC, we curate the record of the Single Audit.
+
+In (Federal) Fiscal Year 2024, we migrated data from 2016-2022 from Census to GSA. This was a significant effort, and we users of our data read about that that transition to understand what it might mean for you.
+
+You can read more about:
+
+* [Validation](validations/): How we validate the data we collect
+* [Migration](migration/): The migration of FAC data from Census to GSA
+* [Concerns](concerns/): Known concerns and inconsistencies in the data
+* [Curation](curation/): The curation process
+* [Curation Log](curation-log/): The log of changes made
+
diff --git a/src/data/migration.md b/src/data/migration.md
new file mode 100644
index 00000000..711fdad4
--- /dev/null
+++ b/src/data/migration.md
@@ -0,0 +1,50 @@
+---
+layout: sidenav.njk
+title: Historic data migration
+meta:
+ name: Historic data migration
+ description: Documentation about the data migration from Census to GSA.
+---
+
+# {{title}}
+
+A critical part of the move of the Clearinghouse from Census to GSA was the migration of the data. GSA began collecting Single Audits in October of 2023; we completed migration of 98% of the data from Census in February of 2024, and the final 2% in July of 2024.
+
+The GSA team treated this work as an act of [data curation](https://en.wikipedia.org/wiki/Data_curation). Instead of copying the data over "as-is," we instead worked to annotate and, where possible, improve the data through the migration process.
+
+## Migration process
+
+In the simplest possible terms: we re-submitted all prior audits collected at Census to the GSA Clearinghouse.
+
+1. We began with the data from an old SF-SAC record from the Census data.
+2. We constructed a new, virtual SF-SAC submission, using the [workbook templates]({{config.baseUrl}}audit-resources/sf-sac/) that auditors and auditees use for their data entry at GSA.
+3. We submitted this virtual data collection form to the FAC.
+4. Where the data fell short of our validations in manageable/measurable ways, we annotated the data and migrated it into the FAC.
+5. Where an SF-SAC was too incomplete to consider usable, we set it aside for further processing.
+
+This work was completely automated, from the analysis through to the annotation. All data was accounted for, regardless of whether it was annotated and accepted, or set aside.
+
+We did our initial migration of data in February of 2024, and 98% of the data (approximately 277K records) were migrated successfully. The remaining 2% (roughly 3200 records) required further processing, and those were integrated into the Clearinghouse in July of 2024.
+
+Two audits were sufficiently incomplete as to not be migrated. We have made provisions for those audit reports to still be found via search, but there is no SF-SAC associated with them.
+
+No part of the historical record was lost in this process.
+
+### Identifiers, old and new
+
+Census had a system by which a `DBKEY` was assigned to each audit as it came in. The `AUDITYEAR` and `DBKEY` effectively formed a unique identifier. This identifier was manually assigned as each audit came in.
+
+GSA's intent is that an audit's report identifier should be automatically assigned. Once assigned, will never change.
+
+When migrating historical records, all audit report packages from Census were given a new-style report ID. For example, Berea College submitted an audit in 2023, and their audit had the record identifier [2023-06-GSAFAC-0000020471](https://app.fac.gov/dissemination/summary/2023-06-GSAFAC-0000020471); this clearly denotes the record as being collected by `GSAFAC`. For their submission in 2022 was given the identifier [2022-06-CENSUS-0000091651](https://app.fac.gov/dissemination/summary/2022-06-CENSUS-0000091651), which clearly denotes it as a record that originated with Census.
+
+In constructing new-style identifiers for `CENSUS` records, we used the `AUDITYEAR` for the first portion of the id (e.g. `2022`), and the `DBKEY` for the last portion of the report id (e.g. `0000091651`).
+
+### Transforms and annotations
+
+It is possible to see the annotation `GSA_MIGRATION` in some records at this time (last updated February 2024). The GSA team is working, as part of this curatorial work, to make the original records available for reference where this annotation was applied.
+
+{#
+* [General data transformations]({{ "/data/general-transforms/" | absoluteUrl }})| Census | +GSA | +Data type | +
|---|---|---|
| {{line.Census}} | +{{line.GSA}} | +{{line.Transformations | safe }} | +
† The dissemination logic converts booleans to a “yes” or “no” text instead, this change is unrelated to the migration logic.
+ +††: The dissemination logic converts this date from utc time zone to american samoa zone. This is transformation is unrelated to the migration logic
+ {% endif %} +These are transformations that were applied wholesale to files as data was migrated from Census to GSA. +* [Specific field transformations]({{"/data/specific-transforms/" | absoluteUrl }})
These are transformations applied to specific portions of the data. For example, a Census record that is missing an entire portion of the form required specialized handling. +#} \ No newline at end of file diff --git a/src/data/specific-transforms.md b/src/data/specific-transforms.md new file mode 100644 index 00000000..eb50710f --- /dev/null +++ b/src/data/specific-transforms.md @@ -0,0 +1,9 @@ +--- +layout: sidenav.njk +title: Specific migration transforms +meta: + name: Specific migration transforms + description: Documentation about the data migration from Census to GSA. +--- + +# {{title}} diff --git a/src/data/validations/cross-validations.md b/src/data/validations/cross-validations.md new file mode 100644 index 00000000..e1367994 --- /dev/null +++ b/src/data/validations/cross-validations.md @@ -0,0 +1,61 @@ +--- +layout: sidenav.njk +title: SF-SAC cross-validations +meta: + name: SF-SAC cross-validations + description: Documentation about the data migration from Census to GSA. +baseurl: https://github.com/GSA-TTS/FAC/tree/414f9932b3797de68c7271ba0b471c9b834062ce/backend/audit/cross_validation +--- + +# {{title}} + +These validations are the penultimate line of defense before a submission is considered complete. They are the validations that look across all of the form sections, and attempt to help auditors and auditees submit consistent, correct data. + +## Why cross-validations? + +The SF-SAC expects grantees and auditors to report on data that is interconnected. For example: + +1. A grantee might have an award from an agency. They need to report ont that award. +2. If the auditor found something of concern, then there are findings that have to be described. Findings are associated with awards. +3. A finding will have a corrective action plan. That plan is described in reference to both an award and a finding. + +These connections are critical to oversight officials; for that reason, we do our best to help auditees and auditors describe these connections in the SF-SAC correctly. The cross-validations are how we support users in this kind of inter-form data entry. + +## What are the cross-validations? + +There are fewer cross-validations than intake validations, but they play a role every bit as critical. For example: + +* The check `auditee_ueis_match` makes sure that every form section has the same UEI in it. This way, if an auditor is supporting multiple submissions, we don't actually end up with form sections from different grantees. +* The check `check_certifying_contacts` make sure that one person did not attempt to certify as both the auditor and auditee certifying officials. +* The check `tribal_data_sharing_consent` makes sure that the consent to distribute the audit report was given if the entity is marked as being a Tribe or Tribal Organization. + +Below is a list of all of the cross validations applied to the SF-SAC as of July 2024. + +
+
+
+
+
+
+## From intake to dissemination
+
+Once these validations pass, the auditor and auditee can certify the correctness of their submission, and the auditee can perform the final submission. This is, unto itself, a final kind of validation. See how the [dissemination validations]({{ "../dissemination-validations/" | absoluteUrl }}) play a final role.
diff --git a/src/data/validations/dissemination-validations.md b/src/data/validations/dissemination-validations.md
new file mode 100644
index 00000000..a699eec8
--- /dev/null
+++ b/src/data/validations/dissemination-validations.md
@@ -0,0 +1,34 @@
+---
+layout: sidenav.njk
+title: FAC dissemination validations
+meta:
+ name: FAC dissemination validations
+ description: Documentation about the data migration from Census to GSA.
+---
+
+# {{title}}
+
+
+Dissemination| Validation | +Description | +
|---|---|
| {{check.check}} | +{{check.description}} | +
+Valid and free from failure
+Except when it's not.
+ + +The dissemination of data serves as one final check that we have all of the data we expect, and that it is of the correct type. It is unlikely at this point that we will detect an error. However, if we do, the dissemination of the data fails, allowing the FAC team and grantee/auditor understand why this happened, and resolve the error before passing data on to agencies and the public. + + +## From intake to validation + +The last step in an SF-SAC submission is to go from these [internal tables to our external dissemination tables](https://github.com/GSA-TTS/FAC/blob/44ae639f70c4552fa584f7042e0c116b0cd93d29/backend/audit/intake_to_dissemination.py). From a systems perspective, we continue to proceed defensively, making no assumptions that all is well with the submission. + +The FAC has an internal database for storing a submission; this is the JSON representation that reflects the shape of the SF-SAC. Once a submission is complete, we convert this into another shape for dissemination. Long-time users of the FAC will recognize this as the `general` table, the `federal_awards` table, the `findings` table, and so on. Our [API]({{"/api" | absoluteUrl}}) is structured around this shape, for consistency of experience for our users. + + +In converting data from our internal to external representation, we make sure all of the data is where we expect it. And, if the wrong *kind* of data is somewhere, it will fail to be written to our dissemination tables. In this regard, we make sure that one of two things is true: + +1. Everything is in the correct place, and can be written out for agencies to begin their work, or +2. Everything isn't, and the submission fails. + +We have not seen this backstop triggered for a very long time. However, it is the final set of checks that are performed on the SF-SAC before we consider it a complete submission. + +*Back to the [FAC data overview]({{"/data" | absoluteUrl}})*. \ No newline at end of file diff --git a/src/data/validations/index.md b/src/data/validations/index.md new file mode 100644 index 00000000..617632c6 --- /dev/null +++ b/src/data/validations/index.md @@ -0,0 +1,51 @@ +--- +layout: sidenav.njk +title: FAC data validations +meta: + name: FAC data validations + description: Documentation about the data migration from Census to GSA. +--- + +# {{title}} + +The data coming into the FAC is validated in five distinct ways. + +## In-form validations + +The SF-SAC is made up of a collection of web forms and spreadsheets that are filled out by grantees and auditors. Within both the web forms and spreadsheets, we have input restrictions and validations that help people complete the forms correctly. + +Because it is difficult to add validations to spreadsheets in a way that works everywhere, this is one of the simplest validation layers in the FAC. + +## Intake validations + +The second layer of data validation is made up of many {# [intake validations]({{"intake-validations/" | absoluteUrl }}) #} intake validations. Here, we transform the spreadsheet or form a user submits into a shape that is more manageable for valiation. We then apply many validations, one after another, to help ensure data quality. Where we find errors, we do our best to provide error messages that guide people to a correct submission. + +{# +Read more about our [intake validations]({{"intake-validations/" | absoluteUrl }}). +#} + +## Schema validations + +A *schema* describes something's *form* or *shape*. For example, we might say the schema for a zip code is five digits followed by an optional hyphen and four more digits. A *schema validation* is where we look at a piece of data, and ask if everything about it has the right shape. + +Every form section in the SF-SAC has a shape. That shape is made up of the fields (names, zip codes, email addresses), and each piece of data in each field has a shape of its own. The SF-SAC schema validations make sure that a submission, after going through our intake validations, ends up in the right shape for storage. + +{# +Read more about our [schema validations]({{"schema-validations/" | absoluteUrl }}). +#} + +## Cross-validations + +Because the SF-SAC is made up of multiple sections, we have to make sure the sections make sense with respect to each-other. Cross-validations look at the submission as a whole, and make sure everything is consistent *between* the SF-SAC sections. + +{# +Read more about our [cross-validations]({{"cross-validations/" | absoluteUrl }}). +#} + +## Dissemination validations + +The final step is to make sure that everything is ready for sharing with agencies and the public. This means we make sure every piece of data we're expecting as part of an SF-SAC submission is present and accounted for. Our goal is to block as many errors as possible; if we don't, then we'll have to [document the errors that get past us and fix them]({{"../curation/" | absoluteUrl}}). + +{# +Read more about our [dissemination validations]({{"dissemination-validations/" | absoluteUrl }}). +#} \ No newline at end of file diff --git a/src/data/validations/intake-validations.md b/src/data/validations/intake-validations.md new file mode 100644 index 00000000..853ea7a2 --- /dev/null +++ b/src/data/validations/intake-validations.md @@ -0,0 +1,76 @@ +--- +layout: sidenav.njk +title: SF-SAC intake validations +meta: + name: SF-SAC intake validations + description: Documentation about the data migration from Census to GSA. +# baseurl: https://github.com/GSA-TTS/FAC/tree/44ae639f70c4552fa584f7042e0c116b0cd93d29/backend/audit/intakelib/checks +baseurl: https://github.com/GSA-TTS/FAC/blob/414f9932b3797de68c7271ba0b471c9b834062ce/backend/audit/intakelib/checks/ +--- + +# {{title}} + +Intake validations are the first layer of validation that the SF-SAC undergoes as it is uploaded to the FAC. In order to validate SF-SAC form sections, we do three things: + +1. Convert the spreadsheet into JSON +2. Validate the data in this intermediate JSON form +3. Convert the data to a meaningful semantic representation + +Each of these steps helps ensure a better submission to the FAC. + + +### Reporting errors + +Intake validations are critical for several reasons. + +1. **They are user-centered**.
Each validation produces errors that are human-friendly. They are what help auditors and grantees fix errors they may have made. Our messages specify errors down to the specific cell, and are refined for usability based on user feedback. +2. **They are self-contained**.
Each validation does only one thing. This helps the GSA team with maintaining the validations. +3. **They are comprehensive**.
The intake validations are intended to cover all possible error conditions we might encounter. In the event that an auditor or grantee finds a new, surprising way to enter data into our forms... we add a new intake validation. + +Ultimately, our goal is for the intake validations to provide a good user experience when part of the SF-SAC is completed incorrectly while simultaneously ensuring that we colelct quality data via the form. + +## Step 1: Convert the spreadsheet into JSON + +Spreadsheets are first converted into JSON, or [Javascript Object Notation]({{glossary.sites.json.url}}). Internally, this is referred to as the *intermediate representation*. This intermediate representation is a small language that uses named ranges to define regions of data. We describe this representation more fully as part of our [intakelib](https://github.com/GSA-TTS/FAC/blob/414f9932b3797de68c7271ba0b471c9b834062ce/backend/audit/intakelib/README.md) documentation. + +This conversion is a validation of its own kind; if we cannot convert the input to our intermediate representation, it is obviously not an SF-SAC upload! The SF-SAC forms must be saved in an Excel 2007 XML-based format (or `xlsx`). We chose this because many current spreadsheet manipulation tools (e.g. [LibreOffice]({{glossary.sites.libreoffice.url}})) are capable of saving data in this format. This means that it is possible to prepare an SF-SAC submission using free and open source software. + +## Step 2: Validate the data + +The intermediate form makes the data validation easier. Every section of the SF-SAC has a common set of validations applied. For example, we make sure every form section has a UEI defined. Then, we apply section-specific validations. Federal Awards is the most complex form section, and therefore many of our data validations are applied to that portion of the submission. + +
+
+
+
+
+
+## Step 3: Convert the data to a meaningful form
+
+After intake validations are applied, we then transform the data into its storage form. This is another JSON representation, but it is more meaningful. Instead of representing the spreadsheet, it represents the meaningful connections between portions of the data. This form is then further validated by our [schema validations]({{ "../schema-validations/" | absoluteUrl }}).
\ No newline at end of file
diff --git a/src/data/validations/schema-validations.md b/src/data/validations/schema-validations.md
new file mode 100644
index 00000000..853a42ae
--- /dev/null
+++ b/src/data/validations/schema-validations.md
@@ -0,0 +1,158 @@
+---
+layout: sidenav.njk
+title: SF-SAC schema validations
+meta:
+ name: SF-SAC schema validations
+ description: Documentation about the data migration from Census to GSA.
+---
+
+# {{title}}
+
+Once we are done with our [intake validations]({{"../intake-validations/" | absoluteUrl}}), we convert our spreadsheet-oriented data representation into something that better reflects the meaning of the data we're collecting. But, before we store that data, we apply a *schema validation* to that data.
+
+Schema validations involve validating the overall shape of the data, from the big picture all the way down to the details.
+
+## Data has a shape?
+
+All data has a shape. Simple data has simple shapes.
+
+* We could say that counting numbers are just digits. `3`, `5`, and `8` are all counting numbers.
+* We might say that currency has a currency indicator, some number of digits, a decimal point, and two digits after the decimal point. For example, `$3.14` and | Form section | +Validation | +Description | +
|---|---|---|
| {{section.form}} | ++ | + |
| + | {{check.check}} | +{{check.description}} | +
£1.37 would then both be examples of numbers that represent currency.
+* A zip code might be described as five digits, and sometimes it is allowed to have a hyphen and another four digits. When we describe zip codes as having this shape, both `40404` and `02492-1200` are zip codes.
+
+More complex data has more complex shapes. In some parts of the world, people think names have three parts: a first name, a middle name, and a surname.
+
+```
+{
+ "first_name": "Alexander",
+ "middle_name": "",
+ "surname": "Hamilton"
+}
+```
+
+Oh no! We found someone without a middle name! Should we... leave it blank? Do we validate that name as... correct? (This is the kind of question the FAC team has to ask *all the time*.) The point is that a *name* is something that has *parts*, in a way that numbers usually do not.
+
+In the same spirit, the FAC converts the data collected via spreadsheet into a structure that gives meaningful names to each part of the data, and groups parts of the data where it makes sense to. Below is an example of part of one submission's Federal award data.
+
+```
+"FederalAwards": {
+ "auditee_uei": "ABCDEFGHIJKL",
+ "federal_awards": [
+ {
+ "cluster": {
+ "cluster_name": "RESEARCH AND DEVELOPMENT",
+ "cluster_total": 3419734
+ },
+ "program": {
+ "is_major": "Y",
+ "program_name": "FISHERIES DEVELOPMENT AND UTILIZATION
+ RESEARCH AND DEVELOPMENT GRANTS AND
+ COOPERATIVE AGREEMENTS PROGRAM",
+ "amount_expended": 32261,
+ "audit_report_type": "U",
+ "federal_agency_prefix": "11",
+ "federal_program_total": 209726,
+ "three_digit_extension": "427",
+ "number_of_audit_findings": 0,
+ "additional_award_identification": "NA15NMF4270262"
+ },
+ "subrecipients": {
+ "is_passed": "N"
+ },
+ "award_reference": "AWARD-0001",
+ "loan_or_loan_guarantee": {
+ "is_guaranteed": "N"
+ },
+ "direct_or_indirect_award": {
+ "is_direct": "Y"
+ }
+ },
+ ...
+```
+
+This representation, again using the [Javascript Object Notation](https://www.json.org/json-en.html), connects part of the data in meaningful ways. For example, we cluster information about the cluster and total cluster expenditures separate from information about the specific Federal program that the funding was issued under. Every form section has its own JSON representation, which reflects the SF-SAC and the meaningful connections between the data in that specific section.
+
+## Validating the shape
+
+The FAC uses a technology called [JSON Schemas](https://json-schema.org/) to validate this complex data shape. JSON Schemas give us a way of saying "we expect a Federal awards sheet to have a UEI, a list of awards, and a total expended." The schema below does exactly that.
+
+```
+local FederalAwards = Types.object {
+ additionalProperties: false,
+ properties: {
+ auditee_uei: Base.Compound.UniqueEntityIdentifier,
+ federal_awards: Types.array {
+ items: FederalAwardEntry,
+ },
+ total_amount_expended: Types.number,
+ },
+ required: ['auditee_uei', 'total_amount_expended'],
+ title: 'FederalAward',
+ version: 20230408,
+};
+```
+
+This tiny bit of the schema tells us that a valid JSON document representing Federal Awards will have three pieces:
+
+1. An `auditee_uei`, which will follow rules for what makes a valid `UniqueEntityIdentifier`
+2. A list of `federal_awards`, where each element of the list will look like a `FederalAwardEntry`
+3. A `total_amount_expended`, which should be a number.
+
+The total expended has a simple shape; it is a number. The `FederalAwardEntry` is yet another complex shape that has to be validated; if any one award entry does not validate, the entire thing fails. Similarly, we have to break the UEI down, because UEIs are actually pretty complex, even though they are only 12 characters long. Here are all of the rules that have to be applied to validate a UEI:
+
+```
+local REGEX_UEI_ALPHA = 'A-HJ-NP-Z';
+local REGEX_UEI_LEADING_CLOISTER = '[' + REGEX_UEI_ALPHA + '1-9]';
+local REGEX_UEI_BODY_CLOISTER = '[' + REGEX_UEI_ALPHA + '0-9]';
+
+local type_uei = Types.string {
+ allOf: [
+ // Is a string
+ {
+ minLength: 12,
+ maxLength: 12,
+ },
+ {
+ pattern: '^'
+ + REGEX_UEI_LEADING_CLOISTER
+ + REGEX_UEI_BODY_CLOISTER
+ + '+$',
+ },
+ // Does not have 9 digits in a row
+ {
+ pattern: '^(?!'
+ + REGEX_UEI_LEADING_CLOISTER
+ + '+'
+ + REGEX_UEI_BODY_CLOISTER
+ + '*?[0-9]{9})'
+ + REGEX_UEI_BODY_CLOISTER
+ + '*$',
+ },
+ // Does not start with 9 digits in a row
+ {
+ pattern: '^(?![0-9]{9})',
+ },
+ ],
+};
+```
+
+That UEI validation makes sure that UEIs entered into our system follow the rules defined for UEIs by SAM.gov. First, we make sure it is a string of length 12. Then, we make sure it starts with a valid letter. Moreso, we make sure there are not 9 digits in a row in the middle of the UEI; this is to prevent confusion with EINs and SSNs. Every piece of data coming into the FAC undergoes this level of scrutiny via our JSON Schema validations.
+
+The [complete schema for the Federal Awards](https://github.com/GSA-TTS/FAC/blob/826167b1e4142f6723177a161e60858053a99c16/backend/schemas/source/sections/FederalAwards.schema.jsonnet) section is several hundred lines long. It specifies everything we expect to see in this section of the form. We have a schema for every part of the data that comes into the FAC.
+
+Every section of the SF-SAC has a schema just like this, making sure that the shape of the data in every section is validated all the way down.
+
+## "You shall not pass"
+
+When the GSA FAC first opened its doors, we relied heavily on our JSON Schemas. They performed *marvelously*. We blocked all kinds of submissions.
+
+However, the error messages were *horrible*. Completely incomprehensible, sometimes even to the GSA FAC team. This is why the [intake validations](intake-validations/) came into being. At this point, the GSA team expects:
+
+1. A user should never see an error from a JSON Schema. If that happens, we add a new intake validation with a user-centered, helpful error message.
+2. JSON Schemas should be our backstop; they make sure, after transforming our data, that we made no mistakes.
+
+These expectations have served the team well. Users get good support from the messages provided by our intake validations, and the schemas have prevented us from storing data that is "wrong" in shape.
+
+## Cross-validations
+
+Once we have a complete submission, we check for correctness across the entire submission. We call these [cross-validations]({{ "../cross-validations/" | absoluteUrl }}).
\ No newline at end of file
diff --git a/src/updates/2023-09-25-updates.md b/src/updates/2023-09-25-updates.md
new file mode 100644
index 00000000..7f55e32f
--- /dev/null
+++ b/src/updates/2023-09-25-updates.md
@@ -0,0 +1,25 @@
+---
+layout: home.njk
+title: FAC Updates - Week of Sept 25, 2023
+date: 2023-09-25
+tags: updates
+---
+
+# {{title}}
+
+We're excited to see completed audits coming in and data exporting via the FAC API. That said, we know there have been some issues. We're doing our best to provide help center support for those as fast as possible.
+
+## Items of note:
+
+- Some auditors and auditees experienced errors when trying to validate their UEIs. We identified the cause of the issue with sam.gov and believe we have implemented a fix.
+- We are now collecting single audit submissions from Indian Tribes and Tribal organizations.
+- Our engineering team improved workbook validations and addressed a bug in the FAC API.
+
+## What's next?
+
+We know that not every tool you are used to is currently available, and we are working quickly to add features in the coming months.
+
+Our goal is to be transparent and share our areas of focus. You can see what features are coming up in [our backlog](https://github.com/orgs/GSA-TTS/projects/13/views/1).
+
+We are currently working on [search](https://github.com/GSA-TTS/FAC/issues/2236). This will give Federal and public users the ability to find and download completed audits. We track our [day-to-day work on our task board](https://github.com/orgs/GSA-TTS/projects/11).
+
\ No newline at end of file
diff --git a/src/updates/2023-10-09-updates.md b/src/updates/2023-10-09-updates.md
new file mode 100644
index 00000000..e9b52b29
--- /dev/null
+++ b/src/updates/2023-10-09-updates.md
@@ -0,0 +1,32 @@
+---
+layout: home.njk
+title: FAC Updates - Week of Oct 9, 2023
+date: 2023-10-09
+tags: updates
+---
+
+# {{title}}
+
+We released the GSA FAC as a minimal viable product. That means we are continuing to work and improve the FAC. We work in [an agile manner](https://asana.com/resources/agile-methodology).
+
+The GSA FAC team worked to minimize change between submission systems. We have worked closely with auditors, auditees, and Federal users to achieve this.
+
+## Items of note:
+
+- We've updated [our UEI validation criteria](https://support.fac.gov/hc/en-us/articles/20299116907149-UEI-validation) to only require that an entity's UEI exists. This should reduce the number of UEI errors users experience.
+- We know some users have found the error messages for workbook validation confusing. We are working to improve these messages.
+
+## What's next?
+
+We know that not every tool you are used to is currently available, and we are working quickly to add features in the coming months.
+
+Our goal is to be transparent and share our areas of focus. You can see what features are coming up in [our backlog](https://github.com/orgs/GSA-TTS/projects/13/views/1).
+
+We are currently working on [search](https://github.com/GSA-TTS/FAC/issues/2236). This will give Federal and public users the ability to find and download completed audits. Our goal is to have a first version of search and download available by the end of October.
+
+Our other feature priorities are:
+- Adding the ability to unlock audit submissions to edit and then re-lock for certification instead of starting a new submission.
+- Adding the ability to add, remove, or change auditors and certifying officials who are attached to the audit.
+- Adding the ability to resubmit a previously submitted and completed audit.
+- Adding email notifications to alert auditees and auditors that a step is ready for them to complete.
+
\ No newline at end of file
diff --git a/src/updates/2023-10-16-updates.md b/src/updates/2023-10-16-updates.md
new file mode 100644
index 00000000..6a83ee8c
--- /dev/null
+++ b/src/updates/2023-10-16-updates.md
@@ -0,0 +1,28 @@
+---
+layout: home.njk
+title: FAC Updates - Week of Oct 16, 2023
+date: 2023-10-16
+tags: updates
+---
+
+# {{title}}
+
+We released the GSA FAC as a minimal viable product. That means we are continuing to work and improve the FAC. We work in [an agile manner](https://asana.com/resources/agile-methodology).
+
+## Items of note:
+
+- All UEIs are validating.
+- We made significant improvements to workbook validations and their associated instructions.
+
+If you previously encountered challenges submitting your workbooks, we recommend you try again.
+
+## What's next?
+
+We are currently working on [search](https://github.com/GSA-TTS/FAC/issues/2236). We hope to demonstrate a prototype to Federal partners this week for early feedback. We anticipate having the ability to search for and download audits by the end of October.
+
+Our pending feature priorities are:
+- Adding the ability to unlock audit submissions to edit and then re-lock for certification instead of starting a new submission.
+- Adding the ability to add, remove, or change auditors and certifying officials who are attached to the audit.
+- Adding the ability to resubmit a previously submitted and completed audit.
+- Adding email notifications to alert auditees and auditors that a step is ready for them to complete.
+
\ No newline at end of file
diff --git a/src/updates/2023-10-23-updates.md b/src/updates/2023-10-23-updates.md
new file mode 100644
index 00000000..055e06de
--- /dev/null
+++ b/src/updates/2023-10-23-updates.md
@@ -0,0 +1,146 @@
+---
+layout: home.njk
+title: FAC Updates - Week of Oct 23, 2023
+date: 2023-10-23
+tags: updates
+---
+
+# {{title}}
+
+The FAC participated in the Single Audit Roundtable, providing an update on our work to date and planned next steps, and used the opportunity to discuss pressing needs for auditors, resolution officials, and IGs.
+
+We released the GSA FAC as a minimal viable product. That means we are continuing to work and improve the FAC.
+
+## What we delivered
+
+We work in [an agile manner](https://asana.com/resources/agile-methodology). That means we have a long-term strategy, medium-term features we work to deliver, and make continuous improvement and bug fixes to the existing product. We update our system through pull requests, or PRs.
+
+- Features add new capabilities or documentation to the FAC
+- Improvements take existing code or text and increase its quality
+- Validations increase the integrity of the data collected or disseminated by the FAC.
+
+| PR | +Type | +Description | +
|---|---|---|
| 2569 | ++ Feature + | +Add system command for generating test data in production environments | +
| 2572 | ++ Feature + | +Allow users to unlock submissions once locked | +
| 2610 | ++ Feature + | +Create backup of media files | +
| 2617 | ++ Feature + | +Test data for historical data migration | +
| 2627 | ++ Feature + | +Prepare workbook generator for use in historical migration | +
| 2563 | ++ Improvement + | +Test data generator improvements to align with data intake checks | +
| 2568 | ++ Improvement + | +Code simplification in end-to-end test environment | +
| 2571 | ++ Improvement + | +Improve data generation command for reuse across environments | +
| 2595 | ++ Improvement + | +Copy improvements and styling for submission unlock | +
| 2596 | ++ Improvement + | +Remove unused application code | +
| 2613 | ++ Improvement + | +Fix PDF download infrastructure | +
| 2624 | ++ Improvement + | +Update software versions for deployment automations | +
| 2586 | ++ Validation + | +Assert all data ranges expected are present in workbooks | +
| PR | +Type | +Description | +
|---|---|---|
| 2725 | ++ Feature + | +Foundation for historic data migration | +
| 2681 | ++ Feature + | +Reporting infrastructure rollout | +
| 2764 | ++ Improvement + | +Fixes to workbook generation for historic data migration | +
| 2724 | ++ Improvement + | +Improvements to the initial release of search | +
| 2734 | ++ Improvement + | +Fixes to search on ALNs | +
| 2733 | ++ Improvement + | +Consistent linking of search from fac.gov and application | +
| 2728 | ++ Improvement + | +Cleanup of documents regarding product decisions | +
| 2683 | ++ Improvement + | +Pagination in search results | +
| 2663 | ++ Improvement + | +Updating privacy ACLs after AWS S3 changes | +
| 2642 | ++ Improvement + | +Inexact matches on search fields | +
| 2651 | ++ Validation + | +Placeholder for future workbook version checks | +
| 2640 | ++ Validation + | +Improved validation around spaces in T3 | +
| 2766 | ++ Decision + | +User permission design for tribal audit access | +
| 2749 | ++ Decision + | +New process for product decision making | +
| 2750 | ++ Decision + | +Revision to decision making process | +
| 2686 | ++ Debt + | +Removed “cms,” unused portions of code | +
+
+
+The Clearinghouse typically sees between 700 and 900 audit submissions per day at the end of March. In 2022, the Census Bureau saw close to 2000 audits submitted in one day.
+
+We're doing many things to prepare for this submission volume. Since our launch, we've run two "copies" of the FAC to handle submissions. This means that, if two people are uploading audits to the FAC, one might get "FAC 1," and the other "FAC 2," so they both experience good web performance. As we go into March, we are increasing that number to 10, and can double that to 20 if necessary. This way, if 10 people are uploading data at the same time, they will each experience the same level of performance.
+
+This is possible because the FAC is built on top of technologies like [cloud.gov](https://cloud.gov) and [Login.gov](https://login.gov).
+
+## What's next?
+
+We're keeping our focus on March. Looking ahead to the next few months, we'll be:
+
+- Focusing on performance improvements and growing our team’s capacity for user support.
+
+- Providing documentation on the migration of historical and how to use it.
+
+- Improving search to support Federal oversight work and the public’s use of FAC data.
diff --git a/src/updates/2024-04-15-updates.md b/src/updates/2024-04-15-updates.md
new file mode 100644
index 00000000..a13e5618
--- /dev/null
+++ b/src/updates/2024-04-15-updates.md
@@ -0,0 +1,46 @@
+---
+layout: home.njk
+title: FAC Updates - Week of April 15, 2024
+date: 2024-04-15
+meta:
+ name: FAC Updates - Week of April 15, 2024
+ description: Read historical updates from the Federal Audit Clearinghouse.
+tags: updates
+---
+
+# {{title}}
+
+### March peak season
+
+The FAC received 5,403 single audit submissions between March 25 and April 1. During this time, [our helpdesk](https://support.fac.gov/hc/en-us/requests/new) resolved more than 400 tickets, most via one-touch and in less than 24 hours.
+
+
+ 
+ 
+ 
+ 
+
+
+ 
+ 
+ 
+
+
+
+
+
+As the rate of submissions tapers off, the team is returning its focus to feature delivery and site maintenance.
+
+## Tribal API
+
+Tribal audits are now available to authorized users [via the API]({{ config.baseUrl }}api/tribal). Documentation on this process has been shared with interested agencies. If you would like to learn more about access to the API, please contact the FAC helpdesk.
+
+## Advanced search
+
+At the end of March, we experienced a performance outage for the FAC search. To resolve this, the team created [an advanced search option](https://app.fac.gov/dissemination/search/advanced/) to better serve audit resolution officials. A more [basic search option](https://app.fac.gov/dissemination/search/) is also still available for quicker confirmation of submitted reports.
+
+During the outage, we uploaded [daily findings workbooks]({{ config.baseUrl }}status/findings) to support resolution work. This page will remain available until May.
+
+## What's next?
+
+### Resubmission
+
+The team is beginning the work on single audit resubmission. We're in the early stages of user interviews to determine the feature requirements. We're also talking with our agency partners about the resubmission process and how its triggered.
+
+### May code freeze
+
+The FAC will undergo a code freeze from 5/17 - 5/31 to allow for security penetration testing. This testing is a required part of our authority to operate. During this time, we won't be making any updates to the FAC.
diff --git a/src/updates/2024-05-13-updates.md b/src/updates/2024-05-13-updates.md
new file mode 100644
index 00000000..314fc75f
--- /dev/null
+++ b/src/updates/2024-05-13-updates.md
@@ -0,0 +1,23 @@
+---
+layout: home.njk
+title: FAC Updates - Week of May 13, 2024
+date: 2024-05-13
+meta:
+ name: FAC Updates - Week of May 13, 2024
+ description: Read historical updates from the Federal Audit Clearinghouse.
+tags: updates
+---
+
+# {{title}}
+
+## Resubmission
+
+We're continuing to work with our agency partners and OMB on single audit resubmission. We're conducting user interviews with inspectors general and audit resolution officials and consulting with OMB on policy.
+
+## Historical data migration
+
+The team continues to work on a remediation plan for the roughly 1,600 historical audits that failed the initial migration. These audits have a variety of data quality issues that we hope to resolve over the next few sprints
+
+## May code freeze
+
+The FAC will undergo a code freeze from 5/17 - 5/31 to allow for security penetration testing. This testing is a required part of our authority to operate. During this time, we won't be making any updates to the FAC.
diff --git a/src/updates/2024-06-10-updates.md b/src/updates/2024-06-10-updates.md
new file mode 100644
index 00000000..ad7b6fa9
--- /dev/null
+++ b/src/updates/2024-06-10-updates.md
@@ -0,0 +1,25 @@
+---
+layout: home.njk
+title: FAC Updates - Week of June 10, 2024
+date: 2024-06-10
+tags: updates
+---
+
+# {{title}}
+
+## Security testing
+
+The FAC completed our compliance-required security testing earlier this month and received a positive report from our oversight body. With this testing complete, the FAC has now done the majority of our Authorization to Operate (ATO) assessment. We're now working with our assessment team to complete the remaining documentation requirements. The ATO process identifies areas of potential security risk and ensures we've taken steps to reduce or eliminate those risks. It also documents our policies for keeping the FAC up-to-date. You can read more about the ATO process on [digital.gov](http://digital.gov/).
+
+## Historical data migration
+
+The team is closing in on final remediation for the historical audits that failed initial migration. These audits have a variety of data quality issues that don't meet the FAC's validation standards. For example, some historical audits are missing contact information for certifying officials; others incorrectly report total federal program expenditures. We follow a standardized process to address each of these issues. First, we describe the issue and analyze its prevalence in the dataset. Next, the team brainstorms ways to fix the issue, such as calculating the correct total for a summary field or using a placeholder value where information is missing. Finally, we apply the remediation to the problematic data. If you would like to learn more, visit [this GitHub issue](https://github.com/GSA-TTS/FAC/issues/3364) where we are tracking the work.
+
+## Fixes and improvements
+
+While the ATO team conducted the security testing, our engineering team continued to work on bug fixes and other site improvements. In the last month, the team:
+- added a timer to the search results page
+- improved accessibility testing
+- fixed a bug related to sorting search results by cog/over
+
+For more information on these fixes and more, see [the FAC's June 6th release](https://github.com/GSA-TTS/FAC/releases/tag/v1.20240606).
diff --git a/src/updates/atom.njk b/src/updates/atom.njk
new file mode 100644
index 00000000..bc3b89a3
--- /dev/null
+++ b/src/updates/atom.njk
@@ -0,0 +1,33 @@
+---
+permalink: "/feeds/atom/updates.xml"
+eleventyExcludeFromCollections: true
+metadata:
+ title: "Updates from the FAC"
+ description: "Periodic updates from the Federal Audit Clearinghouse"
+ language: en
+ base: "https://fac.gov/updates"
+ author:
+ name: "Clearinghouse staff"
+---
+
+
+ 
+
+
+ 
+
+-
+{% for item in collections.updates | reverse %}
+ {% if loop.index0 > count %}
+
- {{ item.data.title }} + {% endif %} +{% endfor %} +