Skip to content

Commit

Permalink
Remove pending guidelines
Browse files Browse the repository at this point in the history
  • Loading branch information
Iinh committed Feb 28, 2024
1 parent fae45a0 commit 018c3ce
Show file tree
Hide file tree
Showing 3 changed files with 111 additions and 87 deletions.
56 changes: 34 additions & 22 deletions docs/adr/0005-serve-from-bq.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,14 @@
# Use BigQuery for metrics data

-Status:
-Deciders: Eduardo Filho, Linh Nguyen, Rob Miller
-Date: 2023-01-19
-Status: -Deciders: Eduardo Filho, Linh Nguyen, Rob Miller -Date: 2023-01-19

## Context and Problem Statement

Postgres acts as both a historical database and a cache for GLAM aggregated data processed in BigQuery. We can remove Postgres from the stack and make GLAM serve data directly from BigQuery, significantly reducing operational burden, reducing some costs and making data available faster, while maintaining, and possibly improving, data read performance.
Postgres acts as both a historical database and a cache for GLAM aggregated data
processed in BigQuery. We can remove Postgres from the stack and make GLAM serve
data directly from BigQuery, significantly reducing operational burden, reducing
some costs and making data available faster, while maintaining, and possibly
improving, data read performance.

## Decision Drivers

Expand All @@ -20,39 +22,49 @@ Postgres acts as both a historical database and a cache for GLAM aggregated data
- Keep aggregation data in Postgres
- Move aggregation data into a different OLTP system


## Decision Outcome

We decided to move GLAM aggregates from Postgres into BigQuery for historical and serving purposes. After running a Proof of Concept with properly backfilled and partitioned tables, it was shown that BigQuery table read times are comparable (sometimes faster) to current Postgres times for the current volume of data. Also, updating BigQuery tables takes only 3 minutes at most, compared to the 15 hours taken by the current glam_glean_imports job that updates Postgres.
We decided to move GLAM aggregates from Postgres into BigQuery for historical
and serving purposes. After running a Proof of Concept with properly backfilled
and partitioned tables, it was shown that BigQuery table read times are
comparable (sometimes faster) to current Postgres times for the current volume
of data. Also, updating BigQuery tables takes only 3 minutes at most, compared
to the 15 hours taken by the current glam_glean_imports job that updates
Postgres.

## Pros and Cons of the Options

### Move GLAM aggregates to BigQuery

This idea involves creating GLAM aggregates tables in their final formats (i.e. ready to be served by GLAM) on BigQuery and replacing Django ORM calls to Postgres with BigQuery read api calls. The BigQuery tables are updated on a daily basis from the result tables of GLAM ETL. This is a low-hanging fruit because of the small amount of changes needed.
This idea involves creating GLAM aggregates tables in their final formats (i.e.
ready to be served by GLAM) on BigQuery and replacing Django ORM calls to
Postgres with BigQuery read api calls. The BigQuery tables are updated on a
daily basis from the result tables of GLAM ETL. This is a low-hanging fruit
because of the small amount of changes needed.

#### Pros/Cons

Pro: Serving tables are updated 300x faster (3 minutes vs 15 hours).
Pro: GLAM history of aggregates on BigQuery, making data analysis/debugging/PoC much easier.
Pro: Local/Dev/Stage environments can consume the same data as Prod. Local environments don’t need to import data locally and always have access to all the data instead of only 3 last versions.
Pro: Data migrations will be much faster.
Pro: No need to get DSRE involved for long-running data migrations.

Con: Some small code changes needed in GLAM.
Con: Will lose structured Django data migration.
Pro: Serving tables are updated 300x faster (3 minutes vs 15 hours). Pro: GLAM
history of aggregates on BigQuery, making data analysis/debugging/PoC much
easier. Pro: Local/Dev/Stage environments can consume the same data as Prod.
Local environments don’t need to import data locally and always have access to
all the data instead of only 3 last versions. Pro: Data migrations will be much
faster. Pro: No need to get DSRE involved for long-running data migrations.

Con: Some small code changes needed in GLAM. Con: Will lose structured Django
data migration.

### Keep GLAM aggregates in Postgres

Status quo.
Pro: Nothing to do
Con: Missing out on every pro from Moving GLAM aggregates to BigQuery

Status quo. Pro: Nothing to do Con: Missing out on every pro from Moving GLAM
aggregates to BigQuery

### Move GLAM aggregates to another OLTP system

Investigate different database systems, put together PoCs for each of them and assess if there’s a better alternative than Postgres.
Investigate different database systems, put together PoCs for each of them and
assess if there’s a better alternative than Postgres.

Pro: We might find something that’s much better for GLAM.
Con: Significant amount of work to set up PoCs for different DB systems. From hosting to table configuration and optimization, to importing data and connecting to GLAM, this is not trivial and we didn’t want to spend too much time on this.
Pro: We might find something that’s much better for GLAM. Con: Significant
amount of work to set up PoCs for different DB systems. From hosting to table
configuration and optimization, to importing data and connecting to GLAM, this
is not trivial and we didn’t want to spend too much time on this.
141 changes: 77 additions & 64 deletions src/components/CookieBanner.svelte
Original file line number Diff line number Diff line change
@@ -1,70 +1,83 @@
<script>
import { onMount } from 'svelte';
let consentGiven = false;
let settingsOpen = false; // To toggle the settings view
onMount(() => {
consentGiven = localStorage.getItem('cookieConsent') === 'true';
});
function giveConsent() {
localStorage.setItem('cookieConsent', 'true');
consentGiven = true;
import { onMount } from 'svelte';
let consentGiven = false;
onMount(() => {
consentGiven = localStorage.getItem('cookieConsent') === 'true';
});
/* eslint-disable */
function initializeGA4() {
// Load the GA4 script dynamically
var script = document.createElement('script');
script.async = true;
script.src =
'https://www.googletagmanager.com/gtag/js?id={{ GA_TRACKING_ID }}';
document.head.appendChild(script);
// Initialize GA4 tracking
window.dataLayer = window.dataLayer || [];
function gtag() {
dataLayer.push(arguments);
}
function rejectCookies() {
localStorage.setItem('cookieConsent', 'false');
consentGiven = false;
}
function toggleSettings() {
settingsOpen = !settingsOpen;
}
</script>

{#if !consentGiven}
gtag('js', new Date());
gtag('config', '{{ GA_TRACKING_ID }}');
}
/* eslint-enable */
function giveConsent() {
localStorage.setItem('cookieConsent', 'true');
consentGiven = true;
}
function rejectCookies() {
localStorage.setItem('cookieConsent', 'false');
consentGiven = false;
}
$: if (consentGiven) {
initializeGA4();
}
</script>

<style>
.cookie-banner {
position: fixed;
bottom: 0;
left: 0;
width: 100%;
background-color: #f4f4f4;
padding: 20px;
text-align: center;
box-shadow: 0 -2px 4px rgba(0, 0, 0, 0.1);
}
.cookie-banner p {
margin: 0 0 10px 0;
}
.cookie-banner button {
padding: 10px 20px;
background-color: #007bff;
color: white;
border: none;
cursor: pointer;
margin-right: 10px;
}
.cookie-settings {
margin-top: 20px;
background-color: #e9ecef;
padding: 10px;
border-radius: 5px;
}
</style>

{#if !consentGiven}
<div class="cookie-banner">
<p>We use cookies to enhance your experience. By continuing to visit this site you agree to our use of cookies.</p>
<button on:click={toggleSettings}>Manage cookie settings</button>
<p>
We use cookies to enhance your experience, by continuing to visit this
site you agree to our use of cookies. <a href="https://www.mozilla.org/en-US/privacy/websites/#cookies" target="_blank">Learn More</a>.
</p>
<button on:click={giveConsent}>Accept</button>
<button on:click={rejectCookies}>Reject</button>

{#if settingsOpen}
<div class="cookie-settings">
<!-- Placeholder for detailed settings from Mozilla org-->
<p>Cookie settings panel...</p>
</div>
{/if}
</div>
{/if}

<style>
.cookie-banner {
position: fixed;
bottom: 0;
left: 0;
width: 100%;
background-color: #f4f4f4;
padding: 20px;
text-align: center;
box-shadow: 0 -2px 4px rgba(0,0,0,0.1);
}
.cookie-banner p {
margin: 0 0 10px 0;
}
.cookie-banner button {
padding: 10px 20px;
background-color: #007bff;
color: white;
border: none;
cursor: pointer;
margin-right: 10px;
}
.cookie-settings {
margin-top: 20px;
background-color: #e9ecef;
padding: 10px;
border-radius: 5px;
}
</style>
{/if}
1 change: 0 additions & 1 deletion src/routing/pages/Home.svelte
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,6 @@
}
</style>


<div class="graphic-body__content">
<CookieBanner />
<div>
Expand Down

0 comments on commit 018c3ce

Please sign in to comment.