Add max queue depth limits #1376
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
WalkthroughThis pull request introduces several changes primarily focused on enhancing queue management in the web application. It adds new properties to the Changes
Possibly related PRs
Suggested reviewers
Poem
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
commit: |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Outside diff range and nitpick comments (14)
packages/database/prisma/migrations/20241003002757_add_max_queue_sizes_to_org/migration.sql (1)
1-3: LGTM! Consider adding default values.The migration successfully adds the required columns for maximum queue sizes in deployed and development environments, aligning with the PR objectives. The use of INTEGER type is appropriate, and the naming is clear and consistent.
Consider setting default values for these columns to simplify application logic. For example:
ALTER TABLE "Organization" ADD COLUMN "maximumDeployedQueueSize" INTEGER DEFAULT NULL, ADD COLUMN "maximumDevQueueSize" INTEGER DEFAULT 100;This suggestion sets no limit for deployed environments by default (NULL) and a default limit of 100 for dev environments. Adjust these values based on your specific requirements.
apps/webapp/app/v3/marqs/types.ts (2)
29-29: Approve new methods with a suggestion for documentationThe new methods
envQueueKeyandenvQueueKeyFromQueueare well-aligned with the PR objective of implementing environment-wide queue depth limits. They follow the existing naming conventions and return types.Consider adding JSDoc comments to these new methods to explain their purpose and usage, similar to other methods in this interface. For example:
/** * Generates a key for the environment queue. * @param env The authenticated environment * @returns A string representing the environment queue key */ envQueueKey(env: AuthenticatedEnvironment): string; /** * Generates an environment queue key from a given queue name. * @param queue The name of the queue * @returns A string representing the environment queue key */ envQueueKeyFromQueue(queue: string): string;Also applies to: 48-48
Line range hint
89-97: Approve MessagePayload update with a minor suggestionThe update to
MessagePayloadusing Zod schema validation is a significant improvement. It enhances type safety and provides runtime validation, which aligns well with best practices for robust type definitions.Consider adding a more specific type for the
datafield instead of usingz.record(z.unknown()). If the structure of the data is known, it would be beneficial to define it more precisely. If it truly can contain any data, consider usingz.any()instead ofz.unknown()for better type inference in TypeScript. For example:data: z.record(z.any()),or if the structure is known:
data: z.object({ // Define known fields here }),apps/webapp/app/v3/marqs/marqsKeyProducer.server.ts (1)
137-139: LGTM! Consider usingenvKeySectionfor consistency.The implementation looks good and follows the pattern of other key generation methods in the class. For improved consistency with other methods like
envCurrentConcurrencyKey, consider using theenvKeySectionprivate method.Here's a suggested minor refactor for consistency:
envQueueKey(env: AuthenticatedEnvironment): string { return [this.envKeySection(env.id), constants.QUEUE_PART].join(":"); }This change makes the method more consistent with other methods in the class and potentially more maintainable if the
envKeySectionlogic needs to be updated in the future.references/v3-catalog/src/trigger/simple.ts (3)
213-218: Add a comment explaining the purpose of this task.The
retryTaskis correctly implemented to always throw an error. However, it would be beneficial to add a comment explaining that this task is intended for testing retry functionality.Consider adding a comment like this:
export const retryTask = task({ id: "retry-task", // This task is designed to always fail, used for testing retry functionality run: async (payload: any) => { throw new Error("This task will always fail"); }, });
220-227: LGTM: Add a comment explaining the purpose of this task.The
maximumQueueDepthParenttask is correctly implemented to trigger the child task three times, which aligns with the PR objective of testing queue depth limits.Consider adding a comment explaining the purpose of this task:
export const maximumQueueDepthParent = task({ id: "maximum-queue-depth-parent", // This task triggers the child task multiple times to test queue depth limits run: async (payload: any) => { await maximumQueueDepthChild.trigger({}); await maximumQueueDepthChild.trigger({}); await maximumQueueDepthChild.trigger({}); }, });
229-237: LGTM: Well-implemented task for testing queue depth.The
maximumQueueDepthChildtask is correctly implemented with a concurrency limit and a delay to simulate a long-running task. This is excellent for testing queue depth limits.Consider adding a comment explaining the purpose of this task:
export const maximumQueueDepthChild = task({ id: "maximum-queue-depth-child", queue: { concurrencyLimit: 1, }, // This task simulates a long-running operation to test queue depth limits run: async (payload: any) => { await setTimeoutP(10_000); }, });apps/webapp/app/env.server.ts (1)
217-218: LGTM! Consider adding comments for clarity.The addition of
MAXIMUM_DEV_QUEUE_SIZEandMAXIMUM_DEPLOYED_QUEUE_SIZEas optional integer properties is well-implemented and consistent with the existing schema structure. These new properties align with the PR objectives to introduce limits on the number of queued task runs.For improved clarity and maintainability, consider adding comments to describe the purpose and expected values for these new properties. For example:
/** Maximum number of queued task runs for development environments. */ MAXIMUM_DEV_QUEUE_SIZE: z.coerce.number().int().optional(), /** Maximum number of queued task runs for deployed environments. Can be set to a higher value or left undefined for unlimited queue size. */ MAXIMUM_DEPLOYED_QUEUE_SIZE: z.coerce.number().int().optional(),packages/database/prisma/schema.prisma (2)
117-119: LGTM! Consider adding comments for the new fields.The addition of
maximumDevQueueSizeandmaximumDeployedQueueSizefields to theOrganizationmodel aligns well with the PR objectives. These optionalIntfields will allow for flexible queue size limits in different environments.Consider adding comments to describe the purpose and behavior of these new fields, similar to the comment on the
maximumSchedulesLimitfield. This would improve code readability and maintainability. For example:/// This is deprecated and will be removed in the future maximumSchedulesLimit Int @default(5) + /// Maximum number of queued task runs for development environments maximumDevQueueSize Int? + /// Maximum number of queued task runs for deployed environments maximumDeployedQueueSize Int?
Line range hint
1-1921: Consider future schema improvementsWhile the current changes are appropriate, here are some suggestions for future improvements to the schema:
Cleanup of deprecated elements: The
InvitationCodemodel and some fields (likemaximumSchedulesLimit) are marked as deprecated. Consider planning for their removal in a future update.Reorganization of the
Organizationmodel: As this model grows, it might be beneficial to group related fields or consider splitting some functionality into separate models.Consistency in default values: Some
Intfields have default values while others don't. It might be worth reviewing if default values should be applied consistently.These suggestions are not critical for the current PR but could improve the overall schema design and maintainability in the future.
apps/webapp/app/v3/marqs/index.server.ts (4)
Line range hint
1358-1381: Update KEYS comment inenqueueMessageRedis command to includeenvQueueIn the
enqueueMessageRedis command definition, the KEYS comment does not includeenvQueue(KEYS[7]). Please update the KEYS comment to includeenvQueuefor clarity.Apply this diff to correct the KEYS comment:
--- lua: ` --- -- Keys: queue, parentQueue, messageKey, concurrencyKey, envCurrentConcurrencyKey, orgCurrentConcurrencyKey +++ -- Keys: queue, parentQueue, messageKey, concurrencyKey, envCurrentConcurrencyKey, orgCurrentConcurrencyKey, envQueue local queue = KEYS[1]
Line range hint
1398-1454: Update KEYS comment indequeueMessageRedis command to includeenvQueueKeyIn the
dequeueMessageRedis command definition, the KEYS comment does not includeenvQueueKey(KEYS[9]). Please update the KEYS comment to includeenvQueueKeyfor clarity.Apply this diff to correct the KEYS comment:
--- lua: ` --- -- Keys: childQueue, parentQueue, concurrencyLimitKey, envConcurrencyLimitKey, orgConcurrencyLimitKey, currentConcurrencyKey, envCurrentConcurrencyKey, orgCurrentConcurrencyKey +++ -- Keys: childQueue, parentQueue, concurrencyLimitKey, envConcurrencyLimitKey, orgConcurrencyLimitKey, currentConcurrencyKey, envCurrentConcurrencyKey, orgCurrentConcurrencyKey, envQueueKey local childQueue = KEYS[1]
Line range hint
1491-1515: Update KEYS comment inacknowledgeMessageRedis command to includeenvQueueKeyIn the
acknowledgeMessageRedis command, the KEYS comment is missingenvQueueKey(KEYS[8]). Please update the comment to reflect all KEYS used.Apply this diff to correct the KEYS comment:
--- lua: ` --- -- Keys: parentQueue, messageKey, messageQueue, visibilityQueue, concurrencyKey, envCurrentConcurrencyKey, orgCurrentConcurrencyKey +++ -- Keys: parentQueue, messageKey, messageQueue, visibilityQueue, concurrencyKey, envCurrentConcurrencyKey, orgCurrentConcurrencyKey, envQueueKey local parentQueue = KEYS[1]
Line range hint
1535-1571: Update KEYS comment innackMessageRedis command to includeenvQueueKeyIn the
nackMessageRedis command, the KEYS comment does not includeenvQueueKey(KEYS[8]). Please update the comment to include all KEYS used for consistency.Apply this diff to correct the KEYS comment:
--- lua: ` --- -- Keys: messageKey, childQueueKey, parentQueueKey, concurrencyKey, envConcurrencyKey, orgConcurrencyKey, visibilityQueue +++ -- Keys: messageKey, childQueueKey, parentQueueKey, concurrencyKey, envConcurrencyKey, orgConcurrencyKey, visibilityQueue, envQueueKey local messageKey = KEYS[1]
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (9)
- apps/webapp/app/env.server.ts (1 hunks)
- apps/webapp/app/v3/marqs/index.server.ts (20 hunks)
- apps/webapp/app/v3/marqs/marqsKeyProducer.server.ts (1 hunks)
- apps/webapp/app/v3/marqs/types.ts (2 hunks)
- apps/webapp/app/v3/queueSizeLimits.server.ts (1 hunks)
- apps/webapp/app/v3/services/triggerTask.server.ts (2 hunks)
- packages/database/prisma/migrations/20241003002757_add_max_queue_sizes_to_org/migration.sql (1 hunks)
- packages/database/prisma/schema.prisma (1 hunks)
- references/v3-catalog/src/trigger/simple.ts (2 hunks)
🔇 Additional comments (18)
apps/webapp/app/v3/marqs/types.ts (1)
Line range hint
1-108: Summary: Changes align well with PR objectives and improve type safetyThe modifications to this file effectively support the PR objective of implementing environment-wide queue depth limits. The new methods in the
MarQSKeyProducerinterface and the updatedMessagePayloadtype using Zod schema validation significantly enhance the type safety and capabilities of the message queue system.These changes are well-implemented and consistent with the existing codebase. The suggestions provided are minor and aimed at further improving documentation and type precision.
apps/webapp/app/v3/marqs/marqsKeyProducer.server.ts (1)
131-139: New methods align with PR objectives for queue depth management.The addition of
envQueueKeyFromQueueandenvQueueKeymethods to theMarQSShortKeyProducerclass aligns well with the PR objectives. These methods provide the necessary functionality to generate keys for environment-specific queues, which is crucial for implementing limits on the number of queued task runs within the system.The new methods will allow for:
- Identifying and managing queues specific to each environment.
- Potentially setting different maximum queue lengths for development and deployed environments.
This implementation supports the goal of enhancing the management of queued tasks by establishing new constraints, which should improve overall system performance and resource allocation.
references/v3-catalog/src/trigger/simple.ts (2)
5-5: LGTM: New import for promise-based setTimeout.The import of
setTimeoutfromnode:timers/promisesis correctly added and appropriately renamed tosetTimeoutP. This distinction from the globalsetTimeoutis a good practice.
5-5: Summary: Excellent implementation for testing max queue depth limits.The changes in this file align well with the PR objectives of adding max queue depth limits. The new tasks (
retryTask,maximumQueueDepthParent, andmaximumQueueDepthChild) work together to create scenarios that will effectively test the queue depth functionality. The implementation is clean and well-structured.A few minor suggestions:
- Add comments to each new task explaining their specific purposes in testing queue depth limits.
- Consider adding some logging or return values in the
maximumQueueDepthParentandmaximumQueueDepthChildtasks to help with debugging and monitoring during tests.Overall, great job on implementing these changes!
Also applies to: 213-237
packages/database/prisma/schema.prisma (1)
117-119: Verify implementation of queue size limitsWhile the schema changes look good, it's important to ensure that these new queue size limits are properly implemented and enforced in the application code.
Please run the following script to check for any references to queue size limits in the codebase:
This will help identify areas where the new limits should be implemented or where existing code might need to be updated to use these new fields.
apps/webapp/app/v3/queueSizeLimits.server.ts (4)
5-9: Type definition forQueueSizeGuardResultis appropriateThe
QueueSizeGuardResulttype is well-defined and clearly represents the result of the queue size check.
17-19: Ensure undefinedmaximumSizeimplies unlimited queueReturning
{ isWithinLimits: true }whenmaximumSizeisundefinedimplies there are no limits on the queue size. Please confirm that this behavior aligns with the intended policy for environments without a defined maximum queue size.
21-23: Review handling whenmarqsis not providedWhen
marqsis not provided, the function returns{ isWithinLimits: true, maximumSize }, potentially bypassing the queue size check. If the absence ofmarqsshould still enforce limits, consider handling this case differently.
34-40: FunctiongetMaximumSizeForEnvironmentcorrectly retrieves queue size limitsThe function accurately obtains the maximum queue size based on the environment type, considering both organization-specific settings and environment variables.
apps/webapp/app/v3/marqs/index.server.ts (9)
142-144:lengthOfEnvQueuemethod implementation is correctThe new method
lengthOfEnvQueuecorrectly retrieves the length of the environment queue usingzcard.
Line range hint
1081-1085:#callEnqueueMessagemethod correctly includesenvQueueparameterThe addition of
this.keys.envQueueKeyFromQueue(message.queue)to theenqueueMessagecall is appropriate and aligns with the updated command definition.
Line range hint
1119-1124:#callDequeueMessagemethod correctly includesenvQueueKeyparameterThe inclusion of
this.keys.envQueueKeyFromQueue(messageQueue)as a parameter in thedequeueMessagecall is correct.
Line range hint
1196-1201:#callAcknowledgeMessagemethod includes new parameter appropriatelyThe addition of
this.keys.envQueueKeyFromQueue(messageQueue)to theacknowledgeMessagecall is consistent with the changes and correctly updates the method parameters.
Line range hint
1244-1251:#callNackMessagemethod includesenvQueueKeyparameter correctlyThe updated call to
nackMessagewith the addedenvQueueKeyparameter is appropriate and aligns with the method's requirements.
Line range hint
1754-1760:enqueueMessageinterface definition updated correctlyThe
RedisCommanderinterface methodenqueueMessagehas been updated to include theenvQueueparameter, matching its usage.
Line range hint
1771-1778:dequeueMessageinterface definition updated correctlyThe
dequeueMessagemethod in theRedisCommanderinterface correctly includes theenvQueueKeyparameter.
Line range hint
1793-1798:acknowledgeMessageinterface definition updated correctlyThe
acknowledgeMessagemethod signature has been properly updated to include theenvQueueKeyparameter.
Line range hint
1807-1814:nackMessageinterface definition updated correctlyThe
nackMessagemethod in theRedisCommanderinterface correctly includes theenvQueueKeyparameter.
matt-aitken
approved these changes
Oct 3, 2024
This was referenced Oct 7, 2024
samejr
added a commit
that referenced
this pull request
Oct 8, 2024
commit 886429b Author: Matt Aitken <[email protected]> Date: Tue Oct 8 18:36:46 2024 +0100 Removed emails, @trigger.dev/database and @trigger.dev/otlp-importer from changesets config commit f65157a Author: Matt Aitken <[email protected]> Date: Tue Oct 8 18:31:27 2024 +0100 Lockfile with run-engine removed commit 3d67bb8 Author: Matt Aitken <[email protected]> Date: Tue Oct 8 18:24:31 2024 +0100 Removed run-engine from the webapp package.json/tsconfig commit d30e971 Author: Matt Aitken <[email protected]> Date: Tue Oct 8 18:06:04 2024 +0100 Dockerfile fix because the database package has been moved commit f2babbf Author: Matt Aitken <[email protected]> Date: Tue Oct 8 09:41:22 2024 -0700 Internal packages (testcontainers, redis-worker and zod-worker) (#1392) * Some notes on the new run engine * lockfile with setup for the run engine * Documenting where TaskRun is currently mutated, to try figure out the shape of the new system * Added notes about how triggering currently works * Details about when triggering happens * Lots of notes about waitpoints * Started scaffolding the RunEngine * Sketch of Prisma waitpoint schema while it’s fresh in my mind * Got Prisma working with testcontainers * Use beforeEach/afterEach * Simple Prisma and Redis test * Return Redis options instead of a client * Simplified things * A very simple FIFO pull-based queue to check the tests working properly * Use vitest extend * Separate redis, postgres and combined tests for faster testing * Some fixes and test improvements * Pass a logger into the queue * A queue processor that processes items from the given queue as fast as it can * Test for retrying an item that wasn’t processed * First draft of waitpoints in the Prisma schema * Remove the custom logger from the test * Added a completedAt to Waitpoint * Notes on the flow for an execution starting * Added redlock, moved some files around * Starting point for the TaskRunExecutionSnapshot table * Added relationships to TaskRunExecutionSnapshot * Change some tsconfig * Moved some things around * Added some packages * WIP on the RunQueue * Fix for some imports * Key producer with some tests * Removed the nv type from the keys… it’s not useful to do global queries * Passing unit tests for all the public key producer functions * Some basic tests passing for the RunQueue * Simple enqueue test working * Enqueue and dequeue for dev is working * Don’t log everything during the tests * Enqueuing/dequeuing from the shared queue is working * Tests for getting a shared queue * The key producer sharedQueue can now be named, to allow multiple separate queues * The key producer uses the name of the queue as the input * Extra info in the Prisma schema * Dequeuing a message gets the payload and sets the task concurrency all in one Lua script * Adding more keys so we can read the concurrency from the queue * Setting the concurrency with dequeue and enquque is working * Improved the tests and fixed some bugs * Acking is resetting the concurrencies * Check the key has been removed after acking * Nacking is working * Changed the package to CommonJS + Node10 so it works with Redlock * Moved the database, otel and emails packages to be in internal-packages * Moved some Prisma code to the database package * Started using the RunEngine for triggering * Progress on run engine triggering, first waitpoint code * Create a delay waitpoint * Moved ZodWorker to an internal package so it can be used in the run engine as well as the webapp * Web app now uses the zod worker package * Added parseNaturalLanguageDuration to core/apps * internal-packages/zod-worker in the lockfile * Pass in the master queue, remove old rebalance workers code * Add masterQueue to TaskRun * Fixed the tests * Moved waitpoint code into the run engine, also the zod worker * Completing waitpoints * An experiment to create a new test container with environment * More changes to triggering * Started testing triggering * Test for a run getting triggered and being enqueued * Removed dequeueMessageInEnv * Update dev queue tests to use the shared queue function * Schema changes for TaskRunExecutionSnapshot * First execution snapshot when the run is created. Dequeue run function added to the engine * Separate internal package for testcontainers so they can be used elsewhere * Remove the simple queue and testcontainers from the run-engine. They’re going to be separate * Fix for the wrong path to the Prisma schem,a * Added the testcontainers package to the run-engine * redis-worker package, just a copy of the simple queue for now * The queue now uses Lua to enqueue dequeue * The queue now has a catalog and an invisible period after dequeuing * Added a visibility timeout and acking, with tests * Added more Redis connection logging, deleted todos * Visibility timeouts are now defined on the catalog and can be overridden when enqueuing * Dequeue multiple items at once * Test for dequeuing multiple items * Export some types to be used elsewhere * Partial refactor of the processor * First stab at a worker with concurrency and NodeWorkers * Don’t have a default visibility timeout in the queue * Worker setup and processing items in a simple test * Process jobs in parallel with retrying * Get the attempt when dequeuing * Workers do exponential backoff * Moved todos * DLQ functionality * DLQ tests * Same cluster for all keys in the same queue * Added DLQ tests * Whitespace * Redis pubsub to redrive from the worker * Fixed database paths * Fix for path to zod-worker * Fixes for typecheck errors, mostly with TS versions and module resolution * Redlock required a patch * Moved the new DB migrations to the new database package folder * Remove the run-engine package * Remove the RunEngine prisma schema changes * Delete triggerTaskV2 * Remove zodworker test script (no tests) * Update test-containers readme * Generate the client first * Use a specific version of the prisma package * Generate the prisma client before running the unit tests commit fc60947 Author: Dan <[email protected]> Date: Tue Oct 8 14:36:03 2024 +0100 Supabase database webhook example upgrade (#1386) * Added overview for guides and examples section and split them all out * New supabase guide wip * Updated images and improved docs * Trimmed the supabase prereqs * Supabase guide wip * more updates * Replaced old database webhook guide * Created one intro page and removed snippets * Updated guide sidebar titles * Code updates * More improvements * Updates and added images * Compressed image * Updated guides descriptions and edge function basic * Removed bold * Updated redirects * Fixed broken links * Updated intro commit 07f82ea Author: nicktrn <[email protected]> Date: Tue Oct 8 13:28:54 2024 +0100 Release 3.0.11 commit 13ebfcc Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Date: Tue Oct 8 13:24:38 2024 +0100 chore: Update version for release (#1381) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> commit 2a04d17 Author: nicktrn <[email protected]> Date: Tue Oct 8 09:24:23 2024 +0100 Simplify showLogs expression commit 002ae4b Author: nicktrn <[email protected]> Date: Tue Oct 8 09:22:29 2024 +0100 Fix dotenv overrides for dev runs (#1388) * override dashboard dev env vars with local .env * add changeset * add simple task for testing env vars commit 047cb00 Author: nicktrn <[email protected]> Date: Tue Oct 8 09:22:05 2024 +0100 Disable schedules for deleted orgs on next tick (#1383) * disable schedules for deleted orgs * add debug logs commit 2c014f7 Author: James Ritchie <[email protected]> Date: Sun Oct 6 13:02:00 2024 -0700 Override log retention (#1385) * set full log retention as admin * If run.logsDeletedAt is set, don’t bother getting the trace commit a69e04f Author: nicktrn <[email protected]> Date: Sat Oct 5 14:18:58 2024 +0100 Include push output in logs for self-hosted deploys (#1382) * include push output in logs * changeset commit c5488df Author: nicktrn <[email protected]> Date: Sat Oct 5 13:12:47 2024 +0100 Fix CLI downgrade check (#1380) * fix downgrade detection * remove unused semver package from webapp * add changeset commit 1caec27 Author: Eric Allam <[email protected]> Date: Fri Oct 4 15:33:35 2024 -0700 docs: Max duration (#1379) * maxDuration docs * Update the init command to set the maxDuration and include a commented out maxDuration in the config file commit e14c954 Author: Eric Allam <[email protected]> Date: Fri Oct 4 15:02:05 2024 -0700 Release 3.0.10 commit 8e61f5d Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Date: Fri Oct 4 14:59:07 2024 -0700 chore: Update version for release (#1378) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> commit 08db565 Author: Eric Allam <[email protected]> Date: Thu Oct 3 12:38:25 2024 -0700 improve the timed out description commit 6d08842 Author: Eric Allam <[email protected]> Date: Thu Oct 3 12:43:26 2024 -0700 feat: Add maxDuration to tasks (#1377) * WIP * Get max duration working on deployed runs * Actually set the timed out runs to status = TIMED_OUT * The client status for TIMED_OUT is now MAX_DURATION_EXCEEDED * New TimedOutIcon * Added new timedout icon * Add ability to opt-out of maxDuration with timeout.None * MAX_DURATION_EXCEEDED -> TIMED_OUT * changeset * Improved styling for the status tooltip content --------- Co-authored-by: James Ritchie <[email protected]> commit 665ccf8 Author: nicktrn <[email protected]> Date: Thu Oct 3 12:33:18 2024 +0100 Update github actions and self-hosting docs commit 1ff7b86 Author: Eric Allam <[email protected]> Date: Wed Oct 2 18:26:36 2024 -0700 Add max queue depth limits (#1376) * Add runs to an env queue, as well as the actual queue * Add queue size limit guard on triggering tasks commit c531a9d Author: Eric Allam <[email protected]> Date: Wed Oct 2 15:30:39 2024 -0700 fix: cleanup ttl expire run graphile jobs (#1373) * fix: remove ttl expire run graphile jobs when a run is started or completed * Update expireEnqueuedRun.server.ts commit 0bf500f Author: Matt Aitken <[email protected]> Date: Wed Oct 2 15:30:16 2024 -0700 Prioritize finishing waited runs (#1375) * If a tree node is missing, estimate the size as zero * Task to test prioritizing finishing existing runs after triggerAndWaits * When requeuing a run with a checkpoint, put it in the queue with the parent run time so it’s correctly prioritized * The same change but if there’s no checkpoint
samejr
added a commit
that referenced
this pull request
Oct 8, 2024
commit 886429b Author: Matt Aitken <[email protected]> Date: Tue Oct 8 18:36:46 2024 +0100 Removed emails, @trigger.dev/database and @trigger.dev/otlp-importer from changesets config commit f65157a Author: Matt Aitken <[email protected]> Date: Tue Oct 8 18:31:27 2024 +0100 Lockfile with run-engine removed commit 3d67bb8 Author: Matt Aitken <[email protected]> Date: Tue Oct 8 18:24:31 2024 +0100 Removed run-engine from the webapp package.json/tsconfig commit d30e971 Author: Matt Aitken <[email protected]> Date: Tue Oct 8 18:06:04 2024 +0100 Dockerfile fix because the database package has been moved commit f2babbf Author: Matt Aitken <[email protected]> Date: Tue Oct 8 09:41:22 2024 -0700 Internal packages (testcontainers, redis-worker and zod-worker) (#1392) * Some notes on the new run engine * lockfile with setup for the run engine * Documenting where TaskRun is currently mutated, to try figure out the shape of the new system * Added notes about how triggering currently works * Details about when triggering happens * Lots of notes about waitpoints * Started scaffolding the RunEngine * Sketch of Prisma waitpoint schema while it’s fresh in my mind * Got Prisma working with testcontainers * Use beforeEach/afterEach * Simple Prisma and Redis test * Return Redis options instead of a client * Simplified things * A very simple FIFO pull-based queue to check the tests working properly * Use vitest extend * Separate redis, postgres and combined tests for faster testing * Some fixes and test improvements * Pass a logger into the queue * A queue processor that processes items from the given queue as fast as it can * Test for retrying an item that wasn’t processed * First draft of waitpoints in the Prisma schema * Remove the custom logger from the test * Added a completedAt to Waitpoint * Notes on the flow for an execution starting * Added redlock, moved some files around * Starting point for the TaskRunExecutionSnapshot table * Added relationships to TaskRunExecutionSnapshot * Change some tsconfig * Moved some things around * Added some packages * WIP on the RunQueue * Fix for some imports * Key producer with some tests * Removed the nv type from the keys… it’s not useful to do global queries * Passing unit tests for all the public key producer functions * Some basic tests passing for the RunQueue * Simple enqueue test working * Enqueue and dequeue for dev is working * Don’t log everything during the tests * Enqueuing/dequeuing from the shared queue is working * Tests for getting a shared queue * The key producer sharedQueue can now be named, to allow multiple separate queues * The key producer uses the name of the queue as the input * Extra info in the Prisma schema * Dequeuing a message gets the payload and sets the task concurrency all in one Lua script * Adding more keys so we can read the concurrency from the queue * Setting the concurrency with dequeue and enquque is working * Improved the tests and fixed some bugs * Acking is resetting the concurrencies * Check the key has been removed after acking * Nacking is working * Changed the package to CommonJS + Node10 so it works with Redlock * Moved the database, otel and emails packages to be in internal-packages * Moved some Prisma code to the database package * Started using the RunEngine for triggering * Progress on run engine triggering, first waitpoint code * Create a delay waitpoint * Moved ZodWorker to an internal package so it can be used in the run engine as well as the webapp * Web app now uses the zod worker package * Added parseNaturalLanguageDuration to core/apps * internal-packages/zod-worker in the lockfile * Pass in the master queue, remove old rebalance workers code * Add masterQueue to TaskRun * Fixed the tests * Moved waitpoint code into the run engine, also the zod worker * Completing waitpoints * An experiment to create a new test container with environment * More changes to triggering * Started testing triggering * Test for a run getting triggered and being enqueued * Removed dequeueMessageInEnv * Update dev queue tests to use the shared queue function * Schema changes for TaskRunExecutionSnapshot * First execution snapshot when the run is created. Dequeue run function added to the engine * Separate internal package for testcontainers so they can be used elsewhere * Remove the simple queue and testcontainers from the run-engine. They’re going to be separate * Fix for the wrong path to the Prisma schem,a * Added the testcontainers package to the run-engine * redis-worker package, just a copy of the simple queue for now * The queue now uses Lua to enqueue dequeue * The queue now has a catalog and an invisible period after dequeuing * Added a visibility timeout and acking, with tests * Added more Redis connection logging, deleted todos * Visibility timeouts are now defined on the catalog and can be overridden when enqueuing * Dequeue multiple items at once * Test for dequeuing multiple items * Export some types to be used elsewhere * Partial refactor of the processor * First stab at a worker with concurrency and NodeWorkers * Don’t have a default visibility timeout in the queue * Worker setup and processing items in a simple test * Process jobs in parallel with retrying * Get the attempt when dequeuing * Workers do exponential backoff * Moved todos * DLQ functionality * DLQ tests * Same cluster for all keys in the same queue * Added DLQ tests * Whitespace * Redis pubsub to redrive from the worker * Fixed database paths * Fix for path to zod-worker * Fixes for typecheck errors, mostly with TS versions and module resolution * Redlock required a patch * Moved the new DB migrations to the new database package folder * Remove the run-engine package * Remove the RunEngine prisma schema changes * Delete triggerTaskV2 * Remove zodworker test script (no tests) * Update test-containers readme * Generate the client first * Use a specific version of the prisma package * Generate the prisma client before running the unit tests commit fc60947 Author: Dan <[email protected]> Date: Tue Oct 8 14:36:03 2024 +0100 Supabase database webhook example upgrade (#1386) * Added overview for guides and examples section and split them all out * New supabase guide wip * Updated images and improved docs * Trimmed the supabase prereqs * Supabase guide wip * more updates * Replaced old database webhook guide * Created one intro page and removed snippets * Updated guide sidebar titles * Code updates * More improvements * Updates and added images * Compressed image * Updated guides descriptions and edge function basic * Removed bold * Updated redirects * Fixed broken links * Updated intro commit 07f82ea Author: nicktrn <[email protected]> Date: Tue Oct 8 13:28:54 2024 +0100 Release 3.0.11 commit 13ebfcc Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Date: Tue Oct 8 13:24:38 2024 +0100 chore: Update version for release (#1381) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> commit 2a04d17 Author: nicktrn <[email protected]> Date: Tue Oct 8 09:24:23 2024 +0100 Simplify showLogs expression commit 002ae4b Author: nicktrn <[email protected]> Date: Tue Oct 8 09:22:29 2024 +0100 Fix dotenv overrides for dev runs (#1388) * override dashboard dev env vars with local .env * add changeset * add simple task for testing env vars commit 047cb00 Author: nicktrn <[email protected]> Date: Tue Oct 8 09:22:05 2024 +0100 Disable schedules for deleted orgs on next tick (#1383) * disable schedules for deleted orgs * add debug logs commit 2c014f7 Author: James Ritchie <[email protected]> Date: Sun Oct 6 13:02:00 2024 -0700 Override log retention (#1385) * set full log retention as admin * If run.logsDeletedAt is set, don’t bother getting the trace commit a69e04f Author: nicktrn <[email protected]> Date: Sat Oct 5 14:18:58 2024 +0100 Include push output in logs for self-hosted deploys (#1382) * include push output in logs * changeset commit c5488df Author: nicktrn <[email protected]> Date: Sat Oct 5 13:12:47 2024 +0100 Fix CLI downgrade check (#1380) * fix downgrade detection * remove unused semver package from webapp * add changeset commit 1caec27 Author: Eric Allam <[email protected]> Date: Fri Oct 4 15:33:35 2024 -0700 docs: Max duration (#1379) * maxDuration docs * Update the init command to set the maxDuration and include a commented out maxDuration in the config file commit e14c954 Author: Eric Allam <[email protected]> Date: Fri Oct 4 15:02:05 2024 -0700 Release 3.0.10 commit 8e61f5d Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Date: Fri Oct 4 14:59:07 2024 -0700 chore: Update version for release (#1378) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> commit 08db565 Author: Eric Allam <[email protected]> Date: Thu Oct 3 12:38:25 2024 -0700 improve the timed out description commit 6d08842 Author: Eric Allam <[email protected]> Date: Thu Oct 3 12:43:26 2024 -0700 feat: Add maxDuration to tasks (#1377) * WIP * Get max duration working on deployed runs * Actually set the timed out runs to status = TIMED_OUT * The client status for TIMED_OUT is now MAX_DURATION_EXCEEDED * New TimedOutIcon * Added new timedout icon * Add ability to opt-out of maxDuration with timeout.None * MAX_DURATION_EXCEEDED -> TIMED_OUT * changeset * Improved styling for the status tooltip content --------- Co-authored-by: James Ritchie <[email protected]> commit 665ccf8 Author: nicktrn <[email protected]> Date: Thu Oct 3 12:33:18 2024 +0100 Update github actions and self-hosting docs commit 1ff7b86 Author: Eric Allam <[email protected]> Date: Wed Oct 2 18:26:36 2024 -0700 Add max queue depth limits (#1376) * Add runs to an env queue, as well as the actual queue * Add queue size limit guard on triggering tasks commit c531a9d Author: Eric Allam <[email protected]> Date: Wed Oct 2 15:30:39 2024 -0700 fix: cleanup ttl expire run graphile jobs (#1373) * fix: remove ttl expire run graphile jobs when a run is started or completed * Update expireEnqueuedRun.server.ts commit 0bf500f Author: Matt Aitken <[email protected]> Date: Wed Oct 2 15:30:16 2024 -0700 Prioritize finishing waited runs (#1375) * If a tree node is missing, estimate the size as zero * Task to test prioritizing finishing existing runs after triggerAndWaits * When requeuing a run with a checkpoint, put it in the queue with the parent run time so it’s correctly prioritized * The same change but if there’s no checkpoint
samejr
added a commit
that referenced
this pull request
Oct 10, 2024
* Checkbox component can have its label styles * Improved the dialog footer * Handle sending feedback to Slack using Plain * WIP making the modal conditional * Moved the Plain form action into the select plan file * removed comment * Show a confirmation diaglog if you’re downgrading from Pro to Hobby * Downgrading to Hobby works * Use redirectWithErrorMessage instead of throw error * The cancel form now submits the data correctly * Modals don’t trigger when you upgrade * Copy improvements * Added a tooltip to explain the link to the pricing page * Squashed commit of the following: commit 886429b Author: Matt Aitken <[email protected]> Date: Tue Oct 8 18:36:46 2024 +0100 Removed emails, @trigger.dev/database and @trigger.dev/otlp-importer from changesets config commit f65157a Author: Matt Aitken <[email protected]> Date: Tue Oct 8 18:31:27 2024 +0100 Lockfile with run-engine removed commit 3d67bb8 Author: Matt Aitken <[email protected]> Date: Tue Oct 8 18:24:31 2024 +0100 Removed run-engine from the webapp package.json/tsconfig commit d30e971 Author: Matt Aitken <[email protected]> Date: Tue Oct 8 18:06:04 2024 +0100 Dockerfile fix because the database package has been moved commit f2babbf Author: Matt Aitken <[email protected]> Date: Tue Oct 8 09:41:22 2024 -0700 Internal packages (testcontainers, redis-worker and zod-worker) (#1392) * Some notes on the new run engine * lockfile with setup for the run engine * Documenting where TaskRun is currently mutated, to try figure out the shape of the new system * Added notes about how triggering currently works * Details about when triggering happens * Lots of notes about waitpoints * Started scaffolding the RunEngine * Sketch of Prisma waitpoint schema while it’s fresh in my mind * Got Prisma working with testcontainers * Use beforeEach/afterEach * Simple Prisma and Redis test * Return Redis options instead of a client * Simplified things * A very simple FIFO pull-based queue to check the tests working properly * Use vitest extend * Separate redis, postgres and combined tests for faster testing * Some fixes and test improvements * Pass a logger into the queue * A queue processor that processes items from the given queue as fast as it can * Test for retrying an item that wasn’t processed * First draft of waitpoints in the Prisma schema * Remove the custom logger from the test * Added a completedAt to Waitpoint * Notes on the flow for an execution starting * Added redlock, moved some files around * Starting point for the TaskRunExecutionSnapshot table * Added relationships to TaskRunExecutionSnapshot * Change some tsconfig * Moved some things around * Added some packages * WIP on the RunQueue * Fix for some imports * Key producer with some tests * Removed the nv type from the keys… it’s not useful to do global queries * Passing unit tests for all the public key producer functions * Some basic tests passing for the RunQueue * Simple enqueue test working * Enqueue and dequeue for dev is working * Don’t log everything during the tests * Enqueuing/dequeuing from the shared queue is working * Tests for getting a shared queue * The key producer sharedQueue can now be named, to allow multiple separate queues * The key producer uses the name of the queue as the input * Extra info in the Prisma schema * Dequeuing a message gets the payload and sets the task concurrency all in one Lua script * Adding more keys so we can read the concurrency from the queue * Setting the concurrency with dequeue and enquque is working * Improved the tests and fixed some bugs * Acking is resetting the concurrencies * Check the key has been removed after acking * Nacking is working * Changed the package to CommonJS + Node10 so it works with Redlock * Moved the database, otel and emails packages to be in internal-packages * Moved some Prisma code to the database package * Started using the RunEngine for triggering * Progress on run engine triggering, first waitpoint code * Create a delay waitpoint * Moved ZodWorker to an internal package so it can be used in the run engine as well as the webapp * Web app now uses the zod worker package * Added parseNaturalLanguageDuration to core/apps * internal-packages/zod-worker in the lockfile * Pass in the master queue, remove old rebalance workers code * Add masterQueue to TaskRun * Fixed the tests * Moved waitpoint code into the run engine, also the zod worker * Completing waitpoints * An experiment to create a new test container with environment * More changes to triggering * Started testing triggering * Test for a run getting triggered and being enqueued * Removed dequeueMessageInEnv * Update dev queue tests to use the shared queue function * Schema changes for TaskRunExecutionSnapshot * First execution snapshot when the run is created. Dequeue run function added to the engine * Separate internal package for testcontainers so they can be used elsewhere * Remove the simple queue and testcontainers from the run-engine. They’re going to be separate * Fix for the wrong path to the Prisma schem,a * Added the testcontainers package to the run-engine * redis-worker package, just a copy of the simple queue for now * The queue now uses Lua to enqueue dequeue * The queue now has a catalog and an invisible period after dequeuing * Added a visibility timeout and acking, with tests * Added more Redis connection logging, deleted todos * Visibility timeouts are now defined on the catalog and can be overridden when enqueuing * Dequeue multiple items at once * Test for dequeuing multiple items * Export some types to be used elsewhere * Partial refactor of the processor * First stab at a worker with concurrency and NodeWorkers * Don’t have a default visibility timeout in the queue * Worker setup and processing items in a simple test * Process jobs in parallel with retrying * Get the attempt when dequeuing * Workers do exponential backoff * Moved todos * DLQ functionality * DLQ tests * Same cluster for all keys in the same queue * Added DLQ tests * Whitespace * Redis pubsub to redrive from the worker * Fixed database paths * Fix for path to zod-worker * Fixes for typecheck errors, mostly with TS versions and module resolution * Redlock required a patch * Moved the new DB migrations to the new database package folder * Remove the run-engine package * Remove the RunEngine prisma schema changes * Delete triggerTaskV2 * Remove zodworker test script (no tests) * Update test-containers readme * Generate the client first * Use a specific version of the prisma package * Generate the prisma client before running the unit tests commit fc60947 Author: Dan <[email protected]> Date: Tue Oct 8 14:36:03 2024 +0100 Supabase database webhook example upgrade (#1386) * Added overview for guides and examples section and split them all out * New supabase guide wip * Updated images and improved docs * Trimmed the supabase prereqs * Supabase guide wip * more updates * Replaced old database webhook guide * Created one intro page and removed snippets * Updated guide sidebar titles * Code updates * More improvements * Updates and added images * Compressed image * Updated guides descriptions and edge function basic * Removed bold * Updated redirects * Fixed broken links * Updated intro commit 07f82ea Author: nicktrn <[email protected]> Date: Tue Oct 8 13:28:54 2024 +0100 Release 3.0.11 commit 13ebfcc Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Date: Tue Oct 8 13:24:38 2024 +0100 chore: Update version for release (#1381) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> commit 2a04d17 Author: nicktrn <[email protected]> Date: Tue Oct 8 09:24:23 2024 +0100 Simplify showLogs expression commit 002ae4b Author: nicktrn <[email protected]> Date: Tue Oct 8 09:22:29 2024 +0100 Fix dotenv overrides for dev runs (#1388) * override dashboard dev env vars with local .env * add changeset * add simple task for testing env vars commit 047cb00 Author: nicktrn <[email protected]> Date: Tue Oct 8 09:22:05 2024 +0100 Disable schedules for deleted orgs on next tick (#1383) * disable schedules for deleted orgs * add debug logs commit 2c014f7 Author: James Ritchie <[email protected]> Date: Sun Oct 6 13:02:00 2024 -0700 Override log retention (#1385) * set full log retention as admin * If run.logsDeletedAt is set, don’t bother getting the trace commit a69e04f Author: nicktrn <[email protected]> Date: Sat Oct 5 14:18:58 2024 +0100 Include push output in logs for self-hosted deploys (#1382) * include push output in logs * changeset commit c5488df Author: nicktrn <[email protected]> Date: Sat Oct 5 13:12:47 2024 +0100 Fix CLI downgrade check (#1380) * fix downgrade detection * remove unused semver package from webapp * add changeset commit 1caec27 Author: Eric Allam <[email protected]> Date: Fri Oct 4 15:33:35 2024 -0700 docs: Max duration (#1379) * maxDuration docs * Update the init command to set the maxDuration and include a commented out maxDuration in the config file commit e14c954 Author: Eric Allam <[email protected]> Date: Fri Oct 4 15:02:05 2024 -0700 Release 3.0.10 commit 8e61f5d Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Date: Fri Oct 4 14:59:07 2024 -0700 chore: Update version for release (#1378) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> commit 08db565 Author: Eric Allam <[email protected]> Date: Thu Oct 3 12:38:25 2024 -0700 improve the timed out description commit 6d08842 Author: Eric Allam <[email protected]> Date: Thu Oct 3 12:43:26 2024 -0700 feat: Add maxDuration to tasks (#1377) * WIP * Get max duration working on deployed runs * Actually set the timed out runs to status = TIMED_OUT * The client status for TIMED_OUT is now MAX_DURATION_EXCEEDED * New TimedOutIcon * Added new timedout icon * Add ability to opt-out of maxDuration with timeout.None * MAX_DURATION_EXCEEDED -> TIMED_OUT * changeset * Improved styling for the status tooltip content --------- Co-authored-by: James Ritchie <[email protected]> commit 665ccf8 Author: nicktrn <[email protected]> Date: Thu Oct 3 12:33:18 2024 +0100 Update github actions and self-hosting docs commit 1ff7b86 Author: Eric Allam <[email protected]> Date: Wed Oct 2 18:26:36 2024 -0700 Add max queue depth limits (#1376) * Add runs to an env queue, as well as the actual queue * Add queue size limit guard on triggering tasks commit c531a9d Author: Eric Allam <[email protected]> Date: Wed Oct 2 15:30:39 2024 -0700 fix: cleanup ttl expire run graphile jobs (#1373) * fix: remove ttl expire run graphile jobs when a run is started or completed * Update expireEnqueuedRun.server.ts commit 0bf500f Author: Matt Aitken <[email protected]> Date: Wed Oct 2 15:30:16 2024 -0700 Prioritize finishing waited runs (#1375) * If a tree node is missing, estimate the size as zero * Task to test prioritizing finishing existing runs after triggerAndWaits * When requeuing a run with a checkpoint, put it in the queue with the parent run time so it’s correctly prioritized * The same change but if there’s no checkpoint * Squashed commit of the following: commit 886429b Author: Matt Aitken <[email protected]> Date: Tue Oct 8 18:36:46 2024 +0100 Removed emails, @trigger.dev/database and @trigger.dev/otlp-importer from changesets config commit f65157a Author: Matt Aitken <[email protected]> Date: Tue Oct 8 18:31:27 2024 +0100 Lockfile with run-engine removed commit 3d67bb8 Author: Matt Aitken <[email protected]> Date: Tue Oct 8 18:24:31 2024 +0100 Removed run-engine from the webapp package.json/tsconfig commit d30e971 Author: Matt Aitken <[email protected]> Date: Tue Oct 8 18:06:04 2024 +0100 Dockerfile fix because the database package has been moved commit f2babbf Author: Matt Aitken <[email protected]> Date: Tue Oct 8 09:41:22 2024 -0700 Internal packages (testcontainers, redis-worker and zod-worker) (#1392) * Some notes on the new run engine * lockfile with setup for the run engine * Documenting where TaskRun is currently mutated, to try figure out the shape of the new system * Added notes about how triggering currently works * Details about when triggering happens * Lots of notes about waitpoints * Started scaffolding the RunEngine * Sketch of Prisma waitpoint schema while it’s fresh in my mind * Got Prisma working with testcontainers * Use beforeEach/afterEach * Simple Prisma and Redis test * Return Redis options instead of a client * Simplified things * A very simple FIFO pull-based queue to check the tests working properly * Use vitest extend * Separate redis, postgres and combined tests for faster testing * Some fixes and test improvements * Pass a logger into the queue * A queue processor that processes items from the given queue as fast as it can * Test for retrying an item that wasn’t processed * First draft of waitpoints in the Prisma schema * Remove the custom logger from the test * Added a completedAt to Waitpoint * Notes on the flow for an execution starting * Added redlock, moved some files around * Starting point for the TaskRunExecutionSnapshot table * Added relationships to TaskRunExecutionSnapshot * Change some tsconfig * Moved some things around * Added some packages * WIP on the RunQueue * Fix for some imports * Key producer with some tests * Removed the nv type from the keys… it’s not useful to do global queries * Passing unit tests for all the public key producer functions * Some basic tests passing for the RunQueue * Simple enqueue test working * Enqueue and dequeue for dev is working * Don’t log everything during the tests * Enqueuing/dequeuing from the shared queue is working * Tests for getting a shared queue * The key producer sharedQueue can now be named, to allow multiple separate queues * The key producer uses the name of the queue as the input * Extra info in the Prisma schema * Dequeuing a message gets the payload and sets the task concurrency all in one Lua script * Adding more keys so we can read the concurrency from the queue * Setting the concurrency with dequeue and enquque is working * Improved the tests and fixed some bugs * Acking is resetting the concurrencies * Check the key has been removed after acking * Nacking is working * Changed the package to CommonJS + Node10 so it works with Redlock * Moved the database, otel and emails packages to be in internal-packages * Moved some Prisma code to the database package * Started using the RunEngine for triggering * Progress on run engine triggering, first waitpoint code * Create a delay waitpoint * Moved ZodWorker to an internal package so it can be used in the run engine as well as the webapp * Web app now uses the zod worker package * Added parseNaturalLanguageDuration to core/apps * internal-packages/zod-worker in the lockfile * Pass in the master queue, remove old rebalance workers code * Add masterQueue to TaskRun * Fixed the tests * Moved waitpoint code into the run engine, also the zod worker * Completing waitpoints * An experiment to create a new test container with environment * More changes to triggering * Started testing triggering * Test for a run getting triggered and being enqueued * Removed dequeueMessageInEnv * Update dev queue tests to use the shared queue function * Schema changes for TaskRunExecutionSnapshot * First execution snapshot when the run is created. Dequeue run function added to the engine * Separate internal package for testcontainers so they can be used elsewhere * Remove the simple queue and testcontainers from the run-engine. They’re going to be separate * Fix for the wrong path to the Prisma schem,a * Added the testcontainers package to the run-engine * redis-worker package, just a copy of the simple queue for now * The queue now uses Lua to enqueue dequeue * The queue now has a catalog and an invisible period after dequeuing * Added a visibility timeout and acking, with tests * Added more Redis connection logging, deleted todos * Visibility timeouts are now defined on the catalog and can be overridden when enqueuing * Dequeue multiple items at once * Test for dequeuing multiple items * Export some types to be used elsewhere * Partial refactor of the processor * First stab at a worker with concurrency and NodeWorkers * Don’t have a default visibility timeout in the queue * Worker setup and processing items in a simple test * Process jobs in parallel with retrying * Get the attempt when dequeuing * Workers do exponential backoff * Moved todos * DLQ functionality * DLQ tests * Same cluster for all keys in the same queue * Added DLQ tests * Whitespace * Redis pubsub to redrive from the worker * Fixed database paths * Fix for path to zod-worker * Fixes for typecheck errors, mostly with TS versions and module resolution * Redlock required a patch * Moved the new DB migrations to the new database package folder * Remove the run-engine package * Remove the RunEngine prisma schema changes * Delete triggerTaskV2 * Remove zodworker test script (no tests) * Update test-containers readme * Generate the client first * Use a specific version of the prisma package * Generate the prisma client before running the unit tests commit fc60947 Author: Dan <[email protected]> Date: Tue Oct 8 14:36:03 2024 +0100 Supabase database webhook example upgrade (#1386) * Added overview for guides and examples section and split them all out * New supabase guide wip * Updated images and improved docs * Trimmed the supabase prereqs * Supabase guide wip * more updates * Replaced old database webhook guide * Created one intro page and removed snippets * Updated guide sidebar titles * Code updates * More improvements * Updates and added images * Compressed image * Updated guides descriptions and edge function basic * Removed bold * Updated redirects * Fixed broken links * Updated intro commit 07f82ea Author: nicktrn <[email protected]> Date: Tue Oct 8 13:28:54 2024 +0100 Release 3.0.11 commit 13ebfcc Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Date: Tue Oct 8 13:24:38 2024 +0100 chore: Update version for release (#1381) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> commit 2a04d17 Author: nicktrn <[email protected]> Date: Tue Oct 8 09:24:23 2024 +0100 Simplify showLogs expression commit 002ae4b Author: nicktrn <[email protected]> Date: Tue Oct 8 09:22:29 2024 +0100 Fix dotenv overrides for dev runs (#1388) * override dashboard dev env vars with local .env * add changeset * add simple task for testing env vars commit 047cb00 Author: nicktrn <[email protected]> Date: Tue Oct 8 09:22:05 2024 +0100 Disable schedules for deleted orgs on next tick (#1383) * disable schedules for deleted orgs * add debug logs commit 2c014f7 Author: James Ritchie <[email protected]> Date: Sun Oct 6 13:02:00 2024 -0700 Override log retention (#1385) * set full log retention as admin * If run.logsDeletedAt is set, don’t bother getting the trace commit a69e04f Author: nicktrn <[email protected]> Date: Sat Oct 5 14:18:58 2024 +0100 Include push output in logs for self-hosted deploys (#1382) * include push output in logs * changeset commit c5488df Author: nicktrn <[email protected]> Date: Sat Oct 5 13:12:47 2024 +0100 Fix CLI downgrade check (#1380) * fix downgrade detection * remove unused semver package from webapp * add changeset commit 1caec27 Author: Eric Allam <[email protected]> Date: Fri Oct 4 15:33:35 2024 -0700 docs: Max duration (#1379) * maxDuration docs * Update the init command to set the maxDuration and include a commented out maxDuration in the config file commit e14c954 Author: Eric Allam <[email protected]> Date: Fri Oct 4 15:02:05 2024 -0700 Release 3.0.10 commit 8e61f5d Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Date: Fri Oct 4 14:59:07 2024 -0700 chore: Update version for release (#1378) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> commit 08db565 Author: Eric Allam <[email protected]> Date: Thu Oct 3 12:38:25 2024 -0700 improve the timed out description commit 6d08842 Author: Eric Allam <[email protected]> Date: Thu Oct 3 12:43:26 2024 -0700 feat: Add maxDuration to tasks (#1377) * WIP * Get max duration working on deployed runs * Actually set the timed out runs to status = TIMED_OUT * The client status for TIMED_OUT is now MAX_DURATION_EXCEEDED * New TimedOutIcon * Added new timedout icon * Add ability to opt-out of maxDuration with timeout.None * MAX_DURATION_EXCEEDED -> TIMED_OUT * changeset * Improved styling for the status tooltip content --------- Co-authored-by: James Ritchie <[email protected]> commit 665ccf8 Author: nicktrn <[email protected]> Date: Thu Oct 3 12:33:18 2024 +0100 Update github actions and self-hosting docs commit 1ff7b86 Author: Eric Allam <[email protected]> Date: Wed Oct 2 18:26:36 2024 -0700 Add max queue depth limits (#1376) * Add runs to an env queue, as well as the actual queue * Add queue size limit guard on triggering tasks commit c531a9d Author: Eric Allam <[email protected]> Date: Wed Oct 2 15:30:39 2024 -0700 fix: cleanup ttl expire run graphile jobs (#1373) * fix: remove ttl expire run graphile jobs when a run is started or completed * Update expireEnqueuedRun.server.ts commit 0bf500f Author: Matt Aitken <[email protected]> Date: Wed Oct 2 15:30:16 2024 -0700 Prioritize finishing waited runs (#1375) * If a tree node is missing, estimate the size as zero * Task to test prioritizing finishing existing runs after triggerAndWaits * When requeuing a run with a checkpoint, put it in the queue with the parent run time so it’s correctly prioritized * The same change but if there’s no checkpoint * Revert "Squashed commit of the following:" This reverts commit b837b5a. * Removed console logs * cleaned up conditionals * Unlock free plan state * Fixed subscribe button if you’re already github verified * Simplified the downgrade reasons logic * made periodEnd required --------- Co-authored-by: nicktrn <[email protected]>
This was referenced Nov 22, 2024
Merged
This was referenced Nov 29, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We currently have no limits on the amount of queued task runs and this PR adds some limits on a per-environment basis. To achieve this, I added the concept of an "env" queue that will hold the queued task run IDs, very similar to our normal marqs queues, but this is across the entire environment.
This PR also allows limiting the max queue length for dev separately from deployed environments. This will allow adding stricter limits to dev and allow deployed environments either unlimited queue size or very very high.
Summary by CodeRabbit
Release Notes
New Features
Bug Fixes
Documentation