Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: WIP experimenting with Smartview service interface #86

Draft
wants to merge 8 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
83 changes: 83 additions & 0 deletions .github/workflows/sync-smartview.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
name: Sync smartview API spec

on:
workflow_dispatch:

jobs:
sync:
strategy:
matrix:
go-version: [1.19.x]
os: [ubuntu-latest]
runs-on: ${{ matrix.os }}
steps:
- name: Get current date
id: date
run: echo "date=$(date +'%Y-%m-%d')" >> "$GITHUB_OUTPUT"
- name: Install Go
uses: actions/setup-go@v5
with:
go-version: ${{ matrix.go-version }}
- name: Checkout code
uses: actions/checkout@v4
- name: GitHub user
run: |
# https://api.github.com/users/github-actions[bot]
git config user.name 'github-actions[bot]'
git config user.email '41898282+github-actions[bot]@users.noreply.github.com'
- name: Fetch latest spec
id: fetch
run: |
make -f Makefile.smartview fetch
git add spec/services/smartview
echo `git commit -m 'sync: fetch ${{ steps.date.outputs.date }} spec and apply patches'`
- name: Apply spec patches
id: patch
if: ${{ always() && steps.fetch.conclusion == 'success' }}
run: |
make -f Makefile.smartview patch
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll want to combine before patching. We would want to be careful about the order of the source URLs being combined so that the fetched+combined spec can be patched against.

git add spec/services/smartview
echo `git commit -m 'sync: patch spec with ${{ steps.date.outputs.date }} spec'`
- name: Generate code
id: generate
if: ${{ always() && steps.patch.conclusion == 'success' }}
run: |
make -f Makefile.smartview generate
git add services/smartview
echo `git commit -m 'sync: generate client with ${{ steps.date.outputs.date }} spec'`
- name: Create Pull Request
id: cpr
uses: peter-evans/create-pull-request@v7
if: ${{ always() && steps.fetch.conclusion == 'success' }}
with:
branch: sync/gh
branch-suffix: timestamp
author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
commit-message: "sync: uncommitted changes detected when opening PR"
title: "feat: API Sync by GitHub Action for @${{ github.event.sender.login }}"
body: |
This API Sync PR was triggered by @${{ github.event.sender.login }} through [GitHub Actions workflow_displatch](https://github.com/equinix/equinix-sdk-go/actions?query=event%3Aworkflow_dispatch)
on ${{ steps.date.outputs.date }}.

* latest Swagger is fetched
* patches have been applied
* generated client has been updated
delete-branch: true
draft: ${{ steps.patch.conclusion == 'failure' || steps.generate.conclusion == 'failure' }}
- name: Comment for failed patch
uses: mshick/add-pr-comment@v2
if: ${{ always() && steps.patch.conclusion == 'failure' && steps.cpr.conclusion == 'success' }}
with:
issue: ${{ steps.cpr.outputs.pull-request-number }}
message: Failed to patch latest spec. Someone with write access must fix this PR manually and then convert it from Draft status to Ready for Review.
- name: Comment for failed generate
uses: mshick/add-pr-comment@v2
if: ${{ always() && steps.generate.conclusion == 'failure' && steps.cpr.conclusion == 'success' }}
with:
issue: ${{ steps.cpr.outputs.pull-request-number }}
message: Failed to generate code from latest patched spec. Someone with write access must fix this PR manually and then convert it from Draft status to Ready for Review.
- name: Check outputs
if: ${{ always() && steps.cpr.conclusion == 'success' }}
run: |
echo "Pull Request Number - ${{ steps.cpr.outputs.pull-request-number }}"
echo "Pull Request URL - ${{ steps.cpr.outputs.pull-request-url }}"
5 changes: 4 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,12 @@ USER_AGENT=${GIT_REPO}/${PACKAGE_VERSION}

OPENAPI_IMAGE_TAG=v7.4.0
OPENAPI_IMAGE=openapitools/openapi-generator-cli:${OPENAPI_IMAGE_TAG}
GOSWAGGER_IMAGE_TAG=v0.31.0
GOSWAGGER_IMAGE=ghcr.io/go-swagger/go-swagger:${GOSWAGGER_IMAGE_TAG}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

go-swagger was the tool I ultimately settled on after trying a number of others. The common failing in other "combine" tools was that they left $ref: spec_1.yaml#/... in the combined spec documents. The go-swagger flatten feature (or attribute of combining) is what was missing in the other tools.

CRI=docker # nerdctl
CRI_COMMAND_BASE=${CRI} run --rm -u ${CURRENT_UID}:${CURRENT_GID} $(DOCKER_EXTRA_ARGS)
OPENAPI_GENERATOR=${CRI_COMMAND_BASE} -v $(CURDIR):/local ${OPENAPI_IMAGE}
GOSWAGGER=${CRI_COMMAND_BASE} -v $(CURDIR):/local -w /local ${GOSWAGGER_IMAGE}
OPENAPI_GENERATOR=${CRI_COMMAND_BASE} -v $(CURDIR):/local -w /local ${OPENAPI_IMAGE}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added -w /local because the argument references sometimes lead with /local and sometimes didn't. As I moved between local tools and docker tools the /local part of the argument list needed to be removed. Working out of /local means we don't need to lead with /local, but it doesn't hurt to have it.

SPEC_FETCHER=${CRI_COMMAND_BASE} -v $(CURDIR):/workdir --entrypoint sh mikefarah/yq:4.30.8 script/download_spec.sh
MIN_GO_VERSION=1.19
GO_CMD=${CRI_COMMAND_BASE} -v $(CURDIR):/workdir -w /workdir -e GOCACHE=/tmp/.cache golang:${MIN_GO_VERSION}
Expand Down
90 changes: 90 additions & 0 deletions Makefile.smartview
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
.PHONY: all pull fetch patch combine generate clean codegen mod docs move-other patch-post fmt test stage validate

include Makefile

PACKAGE_NAME=smartview
# OAS2:
SOURCE_URLS=https://developer.equinix.com/sites/default/files/smartview-v2-catalog-environmental_v2_0.json \
https://developer.equinix.com/sites/default/files/2024-09/power_v1.json \
https://developer.equinix.com/sites/default/files/2024-09/environmental_v1.json \
https://developer.equinix.com/sites/default/files/smartview-v2-catalog-feed-subscription_v2_3.json \
https://developer.equinix.com/sites/default/files/smartview-v1-catalog-alert_v1_0.yaml \
https://developer.equinix.com/sites/default/files/smartview-v1-catalog-assets_v1_1.json \
https://developer.equinix.com/sites/default/files/smartview-v1-catalog-hierarchy_v1_2.json
# TODO(OAS3): https://developer.equinix.com/sites/default/files/smartview-v2-catalog-system-alerts_v2_2.json
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OAS3 spec was excluded.

SPEC_ROOT_FILE=swagger.yaml

CODE_DIR=${CODE_BASE_DIR}/${PACKAGE_NAME}
TEMPLATE_DIR=${TEMPLATE_BASE_DIR}/${PACKAGE_NAME}
SPEC_FETCHED_DIR=${SPEC_BASE_DIR}/${PACKAGE_NAME}/oas3.fetched
SPEC_PATCH_DIR=${SPEC_BASE_DIR}/${PACKAGE_NAME}/patches
SPEC_PATCHED_DIR=${SPEC_BASE_DIR}/${PACKAGE_NAME}/oas3.patched
MERGED_SPEC_FILE=${SPEC_BASE_DIR}/${PACKAGE_NAME}/merged-spec.yaml
SPEC_FETCHER=curl

all: pull fetch patch combine generate stage

generate: clean codegen remove-unused patch-post mod fmt test

pull:
${CRI} pull ${OPENAPI_IMAGE}

# Fetch multiple source URLs into SPEC_FETCHED_DIR with numeric suffixes
fetch:
mkdir -p ${SPEC_FETCHED_DIR}
rm -f ${SPEC_FETCHED_DIR}/*
$(eval INDEX=1)
for URL in ${SOURCE_URLS}; do \
${SPEC_FETCHER} \
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For whatever reason, these curl requests are failing. I believe this was a problem we had back in early versions of metal-go that was worked-around by adding a meaningful User-Agent. No such luck this time.

-H 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36 Edg/131.0.0.0' \
-o ${SPEC_FETCHED_DIR}/spec_$$INDEX.yaml $$URL; \
INDEX=$$((INDEX + 1)); \
displague marked this conversation as resolved.
Show resolved Hide resolved
done

patch:
rm -rf ${SPEC_PATCHED_DIR}
cp -r ${SPEC_FETCHED_DIR} ${SPEC_PATCHED_DIR}

for diff in $(shell set -x; find ${SPEC_PATCH_DIR} -name '*.patch' | sort -n); do \
patch --no-backup-if-mismatch -N -t -p1 -i $$diff; \
done

combine:
${GOSWAGGER} mixin ${SPEC_FETCHED_DIR}/spec* --output=${SPEC_FETCHED_DIR}/merged-spec.yaml --format=yaml || true
${GOSWAGGER} flatten --with-flatten=minimal ${SPEC_FETCHED_DIR}/merged-spec.yaml --output=${SPEC_FETCHED_DIR}/${SPEC_ROOT_FILE} --format=yaml
# rm ${SPEC_FETCHED_DIR}/merged-spec.yaml ${SPEC_FETCHED_DIR}/spec*
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll want to remove the source spec files when they can be fetched reliably.


patch-post:
# patch is idempotent, always starting with the generated files
for diff in $(shell find patches/services/${PACKAGE_NAME} -name \*.patch | sort -n); do \
patch --no-backup-if-mismatch -N -t -p1 -i $$diff; \
done

clean:
rm -rf $(CODE_DIR) ${MERGED_SPEC_FILE}

codegen:
${OPENAPI_GENERATOR} generate -g go \
--package-name ${PACKAGE_NAME} \
--additional-properties=useOneOfDiscriminatorLookup=true \
--http-user-agent "${USER_AGENT}" \
-p packageVersion=${PACKAGE_VERSION} \
--git-user-id ${GIT_ORG} \
--git-repo-id ${GIT_REPO}/services \
-c /local/config/openapi-generator.json \
-t /local/${TEMPLATE_DIR} \
-o /local/${CODE_DIR} \
--skip-validate-spec \
--openapi-normalizer FIX_DUPLICATED_OPERATIONID=true \
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FIX_DUPLICATED_OPERATIONID may not be needed once the duplicated spec_.yaml / spec_2.yaml is removed.

-i /local/${SPEC_FETCHED_DIR}/${SPEC_ROOT_FILE}

validate:
${OPENAPI_GENERATOR} validate \
--recommend \
-i /local/${SPEC_FETCHED_DIR}/${SPEC_ROOT_FILE}

remove-unused:
rm -rf ${CODE_DIR}/api \
${CODE_DIR}/.travis.yml \
${CODE_DIR}/git_push.sh \
${CODE_DIR}/.openapi-generator
Empty file.
24 changes: 24 additions & 0 deletions services/smartview/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# Compiled Object files, Static and Dynamic libs (Shared Objects)
*.o
*.a
*.so

# Folders
_obj
_test

# Architecture specific extensions/prefixes
*.[568vq]
[568vq].out

*.cgo1.go
*.cgo2.c
_cgo_defun.c
_cgo_gotypes.go
_cgo_export.*

_testmain.go

*.exe
*.test
*.prof
23 changes: 23 additions & 0 deletions services/smartview/.openapi-generator-ignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# OpenAPI Generator Ignore
# Generated by openapi-generator https://github.com/openapitools/openapi-generator

# Use this file to prevent files from being overwritten by the generator.
# The patterns follow closely to .gitignore or .dockerignore.

# As an example, the C# client generator defines ApiClient.cs.
# You can make changes and tell OpenAPI Generator to ignore just this file by uncommenting the following line:
#ApiClient.cs

# You can match any string of characters against a directory, file or extension with a single asterisk (*):
#foo/*/qux
# The above matches foo/bar/qux and foo/baz/qux, but not foo/bar/baz/qux

# You can recursively match patterns against a directory, file or extension with a double asterisk (**):
#foo/**/qux
# This matches foo/bar/qux, foo/baz/qux, and foo/bar/baz/qux

# You can also negate patterns with an exclamation (!).
# For example, you can ignore all files in a docs folder with the file extension .md:
#docs/*.md
# Then explicitly reverse the ignore rule for a single file:
#!docs/README.md
Loading
Loading