[BUG] Native Plugin Installation request for .asc and OpenSearch repo has the wrong Public Key #996

peterzhuamazon · 2021-07-21T22:34:04Z

Describe the bug

Two issues:

When install the native plugin directly, opensearch-plugin request to use .asc file for validation instead of .sig file, which we included in the signing and release process.

$ ./opensearch-plugin install repository-s3
-> Installing repository-s3
-> Downloading repository-s3 from opensearch
[=================================================] 100%
-> Failed installing repository-s3
-> Rolling back repository-s3
-> Rolled back repository-s3
Exception in thread "main" java.io.IOException: Server returned HTTP response code: 403 for URL: https://artifacts.opensearch.org/releases/plugins/repository-s3/1.0.0/repository-s3-1.0.0.zip.asc
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1932)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1528)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:224)
	at java.base/java.net.URL.openStream(URL.java:1167)
	at org.opensearch.plugins.InstallPluginCommand.urlOpenStream(InstallPluginCommand.java:482)
	at org.opensearch.plugins.InstallPluginCommand.verifySignature(InstallPluginCommand.java:622)
	at org.opensearch.plugins.InstallPluginCommand.downloadAndValidate(InstallPluginCommand.java:600)
	at org.opensearch.plugins.InstallPluginCommand.download(InstallPluginCommand.java:306)
	at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:260)
	at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:237)
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:99)
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:140)
	at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:104)
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:140)
	at org.opensearch.cli.Command.main(Command.java:103)
	at org.opensearch.plugins.PluginCli.main(PluginCli.java:60)

OpenSearch includes the wrong Public Key here:
05a6fde

The correct public key is here:
https://opensearch.org/verify-signatures.html
https://artifacts.opensearch.org/publickeys/opensearch.pgp

To Reproduce
See Above

Expected behavior
Install success

Plugins
analysis-icu
analysis-kuromoji
analysis-nori
analysis-phonetic
analysis-smartcn
analysis-stempel
analysis-ukrainian
discovery-azure-classic
discovery-ec2
discovery-gce
ingest-attachment
mapper-annotated-text
mapper-murmur3
mapper-size
repository-azure
repository-gcs
repository-hdfs
repository-s3
store-smb
transport-nio

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

OS: [e.g. iOS]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

The text was updated successfully, but these errors were encountered:

saratvemulapalli · 2021-07-21T22:50:53Z

Answering questions in order:

OpenSearch expects the plugins to be signed by .asc and specifically looks for that file.
Ref:

OpenSearch/distribution/tools/plugin-cli/src/main/java/org/opensearch/plugins/InstallPluginCommand.java

Line 615 in 4906a3c

void verifySignature(final Path zip, final String urlString) throws IOException, PGPException {

We can definitely discuss .asc vs .sig.
The key committed to the repository was in PR 05a6fde

Given that I see these as next steps:

See if there is a short term path for GA customers: i.e without changing the key which was already bundled, can we sign the plugins using the key which was bundled and publish them.
Longer term: Decide .asc vs .sig and if needed update OpenSearch with the new key for next release (v1.1.0)

peterzhuamazon · 2021-07-22T00:14:03Z

Consider not even knowing what key is 05a6fde I would say we wont even able to retrieve the private key to sign.
We dont have to change .asc to .sig but the conversation can still go on. The next release we should bundle the correct public key and use .asc/.sig to validate.
Dont really think we can unblock 1.0.0 GA if the bundled public key is not correct as described in 1.

nknize · 2021-07-22T02:19:02Z

The key from 05a6fde was generated back on April 9th for the public release build to pass. This was communicated back then when the pgp key was created and registered in the MIT keystore and was supposed to be the key used for signing the artifacts. Seems that was forgotten somewhere between April 9th and whenever https://opensearch.org/verify-signatures.html was generated so a different key was created. We can open a PR to use the new key but it would probably be a good idea to revoke the one that was initially created. /cc @camerski

nknize · 2021-07-22T02:21:11Z

Consider not even knowing what key is 05a6fde I would say we wont even able to retrieve the private key to sign.

I've got all the info for the original key if you want to use that one instead of going through the process of revoking the original one.

peterzhuamazon · 2021-07-22T17:30:26Z

Consider not even knowing what key is 05a6fde I would say we wont even able to retrieve the private key to sign.

I've got all the info for the original key if you want to use that one instead of going through the process of revoking the original one.

Sure, lets sync up on this one. Thanks.

dblock · 2021-08-03T16:48:14Z

What are we doing about this?

minalsha · 2021-08-10T19:25:16Z

@peterzhuamazon/@camerski : has this issue been resolved? If not, could you move this issue to the right project?

peterzhuamazon · 2021-08-10T20:57:01Z

Hi @nknize can we have a talk on this? Thanks.

peterzhuamazon · 2021-08-10T21:16:43Z

Apparently it is only doing this for official plugins:

OpenSearch/distribution/tools/plugin-cli/src/main/java/org/opensearch/plugins/InstallPluginCommand.java

Line 599 in 4906a3c

if (officialPlugin) {

The plugins.txt is generated and save somewhere so all the native plugins are going through the verification:

OpenSearch/server/build.gradle

Line 210 in 7d18eaf

File pluginsFile = new File(buildDir, 'generated-resources/plugins.txt')

We need to discuss more on how to properly resolve this issue.

nknize · 2021-08-10T21:29:40Z

Here's the original key registered with pgp.mit.edu that matches the public_key.asc file in PR #512.

What do you want to do? Use it, or revoke it in favor of the other one (4096R/67F3A6AA) that was created?

camerski · 2021-08-11T01:27:50Z

As we discussed today, the preferred solution for OpenSearch 1.1.0 and beyond is to change the signature check to use the user's normal PGP keyring so they can trust whichever public keys they choose, and add an option to the plugin installer to skip signature verification.

For existing users we are discussing a couple of options for how to get them unblocked until 1.1.0 is ready. The options are:

Create a second set of signatures for the core plugins using the hard-coded key from PR Update Plugin Signing Key #512 and publish those.
In OpenSearch 1.0.1, sign the core plugins using the main OpenSearch signing key and update the hard-coded key to the correct public master key. No support for unblocking users of 1.0.0 with this option.
Wait for 1.1.0 (not a great option, obviously, but ruling it out should be a decision, not an assumption).
Add the skip-signature-verification argument to the plugin installer in OpenSearch 1.0.1

Any other options? Thoughts?

AmiStrn · 2021-08-31T11:28:16Z

@camerski
What is the status of this issue?
We could really use this fix, as we need to be using these plugins to use v1.0.0 or v1.0.1 but definitely cant wait for v1.1

As we discussed today

Where was that discussed? was it in a thread in the forums? Was it in an internal AWS meeting perhaps?

For existing users we are discussing a couple of options for how to get them unblocked until 1.1.0 is ready

a) please discuss in the open, as this is an open-source project that is supposedly "community driven" and "community-owned".
b) What would you suggest to community members that require this fix in order to upgrade to opensearch? Should they stop their upgrade efforts because of this? If so, we are going to need to add a breaking change or notice to v1.0.0 so users don't upgrade from a perfectly good ES cluster to one that right away breaks...

CEHENKLE · 2021-08-31T18:43:32Z

@bbarani @peterzhuamazon Hey Peter, I've assigned this to you, but if you're the wrong person, please feel free to assign. Can you answer @AmiStrn's question?

Thanks!
/C

peterzhuamazon · 2021-09-01T20:08:42Z

Hi @CEHENKLE I think @nknize already have all the information and discussion with us.
Per @camerski we have already proposed the possible solutions both in short term and long term.
Thanks.

CEHENKLE · 2021-09-07T18:46:50Z

Talking to @nknize, it sounds like the only thing left to do is to update the public key to match what Marc published on pgp.mit.edu

@bbarani I don't believe we have permissions do to that -- can you assign someone in infra to do this?

Thanks!

/C

peterzhuamazon · 2021-09-07T19:03:12Z

Hi @CEHENKLE,

Nick need to add either a skip signature check param, or implement read public key from keyring.
If 1 not doable for the next release, then we need to replace the existing public key save in OpenSearch repo to the one actual public key we use and posted on opensearch.org (https://artifacts.opensearch.org/publickeys/opensearch.pgp)
When generating all the new native plugin zips, we need to convert the sig files to .asc so the existing functionalities can verify the signatures.

Thanks.

bbarani · 2021-09-07T22:07:10Z

@nknize @CEHENKLE Please let us know if there is a plan to fix this in 1.1 release. This is not tied to 1.1 release but we would like to close this out as soon as possible to unblock community from installing those plugins.

nknize · 2021-09-08T16:03:18Z

Looking into this and will update labels shortly.

andresbravog · 2021-10-15T09:20:02Z

This is still happening in tarball 1.1.0 release.

AmiStrn · 2021-10-15T09:21:12Z

This is still happening in tarball 1.1.0 release.

Using same reproduce steps?

andresbravog · 2021-10-15T10:24:37Z

Sorry for the lack of description, this is the setup script I'm using:

export OPENSEARCH_VERSION=1.1.0 && jdk_switcher home openjdk11 && jdk_switcher use openjdk11
OPENSEARCH_VERSION=${OPENSEARCH_VERSION:="1.1.0"}
OPENSEARCH_PORT=${OPENSEARCH_PORT:="9333"}
OPENSEARCH_DIR=${OPENSEARCH_DIR:="$HOME/opensearch"}


OPENSEARCH_DL_URL="https://artifacts.opensearch.org/releases/bundle/opensearch/${OPENSEARCH_VERSION}/opensearch-${OPENSEARCH_VERSION}-linux-x64.tar.gz"
OPENSEARCH_PLUGIN_BIN="${OPENSEARCH_DIR}/opensearch-${OPENSEARCH_VERSION}/bin/opensearch-plugin"

set -e

function wait_for_opensearch() {
    echo -n "Waiting for Opensearch to come up..."
    while true; do
	curl -XGET -s http://localhost:${OPENSEARCH_PORT}/ --insecure >/dev/null && break
	sleep 1
	echo -n .
    done
    echo " OK."
}

rm -rf "${OPENSEARCH_DIR}"
mkdir -p "${OPENSEARCH_DIR}"
wget "${OPENSEARCH_DL_URL}"
tar -zxf "opensearch-${OPENSEARCH_VERSION}-linux-x64.tar.gz" --directory "${OPENSEARCH_DIR}"

eval "${OPENSEARCH_PLUGIN_BIN} remove opensearch-security"
eval "${OPENSEARCH_PLUGIN_BIN} install https://artifacts.opensearch.org/releases/plugins/analysis-kuromoji/1.0.0/analysis-kuromoji-1.0.0.zip"
eval "${OPENSEARCH_PLUGIN_BIN} install https://artifacts.opensearch.org/releases/plugins/analysis-icu/1.0.0/analysis-icu-1.0.0.zip"
eval "${OPENSEARCH_PLUGIN_BIN} install https://artifacts.opensearch.org/releases/plugins/analysis-phonetic/1.0.0/analysis-phonetic-1.0.0.zip"
eval "${OPENSEARCH_PLUGIN_BIN} install https://artifacts.opensearch.org/releases/plugins/analysis-smartcn/1.0.0/analysis-smartcn-1.0.0.zip"
eval "${OPENSEARCH_PLUGIN_BIN} install https://artifacts.opensearch.org/releases/plugins/analysis-nori/1.0.0/analysis-nori-1.0.0.zip"

bash -c "${OPENSEARCH_DIR}/opensearch-${OPENSEARCH_VERSION}/opensearch-tar-install.sh -Ecluster.name=opensearch-cluster -Enode.name=opensearch-node1 -Ehttp.host=0.0.0.0 -Ediscovery.type=single-node -Ehttp.port=${OPENSEARCH_PORT} 2>&1 >/dev/null" >/dev/null & disown
wait_for_opensearch

It fails to install plugins with the same 403 error

-> Installing https://artifacts.opensearch.org/releases/plugins/analysis-icu/1.0.0/analysis-icu-1.0.0.zip
-> Downloading https://artifacts.opensearch.org/releases/plugins/analysis-icu/1.0.0/analysis-icu-1.0.0.zip
-> Failed installing https://artifacts.opensearch.org/releases/plugins/analysis-icu/1.0.0/analysis-icu-1.0.0.zip
-> Rolling back https://artifacts.opensearch.org/releases/plugins/analysis-icu/1.0.0/analysis-icu-1.0.0.zip
-> Rolled back https://artifacts.opensearch.org/releases/plugins/analysis-icu/1.0.0/analysis-icu-1.0.0.zip
Exception in thread "main" java.io.IOException: Server returned HTTP response code: 403 for URL: https://artifacts.opensearch.org/releases/plugins/analysis-icu/1.0.0/analysis-icu-1.0.0.zip
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1924)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:250)
	at org.opensearch.plugins.InstallPluginCommand.downloadZip(InstallPluginCommand.java:438)
	at org.opensearch.plugins.InstallPluginCommand.download(InstallPluginCommand.java:328)
	at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:260)
	at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:237)
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:99)
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:140)
	at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:104)
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:140)
	at org.opensearch.cli.Command.main(Command.java:103)
	at org.opensearch.plugins.PluginCli.main(PluginCli.java:60)

Here is a DockerFile that fully reproduces the issue

FROM ubuntu:bionic

# Fix issue with TZ and dpkg not allowing to apt-get update or install new
# packages
ENV TZ 'Etc/UTC'
RUN echo $TZ > /etc/timezone && \
  apt-get update && apt-get install -y tzdata && \
  rm /etc/localtime && \
  ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && \
  dpkg-reconfigure -f noninteractive tzdata && \
  apt-get clean

  # install standard packages
RUN apt-get update && apt-get install -y \
    build-essential \
    curl \
    libapr1-dev \
    libaprutil1-dev \
    libcgi-session-perl \
    libcurl4 \
    libimage-exiftool-perl \
    libpq5 \
    libpq-dev \
    libreadline7 \
    libreadline-dev \
    libssl-dev \
    libxml2-dev \
    libxslt-dev \
    libyaml-dev \
    libldap2-dev \
    libidn11-dev \
    openssl \
    software-properties-common \
    ssl-cert \
    unison \
    vim \
    wget \
    zlib1g \
    zlib1g-dev \
    idn

ENV OPENSEARCH_VERSION "1.1.0"
ENV OPENSEARCH_PORT "9333"
ENV OPENSEARCH_DIR "/opensearch"

ENV OPENSEARCH_DL_URL "https://artifacts.opensearch.org/releases/bundle/opensearch/${OPENSEARCH_VERSION}/opensearch-${OPENSEARCH_VERSION}-linux-x64.tar.gz"
ENV OPENSEARCH_PLUGIN_BIN "${OPENSEARCH_DIR}/opensearch-${OPENSEARCH_VERSION}/bin/opensearch-plugin"

RUN rm -rf "${OPENSEARCH_DIR}"
RUN mkdir -p "${OPENSEARCH_DIR}"
RUN wget "${OPENSEARCH_DL_URL}"
RUN tar -zxf "opensearch-${OPENSEARCH_VERSION}-linux-x64.tar.gz" --directory "${OPENSEARCH_DIR}"

RUN eval "${OPENSEARCH_PLUGIN_BIN} remove opensearch-security"
RUN eval "${OPENSEARCH_PLUGIN_BIN} install https://artifacts.opensearch.org/releases/plugins/analysis-kuromoji/1.0.0/analysis-kuromoji-1.0.0.zip"

CMD "${OPENSEARCH_DIR}/opensearch-${OPENSEARCH_VERSION}/opensearch-tar-install.sh -Ecluster.name=opensearch-cluster -Enode.name=opensearch-node1 -Ehttp.host=0.0.0.0 -Ediscovery.type=single-node -Ehttp.port=${OPENSEARCH_PORT}"

peterzhuamazon · 2021-10-15T16:32:01Z

You are installing 1.0.0 version of plugin on 1.1.0 which have different sig.

https://artifacts.opensearch.org/releases/plugins/analysis-kuromoji/1.0.0/

peterzhuamazon · 2021-10-15T16:32:38Z

You can just opensearch-plugin install analysis-kuromoji and it will auto pick up the 1.1.0 version.

andresbravog · 2021-10-17T12:32:54Z

Thanks, @peterzhuamazon that fixed the error.

peterzhuamazon added bug Something isn't working untriaged Beta labels Jul 21, 2021

peterzhuamazon mentioned this issue Jul 21, 2021

Pre-compile and host native plugins on artifacts.opensearch.org opensearch-project/opensearch-build#86

Closed

saratvemulapalli removed the untriaged label Jul 21, 2021

dblock removed the Beta label Jul 22, 2021

minalsha added the stalled Issues that have stalled label Aug 10, 2021

gaiksaya mentioned this issue Aug 11, 2021

[BUG] Problem installing plugins from artifacts.opensearch.org opensearch-project/opensearch-build#188

Closed

peterzhuamazon mentioned this issue Aug 30, 2021

[BUG] 403 for URL: https://artifacts.opensearch.org/releases/plugins/repository-s3/1.0.0/repository-s3-1.0.0.zip.asc #1169

Closed

CEHENKLE assigned peterzhuamazon Aug 31, 2021

CEHENKLE assigned bbarani Sep 7, 2021

bbarani unassigned peterzhuamazon and bbarani Sep 7, 2021

bbarani assigned nknize and CEHENKLE Sep 7, 2021

nknize mentioned this issue Sep 9, 2021

[Bug] Fix InstallPluginCommand to use proper key signatures #1233

Merged

adnapibar closed this as completed in #1233 Sep 10, 2021

peternied mentioned this issue Sep 14, 2021

[BUG] Install of analysis-icu plugin fails in beta1 opensearch-project/opensearch-build#180

Closed

peterzhuamazon mentioned this issue Oct 15, 2021

Weekly Community Engagement on 2021/10/12 - 2021/10/25 opensearch-project/opensearch-build#720

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] Native Plugin Installation request for .asc and OpenSearch repo has the wrong Public Key #996

[BUG] Native Plugin Installation request for .asc and OpenSearch repo has the wrong Public Key #996

peterzhuamazon commented Jul 21, 2021 •

edited

Loading

saratvemulapalli commented Jul 21, 2021 •

edited

Loading

peterzhuamazon commented Jul 22, 2021

nknize commented Jul 22, 2021

nknize commented Jul 22, 2021

peterzhuamazon commented Jul 22, 2021

dblock commented Aug 3, 2021

minalsha commented Aug 10, 2021

peterzhuamazon commented Aug 10, 2021

peterzhuamazon commented Aug 10, 2021

nknize commented Aug 10, 2021

camerski commented Aug 11, 2021

AmiStrn commented Aug 31, 2021 •

edited

Loading

CEHENKLE commented Aug 31, 2021

peterzhuamazon commented Sep 1, 2021

CEHENKLE commented Sep 7, 2021

peterzhuamazon commented Sep 7, 2021 •

edited

Loading

bbarani commented Sep 7, 2021

nknize commented Sep 8, 2021

andresbravog commented Oct 15, 2021

AmiStrn commented Oct 15, 2021

andresbravog commented Oct 15, 2021 •

edited

Loading

peterzhuamazon commented Oct 15, 2021

peterzhuamazon commented Oct 15, 2021

andresbravog commented Oct 17, 2021

[BUG] Native Plugin Installation request for .asc and OpenSearch repo has the wrong Public Key #996

[BUG] Native Plugin Installation request for .asc and OpenSearch repo has the wrong Public Key #996

Comments

peterzhuamazon commented Jul 21, 2021 • edited Loading

saratvemulapalli commented Jul 21, 2021 • edited Loading

peterzhuamazon commented Jul 22, 2021

nknize commented Jul 22, 2021

nknize commented Jul 22, 2021

peterzhuamazon commented Jul 22, 2021

dblock commented Aug 3, 2021

minalsha commented Aug 10, 2021

peterzhuamazon commented Aug 10, 2021

peterzhuamazon commented Aug 10, 2021

nknize commented Aug 10, 2021

camerski commented Aug 11, 2021

AmiStrn commented Aug 31, 2021 • edited Loading

CEHENKLE commented Aug 31, 2021

peterzhuamazon commented Sep 1, 2021

CEHENKLE commented Sep 7, 2021

peterzhuamazon commented Sep 7, 2021 • edited Loading

bbarani commented Sep 7, 2021

nknize commented Sep 8, 2021

andresbravog commented Oct 15, 2021

AmiStrn commented Oct 15, 2021

andresbravog commented Oct 15, 2021 • edited Loading

peterzhuamazon commented Oct 15, 2021

peterzhuamazon commented Oct 15, 2021

andresbravog commented Oct 17, 2021

peterzhuamazon commented Jul 21, 2021 •

edited

Loading

saratvemulapalli commented Jul 21, 2021 •

edited

Loading

AmiStrn commented Aug 31, 2021 •

edited

Loading

peterzhuamazon commented Sep 7, 2021 •

edited

Loading

andresbravog commented Oct 15, 2021 •

edited

Loading