Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] [Sftp Reader] If there are wildcard file names in the sftp directory and there happen to be other matching file names, a uniqueness error will occur #8500

Open
3 tasks done
e-mhui opened this issue Jan 11, 2025 · 0 comments · May be fixed by #8501
Open
3 tasks done

[Bug] [Sftp Reader] If there are wildcard file names in the sftp directory and there happen to be other matching file names, a uniqueness error will occur #8500

e-mhui opened this issue Jan 11, 2025 · 0 comments · May be fixed by #8501
Labels
bug

Comments

@e-mhui
Copy link

e-mhui commented Jan 11, 2025

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

If there are wildcard file names in the sftp directory and there happen to be other matching file names, a uniqueness error will occur.
For example, there are two files, a * b.txt and abc.txt, in the /data directory. When configuring/data/a * b.txt, a uniqueness error occurs.
Clipboard_Screenshot_1736600413

SeaTunnel Version

dev

SeaTunnel Config

env {
  parallelism = 1
  job.mode = "BATCH"
}

source {
  SftpFile {
    host = ""
    port = xxx
    user = xxx
    password = xxx
    path = "/data/a*b.txt"
    file_format_type = "text"
    field_delimiter = ","
    schema = {
      fields {
        c_int = int
        c_string = string
      }
    }
  }
}

transform {
}

sink {
  console {
  }
}

Running Command

`org.apache.seatunnel.example.engine.SeaTunnelEngineLocalExample`


public static void main(String[] args)
            throws FileNotFoundException, URISyntaxException, CommandException {
        String configurePath = args.length > 0 ? args[0] : "/examples/sftp_to_console.conf";
        String configFile = getTestConfigFile(configurePath);
        ClientCommandArgs clientCommandArgs = new ClientCommandArgs();
        clientCommandArgs.setConfigFile(configFile);
        clientCommandArgs.setCheckConfig(false);
        clientCommandArgs.setJobName(Paths.get(configFile).getFileName().toString());
        // Change Execution Mode to CLUSTER to use client mode, before do this, you should start
        // SeaTunnelEngineClusterServerExample
        clientCommandArgs.setMasterType(MasterType.LOCAL);
        SeaTunnel.run(clientCommandArgs.buildCommand());
    }

Error Exception

Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: org.apache.seatunnel.connectors.seatunnel.file.exception.FileConnectorException: ErrorCode:[FILE-08], ErrorDescription:[File read failed] - Read data from this file [default.default.default_sftp://xxxx/data/aa*bbb.txt] failed
    at org.apache.seatunnel.connectors.seatunnel.file.source.reader.MultipleTableFileSourceReader.pollNext(MultipleTableFileSourceReader.java:85)
    at org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle.collect(SourceFlowLifeCycle.java:159)
    at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.collect(SourceSeaTunnelTask.java:127)
    at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:169)
    at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.call(SourceSeaTunnelTask.java:132)
    at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:694)
    at org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:1019)
    at org.apache.seatunnel.api.tracing.MDCRunnable.run(MDCRunnable.java:43)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: 4: /data/aa*bbb.txt is not unique: [/data/aa123bbb.txt, /data/aa*bbb.txt]
    at org.apache.seatunnel.connectors.seatunnel.file.sftp.system.SFTPFileSystem.open(SFTPFileSystem.java:495)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:899)
    at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy.lambda$getInputStream$11(HadoopFileSystemProxy.java:195)
    at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy.execute(HadoopFileSystemProxy.java:327)
    at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy.getInputStream(HadoopFileSystemProxy.java:195)
    at org.apache.seatunnel.connectors.seatunnel.file.source.reader.AbstractReadStrategy.resolveArchiveCompressedInputStream(AbstractReadStrategy.java:272)
    at org.apache.seatunnel.connectors.seatunnel.file.source.reader.TextReadStrategy.read(TextReadStrategy.java:71)
    at org.apache.seatunnel.connectors.seatunnel.file.source.reader.MultipleTableFileSourceReader.pollNext(MultipleTableFileSourceReader.java:81)
    ... 12 more

Zeta or Flink or Spark Version

No response

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Hotfix][Connector-V2][SFTP] Add quote to sftp file names with wildcard characters e-mhui/seatunnel
1 participant
@e-mhui