Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

File Stream Adapter ignores speed up factor #2856

Closed
FabriceHansen opened this issue May 13, 2024 · 0 comments · Fixed by #2858
Closed

File Stream Adapter ignores speed up factor #2856

FabriceHansen opened this issue May 13, 2024 · 0 comments · Fixed by #2858
Labels
bug Something isn't working
Milestone
0.95.0

Comments

@FabriceHansen
Copy link

Apache StreamPipes version

dev (current development state)

Affected StreamPipes components

UI

What happened?

We try to replay a CSV file with a File Stream Adapter but it is not working properly.

The target is to replay 1 row every 100ms, store the rsults in a Data Lake and publish in parallel the results to a websocket Sink.

It seems that csv files (we tried different CSV formats, delimiters and options) rows are processed for all lines immediately.

In „Basic Settings“ „Replay Speed“ gives no change. Any changes in „Speed Up Factor“ result in the same result.

We tried to add a column with milliseconds and convert to DateNumber and mark it as TimeStamp but this was not working.

The UI is losing after saving several options such as:

  • Selected file
  • Timestamp conversion options Milliseconds/Seconds

There is no help available for this Adapter.

How to reproduce?

No response

Expected behavior

No response

Additional technical information

No response

Are you willing to submit a PR?

Yes, I am willing to submit a PR!
@FabriceHansen FabriceHansen added the bug Something isn't working label May 13, 2024
@bossenti bossenti added this to the 0.95.0 milestone May 13, 2024
@bossenti bossenti changed the title File Stream Adapter File Stream Adapter ignores speed up factor May 13, 2024
bossenti added a commit that referenced this issue May 13, 2024
@bossenti
fix(#2856): improve handling of invalid UNIX timestamps
856801e
@bossenti bossenti linked a pull request May 13, 2024 that will close this issue
fix(#2856): improve handling of invalid UNIX timestamps #2858
Merged
@bossenti bossenti mentioned this issue May 13, 2024
Merged
bossenti added a commit that referenced this issue May 15, 2024
@bossenti
fix(#2856): improve handling of invalid UNIX timestamps (#2858)
7dd8ca0 
* docs: improve documentation of file stream adapter

* fix(#2856): improve handling of invalid UNIX timestamps

* minor adaptions

* style: fix line length

* fix: make type casting more robust

* fix: improve handling of missing timestamps

* style: add missing whitespace

* tests: adapt E2E test

* style: adapt file formatting
bossenti added a commit that referenced this issue May 15, 2024
@bossenti
fix(#2856): improve handling of invalid UNIX timestamps (#2858)
061ecc1 
* docs: improve documentation of file stream adapter

* fix(#2856): improve handling of invalid UNIX timestamps

* minor adaptions

* style: fix line length

* fix: make type casting more robust

* fix: improve handling of missing timestamps

* style: add missing whitespace

* tests: adapt E2E test

* style: adapt file formatting
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
0.95.0
Development

Successfully merging a pull request may close this issue.

fix(#2856): improve handling of invalid UNIX timestamps apache/streampipes
2 participants
@bossenti @FabriceHansen