Load functionality for multiple .npy files #1388

Reisii · 2024-02-28T12:16:42Z

Due Diligence

General:
- title of the PR is suitable to appear in the Release Notes
Implementation:
- unit tests: all split configurations tested
- unit tests: multiple dtypes tested
- documentation updated where needed

Description

Many smaller .npy-files will be loaded at the same time

Added to one big DNDarray

Issue/s resolved: #900

Changes proposed:

Type of change

New feature

Memory requirements

Performance

Does this change modify the behaviour of other functions? If so, which?

no

github-actions · 2024-02-28T12:21:20Z

Thank you for the PR!

mrfh92 · 2024-02-28T13:12:01Z

heat/core/manipulations.py

@@ -1,6 +1,7 @@
 """
 Manipulation operations for (potentially distributed) `DNDarray`s.
 """
+


I guess this has been added unintentionally

…_multiple_files_into_one_DNDarray

github-actions · 2024-02-28T13:20:17Z

Thank you for the PR!

codecov · 2024-02-28T14:30:38Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.91%. Comparing base (225dc96) to head (b5a06b2).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1388      +/-   ##
==========================================
+ Coverage   91.88%   91.91%   +0.02%     
==========================================
  Files          80       80              
  Lines       11916    11942      +26     
==========================================
+ Hits        10949    10976      +27     
+ Misses        967      966       -1

Flag	Coverage Δ
unit	`91.91% <100.00%> (+0.02%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

mrfh92 · 2024-03-05T08:31:56Z

heat/core/tests/test_io.py

+            crea_array = np.concatenate(x, 0)
+            np.save("data" + str(i), x)
+
+        array1 = ht.load_npy_from_path("/home/nguy_t4/home/heat/heat/core/tests")


Der Pfad wird natürlich nicht gefunden, da die Tests auf den Runnern nicht auf deinem Rechner laufen... du musst hier einen relativen Pfad angeben, vermutlich /heat/core/tests o.ä.

… input

…opened by the function

…got only filename but not path

…_multiple_files_into_one_DNDarray

…iles_into_one_DNDarray' of github.com:helmholtz-analytics/heat into features/900-Improve_load-functionality_load_multiple_files_into_one_DNDarray

mrfh92 · 2024-03-11T13:09:06Z

As the error

ERROR: test_load_npy (heat.core.tests.test_io.TestIO)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/heat/heat/heat/core/tests/test_io.py", line 760, in test_load_npy
    self.assertEqual(load_array.numpy(), int_array)
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/unittest/case.py", line 912, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/unittest/case.py", line 902, in _baseAssertEqual
    if not first == second:
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

tells, comparing two numpy arrays a and b results in an ambiguous result as an array with the same shape as a and b but with entries True or False is returned and, in general, it is difficult to assign a single True/False to such an outcome.

In our case, however, True should be the outcome if and only if the two arrays coincide in every element. Thus, the .all() function needs to be used as follows:

self.assertTrue((a==b).all())

I have made these changes.

…on one process only now

…king

github-actions · 2024-03-19T14:59:13Z

Thank you for the PR!

…e created files after run

github-actions · 2024-03-20T09:54:21Z

Thank you for the PR!

github-actions · 2024-07-02T08:25:59Z

Thank you for the PR!

…iles_into_one_DNDarray' of github.com:helmholtz-analytics/heat into features/900-Improve_load-functionality_load_multiple_files_into_one_DNDarray

github-actions · 2024-07-03T07:19:35Z

Thank you for the PR!

github-actions · 2024-07-03T07:42:46Z

Thank you for the PR!

Reisii · 2024-07-03T09:35:20Z

heat/core/tests/test_io.py

+                np.save(os.path.join(os.getcwd(), "heat/datasets", "float_data"), x)
+            ht.MPI_WORLD.Barrier()
+            with self.assertRaises(RuntimeError):
+                ht.load_npy_from_path("heat/datasets/npy_dummy", dtype=ht.int64, split=0)


You are loading the dummy file from the wrong directory or you saved it in the wrong directory

thx for the hint 👍

now it should work again

github-actions · 2024-07-03T10:37:07Z

Thank you for the PR!

mrfh92

approve to let full CI-matrix run (test combinations of different Python and PyTorch versions)

mrfh92 · 2024-07-03T12:43:03Z

@mtar if you have time can you have a second look into this PR and review? (to follow the convention not to review own students' work)

mrfh92

approve again to let CI-matrix run

github-actions · 2024-07-03T12:57:51Z

Thank you for the PR!

heat/core/io.py

Co-authored-by: Michael Tarnawa <[email protected]>

for more information, see https://pre-commit.ci

github-actions · 2024-07-04T12:47:51Z

Thank you for the PR!

…_multiple_files_into_one_DNDarray

mrfh92

approve to let CI run; also @mtar has reviewed yesterday already

github-actions · 2024-07-04T14:14:35Z

Thank you for the PR!

Load functionality for .npy files

aac6577

mrfh92 reviewed Feb 28, 2024

View reviewed changes

Merge branch 'main' into features/900-Improve_load-functionality_load…

eec33c1

…_multiple_files_into_one_DNDarray

ClaudiaComito added the I/O label Mar 3, 2024

Added pseudo Unittests, not on par in terms of quality and functionality

44b8383

mrfh92 reviewed Mar 5, 2024

View reviewed changes

Reisii and others added 13 commits March 7, 2024 14:33

Added self.asserts in Unittest and some error messages for wrong user…

68dee6a

… input

Added self.asserts in Unittest and some error messages for wrong user…

7919a90

… input

Deleted old test data

72b557e

Datasets will now be created on the datasets direcotry but cannot be …

5ffadb4

…opened by the function

fixed problem with paths: error was in io.py line 1185 since np.load …

df2800a

…got only filename but not path

removed npy files

c91e029

Some exception testing added

b27c8b7

Some exception testing added

afe23fb

Merge branch 'main' into features/900-Improve_load-functionality_load…

1c2658d

…_multiple_files_into_one_DNDarray

Now stuck on test_load_npy and unable to complete

f5ce751

Now stuck on test_load_npy and unable to complete

bb88962

Merge branch 'features/900-Improve_load-functionality_load_multiple_f…

0d902c3

…iles_into_one_DNDarray' of github.com:helmholtz-analytics/heat into features/900-Improve_load-functionality_load_multiple_files_into_one_DNDarray

added comparison of two numpy arrays

6ed3cbd

Reisii added 4 commits March 13, 2024 14:33

Still not the same arrays, however, the creation of the dataset runs …

62175be

…on one process only now

Fixed split problem

9aafa97

Not so pretty solution for failed self.assertTrue() test, but now wor…

ccd4072

…king

Not so pretty solution for failed self.assertTrue() test, but now wor…

063b382

…king

Added a test for floats and different split axis. Test will now delet…

975884d

…e created files after run

Hoppe added 3 commits July 3, 2024 09:03

Merge branch 'features/900-Improve_load-functionality_load_multiple_f…

3428798

…iles_into_one_DNDarray' of github.com:helmholtz-analytics/heat into features/900-Improve_load-functionality_load_multiple_files_into_one_DNDarray

q# neue Datei: heat/datasets/float_data.npy

c519d51

removed files

6ca3540

mrfh92 and others added 2 commits July 3, 2024 09:20

Update test_io.py

77b937b

fixed wrong unindent

f4fa1d6

Reisii commented Jul 3, 2024

View reviewed changes

Update test_io.py

3666a9c

mrfh92 self-requested a review July 3, 2024 12:38

mrfh92 previously approved these changes Jul 3, 2024

View reviewed changes

mrfh92 requested a review from mtar July 3, 2024 12:39

Update test_io.py

ea30011

mrfh92 dismissed their stale review via ea30011 July 3, 2024 12:49

mrfh92 previously approved these changes Jul 3, 2024

View reviewed changes

mtar reviewed Jul 3, 2024

View reviewed changes

heat/core/io.py Outdated Show resolved Hide resolved

Commited changes from mtar

39a2bc3

Co-authored-by: Michael Tarnawa <[email protected]>

Reisii dismissed mrfh92’s stale review via 39a2bc3 July 4, 2024 12:40

[pre-commit.ci] auto fixes from pre-commit.com hooks

94d2882

for more information, see https://pre-commit.ci

Merge branch 'main' into features/900-Improve_load-functionality_load…

b5a06b2

…_multiple_files_into_one_DNDarray

mrfh92 self-requested a review July 4, 2024 14:06

mrfh92 approved these changes Jul 4, 2024

View reviewed changes

mrfh92 merged commit a774559 into main Jul 4, 2024
53 checks passed

mrfh92 deleted the features/900-Improve_load-functionality_load_multiple_files_into_one_DNDarray branch July 4, 2024 14:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Load functionality for multiple .npy files #1388

Load functionality for multiple .npy files #1388

Reisii commented Feb 28, 2024 •

edited by mrfh92

Loading

github-actions bot commented Feb 28, 2024

mrfh92 Feb 28, 2024

github-actions bot commented Feb 28, 2024

codecov bot commented Feb 28, 2024 •

edited

Loading

mrfh92 Mar 5, 2024

mrfh92 commented Mar 11, 2024

github-actions bot commented Mar 19, 2024

github-actions bot commented Mar 20, 2024

github-actions bot commented Jul 2, 2024

github-actions bot commented Jul 3, 2024

github-actions bot commented Jul 3, 2024

Reisii Jul 3, 2024

mrfh92 Jul 3, 2024

mrfh92 Jul 3, 2024

github-actions bot commented Jul 3, 2024

mrfh92 left a comment

mrfh92 commented Jul 3, 2024 •

edited

Loading

mrfh92 left a comment

github-actions bot commented Jul 3, 2024

github-actions bot commented Jul 4, 2024

mrfh92 left a comment

github-actions bot commented Jul 4, 2024

Load functionality for multiple .npy files #1388

Load functionality for multiple .npy files #1388

Conversation

Reisii commented Feb 28, 2024 • edited by mrfh92 Loading

Due Diligence

Description

Changes proposed:

Type of change

Memory requirements

Performance

Does this change modify the behaviour of other functions? If so, which?

github-actions bot commented Feb 28, 2024

mrfh92 Feb 28, 2024

Choose a reason for hiding this comment

github-actions bot commented Feb 28, 2024

codecov bot commented Feb 28, 2024 • edited Loading

Codecov Report

mrfh92 Mar 5, 2024

Choose a reason for hiding this comment

mrfh92 commented Mar 11, 2024

github-actions bot commented Mar 19, 2024

github-actions bot commented Mar 20, 2024

github-actions bot commented Jul 2, 2024

github-actions bot commented Jul 3, 2024

github-actions bot commented Jul 3, 2024

Reisii Jul 3, 2024

Choose a reason for hiding this comment

mrfh92 Jul 3, 2024

Choose a reason for hiding this comment

mrfh92 Jul 3, 2024

Choose a reason for hiding this comment

github-actions bot commented Jul 3, 2024

mrfh92 left a comment

Choose a reason for hiding this comment

mrfh92 commented Jul 3, 2024 • edited Loading

mrfh92 left a comment

Choose a reason for hiding this comment

github-actions bot commented Jul 3, 2024

github-actions bot commented Jul 4, 2024

mrfh92 left a comment

Choose a reason for hiding this comment

github-actions bot commented Jul 4, 2024

Reisii commented Feb 28, 2024 •

edited by mrfh92

Loading

codecov bot commented Feb 28, 2024 •

edited

Loading

mrfh92 commented Jul 3, 2024 •

edited

Loading