Field Calculator, slow performance with 2.1.0-104 #17922

qgib · 2014-01-09T13:20:32Z

Author Name: joe larson (@oeon)
Original Redmine Issue: 9315
Affected QGIS version: master
Redmine category:vectors
Assignee: Matthias Kuhn

using OSGeo4W64, 2.1.0-104 while trying to update a field in a large dataset (~300MB) with the Field Calculator ...very slow performance was experienced.

Another user in IRC, lrssvt also confirmed experiencing this with this dataset (CARTO1_Lin_6CNivelD.zip) available here #17441-23

Related issue(s): #18100 (relates), #18109 (relates)
Redmine related issue(s): 9509, 9519

The text was updated successfully, but these errors were encountered:

qgib · 2014-01-09T13:47:57Z

Author Name: joe larson (@oeon)

This only occurs when Attribute Table is open.

operating_system was changed from Windows 7 to Windows
os_version was changed from to 7

qgib · 2014-01-09T13:59:44Z

Author Name: Nathan Woodrow (@NathanW2)

assigned_to_id was configured as Matthias Kuhn

qgib · 2014-01-09T14:52:43Z

Author Name: Jürgen Fischer (@jef-n)

Fixed in changeset "ccde424dc14ebc82c97e62d3568b1cf15f3bfda9".

status_id was changed from Open to Closed

qgib · 2014-01-09T21:29:23Z

Author Name: Matthias Kuhn (@m-kuhn)

I've noticed a similar latency recently, but in a release build (i.e. without debug output, so Jürgens commits should not make any differece there). Adding a new calculated column to ~1mio rows was incredibly slow. This was with a build from before the recent changes concerning reliable signalling of attribute changes. I hope to get time to looking into batch updating.

qgib · 2014-01-13T17:42:56Z

Author Name: dr - (dr -)

Also noticed that slow performance occurs only when Attribute Table is open (master, Ubuntu).

operating_system was changed from Windows to
status_id was changed from Closed to Reopened
os_version was changed from 7 to

qgib · 2014-01-26T12:41:59Z

Author Name: Giovanni Manghi (@gioman)

dr - wrote:

Also noticed that slow performance occurs only when Attribute Table is open (master, Ubuntu).

confirmed very slow field calculator performance when table is open.

category_id was configured as Vectors
fixed_version_id was configured as Future Release - High Priority

qgib · 2014-01-26T12:47:08Z

Author Name: Matthias Kuhn (@m-kuhn)

Is it a regression?

qgib · 2014-01-26T13:16:01Z

Author Name: Giovanni Manghi (@gioman)

Matthias Kuhn wrote:

Is it a regression?

it seems faster in qgis 2.0.1

qgib · 2014-01-26T13:30:26Z

Author Name: Matthias Kuhn (@m-kuhn)

I could (partially) revert 5c38775 which could probably help here. But this would in turn reintroduce bug #17882 (updates not pushed to the attribute table)
In the future I would like to have a solution which solves both. For now somebody has to decide, what is more important (or agree on e.g. a threshold feature count for which live-propagating will be turned on/off)

http://lists.osgeo.org/pipermail/qgis-developer/2014-January/030018.html

qgib · 2014-01-29T13:54:40Z

Author Name: Giovanni Manghi (@gioman)

Matthias Kuhn wrote:

I could (partially) revert 5c38775 which could probably help here. But this would in turn reintroduce bug #17882 (updates not pushed to the attribute table)
In the future I would like to have a solution which solves both. For now somebody has to decide, what is more important (or agree on e.g. a threshold feature count for which live-propagating will be turned on/off)

http://lists.osgeo.org/pipermail/qgis-developer/2014-January/030018.html

personally if I have to choose I would prefer speed (in table edits) rather than #17882 fixed.

qgib · 2014-01-30T05:32:36Z

Author Name: Matthias Kuhn (@m-kuhn)

I have created a branch which could potentially fix both issues, but I did not have any time at all to test it.

If somebody is able to test this and give a feedback... It would be great to have these things solved, but didn't even dare to create a pull request so far.

https://github.com/matthias-kuhn/QGIS/tree/tblspeed

qgib · 2014-01-30T12:33:12Z

Author Name: Matthias Kuhn (@m-kuhn)

Fixed in changeset "d378f942a50ad25cc24570e125ea3724e09d79d6".

status_id was changed from Reopened to Closed

qgib · 2014-01-30T12:37:11Z

Author Name: Matthias Kuhn (@m-kuhn)

My propsed fix was based on wrong assumptions, the main reason for the bad performance was not the fix for #17882 but another change introduced for the relations, which led to tons of unnecessary requests. Fixing that brought back an acceptable speed and the proposed fix implemented this morning gave another remarkable speed boost.

Please help testing thiese fixes and report any related bugs or reopen if this issue should still not be solved for any reason.

status_id was changed from Closed to Reopened

qgib · 2014-01-30T12:37:26Z

Author Name: Matthias Kuhn (@m-kuhn)

status_id was changed from Reopened to Closed

qgib · 2014-02-01T12:42:07Z

Author Name: Salvatore Larosa (@slarosa)

Hi Matthias,
I just noticed that with large shapefiles (over ~70k features)
when turn off the edit mode and you choose "close without saving" QGIS freezes (not crashes).

qgib · 2014-02-02T00:08:42Z

Author Name: Giovanni Manghi (@gioman)

Salvatore Larosa wrote:

Hi Matthias,
I just noticed that with large shapefiles (over ~70k features)
when turn off the edit mode and you choose "close without saving" QGIS freezes (not crashes).

Hi Matthias and Salvatore,

are the fixes already committed in master or to test I have to use Matthias branch?

thanks!

qgib · 2014-02-02T00:27:57Z

Author Name: Salvatore Larosa (@slarosa)

Giovanni Manghi wrote:

are the fixes already committed in master or to test I have to use Matthias branch?

Yes, they are in master branch now.

qgib · 2014-02-07T10:42:03Z

Author Name: Giovanni Manghi (@gioman)

Salvatore Larosa wrote:

Hi Matthias,
I just noticed that with large shapefiles (over ~70k features)
when turn off the edit mode and you choose "close without saving" QGIS freezes (not crashes).

very true, when discarding edits it take ages to exit the freeze state.

qgib · 2014-02-09T03:24:06Z

Author Name: Matthias Kuhn (@m-kuhn)

I noticed the same. This is due to the undo operation requesting all the features in single requests AFAICT.

Please note that this also happens with the attribute table closed and should therefore be treated as a separate issue.

See #18109

qgib · 2014-02-09T04:28:32Z

Author Name: Jürgen Fischer (@jef-n)

Matthias Kuhn wrote:

Please note that this also happens with the attribute table closed and should therefore be treated as a separate issue.

See #18109

Giovanni already filed #18100.

qgib · 2017-08-07T23:05:10Z

Author Name: Jürgen Fischer (@jef-n)

description was changed from using OSGeo4W64, 2.1.0-104 while trying to update a field in a large dataset (~300MB) with the Field Calculator ...very slow performance was experienced.

Another user in IRC, lrssvt also confirmed experiencing this with this dataset (CARTO1_Lin_6CNivelD.zip) available here https://issues.qgis.org/issues/8725#note-23 to using OSGeo4W64, 2.1.0-104 while trying to update a field in a large dataset (~300MB) with the Field Calculator ...very slow performance was experienced.

Another user in IRC, lrssvt also confirmed experiencing this with this dataset (CARTO1_Lin_6CNivelD.zip) available here #17441 (comment)

priority_id was changed from Severe/Regression to Low

qgib added Bug Either a bug report, or a bug fix. Let's hope for the latter! Vectors Related to general vector layer handling (not specific data formats) labels May 24, 2019

qgib added this to the Future Release - High Priority milestone May 24, 2019

qgib closed this as completed May 24, 2019

This was referenced May 25, 2019

Discarding edits freezes qgis for a long time #18100

Closed

Rollback operations slow if lots of features changed #18109

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Field Calculator, slow performance with 2.1.0-104 #17922

Field Calculator, slow performance with 2.1.0-104 #17922

qgib commented Jan 9, 2014 •

edited

Loading

qgib commented Jan 9, 2014

qgib commented Jan 9, 2014

qgib commented Jan 9, 2014

qgib commented Jan 9, 2014

qgib commented Jan 13, 2014

qgib commented Jan 26, 2014

qgib commented Jan 26, 2014

qgib commented Jan 26, 2014

qgib commented Jan 26, 2014 •

edited

Loading

qgib commented Jan 29, 2014 •

edited

Loading

qgib commented Jan 30, 2014

qgib commented Jan 30, 2014

qgib commented Jan 30, 2014 •

edited

Loading

qgib commented Jan 30, 2014

qgib commented Feb 1, 2014

qgib commented Feb 2, 2014

qgib commented Feb 2, 2014

qgib commented Feb 7, 2014

qgib commented Feb 9, 2014 •

edited

Loading

qgib commented Feb 9, 2014 •

edited

Loading

qgib commented Aug 7, 2017 •

edited

Loading

Field Calculator, slow performance with 2.1.0-104 #17922

Field Calculator, slow performance with 2.1.0-104 #17922

Comments

qgib commented Jan 9, 2014 • edited Loading

qgib commented Jan 9, 2014

qgib commented Jan 9, 2014

qgib commented Jan 9, 2014

qgib commented Jan 9, 2014

qgib commented Jan 13, 2014

qgib commented Jan 26, 2014

qgib commented Jan 26, 2014

qgib commented Jan 26, 2014

qgib commented Jan 26, 2014 • edited Loading

qgib commented Jan 29, 2014 • edited Loading

qgib commented Jan 30, 2014

qgib commented Jan 30, 2014

qgib commented Jan 30, 2014 • edited Loading

qgib commented Jan 30, 2014

qgib commented Feb 1, 2014

qgib commented Feb 2, 2014

qgib commented Feb 2, 2014

qgib commented Feb 7, 2014

qgib commented Feb 9, 2014 • edited Loading

qgib commented Feb 9, 2014 • edited Loading

qgib commented Aug 7, 2017 • edited Loading

qgib commented Jan 9, 2014 •

edited

Loading

qgib commented Jan 26, 2014 •

edited

Loading

qgib commented Jan 29, 2014 •

edited

Loading

qgib commented Jan 30, 2014 •

edited

Loading

qgib commented Feb 9, 2014 •

edited

Loading

qgib commented Feb 9, 2014 •

edited

Loading

qgib commented Aug 7, 2017 •

edited

Loading