This project is read-only.

Out Of Memory Error

Apr 15, 2011 at 5:45 PM
Edited Apr 15, 2011 at 5:47 PM

I am trying to load a small shapefile (135 MB), read the attribute table, make changes, then save it.

I am getting an 'Out Of Memory' error when saving the FeatureSet back.

Is there a better way to do this to overcome the error, than shown below:

                               
                                Dim myFS As New DotSpatial.Data.FeatureSet

                                myFS.FillAttributes()

                                myFS.Open(myFilename)

                                Dim myDT As DataTable = myFS.DataTable
                              
                                For myRow As Integer = 0 To myDT.Rows.Count - 1

                                    Dim myVal As Object = myDT.Rows(myRow)("ImagePath")

                                    myFolder = Trim(myVal).ToUpper

                                    myNewFolder = (frmMain.ComboBox8.Text & myFolder.Substring(kj)).ToUpper

                                    myDT.Rows(myRow)("ImagePath") = myNewFolder
      
                                Next myRow
                                
                               myFS.Save()

                               myFS.Close()





Apr 19, 2011 at 8:45 AM

It's a know bug not resolved, check out this forum post:

http://dotspatial.codeplex.com/discussions/248083

And in the issue tracker:

http://dotspatial.codeplex.com/workitem/277

It should be hard to resolve...

Apr 19, 2011 at 2:42 PM

You can use either ShapefileFeatureSource class or AttributeTable class to edit attributes in a shapefile without reading in the entire file, making the changes, then writing it out.  I wish I had time right now to put together a code sample.  I have been using those classes successfully, but I don't have a straightforward example.  Both of those classes have methods for retrieving 'pages' of data and updating the file with changes.  There are 3 different flavors of ShapefileFeatureSource (one each for polygons, lines, points), but they all implement IFeatureSource.  The IFeatureSource interface allows you to edit feature geometry as well as attributes.  If you choose to use AttributeTable class, you can edit only attributes.  IFeatureSource uses AttributeTable underneath.  Take a look at implementations of IFeatureSource in DotSpatial.Data.

HTH,

Kyle

Apr 19, 2011 at 3:14 PM

I'm a bit confused . . . in real world GIS data, my shapefile is tiny and insignificant. Are we saying (other than the suggestion above) that DotSpatial cannot open and edit this file normally? Are we also saying that there's a possibility that it never can? If any of these questions are ' yes ' does this not eliminate DotSpatial as a viable GIS platform?

Terry

 

Apr 19, 2011 at 3:22 PM

Hey Terry, Kyle's explanation is how you would handle your shapefile in DotSpatial to edit the attributes in code in a relatively straightforward way without getting the out of memory exception. Bottom line is that you connect to it as a "featuresource" rather than featureset. Give it a try. - Dan

Apr 19, 2011 at 3:58 PM

Here is an example that copies the ID field to the Desc field (ID is a double and Desc is a string).  I think the last row in the table was empty.  That may be a bug, but the DBNull test got around it.

            AttributeTable at = new AttributeTable(@"c:\temp\myfile.dbf");
            int nStart = 0;
            int numRows;
            do
            {
                DataTable dt = at.SupplyPageOfData(nStart, PageSize);
                numRows = dt.Rows.Count;

                foreach(DataRow dr in dt.Rows)
                {
                    object id = dr["ID"];
                    if(DBNull.Value != id)
                        dr["Desc"] = ((double)dr["ID"]).ToString();
                }

                at.SetAttributes(nStart, dt);
                nStart += PageSize;
            } while (numRows == PageSize);

Apr 20, 2011 at 9:01 PM

I have used the AttributeTable method shown above successfully. It took 13 hours to update 67 shapesfiles with about 5000 to 40000 records in each. Is this the correct amount of time required to to do this?

 

To use the ShapefileFeatureSource method, am I on the right track here? I am starting with an empty shapefile with all my fields set already. I now want to fill the columns with data as I add a new row to the shapefile.

Dim myCount As Integer

            Dim myGeometry As DotSpatial.Topology.Point

            Dim myCoordinate As DotSpatial.Topology.Coordinate

            Dim myFeatureSource As DotSpatial.Data.ShapefileFeatureSource

            '****************************************************************
            ' Open The Shapefile And Add The Shape Data.
            '****************************************************************

            myFeatureSource = New DotSpatial.Data.PointShapefileFeatureSource(myFilename)

            myFeatureSource.PageSize = myPageSize

            '**************************************************
            ' Set the values for the point to be inserted.
            '**************************************************

            myGeometry = New DotSpatial.Topology.Point

            myCoordinate = New DotSpatial.Topology.Coordinate

            myCoordinate.X = 100

            myCoordinate.Y = 110

            myCoordinate.Z = 0

??? From here what comes next?

 

Apr 20, 2011 at 11:32 PM

It should be much faster than that.  I will probably eventually run into the same issue and will take a look at performance then.  If you have access to a profiler, you might want to see where it is spending its time.

Use the IFeatureSource.Add method to add a feature:

You can use something along the lines below.  The trick is getting a DataRow.  I usually get one this way, but there's gotta be a better way:
                // Get us a DataRow that we can manipulate
                AttributeTable at = new AttributeTable();
                at.Open("your filename", new List<int>()); // Passing in an empty list will suppress the automatic search for deleted rows.
                DataRow dr = at.Table.NewRow();
It may be OK to have a null DataRow on the feature if you don't have attributes but have never tried that.
 IFeatureSource ifs already supplied                     
IFeature feature = FeatureFromPoints(numPoints, xPts, yPts, feature.FeatureType);
feature.DataRow = dr; 
ifs.Add(feature);
private static IFeature FeatureFromPoints(int count, double[] xPts, double[] yPts, FeatureType featureType) { Coordinate[] coordinates = new Coordinate[count]; for (int j = 0; j < count; j++) coordinates[j] = new Coordinate(xPts[j], yPts[j]); Shape shp = null; switch (featureType) { case FeatureType.Point: shp = new Shape(new Point(coordinates[0])); break; case FeatureType.Line: shp = new Shape(new LineString(coordinates)); break; case FeatureType.Polygon: shp = new Shape(new Polygon(coordinates)); break; } return new Feature(shp); }
Apr 21, 2011 at 8:48 PM
Did you try adjusting your page size? There's probably going to be an optimal size that balances memory use with disc reads. - Dan

On Wed, Apr 20, 2011 at 10:01 PM, tposhea <notifications@codeplex.com> wrote:

From: tposhea

I have used the AttributeTable method shown above successfully. It took 13 hours to update 67 shapesfiles with about 5000 to 40000 records in each. Is this the correct amount of time required to to do this?

To use the ShapefileFeatureSource method, am I on the right track here? I am starting with an empty shapefile with all my fields set already. I now want to fill the columns with data as I add a new row to the shapefile.

Dim myCount As Integer

Dim myGeometry As DotSpatial.Topology.Point

Dim myCoordinate As DotSpatial.Topology.Coordinate

Dim myFeatureSource As DotSpatial.Data.ShapefileFeatureSource

'****************************************************************
' Open The Shapefile And Add The Shape Data.
'****************************************************************

myFeatureSource = New DotSpatial.Data.PointShapefileFeatureSource(myFilename)

myFeatureSource.PageSize = myPageSize

'**************************************************
' Set the values for the point to be inserted.
'**************************************************

myGeometry = New DotSpatial.Topology.Point

myCoordinate = New DotSpatial.Topology.Coordinate

myCoordinate.X = 100

myCoordinate.Y = 110

myCoordinate.Z = 0

??? From here what comes next?

Read the full discussion online.

To add a post to this discussion, reply to this email (DotSpatial@discussions.codeplex.com)

To start a new discussion for this project, email DotSpatial@discussions.codeplex.com

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe or change your settings on codePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at codeplex.com




--
Daniel P. Ames, Ph.D. PE
Associate Professor, Geosciences
Idaho State University - Idaho Falls
amesdani@isu.edu
geology.isu.edu
www.mapwindow.org


Apr 21, 2011 at 11:00 PM

Somewhere between 1,000 and 10,000 should be good.  I usually go with something closer to 1,000 because my applications are already pretty big memory consumers.  But, if all you are doing is transforming the data, you might go closer to 10,000.

Kyle